When Apple acquired Burstly, makers of TestFlight, earlier this year, many were hopeful that Apple was finally ready to provide developers with an easy way to manage beta testing. So naturally, developers responded to Apple’s official announcement of the (re)launch of TestFlight at WWDC with great applause. Since then, many (including Apple) have rejoiced that the days of dealing with UDIDs and provisioning profiles are over. Many already believe that TestFlight spells the end for HockeyApp. But looking at what we know so far about TestFlight, I’m not so sure that’s the case.
The Promise
TestFlight will bring two big changes to ad hoc app distribution as we know it. First, test devices are now managed using email addresses, rather than UDIDs. Second, the test device limit was drastically increased from 100 devices per account to 1,000 users per app. Many developers have been frustrated for some time now with Apple’s 100 device limit, and this has only gotten worse over the years with more new devices being released. The current system of UDIDs and provisioning profiles unnecessarily wastes a lot of developer time, requiring close monitoring and management of test devices and troubleshooting esoteric error messages.
With TestFlight, developers won’t need to ask testers for UDIDs anymore—they will simply send them an email invite from iTunes Connect. Once a user accepts the invite and installs the TestFlight app, they’ll be able to view details for apps, install betas, and provide feedback to developers. Developers will be able to use iTunes connect to upload builds, manage users, and even get insights into how testers are using the app. So far, so good, but here come the caveats.
Internal Testers vs. Beta Testers
TestFlight users will fall into two buckets: internal testers and beta testers. Beta testers will be the ones invited via email and, as previously mentioned, you will be able to have up to 1,000 of them per app. Internal testers will be managed in iTunes Connect by creating an account for each tester; you are limited to 25 of these. Builds for beta testers will need to be reviewed and approved by Apple before testers will have access to them, while internal testers will have access to builds as soon as they are uploaded. As others have already pointed out, reviews will only be required for the initial submission and any builds that contain major changes to the app—minor changes won’t require another review. But this is already a hurdle that developers didn’t previously have to deal with. And for many companies, 25 internal tester slots simply won’t be enough. It’s great to see Apple get rid of the 100-device limit, and for a lot of developers it may be sufficient, but for many the hassle of dealing with the 100-device limit isn’t going away, it’s just being replaced with a similarly frustrating limitation of 25 internal testers.
Supported Platforms
Not surprisingly, TestFlight is only available for iOS 8. While TestFlight originally worked on Android, that support was dropped shortly after Apple acquired Burstly and there’s no indication it will return. There also hasn’t been any mention of TestFlight supporting Mac apps (which Hockey does). So if you’re planning on making an iOS 8-only app, then you’re in good shape. But anybody supporting older iOS versions, Android, or wanting to have a Mac app will need to look elsewhere.
Build History
The demonstration shown in Apple’s iTunes Connect session seems to suggest that testers will only be able to install the latest version of a beta. When a new build is uploaded, the previous build is marked as Inactive. Most of the time this is perfectly fine, but it’s not uncommon for developers and testers to need to install previous builds of an app. You may need to go back and reproduce a bug in an old build, or go back to compare a design element that has changed, or check old builds to determine when a regression was introduced, or to test database migrations from old builds… there are plenty of perfectly valid reasons for needing the ability to install previous builds. Perhaps I’m reading into Apple’s demonstration too much, but if this winds up being the case, Hockey will be able to add this to the list of features it has over TestFlight.
Automated Builds & Continuous Integration
TestFlight builds will be uploaded and managed through iTunes Connect. Unless Apple provides command line tools for this (which there has been no hint of), any developers who use any sort of continuous integration environment for automating builds and uploads will be out of luck. Each update for testers will require a developer to manually build and upload through iTunes Connect. Obviously there are plenty of developers who currently do manual builds, and for them this won’t be a big deal. But for the many of us using CI to upload multiples builds a day, this lack of functionality alone is enough to immediately rule out moving from Hockey to TestFlight.
Crash Reporting
Another big feature that TestFlight will have is crash reporting. Apple has provided crash reporting for App Store apps for a while now, but many have found it to be lacking and utilize 3rd-party solutions instead. Apple has only mentioned TestFlight supporting crash reporting for App Store submissions, not betas. It’s better than no crash reporting at all, but it’s not a complete solution and it’s still subpar to the many 3rd-party solutions currently out there. Oh, and one more thing. Apple said that crash reporting and symbolication will be available later next year.
Support
Last but not least, let’s talk about support. Hockey won me over early on with their beyond-stellar support. I frequently receive responses to my support requests in less than 15 minutes, and at all hours of the day and night. One time I submitted a support request to find out if I could aggregate crash information in a particular way through the Hockey website, but was told it wasn’t currently possible. The next morning Hockey support sent me a Ruby script to accomplish what I wanted using their API. Apple won’t be, and likely has no interest in, providing that kind of support.
Where TestFlight Fits In
Believe it or not, I’m not trying to talk anybody out of using TestFlight. I’m quite glad to see TestFlight; I think it’s a big step in the right direction. TestFlight will be entirely sufficient for many developers’ testing needs, and the fact that those developers will be able to utilize TestFlight instead of relying on, and having to pay for a 3rd-party service, is great news. Even developers who continue to use services like Hockey to fulfill needs unmet by TestFlight will likely benefit from TestFlight. Hockey may make the most sense during development when you’re cranking out builds and rapidly iterating, but as you’re getting ready to ship to the App Store, being able to get your app into the hands of a much larger group before finally shipping to the App Store is a huge plus. The more users you have testing your app on more devices, the greater your ability to catch strange edge cases and eventually ship a more polished app. That’s not only great news for developers, it’s great news for users who easily become collateral damage of insufficient testing.
Maybe down the road Apple will expand TestFlight to compete more aggressively with Hockey. Maybe they’ll even kill off manual management of distribution profiles for good. But looking at what we know about these upcoming changes so far, TestFlight and HockeyApp are two different services that serve two very different needs. TestFlight isn’t going to kill Hockey, it’s going to complement it.
