When Apple acquired Burstly, makers of TestFlight, earlier this year, many were hopeful that Apple was finally ready to provide developers with an easy way to manage beta testing. So naturally, developers responded to Apple’s official announcement of the (re)launch of TestFlight at WWDC with great applause. Since then, many (including Apple) have rejoiced that the days of dealing with UDIDs and provisioning profiles are over. Many already believe that TestFlight spells the end for HockeyApp. But looking at what we know so far about TestFlight, I’m not so sure that’s the case.
TestFlight will bring two big changes to ad hoc app distribution as we know it. First, test devices are now managed using email addresses, rather than UDIDs. Second, the test device limit was drastically increased from 100 devices per account to 1,000 users per app. Many developers have been frustrated for some time now with Apple’s 100 device limit, and this has only gotten worse over the years with more new devices being released. The current system of UDIDs and provisioning profiles unnecessarily wastes a lot of developer time, requiring close monitoring and management of test devices and troubleshooting esoteric error messages.
With TestFlight, developers won’t need to ask testers for UDIDs anymoreâ€”they will simply send them an email invite from iTunes Connect. Once a user accepts the invite and installs the TestFlight app, they’ll be able to view details for apps, install betas, and provide feedback to developers. Developers will be able to use iTunes connect to upload builds, manage users, and even get insights into how testers are using the app. So far, so good, but here come the caveats.
Internal Testers vs. Beta Testers
TestFlight users will fall into two buckets: internal testers and beta testers. Beta testers will be the ones invited via email and, as previously mentioned, you will be able to have up to 1,000 of them per app. Internal testers will be managed in iTunes Connect by creating an account for each tester; you are limited to 25 of these. Builds for beta testers will need to be reviewed and approved by Apple before testers will have access to them, while internal testers will have access to builds as soon as they are uploaded. As others have already pointed out, reviews will only be required for the initial submission and any builds that contain major changes to the appâ€”minor changes won’t require another review. But this is already a hurdle that developers didn’t previously have to deal with. And for many companies, 25 internal tester slots simply won’t be enough. It’s great to see Apple get rid of the 100-device limit, and for a lot of developers it may be sufficient, but for many the hassle of dealing with the 100-device limit isn’t going away, it’s just being replaced with a similarly frustrating limitation of 25 internal testers.
Not surprisingly, TestFlight is only available for iOS 8. While TestFlight originally worked on Android, that support was dropped shortly after Apple acquired Burstly and there’s no indication it will return. There also hasn’t been any mention of TestFlight supporting Mac apps (which Hockey does). So if you’re planning on making an iOS 8-only app, then you’re in good shape. But anybody supporting older iOS versions, Android, or wanting to have a Mac app will need to look elsewhere.
The demonstration shown in Apple’s iTunes Connect session seems to suggest that testers will only be able to install the latest version of a beta. When a new build is uploaded, the previous build is marked as Inactive. Most of the time this is perfectly fine, but it’s not uncommon for developers and testers to need to install previous builds of an app. You may need to go back and reproduce a bug in an old build, or go back to compare a design element that has changed, or check old builds to determine when a regression was introduced, or to test database migrations from old builds… there are plenty of perfectly valid reasons for needing the ability to install previous builds. Perhaps I’m reading into Apple’s demonstration too much, but if this winds up being the case, Hockey will be able to add this to the list of features it has over TestFlight.
Automated Builds & Continuous Integration
TestFlight builds will be uploaded and managed through iTunes Connect. Unless Apple provides command line tools for this (which there has been no hint of), any developers who use any sort of continuous integration environment for automating builds and uploads will be out of luck. Each update for testers will require a developer to manually build and upload through iTunes Connect. Obviously there are plenty of developers who currently do manual builds, and for them this won’t be a big deal. But for the many of us using CI to upload multiples builds a day, this lack of functionality alone is enough to immediately rule out moving from Hockey to TestFlight.
Another big feature that TestFlight will have is crash reporting. Apple has provided crash reporting for App Store apps for a while now, but many have found it to be lacking and utilize 3rd-party solutions instead. Apple has only mentioned TestFlight supporting crash reporting for App Store submissions, not betas. It’s better than no crash reporting at all, but it’s not a complete solution and it’s still subpar to the many 3rd-party solutions currently out there. Oh, and one more thing. Apple said that crash reporting and symbolication will be available later next year.
Last but not least, let’s talk about support. Hockey won me over early on with their beyond-stellar support. I frequently receive responses to my support requests in less than 15 minutes, and at all hours of the day and night. One time I submitted a support request to find out if I could aggregate crash information in a particular way through the Hockey website, but was told it wasn’t currently possible. The next morning Hockey support sent me a Ruby script to accomplish what I wanted using their API. Apple won’t be, and likely has no interest in, providing that kind of support.
Where TestFlight Fits In
Believe it or not, I’m not trying to talk anybody out of using TestFlight. I’m quite glad to see TestFlight; I think it’s a big step in the right direction. TestFlight will be entirely sufficient for many developers’ testing needs, and the fact that those developers will be able to utilize TestFlight instead of relying on, and having to pay for a 3rd-party service, is great news. Even developers who continue to use services like Hockey to fulfill needs unmet by TestFlight will likely benefit from TestFlight. Hockey may make the most sense during development when you’re cranking out builds and rapidly iterating, but as you’re getting ready to ship to the App Store, being able to get your app into the hands of a much larger group before finally shipping to the App Store is a huge plus. The more users you have testing your app on more devices, the greater your ability to catch strange edge cases and eventually ship a more polished app. That’s not only great news for developers, it’s great news for users who easily become collateral damage of insufficient testing.
Maybe down the road Apple will expand TestFlight to compete more aggressively with Hockey. Maybe they’ll even kill off manual management of distribution profiles for good. But looking at what we know about these upcoming changes so far, TestFlight and HockeyApp are two different services that serve two very different needs. TestFlight isn’t going to kill Hockey, it’s going to complement it.
15 replies on “What Developers Should Know About Apple’s TestFlight”
There was also no mention of support for members of the Enterprise developer program. We donâ€™t submit apps through iTunes connect or have to get approval. Iâ€™m pretty doubtful testflight will be of any use in the enterprise.
TestFlight only had Android support for a short time before they were acquired. Crashlytics is shaping up to be tge reak hockey killer if you ask me. The just opened up access to their beta distribution system.
We do nightly test builds – uploading those manually would be a pain. Hopefully they keep the TestFlight upload API.
No mention of Crashlytics? Their beta-distribution service is really slick.
A few things I’m still wondering about… how will GameCenter testing work on TestFlight beta apps? Right now, I’m considering disabling GameCenter support for any beta versions of the app I give to testers, since creating a separate iTunes account for Sandbox’ed GC is such a pain. Also, will other beta test services, such as Hockey or Crashlytics, add in the logging checkpoint feature of TestFlight (which hopefully stays in once supported by Apple!) I really like having that available to me, and has been one of the main reasons I haven’t switched over to Hockey or Crashylytics, both of which have really nice tools and processes.
Forgive me if this question has been answered elsewhere… When do these changes (1000 device limit etc) come into effect? And is this restricted to iOS8 or will we be able to distribute to any device providing we have their email address?
Any information if “old” testflight will be supported ? CI builds upload, iOS 6 support and old versions installations is deal breaker for me.
I know Apple says TestFlight will require iOS 8 or later but does anyone know if this means the devices that it is installed on or whether the app itself must be iOS 8 only? Surely they can’t not support an app that is compatible with 8 and earlier version?
I don’t think TestFlight is killing anything. It will be a nice tool for personal apps to lower my costs, but like the Xcode build bots, unless they support more than iOS, professionals will have to look elsewhere.
Wonder what Ubertesters would do about it. Their app distribution is on par with Hockey or TF but I dont know anyone else who would offer the in-app bug submission SDK. We’re using them pretty intensively in our QA teams…
With this you should be able to upload automatically to iTunes Connect :
Just tried the old testflightapp.com on device running iOS 8.1 BETA and the test versions of my apps cannot be installed. Was perfectly fine on iOS 8.0 – iOS 8.0.2 Is itms-services scheme more restrictive now ?
My biggest cripe is having to submit to Apple for internal testing, I often send out test builds – with major features turned off – to test a certain functionality. Or just tests to my team. There the ease of the old testflight was great.
Good Day! If Testers already own a version of the app then tried the newer version through testflight then another version has been uploaded and goes live, can they update from testflight to the live version? Ex: Tester own live version 1.0 then updated to testflight version 2.0 then a new build let’s say version 3.0 will be uploaded to the appstore. Can they update from version 2 which is in testflight to version 3 that will be in appstore.
Hoping for your kind consideration. Thank you.
We’re using TestFlight for the first time while also running an Android beta via Google Dev Console.
At this very moment I am struggling with user management since over 1300 people signed up for beta testing and we wanted to remove users who hadn’t accepted an invite over the weekend – but! I am now getting a 666 out of 1000 users status on the list of users and when trying to add in 25 people seeing an error “You can only add 1000 testers to a group.”. Go figure!
Really have to start looking into Crashalytics, seems everyone is suggesting it.