A few weeks back there was an abuse of Apple’s Enterprise Developer Program that gathered a lot of attention after it was used to distribute a GameBoy emulator outside of the App Store. Obviously the focus at the time was on the fact that anybody could build an app to distribute outside of the App Store by using one company’s enterprise certificate. What got no attention, as far as I can tell, was the lack of security on the build server.
Secrets of an Error Log
Not quite. When my boss, Jay Graves, went to try out MacBuildServer his build failed as a result of a required variable that had not been set. When a build fails in Xcode, you’ll get output telling you what went wrong so you can try and fix it. Many continuous integration servers, like Jenkins for example, will give you console output from the entire build process so you can see everything that took place and any contributing factors that may have led up to the build failure, as well as the error that ultimately caused it to fail. MacBuildServer provides the same. If a build fails, you’ll get the option to view the error log. One of the first things Jay noticed was at the top of the log, the build server had echoed out the password for the certificate that the server was signing with. We had a laugh about it, but without the certificate itself the password doesn’t do you much good. But what if we could get the certificate?
I created an empty application in Xcode. In my project settings, I navigated to the Build Phases tab for my target. Under this tab you’ll find an item called “Run Script”. This does exactly what you might expect; it will execute any commands you enter as part of a shell script during the build process. I started off with a simple
whoami just to see if it would work. I followed this with a non-existent command in order to guarantee my build would fail right after my script had run so I could see the output. I uploaded my sample project to GitHub, pointed MacBuildServer to it, chose the option to sign with their enterprise certificate and as expected, my build failed. When I clicked to view the error log I was given the server’s output from my build and at the end of it was the output from my
whoami command showing me the user that the build had been run by.
The output of
whoami wasn’t terribly interesting, but what it represented was rather significant. At this point it’s pretty much game over for the build server. I have the ability to run any command that I could as a normal user logged in to the computer. The build failure logs told me everything I needed to know about where the enterprise certificate and corresponding keychain were being stored. Combine this with the certificate password that the build log is already giving and you now have the ability to locally sign any app you want with this company’s enterprise certificate. Of course this is less meaningful now that the certificate has been revoked. However, you could also run any number of other commands to cause harm to the build server or use access to the server to launch other attacks from inside their network. From navigating the filesystem, to copying files, to tampering with other people’s git repos, the possibilities are just about endless. I promptly contacted MacBuildServer about this problem and they have since disabled scripts from Build Phases and Build Rules.
What It Means for Users
While this seems like a lesson for administrators of hosted build servers, this should also serve as a warning to users. Using a hosted build server requires that you trust a third-party with your code, all of your signing credentials, as well as trusting that the IPA you get out the other end has not been tampered with. In the case of this particular service, the GitHub repository you use has to be public, but a number of other hosted build server solutions will build from private repositories. If the server you entrust isn’t properly secured, you may unwittingly be giving access to your code to a malicious party.
Of course, this is true of any third-party service. We often hand sensitive information over to a third party and trust that they have properly secured their services to protect our information. But build servers may inherently have a greater risk of being exploited. The very nature of a build server requires that it downloads remote code that is controlled by a user. Usually this is exactly the kind of action that you would want to try and prevent your users from doing on your servers.
Additionally, build servers are usually set up in trusted environments. When an organization sets up a build server internally, it usually has a trusting nature because the need for convenience outweighs security needs. A build server hosted inside of an organization, on a private network, to compile and run code that was written by developers who work for that company will naturally have less need for security than a build server accessible from the Internet to anybody who wants to sign up for a free account. If you’re building the latter, a fair amount of work will need to go into locking down the build server to limit access; that’s not the environment they’re usually set up for. It’s not impossible, but you need to know what you’re doing.
All of this is to not say that hosted build servers should not be used. Developers should simply be aware of the potential risk involved when trusting a third party like this. Similarly, anybody offering hosted build servers as a service should be aware of the risk involved, and be familiar with the security measures necessary to protect themselves and their users.
This post should also not be construed as a strict criticism of MacBuildServer. When I contacted MacBuildServer about the issues I had found, they were receptive and appreciative to my feedback and promptly addressed the issue. I have not taken the time to investigate other hosted build server solutions, but I would be surprised to learn that other services don’t suffer from their own security problems.
Special thanks to Carl Veazey for his help in Xcode and eternal willingness to answer my stupid questions.
2 replies on “Evaluating the Security of Hosted Build Servers”
I’m founder of another hosted build service â€“ https://hosted-ci.com, so think it would be valuable to write about it’s security here.
Right now we are still in private beta, but situation similar to described above isn’t possible with our service. Each of the paid plan users gets his own private Jenkins instance which runs under a different user, so it isn’t possible to get certificate of other users in a similar manner.
I understand that there is possibility of breaking OS level security mechanisms (i.e. privilege escalation) that may still allow to get data of other users, however this isn’t something under my direct control.
However for public launch I plan to switch to running multiple OS X instances via VMWare Fusion, so that there is better separation between Jenkins instances.
I also would welcome your advice on what I can do to make our system more secure. Drop me a line to email@example.com if you are interested to discuss it.
Outputting passwords (and storing them) and not isolating builds aren’t brilliant moves. It’s a bit sad that this hurts hosted CI (or any hosted cloud service even). I like how some of the hosted ci environments have a /security page explaining their security measures. I encourage any hosted CI environment to do the same: