Category: Uncategorized

Escrow Keybags and the iOS 6 lock screen bug

There was a lot of fuss last week about an alleged iPhone lock screen bug that allowed an attacker to bypass the lock screen and access your iPhone’s filesystem. I wrote an article on iMore explaining that this wasn’t the case and the confusion seemed to be the result of a misunderstanding on how iPhone passcodes work.

As mentioned in the article, when an unlocked iPhone is plugged in to a computer with iTunes running (or a locked phone that you then enter the passcode on), iTunes has a mechanism that allows it to access the device’s filesystem in the future without requiring the user to enter their passcode. I ambiguously referred to this “mechanism” because I had absolutely no clue how it worked. All I knew is that the first time you plug a locked device in to a computer, iTunes will give you an error message. However, once you enter the device’s passcode, iTunes will never prompt you again to enter it on that computer. Whether the phone is locked or not, iTunes (and any other app that can talk to your iPhone) will gladly read files from the device and write new data to it.

Twitter user Hubert was kind enough to send me more info about this process. He sent me an article that looks at how iOS 5 backups work. The article briefly explains the use of Escrow Keybags by iTunes (technically I think usbmux is what uses them) to talk to locked devices after they have been unlocked once. The gist seems to be that there are certificates and keys which are exchanged via iTunes when an iPhone is plugged in and unlocked, which are then stored in an Escrow Keybag on the host computer. The Escrow Keybag for every device you’ve ever plugged in to your machine can be found in /private/var/db/lockdown and is named with the UDID for the device that it is for. When a device is plugged in to your computer in the future, this Escrow Keybag is used to unlock the device without the need for the user to enter the passcode.

The confusion on the alleged lock screen bypass bug came from this. The bug discovered simply made the screen go black with a status bar still visible. If you plugged this device in to a computer that it had previously been plugged in to while unlocked, the computer used the Escrow Keybag in order to read the file system without needing the passcode. This would have happened with or without causing the black screen glitch. If the person who found the glitch had plugged their iPhone into a computer that it had never been connected to, they would have gotten an error from iTunes saying that the device was locked and they would need to enter their passcode.

The original video demonstrating the bug can be found here. A video of the same bug being reproduced, but with the phone plugged in to a computer it has not been unlocked with before can be seen here. My sincerest apologies that I don’t have any block rocking beats accompanying my video.

Kaleidoscope 2

The folks over at Black Pixel have just released Kaleidoscope 2.

Use Kaleidoscope to spot the differences in text, images, and folders. Review and merge changes in seconds with the world’s most advanced file comparison application.

I regularly use Kaleidoscope in my workflow for diffing images, text and files. If you do QA, development or design, you need this app. Go check it out.

Full Disclosure: I have a large bias toward the people at Black Pixel because they’re awesome.

Nobody panic… Dropbox is down

If you’re having problems syncing with Dropbox right now, you’re not alone. Dropbox’s services seem to be experiencing trouble uploading any new files (though viewing existing seems to be working for now). Dropbox’s status page doesn’t give much info, but does report they’re having some technical issues and believe services will be back in the next hour. You can watch for updates on the @DropboxOps Twitter account.

Update 8:10AM MST 1/11/13: @DropboxOps just announced all issues have been fixed and all services should now be working.

Mac Mini Servers: A Cautionary Tale

This post will deviate from the type of content I generally post here. It isn’t really related to QA and instead deals with a recent problem we encountered when upgrading our build servers. I’m posting about it here in hopes that others may save themselves time and trouble from the lesson that we learned. If you’re interested, read on, if not, hopefully I get some more QA posts up over the holidays. If you just want to know what the problem was and how we fixed it, you can scroll right to the bottom.

We recently decided to give our build servers an upgrade. Our existing setup was a Mac mini running Jenkins and 4 Xserves that served as slaves. While the Xserves were great as servers, their product line has been discontinued, they’re noisey, they’re large, and they require more power than Mac minis. And even though redundant networking and power supplies are nice, they’re overkill for our needs. So when we went to update the servers we decided go to with Mac mini servers.

Setting up the first slave took a little bit of time and patience. We decided to start clean and set these up from scratch rather than migrate the existing servers. We had a general idea of what was needed and figured out what pieces we were missing through trial-and-error. Taking notes on steps as we went through the first one allowed for an expedited process on the subsequent servers. When we finally got the first one setup we launched it as a slave on Jenkins, assigned a project to build on it, kicked it off and, with some tweaking, got the build to succeed.

With my set of steps to follow, I quickly ran through the second server and had it up and going in an hour or so. Feeling fairly confident in the process at this point, I decided to try a third server near the end of the day, but this one was a special case. This one was going to replace the server that also ran our VPN. Initially to try and avoid having to re-create all the VPN users and have everybody set up a new password, I set up this Mac mini as a clone of the existing server it would be replacing.  During the cloning process though I began to give things more thought. After talking with my boss we decided that VPN would actually be moved off of the build servers to a more dedicated box. Once the cloning finished, I booted the mini into the recovery menu and did a re-install.

An hour or so later the re-install was complete, but it hadn’t gone quite as I had hoped.  There still seemed to be certain user data and settings that had persisted through the re-install, but it was good enough for now. I ran through my list of steps, skipping a few for software that had persisted through the re-install, and got the server added to Jenkins. Feeling satisfied with my work for the day, I decided to hold off until the next day to do the remaining servers.

The next morning we noticed something strange; Jenkins was showing that two of the new servers had response times of a few seconds while the old servers were showing response times of a few milliseconds. Jenkins was also reporting a clock difference on the two new servers of being a few seconds ahead while the old ones all showed in sync. We also noticed a few builds had failed overnight, but all at different points in the build process. Some of them were even failing in the middle of uploading builds to an external server.

Our first guess was a networking problem, but why? When we created the new build servers we gave them the same names as their predecessors that were to be retired. We also assigned them the same IP addresses, but due to some technical hurdles, the static IPs weren’t being handled by the router. Instead we were telling the machines to use DHCP, but were manually assigning them the IPs we wanted them to have. We wondered if these factors had caused some weird networking problems when trying to reach these new machines that had taken over old hostnames and IPs. While trying to diagnose the issue I  noticed that if I pinged one of the new servers continuously, intermittently several packets would timeout.

Oddly, the third server wasn’t exhibiting any of the symptoms that the other two servers were. I had more or less followed the same steps, but one notable difference is this server had gotten a new hostname and IP since the existing Xserve needed to stay up for now to run our VPN. We were already thinking it seemed like some kind of network issue, and now we had one server that didn’t take over an existing IP and hostname and it wasn’t having any of the same problems.

Now that we had figured out the issue, it was just a matter of moving the other two servers we had already setup to new hostnames and IPs. The change only took a couple minutes and we were back in business. Jenkins was showing the response times as low and the clocks in sync. Everything looked great… until a little while later when I got an email notifying me that a build had failed. I looked and sure enough, it was one of our new Mac minis and like the other failures, it happened in the middle of the build. Looking at the Jenkins interface, the slave had now been marked as down as its response times had gotten so bad. I looked to the third server that had shown the initial success and it was still fine. Once again, I was perplexed.

I tried searching online for any clues on what to do about a drifting clock in Jenkins with no luck. I went to an OS X Server room on IRC and asked for help, but was directed to try the Jenkins room. I went to the Jenkins room for help and nobody had any suggestions. I spent the night mulling things over and was running out of ideas.

The next morning I talked with Casey, one of our developers who had been helping me try to troubleshoot the issue the last couple of days. Casey had an idea; what about the energy saving preferences? I quickly took a look at the two problematic servers and noticed they were set to put the computer to sleep after 30 minutes of inactivity. I checked the third server and this option was disabled. I quickly disabled the option on the other two servers and wished for the best. Initially things looked better, but I had already been fooled several times with other solutions that initially had good results.

We let the servers go for a few hours and this time the good results actually stuck. That was it; Casey figured it out. In retrospect it all makes sense. Well… almost all of it. The build processes didn’t count as activity because they were running in the background; the OS was relying on mouse/keyboard input to tell it the computer was in use. The reason the third server didn’t have any issues is because one of the settings that persisted across the reinstallation was the Xserve’s Energy Saver preferences. The Xserve’s seemed to have shipped with a much saner default of “Never” for the computer sleep option. I still don’t quite understand the symptoms of showing the clock as drifting ahead or why the response time appeared slow rather than just non-existant. Regardless, this was a much harder lesson to learn than it probably should have been. It should have occurred to us much sooner to check the Energy Saver preferences, but I do hope that Apple will change the default on the Mac mini servers to have these preferences disabled and save others the trouble and confusion. I don’t imagine that many people out there running servers want them going to sleep every 30 minutes.

tl;dr: Mac mini servers running as build servers had symptoms of slow response times with Jenkins, Jenkins reporting their system clocks as ahead, and builds failing at random points, but not all the time. This was fixed by changing the Mac mini server Energy Saver preferences from the default of putting it to sleep after 30 minutes of inactivity, to never putting it to sleep.

Apple to Fix IAP Vulnerability in iOS 6

Apple recently announced that the IAP vulnerability discovered earlier this week will be fixed in iOS 6. They have also released documentation for developers outlining best practices that should be taken to ensure they are not affected by this attack moving forward. One interesting bit is that Apple actually instructs developers to make use of private APIs to secure their apps.

Thanks to Matthew Panzarino for the heads up.


Getting Started With Mobile Testing

Over at Ministry of Testing they have put together a nice mind map that serves as a great starting point of things to consider for testing (as well as development and design) when you’re starting a mobile project. I’m hoping at some point somebody will make platform specific ones for Android and iOS. If not… maybe I will. Anyway, head on over to check it out: Getting Started With Mobile Testing – A MindMap


Charles Proxy: Like One of the Family

When it comes to testing slow network conditions, we already covered Network Link Conditioner. But what’s that you say? You’re wondering if there’s a more robust and extensive tool for network testing and debugging? Well I’m glad you finally asked, because as it turns out, there is, and his name is Charles. Charles Proxy is a cross-platform proxy application that can be used to monitor and manipulate traffic. Go download and install the trial so we can get started.

After starting up Charles you’ll need to configure your device to route its traffic through your computer (unless you’re using the sim, in which case you can skip this part). With your computer and iOS device on the same network, go to your iOS device Settings, then go to Wi-Fi and tap on the blue circle with a white > in it to the right of your network. Scroll to the bottom and under HTTP Proxy, tap Manual. For the Server, enter your computer’s IP address and for the port enter 8888. Close Settings and now your device’s traffic should be going through Charles.

As previously discussed, Network Link Conditioner lacks the ability to specify which traffic you want to throttle; it’s an all or nothing deal. So first, let’s look at Charles’ throttling feature. Go up to the Proxy menu and select Throttle Settings. Check “Enable Throttling” and “Only for selected hosts”. The throttle settings aren’t much different than NLC, but the “Only for selected hosts” option is something new to us. Any hosts added to the list at the bottom will be throttled according to your settings and all other traffic will go through the proxy unthrottled. This makes testing poor network conditions much more tolerable as you can now slow down your app’s traffic while still listening to your Dubstep Nickelback playlist on Pandora uninterrupted. So add whatever hosts you want to throttle to that list. You can easily toggle throttling on and off by pressing ⌘-T at any time. So now that you’re set to test your app on a slow network, let’s have a look at what else we can do with Charles.

By now you might be seeing some URLs appear in the Structure (left) pane of Charles’ main window. These are all the URLs of requests that have gone through Charles so far this session. If your window is still blank, then you must not have much background traffic going on (or you’re not recording in which case, click that record button up top that looks like… well, a record button). Expanding any one of the URLs will show which files on that site are being hit. As you select them, in the right pane you can see more information like the request to the server and the response that came back. If your application is using SSL for its connections, Charles has you covered there too. You’ll just need to go install Charles’ SSL certificates for your computer and for your iPhone (iOS 4 instructions also work on iOS 5 & 6).

Once you have the correct certificates installed, you’ll need to explicitly add all hosts that you want to enable SSL proxying for. While you can just add a wildcard to enable SSL proxying for all hosts, there are a number of websites that won’t play nicely with the type of man-in-the-middle attack Charles uses to make SSL proxying possible. Instead it’s recommended that you explicitly add hosts that you’re actually interested in looking at. You can manually add these hosts under Proxy > Proxy Settings > SSL, or if the host you want to proxy already appears in your list of traffic for this session, you can simply right-click on the host and click SSL Proxying. Now any SSL requests made to that host should show up, and the contents of the request and the response should be visible.

The last feature of Charles that I want to touch on right now is the ability to map to local files. Normally your application will make a request and the server will send it a response. Often times for testing, it’s extremely useful to be able to manipulate the server’s response, especially in apps that rely heavily on data from a server. For example, you might want to verify that your application can handle invalid JSON (I’ve previously run into a 3rd-party JSON parser that crashed when it got invalid JSON), but in testing the server might always give you a valid response. Usually to test this I’ll start by having my application make a normal request and receive a normal response. Then in the Structure pane of Charles, find that request, right-click on it and select Save Response. After saving the server’s response locally, you can edit it to your heart’s content. Once you’ve made whatever edits you want, back in Charles right-click on the request again and now select “Map Local…”. In this dialog you can set the local path for the file that you would like Charles to use (the one you just saved and edited) in place of the server’s response. Now the next time your application makes this same request, Charles will hand it your local file instead. You can use this feature to test for things like making sure your app handles long strings of text, large numbers, or blank values. You know, all those scenarios developers say won’t happen, but they always do. You can manage all the requests you have mapped locally under Tools > Map Local…

So there you have it kids. Charles Proxy is a powerful tool for testing and debugging your applications and their traffic. Hopefully this has helped you get started. The Charles Proxy website also has excellent documentation if you find yourself stuck on something or want to learn more about its other features.

Slow Your App’s Roll

Update As of iOS 6, it is now possible to use Network Link Conditioner directly on the device. Additional details can be found in this post.

Note: After upgrading to Mountain Lion, Network Link Conditioner was not working for me. Steven has pointed out in the comments below that this can be resolved by removing and re-installing the prefpane. Thanks for the tip, steven!

One shortcoming of testing in iOS Simulator is you don’t get to test how your app does in real world network scenarios. The solution? Network Link Conditioner.

Network Link Conditioner is included with Lion’s Developer Tools. You can install it by going to Applications > Utilities > Network Link Conditioner and double clicking the prefpane file. If you don’t see the Network Link Conditioner folder, you can open Xcode, go to the Xcode menu > Open Developer Tool > More Developer Tools…, then grab the Hardware IO Tools for Xcode download (Apple developer account required). Once you have it installed, open up System Preferences and you should see Network Link Conditioner listed at the bottom under Other.

After launching it, you’ll see there is a Profile drop-down that lists a number of preconfigured network scenarios including various 3G and EDGE profiles. If you click on Manage Profiles you can also create your own custom conditions in there to suit any needs you might have (for example, simulating a completely dead network where no traffic gets through). Playing around with my own profiles a bit, I found a pretty good mix with bandwidth set to 85kbps and packets dropped set to 15% for both downlink and uplink. I get a pretty good mix of some things loading and others timing out with those settings.

If you’re using iOS Simulator, you don’t need to do anything special to route your traffic through Network Link Conditioner (except turning it on). However, if you’re testing on a device, you’ll need to proxy your device’s traffic through your computer. The details are beyond the scope of this post, but I will say that I use Charles Proxy for this. I’ll cover the specifics in a later post.

A couple of caveats to note when using Network Link Conditioner. The first is it seems to lock itself about every 5 minutes. So you may find that you frequently need to re-enter your password to turn it on and off while testing. The other drawback worth noting is that Network Link Conditioner is system-wide and has no ability to distinguish between traffic. It doesn’t care about the color of your traffic or the content of its character; it treats all your traffic the same. This means when testing very poor conditions you have to say goodbye to IM, email, loading your ticket tracker, and that sweet dubstep remix you’re playing in the background on YouTube. Obviously this isn’t ideal and I hope that Apple will extend functionality in the future to customize the tool’s reach. It’s for this reason that often times I prefer to use Charles Proxy which has a Throttling feature with similar functionality to Network Link Conditioner, and you can customize which traffic you want it to affect. None-the-less, Network Link Conditioner is a great, easy-to-use, free tool that’s extremely useful in uncovering bugs you may have otherwise missed.

Now that we’ve covered the how, let’s talk a little bit about the why. As shocking as it may be to hear this, cell phones aren’t always connected to a speedy, reliable network. Every thing from a weak signal, to slow public wi-fi, to Sprint’s data network can result in poor network performance for your user and some poor conditions for your app to work under. While you can’t control the quality of the network your users are connected to, you can ensure that your app handles the poor conditions as gracefully as possible. It’s important to use a tool like Network Link Conditioner or Charles Proxy (or a terrible cellular connection) to weed out any bugs that may manifest in these poor conditions and make sure the overall user experience is still a good one.

One big no-no to this will help check for is doing network requests on the main thread. If you’ve ever gotten crash reports with an exception code of 0x8badf00d, you may already be familiar with this problem. Essentially if your app makes network requests on the main thread and it occurs on launch, a slow network could mean your app won’t successfully launch in time and watchdog will kill it before your user even gets it open. This type of issue can be very easy to miss during development and QA because your office wi-fi will likely load things quickly enough to avoid this problem. But as soon as a user in the real world with a poor network tries to use your app, the 1-star reviews will start rolling in.

Once you’ve made sure you don’t see any crashes in your application, turn your attention to the user experience. Even under poor conditions, your application should have some kind of messaging to your users about what’s going on. Failing gracefully is important— in your apps and in life.  A big point of frustration for users can be if your app is loading data (especially if it’s taking a long time) and you’re not conveying that to them in some way. When you’re loading data, tell the user. Using iOS’ network activity indicator icon that’s in the status bar is a good start, but often times it’s beneficial to show a loading indicator within the app, either as an overlay or in the portion of the app where you’ll display the data once it’s loaded.

If your data requests timeout or otherwise fail, once again, make sure you’re telling the user (preferably not with an alert dialog). Tweetbot has an implementation for this that I really like. If your pull-to-refresh fails, you get a nice little notification that slides down from the top to let you know and after a few seconds disappears, not requiring any user interaction and not disruptive to the app’s flow. On a related note, putting a timestamp in your pull-to-refresh bar that says when the last successful update from the server occurred can also be very helpful to users. It’s a simple way for users to see how stale their data is.

Finally, be sure to test bad network conditions on a clean install. Your users may very well be on a bad network the first time they launch your app and not have the luxury of some data already being cached. If your app relies heavily (or entirely) on data requests going across a network, your poor user stuck on a flight with no wi-fi should be able to see something other than completely blank views. Even if your user’s network fails, your app doesn’t have to.

Testing Memory Warnings in iOS Applications

A common cause of problems in iOS applications can be memory warnings. In the simplest terms, when the device is running low on available memory, iOS issues a memory warning which tells all running applications to stop being so greedy and to free up any memory they don’t absolutely need (and if that doesn’t free up enough, it will just start ruthlessly slaughtering them one by one). If an application doesn’t handle this properly and does something like release an object from memory without unlinking it, you can get a crash. Often times this will arise when a user has drilled down several views deep, a memory warning occurs, and as the user navigates back up the stack… kaboom! Let’s say a user is in an application, which has a photo albums view, and from there the user selects an album, and in that album they select a photo. So we have something like: Photo Albums view > Album view > Photo view.
Normally when you’d come back to something like the Photo Albums view, it would have a reference stored of which photo album you tapped on so that it could do a fancy deselect animation when you came back up out of your album view. So let’s say you’re on Photo Details, a memory warning gets issued and in an attempt to free up memory, the application releases the contents of the Photo Albums view, including the cell that you tapped and which it is holding a reference to. As you come back out of the stack to the Photo Albums view, the application will try to reference the cell for the album that you tapped, and be like “WTF? That cell doesn’t exist.” Naturally the application will throw a temper tantrum at this point, or in more technical terms, it will crash. This is just one type of scenario that can cause a crash from a memory warning. (I might cover more later, but for now it’s just important to know that nested views are where to look for these issues). Memory warnings, like all relationships, occur naturally in the wild all the time, but it’s so much easier if you can just force them to happen when it’s convenient for you. Fortunately for us, iOS Simulator provides this functionality. You can simulate a memory warning by going to the Hardware menu in iOS Simulator and selecting Simulate Memory Warning (who knows why they named it something so cryptic). You should go ahead and set a keyboard shortcut for this now because you’ll be using it a lot from now on. For this just go to System Preferences, then select the Keyboard preference pane. Then select the Keyboard Shortcuts tab, select Application Shortcuts in the left pane, then click + at the bottom. Set Application to iOS Simulator, Menu Title to Simulator Memory Warning and your shortcut to whatever you want (I went with ⌘-1 because I think memory warnings are A #1). Also remember as you’re finding these crashes to note the path you took through the views to make it happen; no need to give developers another reason to curse your name.

So now you can open up some bugs for your developers with all the crashes you’ve found. Sadly, your developers will get these crashes fixed and resolve your tickets just as you’re getting the hang of cruising around the app jamming on ⌘-1. You’ve spent your penny on the mechanical horse ride, but now it has stopped and Mom’s telling you to get off. Fear not! Crashes are only the most obvious bug to spot with memory warnings. So… what else can we look at?

Using apps should be an immersive, consistent, fluid experience for your users. One consistency that a user expects in an application is that however they leave a view, when they come back, it will be the same. For example, if you’re using the music app and you scroll down the artist list to V and tap on Vanilla Ice, when you come back to the list of artists, you expect the view to still be down in the Vs. If every time a memory warning happened while you were listening to music it meant that the Artists scroll view got reset to the top, that’s a frustrating experience. It’s important to note the different properties of your views and make sure that they are being preserved accordingly after memory warnings.

A couple of other things that are sometimes lost are the currently selected tab if you get a memory warning while a modal view is up as well as tapped cells being deselected when you come back to the view. Sometimes your developers have made sure the app won’t crash, but overlook proper behavior like deselecting a cell. Many users may not notice these details on a conscious level, but this kind of polish is what can make the difference between a good app and a great app.

Sometimes you’ll find yourself needing to test particular views on a device that don’t show up in the simulator. The easiest and quickest option for this is to launch some other applications that will eat up memory. So first plug your device in, then open Xcode and from the Window menu select Organizer. You should see your device over in the left pane with a Console option below it, select that. This is where you can get a behind-the-scenes peek at what’s going on with your device and where we’ll see when a memory warning occurs. Now that you know where to look to see when a memory warning happens, let’s figure out how to make one happen without our simulator.

One way to eat up some memory is to start a recording with Voice Memos. This option has the added luxury of allowing you to run your app in the foreground while the recording is still going and eating up memory. (It can also help you test to make sure your app resizes views properly, but we’ll go into that another time). Safari and Camera are two more apps that act like they’re trying to win Hungry Hungry Hippos when it comes to memory. In fact, for a while (I think on  iOS 5.0 or 5.0.1), if you had Voice Memos recording and tried to take a photo, both apps would crash from using too much memory. What you’ll be looking for in the console output is something like this: unknown Camera[13081] <Warning>: Received memory warning.

When you see your app receive a memory warning like this, you can then go back to your app and make sure everything still looks okay. One downfall to this method is depending on how low your device is on memory, you might find that your app has freed up more memory than you wanted it to. Instead of being taken back to the view you were expecting, you might be greeted by a splash screen indicating your application got completely killed as a result of the memory warnings (playing Infinity Blade tends to have this result). So this might take a little playing around to find the right combination of apps for you.

One final option, if your developers really love you, might be to have them put code in your debug builds that gives you buttons right in the app to generate memory warnings. This is extremely helpful for testing memory warnings on devices, but might not be an option for everybody. If it’s not an option, you might try searching around for somebody who has already written some memory warning code and made it freely available, like this guy. (Note: I haven’t tried this code, just came across it. Let me know if you have luck with it.)

As you can see, memory warning problems can manifest in a number of different ways. Hopefully with the info above, you’ll be catching these before your users do.

(Thanks to Chris Weathers for help with the explanation of memory warnings.)

And So It Begins

After months of contemplation and procrastination, I’m finally going to start this site. In my last year of doing QA on iPhone and Android applications, I’ve come across various tools, resources and information to make my job easier and to help me do it better. Each time I’ve had a problem to solve, I’ve wished there was a site or a community dedicated to mobile testing where I might find solutions or a step in the right direction. And every time I’ve found an answer, I’ve wished I had a place to share it for anybody else tackling the same problem. This site will hopefully make this information easier to find and more accessible to those in the same boat. I also encourage those who come across the site and have questions or something to contribute to reach out so we can either share the discoveries here or help answer questions.