What is performance testing?
Performance testing is the process of determining the speed or effectiveness of a computer, network, software program, or device. The longer any function of an app or a website takes, the more users an app stands to lose. When loading a typical website, for example, nearly half of users expect a page to load within two seconds and may abandon it once three or four have gone on. Users prefer quick, smooth apps over ones that take a long time to load, stutter, or take a moment to respond. Even milder defects can, in aggregate, take a well designed user experience and make it feel like a grind to get anything done.
This is where performance testing comes into play. We use performance testing to validate that the app is responsive and that it meets the user’s expectations.
So what do a few normal numbers look like? Let’s measure a few video apps, just looking at the central processing unit (CPU) usage. The CPU is what does most of your app’s processing. If the CPU usage for a given app is too high, it will start moving slowly and choppily. All of these numbers are from the same iPad Air, testing three streaming apps: 1. YouTube, 2. another common app which performed very badly (I’ll call it Bad App from here on), and 3. an app which L4 built.
The numbers above represent the CPU utilization during different user actions on the phone. Low CPU usage means a better user experience. Note that this scale is from 0% to 200%, not 0% to 100%, as the percentage is usage of the capacity of a single CPU core and the device is dual-core. The bars represent four different user scenarios in the application: general navigation, scrolling through lists on a fresh install, scrolling through prefetched lists, and watching videos. The line represents the spot above which performance degraded badly.
Testing navigation consisted of using the side bars (in the cases of YouTube and Bad App) or the tab bar (in the case of L4’s App) to move between the app’s screens. One of Xcode’s performance monitoring Instruments recorded how much of the device’s resources the app used while doing this. If you would like to get performance metrics on your own apps, or any app for that matter, check out how here. Xcode is the tool Apple provides for iOS and Mac OS development. YouTube performed the best and had no visible hiccups. L4’s App was only just behind it, also showing no visible problems. Bad App, though, had twice the CPU usage and it showed. The app was noticeably slow when transitioning between screens. It was like having an app in claymation.
On the screens with large lists of videos, scrolling was tested both right after opening the app while it was still loading images and then later again after having already scrolled through it once so that the app had already cached the images it needed. Scrolling while loading images tends to take more resources and be less performant than scrolling after they’ve been loaded. Even in the first scroll while loading, YouTube scrolled pretty smoothly. L4’s App had a few minor hiccups, but nothing that would greatly detract from the experience or which made it hard to see video images as they moved by (these issues may be a candidate for further performance improvements). Notably, L4’s App scrolled more columns of videos than YouTube. When Bad App scrolled, it hiccuped several times for long enough that a whole row of videos could go by without the user seeing it. Once all of the images had been loaded, however, Bad App actually showed the lowest performance costs of all three apps. Performing like YouTube here and keeping your CPU low is ideal, but hitting L4’s App’s performance is still quite good for any video app.
For all three apps, playback didn’t really have a high average CPU cost, since playback is handled mostly by the device’s GPU. Little spikes happened during network requests, but nothing performance affecting and the spikes quickly fell back down.
Performance testing is a tool that can help clarify user issues like “it’s slow”, “it’s choppy”, or “it stutters”. New features are great, but not at the cost of bad user experiences. Many products follow a “same or better” performance model to ensure no bugs or performance degradations are introduced that would be very noticeable to a loyal user.
Things to watch out for in your own apps
Poor responsiveness when scrolling can prevent your user from being able to stop where they want in a list. Stuttered scrolling can prevent them from ever seeing the item they were looking for. Making sure your app can scroll well not only makes it look better, but also makes it functionally easier for your user to use it.
Not all of the rest of your navigation animation will have that much functional impact on your user, but slow, choppy animations make an app look unprofessional and are distracting. Keeping the simple things in your app clean and performant will let your users focus on your content instead of being distracted by ugly performance issues.
So there you go! Now you can go make sure your app performs smoothly and cleanly, too!
Image courtesy of Marc-Olivier Jodoin for Unsplash.