Android OS, with its promise of being the Great Equalizer for open standards on smartphones and tablets, is successful when you look at market share: Android accounts for nearly 70% of the world’s mobile operating systems today. Yet the disparity in versions of Android OS, different devices types/sizes/implementations, and varied user experiences create a fragmented and frustrating environment for app developers and QA pros. To address this issue, we’ve built an automated Android test farm. A test farm is a group of devices (mobile phones) that are available for a build system to use for both making builds and test automation.
Before moving on to how awesome it is to have remote control over a plethora of Android devices, let’s start off with why you might need an Android test farm. The most standard reasoning is that testing with real devices helps protect users: if we test and develop on real devices we are likely to run into the same potential issues real users would (before releasing to the app store).
Another large selling point is if the company only owns one of each device, having a shared pool of devices where each device is virtually checked out reduces the chances of losing devices or blocking other team members from accomplishing work. If you are still not convinced, know that having a large device selection for building/automating allows engineers to ensure your app works well while taking into account different operating system levels, screen resolutions, CPU architectures, and other manufacturer differences that may cause issues. Some groups have an automation gate before entering production, where they run all their automation on all devices available before release; this ensures that automated features will work for their users.
In advance of building your own device farm you may want to consider other services that might be better for your team. Amazon and Google both provide an Android device test farm where they handle the devices and all you have to do is upload your app to be tested. Below is a short list of pros and cons for building a test farm in-house or purchasing (renting) devices from a cloud solution:
- Cheapest option — the only costs are a host machine (or VM) and the actual devices in use
- Since the system is wholly owned internally, there is less of a security risk in comparison to the cloud based solutions—it will always be behind your firewall, and won’t be in a network out of your control
- Initial setup can be a blocker for some groups (hopefully no longer with the instruction guide we provide at the end of this post)
- Maintenance of devices—both software (WiFi, battery charge, etc.) and hardware (sometimes batteries explode; maybe don’t use a Samsung Galaxy Note S7?)
- May not have enough devices in house for a dedicated solution
- AWS Test Farm https://aws.amazon.com/device-farm/
- Firebase Test lab (Google) https://firebase.google.com/docs/test-lab/web-ui
- Little to no set up
- Test clouds typically have a much larger selection of devices compared to one’s own test farm; they also might have multiple instances of the same model devices
- There are usage costs associated with these services
- Since the system is NOT wholly owned internally, there is an increased security risk in comparison to the in-house solutions
- There may be a legal concern for some products
- Some of the cloud solutions only support native test frameworks (may not support custom or open source frameworks)
Here at L4 we adhere to the security and privacy practices of our clients and have opted for an in-house device farm. We have prepared an installation guide for building and setting up OpenSTF, your own Android Test Farm. OpenSTF is the real hero here and can take your Android builds/automation to the next level when paired with a continuous integration server. Thanks for sticking around to the end, and happy testing!