SteelKiwi’s Own Device Farm to Test Android Apps

Android app testing can be a pain. While iOS developers are fortunate to have less testing requirements because of fewer supported versions, Android mobile app developers aren’t as lucky.

The Android operating system (OS) is vast, with different screen sizes, resolutions, and capabilities ranging across versions and devices. The list of versions which the system supports (including updates) is much bigger than that of iOS.

As of 2017, 85.2% of the global market share has an Android core powering it. As the demand for Android OS continues to grow in comparison with other operating systems, so does the awareness among developers that all apps need to be sufficiently tested before they are released.

Market share worldwide smartphone shipments by operating system from 2014 to 2021:

Why testing matters

Users interact with apps in a multitude of ways, from pressing buttons to making voice commands to downloading different information onto their devices and much more. Therefore, testing different use cases and user interactions with user interface (UI) components is crucial since most apps are developed iteratively. A comprehensive testing strategy allows developers to detect defects in the app early on, if and when these defects occur.

UI testing can be divided into two major categories:

1. Manual testing, where a human tester performs a set of user operations to check for unexpected results;
2. Automated testing, where specially written programs/scripts generate inputs that feed into the target application under predefined tests to analyze the app’s behavior.

Device farms: fast facts

As new devices became available on the market, the problem of testing all of these devices grew as well. Nokia was the first company to offer a free service for developers in which they could “rent” smartphones, allowing them to access different devices remotely via simulators/emulators to test key use scenarios manually. This testing gave developers evidence-based hope that the software they had built would work on the given device, resulting in more satisfied end users.

The trend caught on, with other companies starting to offer the same type of service to developers to conduct their testing on. As the technical revolution continued, dedicated services began to emerge, offering more tools that are now commonly known as device farms.

Device farms out there, at a glance

A device farm is an app testing service that allows developers and quality assurance (QA) engineers to interact with and test Android, iOS, and web apps. This can be done through real, physical devices that the dev can temporarily get for testing or it can be done through cloud-based access to devices which are used to remotely run and test apps in real time.

Let’s take a quick look at the most popular cloud-based device farms currently available. The following chart features specialized professional device farms which give remote access to power the testing of applications, offering different packages. Some of these also provide an automated option that sends screenshots or logs of the testing conducted so that devs and QA engineers can review and verify the results of the testing.

SteelKiwi’s very own device farm

At SteelKiwi, one of our goals is to help clients with automating their business processes. In order to do that, we leverage automation tools within our own company to increase our team’s efficiency. We have successfully implemented an in-house ERP system to better manage both internal and external processes, and we’ve also built our own device farm to further our testing efforts, without needing to use third-party services like the ones mentioned above.

We are primarily a software development company and so we conduct a lot of testing for ourselves and our clients to make sure every app we develop works as expected. We decided to build our own device farm for three main reasons:

• to streamline internal testing processes in a cost-effective way;
• and to deliver top-quality products by putting a greater emphasis on comprehensive testing;
• to ease the testing workload for our QA team.

The development process behind building our own device farm required specific business logic that fit our goals. We wanted to automate testing as much as possible, especially with regard to UI-related components in the selected app.

The business logic powering our device farm

The business logic we used to create our device farm has three main pillars that include:

• downloading the latest version of code from the repository to the work machine;
• installing the app to the test device automatically;
• running monkey tests on the device automatically.

Before we walk you through the workflow of SteelKiwi’s very own device farm, let us highlight what monkey tests are and why we chose to use them exclusively in the configuration of our device farm.

What is monkey testing?

Monkey testing is a technique in software testing where either a user or program/script checks the app’s behavior by purposefully attempting to crash it. Random inputs such as clicks, touches, or gestures, as well as a number of system-level events, are used to assess (un)expected results.

A monkey test can be run on emulators or physical devices and offers a number of advantages, including:

1. easy setup and execution;
2. quick identification of some out-of-the-box errors;
3. a tool that’s always included in Android software development kits (SDKs), whenever a new version is released;
4. identification of crashes in the code, caused by human error (e.g. the dev forgets to add a screen);
5. a good technique to test the reliability of the software;
6. testing that can be performed immediately after build creation, before starting manual testing for rapid bug detection;
7. an easy way to perform stress testing;
8. automation of certain testing techniques;
9. and cost-effectiveness.

Because of these advantages, we’ve used monkey testing to power our device farm so that the testing we conduct internally, as well as externally for client apps, can be done in a thorough, yet efficient manner.

How the testing workflow works

We have a computer running on MacOS. This is our local machine that we specifically use to access the device farm. An adapter with many USB slots is hooked to this computer so that we can plug in as many phones into this adapter as we need to.

We usually store our projects on GitLab. Whenever we need to run a test, we go to our web interface to input relevant information such as the project name, the right branch in the repository needed to build the actual application state, as taken from GitLab, and we start the build.

Whenever we need to run a test, we go to our web interface to input relevant information that identifies the correct project, and we start the build of the app on our test machine.

Once the app has been installed, the program starts running monkey tests automatically on one device. When the tests are over on one device, the program runs the tests again on the next device, and so on. If an error is found and the app crashes, the error is recorded in the log file. The logs will specify the error type, making it easier for the developer to identify and fix it. Different configurations of monkey tests allow users to schedule flexible, regular testing which can be run at a specific time.

Using an in-house Android device farm

Device farms are a worthy addition to the tools used to help perform automated testing. They help development and QA teams deliver better product quality, while also reducing the amount of human hours spent on testing. Device farms will never be able to substitute manual testing completely, but they can complement any development cycle.

By creating our very own local device farm to test Android apps, we implemented hand-written configurations that allow us to set flexible testing options. These options help us develop more automated testing to continue to improve the services we offer. We are currently working towards improving our current device farm and have plans to start building another device farm to test iOS applications in the future.

If you are looking for a development team to build and test your app, contact us and we'll make sure that your app’s release results in even more satisfied users!