Parallel Test Execution (Gradle)

Introduction

Right now, when we run all of our tests in our framework, they are run sequentially (e.g. one at a time).  This is arguably ok for now as we only have a total of 5 test scenarios (4 base scenarios and 1 proper test), running against only one Android device (emulator) that we setup.

But let’s imagine we have loads of tests in our framework, and those tests need testing across multiple mobile devices. If we ran all those tests against one mobile device at a time, the time it would take to complete would be relatively long.

This is where parallel test execution comes in. By using multiple forks or threads, we can run our tests against multiple mobile devices concurrently (at the same time) to reduce the total amount of time it takes to execute all the tests in the framework.

Back in the earlier days of Appium, we would need to run a separate Appium server for each mobile device we want to run in parallel. Luckily, this is no longer the case, as we can now run only one Appium server that handles multiple sessions for all of our mobile devices.

In today’s modern age, most computers have multiple processors, which we can use to our advantage to run tests on a few different forks or threads at the same time.  If we want to run on loads of forks or threads without making the computer unusable, we can scale up using something like Selenium Grid, and then scale up even further with a cloud based service like SauceLabs. We will cover both of these in later parts of this blog series.

Let’s get started 😀

Running our tests against multiple mobile devices concurrently

Due to the fact that we cannot run multiple tests against the same mobile device at the same time, our best options for parallel test execution approach are either:

• Distribution – Feature files are distributed almost equally on all devices available during a run
• Fragmentation – All Features are run across all available devices

Due to the currently very small size of our test framework, we will go with the Fragmentation approach.

Currently the way our framework is setup, is that in our build.gradle file (Gradle), we specify the testng.xml file that contains all of our test suites/classes.

We will be wanting to run our tests in parallel on the TestRunner level, by defining our TestRunner classes in a build.gradle task we will create, rather than the TestNG level, due to the simple fact that it’s a lot quicker and easier to get it setup. This means we no longer need the testng.xml file!

1. So, firstly, with the project open in IntelliJ, right-click on the ‘testng.xml’ file and select ‘Delete’ and then ‘OK’

Adding another Android device (emulator)

Creating a new Android emulator

1. Launch Android Studio
   • Select ‘Start a new Android Studio Project’
   • Leave all the options as they are by default in the ‘Create New Project’ wizard and keep clicking ‘Next’
   • Click ‘Finish’ on the last screen of the wizard
2. In Android Studio, advance to ‘Tools’ -> ‘AVD Manager’
3. Click ‘Create Virtual Device’
   • Select a suitable device definition (e.g. Phone – Nexus 6P) and click ‘Next’
   • Select a suitable system image / Android OS (e.g. Marshmallow – API Level 27) and click ‘Next’ (usually latest is the best to choose, make sure it is compatible with Appium and download the image if necessary)
   • Give the Android Virtual Device (AVD) a suitable name (e.g. Nexus6pMarshmallow) and click ‘Finish’
     • You can now click the ‘Play’ icon and open the virtual device if you want and play around with it (e.g. try opening Browser in it or another app)
4. From now on, you can also open up the AVD via Terminal (or command prompt).  Open up Terminal and type in the following commands…

   cd $ANDROID_HOME/platform-tools

   emulator -avd yourAvdName

   (replace ‘yourAvdName’ with name of the AVD you made)

Adding another ‘AndroidDriver’ for the new mobile device

1. With the project open in IntelliJ, open up the ‘AndroidAppDriver’ class and add a new ‘public static AndroidDriver’ method to load the new mobile device emulator you just created, like below…

   public static AndroidDriver loadNexus6pMarshmallow() throws IOException {
       AppiumServer.start();
   
       File file = new File("src");
       File fileApp = new File(file, "reddit-official-app-3-12-0.apk"); //set app filepath to /src/[name-of-apk-file]
   
       DesiredCapabilities cap = new DesiredCapabilities();
       cap.setCapability("avd", "Nexus6pMarshmallow"); //set the AVD (Android Virtual Device) to be launched
       cap.setCapability("deviceName", "Nexus6pMarshmallow"); //set the name of the device to be launched (should be same as AVD)
       cap.setCapability("app", fileApp.getAbsolutePath()); //set the app to install and use as the one in the filepath specified above
   
       driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap); //set the AndroidDriver to an Appium session with the above DesiredCapabilities
       return driver;
   }

   The whole thing should look similar to below…

   package utils.drivers;
   
   import io.appium.java_client.android.AndroidDriver;
   import org.openqa.selenium.remote.DesiredCapabilities;
   import utils.appium.AppiumServer;
   import java.io.File;
   import java.io.IOException;
   import java.net.URL;
   
   public class AndroidAppDriver {
   
       private static AndroidDriver driver;
   
       public static AndroidDriver loadNexus5xOreo() throws IOException {
           AppiumServer.start();
   
           File file = new File("src");
           File fileApp = new File(file, "reddit-official-app-3-12-0.apk"); //set app filepath to /src/[name-of-apk-file]
   
           DesiredCapabilities cap = new DesiredCapabilities();
           cap.setCapability("avd", "Nexus5xOreo"); //set the AVD (Android Virtual Device) to be launched
           cap.setCapability("deviceName", "Nexus5xOreo"); //set the name of the device to be launched (should be same as AVD)
           cap.setCapability("app", fileApp.getAbsolutePath()); //set the app to install and use as the one in the filepath specified above
   
           driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap); //set the AndroidDriver to an Appium session with the above DesiredCapabilities
           return driver;
       }
   
       public static AndroidDriver loadNexus6pMarshmallow() throws IOException {
           AppiumServer.start();
   
           File file = new File("src");
           File fileApp = new File(file, "reddit-official-app-3-12-0.apk"); //set app filepath to /src/[name-of-apk-file]
   
           DesiredCapabilities cap = new DesiredCapabilities();
           cap.setCapability("avd", "Nexus6pMarshmallow"); //set the AVD (Android Virtual Device) to be launched
           cap.setCapability("deviceName", "Nexus6pMarshmallow"); //set the name of the device to be launched (should be same as AVD)
           cap.setCapability("app", fileApp.getAbsolutePath()); //set the app to install and use as the one in the filepath specified above
   
           driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap); //set the AndroidDriver to an Appium session with the above DesiredCapabilities
           return driver;
       }
   }

Refactoring the ‘AndroidAppDriver’ class

You might notice here that our two ‘AndroidDriver’ methods have a lot of duplicated code, making it not very DRY (Do not Repeat Yourself). We should fix this by instead creating a shared method where we can pass in arguments for our DesiredCapabilities.

To safely run multiple sessions in the same Appium Server, each Android session needs to be run on a different ‘System Port’:

• systemPort: to communicate to the UiAutomator2 process, Appium utilises an HTTP connection which opens up a port on the host system as well as on the device. The port on the host system must be reserved for a single session, which means that if you’re running multiple sessions on the same host, you’ll need to specify different ports here (for example 8200 for one test thread and 8201 for another). Ports 8200-8299 are used for UiAutomator2, where as ports 8300-8399 are given for Android Espresso.

We also HAVE TO to declare the udid for each device (even emulators). This can be found by starting up an Android emulator and entering into Terminal (or command prompt)…

adb devices

. If we don’t include this capability, the driver will attempt to use the first device in the list returned by ADB. This could result in multiple sessions targeting the same device, which is not good.

1. Let’s make our generic ‘loadDriver’ method that will take in arguments that we pass in to it (‘AVD’ and ‘DeviceName’ are the same so we only need to pass one String for those, and then another String for ‘udid’ and one more int for ‘systemport’)…

   public static AndroidDriver loadDriver(String avdDeviceName, String udid, int systemPort) throws IOException {
       AppiumServer.start();
   
       File file = new File("src");
       File fileApp = new File(file, "reddit-official-app-3-12-0.apk");
   
       DesiredCapabilities cap = new DesiredCapabilities();
       cap.setCapability("avd", avdDeviceName);
       cap.setCapability("deviceName", avdDeviceName);
       cap.setCapability("udid", udid);
       cap.setCapability("systemPort", systemPort);
       cap.setCapability("app", fileApp.getAbsolutePath());
   
       driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap);
       return driver;
   }

   We can now get rid of our other two methods. The whole thing should then look like below…

   package utils.drivers;
   
   import io.appium.java_client.android.AndroidDriver;
   import org.openqa.selenium.remote.DesiredCapabilities;
   import utils.appium.AppiumServer;
   import java.io.File;
   import java.io.IOException;
   import java.net.URL;
   
   public class AndroidAppDriver {
   
       private static AndroidDriver driver;
   
       public static AndroidDriver loadDriver(String avdDeviceName, String udid, int systemPort) throws IOException {
           AppiumServer.start();
   
           File file = new File("src");
           File fileApp = new File(file, "reddit-official-app-3-12-0.apk");
   
           DesiredCapabilities cap = new DesiredCapabilities();
           cap.setCapability("avd", avdDeviceName);
           cap.setCapability("deviceName", avdDeviceName);
           cap.setCapability("udid", udid);
           cap.setCapability("systemPort", systemPort);
           cap.setCapability("app", fileApp.getAbsolutePath());
   
           driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap);
           return driver;
       }
   }

Refactoring our AppiumServer class

Because we can now run multiple sessions in a single Appium server, we only need to startup the Appium server if it is not already running. Let’s refactor our AppiumServer class to check for this and only start it up if it is not already running 🙂

1. Open up the ‘AppiumServer’ class in the ‘utils.appium’ package and refactor it to match below…

   package utils.appium;
   
   import io.appium.java_client.service.local.AppiumDriverLocalService;
   import io.appium.java_client.service.local.AppiumServiceBuilder;
   import io.appium.java_client.service.local.flags.GeneralServerFlag;
   
   public class AppiumServer {
   
       private static AppiumDriverLocalService service;
   
       public static void start() {
           if (checkIfServerIsRunning()) {
               return; 
           }
   
           //Build the Appium service
           AppiumServiceBuilder builder = new AppiumServiceBuilder();
           builder.withIPAddress("127.0.0.1");
           builder.withArgument(GeneralServerFlag.SESSION_OVERRIDE);
           builder.withArgument(GeneralServerFlag.LOG_LEVEL, "error");
   
           //Start the server with the builder if not already running
           service = AppiumDriverLocalService.buildService(builder);
           service.start();
       }
   
       static void stop() {
           service.stop();
       }
   
       //Check if Appium server is running
       private static boolean checkIfServerIsRunning() {
           return service != null;
       }
   }

Refactoring the DriverController class

Because we now have only one generic method that we pass arguments into for loading the Android driver sessions, we only need one generic method in the DriverController class to start the driver session  by calling the loadDriver() method with the arguments we pass into it.

1. Open up the ‘DriverController’ class in the ‘utils.appium’ package and refactor it to look like below…

   package utils.appium;
   
   import io.appium.java_client.android.AndroidDriver;
   import org.apache.logging.log4j.LogManager;
   import org.apache.logging.log4j.Logger;
   import utils.drivers.AndroidAppDriver;
   import java.io.IOException;
   
   public class DriverController {
   
       public static DriverController instance = new DriverController();
   
       AndroidDriver androidDriver;
   
       private static Logger log = LogManager.getLogger(DriverController.class);
   
       public void startAppDriver(String avdDeviceName, String udid, int systemPort) throws IOException {
           if (instance.androidDriver != null) return;
           instance.androidDriver = AndroidAppDriver.loadDriver(avdDeviceName, udid, systemPort);
       }
   
       public void stopAppDriver() {
           if (instance.androidDriver == null) return;
   
           try {
               instance.androidDriver.quit();
           } catch (Exception e) {
               log.error(e + ":: AndroidDriver stop error");
           }
   
           AppiumServer.stop();
           instance.androidDriver = null;
           log.debug(":: AndroidDriver stopped");
       }
   }

Forking

In order for our tests to successfully run across multiple mobile devices in parallel, we need to run multiple forks, with each fork running our tests against a particular device in one session.

It is also possible to run each session in a thread (e.g. by putting our Drivers or DriverController instance in a ThreadLocal<>), however running them in forks is easier.  Each fork uses its own JVM in contrast to Threads. Running multiple JVM’s uses up more resources of course, but is a lot quicker and easier to get started with.

TestRunners

To run separate sessions in each fork, we first need to create TestRunner classes to run all of our feature files against different specified mobile devices. Let’s do that now.

1. In IntelliJ, right-click on the ‘TestRunner’ class and select ‘Refactor’ -> ‘Rename’
   • Rename the class to ‘Nexus5xTestRunner’, and click ‘Refactor’ (and then ‘Do Refactor’ if necessary)
2. Open up the ‘Nexus5xTestRunner’ class and edit it to match below…

   import io.cucumber.testng.AbstractTestNGCucumberTests;
   import io.cucumber.testng.CucumberOptions;
   import org.testng.annotations.AfterTest;
   import org.testng.annotations.BeforeTest;
   import pages.BasePage;
   import utils.appium.DriverController;
   
   import java.io.IOException;
   
   import static pages.Page.instanceOf;
   
   
   @CucumberOptions(
           features = "src/test/resources/features",
           glue = {"utils.hooks", "steps"},
           tags = {"~@Ignore"},
           plugin = {"html:target/cucumber-reports/cucumber-pretty",
                   "json:target/cucumber-reports/CucumberTestReport.json",
                   "rerun:target/cucumber-reports/rerun.txt"
           })
   
   public class Nexus5xTestRunner extends AbstractTestNGCucumberTests {
   
       @BeforeTest //this method gets run first
       public void setUpTest() throws IOException {
           DriverController.instance.startAppDriver("Nexus5xOreo","emulator-5554", 8201);
           instanceOf(BasePage.class).appFullyLaunched();
       }
   
       @AfterTest //tearDown of AppDriver method happens at very end
       public void tearDownTest() {
           DriverController.instance.stopAppDriver();
       }
   }

   Now let’s make another similar TestRunner for our ‘Nexus6P’ mobile device (emulator)

3. In IntelliJ, right-click on the ‘Nexus5xTestRunner’ class in the project explorer and select ‘Copy’
4. Click on the ‘Nexus5xTestRunnerClass’ in the project explorer again and press Ctrl/CMD+V to paste
5. Rename the copied class to ‘Nexus6pTestRunner’ and click OK
6. Open up the ‘Nexus6pTestRunner’ class and refactor it to match below…

   import io.cucumber.testng.AbstractTestNGCucumberTests;
   import io.cucumber.testng.CucumberOptions;
   import org.testng.annotations.AfterTest;
   import org.testng.annotations.BeforeTest;
   import pages.BasePage;
   import utils.appium.DriverController;
   
   import java.io.IOException;
   
   import static pages.Page.instanceOf;
   
   
   @CucumberOptions(
           features = "src/test/resources/features",
           glue = {"utils.hooks", "steps"},
           tags = {"~@Ignore"},
           plugin = {"html:target/cucumber-reports/cucumber-pretty",
                   "json:target/cucumber-reports/CucumberTestReport.json",
                   "rerun:target/cucumber-reports/rerun.txt"
           })
   
   public class Nexus6pTestRunner extends AbstractTestNGCucumberTests {
   
       @BeforeTest //this method gets run first
       public void setUpTest() throws IOException {
           DriverController.instance.startAppDriver("Nexus6pMarshmallow","emulator-5556", 8202);
           instanceOf(BasePage.class).appFullyLaunched();
       }
   
       @AfterTest //tearDown of AppDriver method happens at very end
       public void tearDownTest() {
           DriverController.instance.stopAppDriver();
       }
   }

Editing the build.gradle

We will now edit our ‘build.gradle’ file so that any tasks of type ‘test’ get run in 2 parallel forks. We will then define our test task to use TestNG and to include all of our TestRunner classes (via a glob pattern)

1. In IntelliJ, open up the ‘build.gradle’ file and edit it to match below…

   apply plugin: 'java'
   apply plugin: 'maven'
   
   group = 'com.testifyqa.appiumandroid'
   version = '1.0-SNAPSHOT'
   
   description = """"""
   
   sourceCompatibility = 8.0
   targetCompatibility = 8.0
   tasks.withType(JavaCompile) {
   	options.encoding = 'UTF-8'
   }
   
   repositories {
           
        maven { url "http://repo.maven.apache.org/maven2" }
   }
   dependencies {
       compile(group: 'info.cukes', name: 'cucumber-testng', version:'1.2.5') {
   exclude(module: 'junit')
       }
       compile group: 'net.masterthought', name: 'cucumber-reporting', version:'3.8.0'
       compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version:'3.12.0'
       compile group: 'io.appium', name: 'java-client', version:'6.1.0'
       compile group: 'com.google.guava', name: 'guava', version:'25.0-jre'
       compile group: 'com.google.code.gson', name: 'gson', version:'2.8.4'
       compile group: 'org.apache.logging.log4j', name: 'log4j-core', version:'2.11.0'
       compile group: 'org.apache.logging.log4j', name: 'log4j-api', version:'2.11.0'
       testCompile group: 'info.cukes', name: 'cucumber-java', version:'1.2.5'
       testCompile group: 'info.cukes', name: 'cucumber-jvm-deps', version:'1.0.5'
       testCompile group: 'org.testng', name: 'testng', version:'6.9.8'
   }
   
   tasks.withType(Test) {
       systemProperties = System.getProperties()
       systemProperties.remove("java.endorsed.dirs")
       maxParallelForks = 2
   }
   
   task runTests(type: Test) {
       useTestNG() // specify that we use TestNG instead of JUnit
       include '**/*TestRunner.class'
       outputs.upToDateWhen { false }
   }

Cleaning up the CucumberHooks

Because we now startup and teardown the appropriate drivers in our TestRunner classes, we only really need the CucumberHooks when running a test locally or debugging. Therefore, we should change our tags in there to reflect this.

1. Open up the CucumberHooks class and refactor so it’s similar to below…

   package utils.hooks;
   
   import io.cucumber.java.After;
   import io.cucumber.java.Before;
   import pages.BasePage;
   import utils.appium.DriverController;
   import java.io.IOException;
   
   import static pages.Page.instanceOf;
   
   public class CucumberHooks {
   
       @Before("@DebugNexus5xOreo")
       public void beforeNexus5xOreo() throws IOException {
           DriverController.instance.startAppDriver("Nexus5xOreo","emulator-5554", 8201);
           instanceOf(BasePage.class).appFullyLaunched();
       }
   
       @Before("@DebugNexus6pMarshmallow")
       public void beforeNexus6pMarshmallow() throws IOException {
           DriverController.instance.startAppDriver("Nexus6pMarshmallow", "emulator-5556", 8202);
           instanceOf(BasePage.class).appFullyLaunched();
       }
       
       @After("@DebugNexus5xOreo, @DebugNexus6pMarshmallow")
       public void afterDevices() {
           DriverController.instance.stopAppDriver();
       }
   }

Running our Tests

1. For simplicity, remove all tags in your Feature files and place both ‘@Nexus5xOreo’ and ‘@Nexus6pMarshmallow’ tags at the top of the ‘NavigationScenarios.feature’ file, like below…

   @Nexus6pMarshmallow @Nexus5xOreo
   Feature: Navigation Scenarios
     As a user of Reddit, I can navigate around the Reddit app
   
     Scenario: 01. View the 'Home' tab whilst logged out
       Given the welcome screen has been skipped without logging in
       When I view the Home tab
       Then I see posts and information pertaining to the Home tab

2. Open up Terminal (or Command line) and change directory (cd) into your project root
3. Enter

   gradle runTests

   to run all the tests in parallel 😀