Part 7. Parallel Test Execution (Maven / SureFire)

Parallel Test Execution (Maven)

Introduction

Right now, when we run all of our tests in our framework, they are run sequentially (e.g. one at a time).  This is arguably ok for now as we only have a total of 5 test scenarios (4 base scenarios and 1 proper test), running against only one Android device (emulator) that we setup.

But let’s imagine we have loads of tests in our framework, and those tests need testing across multiple mobile devices. If we ran all those tests against one mobile device at a time, the time it would take to complete would be relatively long.

This is where parallel test execution comes in. By using multiple forks or threads, we can run our tests against multiple mobile devices concurrently (at the same time) to reduce the total amount of time it takes to execute all the tests in the framework.

Back in the earlier days of Appium, we would need to run a separate Appium server for each mobile device we want to run in parallel. Luckily, this is no longer the case, as we can now run only one Appium server that handles multiple sessions for all of our mobile devices.

In today’s modern age, most computers have multiple processors, which we can use to our advantage to run tests on a few different forks or threads at the same time.  If we want to run on loads of forks or threads without making the computer unusable, we can scale up using something like Selenium Grid, and then scale up even further with a cloud based service like SauceLabs. We will cover both of these in later parts of this blog series.

Let’s get started 😀

Running our tests against multiple mobile devices concurrently

Due to the fact that we cannot run multiple tests against the same mobile device at the same time, our best options for parallel test execution approach are either:

  • Distribution – Feature files are distributed almost equally on all devices available during a run
  • Fragmentation – All Features are run across all available devices

Due to the currently very small size of our test framework, we will go with the Fragmentation approach.

Currently the way our framework is setup, is that in our pom.xml file (Maven), we specify the testng.xml file that contains all of our test classes.

We will be wanting to run our tests in parallel on the Cucumber level, utilising the Maven-SureFire plugin, rather than the TestNG level, due to the simple fact that it’s a lot quicker and easier to get it setup. This means we no longer need the testng.xml file!

  1. So, firstly, with the project open in IntelliJ, right-click on the ‘testng.xml’ file and select ‘Delete’ and then ‘OK’

Choosing another iOS device (simulator)

  1. Launch Xcode
  2. Advance to ‘Window’ –> ‘Devices & Simulators’
    • Select the ‘Simulators’ tab and view the list of iOS simulators available for use
      • For the purpose of this blog, we will choose iPhone X (iOS 12.1) as our second simulator

Adding another ‘iOSDriver’ for the new mobile device (simulator)

  1. With the project open in IntelliJ, open up the ‘IOSAppDriver’ class and add a new ‘public static IOSDriver’ method to load the new mobile device simulator you just chose, like below…
    public static IOSDriver loadIphoneX() throws MalformedURLException {
        AppiumServer.start();
    
        File file = new File("src");
        File fileApp = new File(file, "Artistry.app"); //set app filepath to /src/[name-of-app-file]
    
        DesiredCapabilities cap = new DesiredCapabilities();
        cap.setCapability("platformVersion", "12.1"); //set the iOS simulator version to be launched
        cap.setCapability("deviceName", "iPhone X"); //set the name of the device to be launched (should be same as AVD)
        cap.setCapability("automationName", "XCUITest"); //set the automation engine to use as XCUITest
        cap.setCapability("app", fileApp.getAbsolutePath()); //set the app to install and use as the one in the filepath specified above
    
        driver = new IOSDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap); //set the IOSDriver to an Appium session with the above DesiredCapabilities
        return driver;
    }

    The whole thing should look similar to below…

    package utils.drivers;
    
    import io.appium.java_client.ios.IOSDriver;
    import org.openqa.selenium.remote.DesiredCapabilities;
    import utils.appium.AppiumServer;
    import java.io.File;
    import java.net.MalformedURLException;
    import java.net.URL;
    
    public class IOSAppDriver {
    
        private static IOSDriver driver;
    
        public static IOSDriver loadIphone8() throws MalformedURLException {
            AppiumServer.start();
    
            File file = new File("src");
            File fileApp = new File(file, "Artistry.app"); //set app filepath to /src/[name-of-app-file]
    
            DesiredCapabilities cap = new DesiredCapabilities();
            cap.setCapability("platformVersion", "12.1"); //set the iOS simulator version to be launched
            cap.setCapability("deviceName", "iPhone 6s"); //set the name of the device to be launched (should be same as AVD)
            cap.setCapability("automationName", "XCUITest"); //set the automation engine to use as XCUITest
            cap.setCapability("app", fileApp.getAbsolutePath()); //set the app to install and use as the one in the filepath specified above
    
            driver = new IOSDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap); //set the IOSDriver to an Appium session with the above DesiredCapabilities
            return driver;
        }
    
        public static IOSDriver loadIphoneX() throws MalformedURLException {
            AppiumServer.start();
    
            File file = new File("src");
            File fileApp = new File(file, "Artistry.app"); //set app filepath to /src/[name-of-app-file]
    
            DesiredCapabilities cap = new DesiredCapabilities();
            cap.setCapability("platformVersion", "12.1"); //set the iOS simulator version to be launched
            cap.setCapability("deviceName", "iPhone X"); //set the name of the device to be launched (should be same as AVD)
            cap.setCapability("automationName", "XCUITest"); //set the automation engine to use as XCUITest
            cap.setCapability("app", fileApp.getAbsolutePath()); //set the app to install and use as the one in the filepath specified above
    
            driver = new IOSDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap); //set the IOSDriver to an Appium session with the above DesiredCapabilities
            return driver;
        }
    }
Refactoring the ‘IOSAppDriver’ class

You might notice here that our two ‘IOSDriver’ methods have a lot of duplicated code, making it not very DRY (Do not Repeat Yourself). We should fix this by instead creating a shared method where we can pass in arguments for our DesiredCapabilities.

To safely run multiple sessions in the same Appium Server, each iOS session needs to be run on a different ‘WdaLocalPort’:

  • wdaLocalPort: just as with UiAutomator2, the iOS XCUITest driver uses a specific port to communicate with WebDriverAgent running on the iOS device. It’s good to make sure these are unique for each session. Port number defaults to 8100 so good to start around/above this range.

We also HAVE TO to declare the udid for each device (even simulators), unless the deviceName and platformVersion pairings are all uniquely different (which they are, so we will exclude ‘udid’)

  1. Let’s make our generic ‘loadDriver’ method that will take in arguments that we pass in to it (A String for ‘platformVersion’, a String for ‘deviceName’, and then an int for ‘wdaLocalPort’)…
    public static IOSDriver loadDriver(String platformVersion, String deviceName, int wdaLocalPort) throws MalformedURLException {
        AppiumServer.start();
    
        File file = new File("src");
        File fileApp = new File(file, "Artistry.app");
    
        DesiredCapabilities cap = new DesiredCapabilities();
        cap.setCapability("platformVersion", platformVersion);
        cap.setCapability("deviceName", deviceName);
        cap.setCapability("wdaLocalPort", wdaLocalPort);
        cap.setCapability("automationName", "XCUITest");
        cap.setCapability("app", fileApp.getAbsolutePath());
    
        driver = new IOSDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap);
        return driver;
    }

    We can now get rid of our other two methods. The whole thing should then look like below…

    package utils.drivers;
    
    import io.appium.java_client.ios.IOSDriver;
    import org.openqa.selenium.remote.DesiredCapabilities;
    import utils.appium.AppiumServer;
    import java.io.File;
    import java.net.MalformedURLException;
    import java.net.URL;
    
    public class IOSAppDriver {
    
        private static IOSDriver driver;
    
        public static IOSDriver loadDriver(String platformVersion, String deviceName, int wdaLocalPort) throws MalformedURLException {
            AppiumServer.start();
    
            File file = new File("src");
            File fileApp = new File(file, "Artistry.app");
    
            DesiredCapabilities cap = new DesiredCapabilities();
            cap.setCapability("platformVersion", platformVersion);
            cap.setCapability("deviceName", deviceName);
            cap.setCapability("wdaLocalPort", wdaLocalPort);
            cap.setCapability("automationName", "XCUITest");
            cap.setCapability("app", fileApp.getAbsolutePath());
    
            driver = new IOSDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap);
            return driver;
        }
    }

Refactoring our AppiumServer class

Because we can now run multiple sessions in a single Appium server, we only need to startup the Appium server if it is not already running. Let’s refactor our AppiumServer class to check for this and only start it up if it is not already running 🙂

  1. Open up the ‘AppiumServer’ class in the ‘utils.appium’ package and refactor it to match below…
    package utils.appium;
    
    import io.appium.java_client.service.local.AppiumDriverLocalService;
    import io.appium.java_client.service.local.AppiumServiceBuilder;
    import io.appium.java_client.service.local.flags.GeneralServerFlag;
    
    public class AppiumServer {
    
        private static AppiumDriverLocalService service;
    
        public static void start() {
            if (checkIfServerIsRunning()) {
                return; 
            }
    
            //Build the Appium service
            AppiumServiceBuilder builder = new AppiumServiceBuilder();
            builder.withIPAddress("127.0.0.1");
            builder.withArgument(GeneralServerFlag.SESSION_OVERRIDE);
            builder.withArgument(GeneralServerFlag.LOG_LEVEL, "error");
    
            //Start the server with the builder if not already running
            service = AppiumDriverLocalService.buildService(builder);
            service.start();
        }
    
        static void stop() {
            service.stop();
        }
    
        //Check if Appium server is running
        private static boolean checkIfServerIsRunning() {
            return service != null;
        }
    }

Refactoring the DriverController class

Because we now have only one generic method that we pass arguments into for loading the Android driver sessions, we only need one generic method in the DriverController class to start the driver session  by calling the loadDriver() method with the arguments we pass into it.

  1. Open up the ‘DriverController’ class in the ‘utils.appium’ package and refactor it to look like below…
    package utils.appium;
    
    import io.appium.java_client.ios.IOSDriver;
    import org.apache.logging.log4j.LogManager;
    import org.apache.logging.log4j.Logger;
    import utils.drivers.IOSAppDriver;
    import java.io.IOException;
    
    public class DriverController {
    
        public static DriverController instance = new DriverController();
    
        IOSDriver iosDriver;
    
        private static Logger log = LogManager.getLogger(DriverController.class);
    
        public void startAppDriver(String platformVersion, String deviceName, int wdaLocalPort) throws IOException {
            if (instance.iosDriver != null) return;
            instance.iosDriver = IOSAppDriver.loadDriver(platformVersion, deviceName, wdaLocalPort);
        }
    
        public void stopAppDriver() {
            if (instance.iosDriver == null) return;
    
            try {
                instance.iosDriver.quit();
            } catch (Exception e) {
                log.error(e + ":: AndroidDriver stop error");
            }
    
            AppiumServer.stop();
            instance.iosDriver = null;
            log.debug(":: AndroidDriver stopped");
        }
    }

Forking

In order for our tests to successfully run across multiple mobile devices in parallel, we need to run multiple forks, with each fork running our tests against a particular device in one session.

It is also possible to run each session in a thread (e.g. by putting our Drivers or DriverController instance in a ThreadLocal<>), however running them in forks is easier.  Each fork uses its own JVM in contrast to Threads. Running multiple JVM’s uses up more resources of course, but is a lot quicker and easier to get started with.

TestRunners

To run separate sessions in each fork, we first need to create TestRunner classes to run all of our feature files against different specified mobile devices. Let’s do that now.

  1. In IntelliJ, right-click on the ‘TestRunner’ class and select ‘Refactor’ -> ‘Rename’
    • Rename the class to ‘Iphone8TestRunner’, and click ‘Refactor’ (and then ‘Do Refactor’ if necessary)
  2. Open up the ‘Iphone8TestRunner’ class and edit it to match below…
    import io.cucumber.testng.AbstractTestNGCucumberTests;
    import io.cucumber.testng.CucumberOptions;
    import org.testng.annotations.AfterTest;
    import org.testng.annotations.BeforeTest;
    import pages.BasePage;
    import utils.appium.DriverController;
    
    import java.io.IOException;
    
    import static pages.Page.instanceOf;
    
    @CucumberOptions(
            features = "src/test/resources/features",
            glue = {"utils.hooks", "steps"},
            tags = {"~@Ignore"},
            plugin = {"html:target/cucumber-reports/cucumber-pretty",
                    "json:target/cucumber-reports/CucumberTestReport.json",
                    "rerun:target/cucumber-reports/rerun.txt"
            })
    
    public class Iphone8TestRunner extends AbstractTestNGCucumberTests {
    
        @BeforeTest //this method gets run first 
        public void setUpTest() throws IOException { 
            DriverController.instance.startAppDriver("12.1","iPhone 8", 8101); 
            instanceOf(BasePage.class).appFullyLaunched();
        }
        
        @AfterTest // tearDown of AppDriver method happens at very end
        public void tearDownTest() {
            DriverController.instance.stopAppDriver();
        }
    }

    Now let’s make another similar TestRunner for our ‘Nexus6P’ mobile device (emulator)

  3. In IntelliJ, right-click on the ‘Nexus5xTestRunner’ class in the project explorer and select ‘Copy’
  4. Click on the ‘Iphone8TestRunner’ class in the project explorer again and press Ctrl/CMD+V to paste
  5. Rename the copied class to ‘IphoneXTestRunner’ and click OK
  6. Open up the ‘IphoneXTestRunner’ class and refactor it to match below…
    import io.cucumber.testng.AbstractTestNGCucumberTests;
    import io.cucumber.testng.CucumberOptions;
    import org.testng.annotations.AfterTest;
    import org.testng.annotations.BeforeTest;
    import pages.BasePage;
    import utils.appium.DriverController;
    
    import java.io.IOException;
    
    import static pages.Page.instanceOf;
    
    @CucumberOptions(
            features = "src/test/resources/features",
            glue = {"utils.hooks", "steps"},
            tags = {"~@Ignore"},
            plugin = {"html:target/cucumber-reports/cucumber-pretty",
                    "json:target/cucumber-reports/CucumberTestReport.json",
                    "rerun:target/cucumber-reports/rerun.txt"
            })
    
    public class IphoneXTestRunner extends AbstractTestNGCucumberTests {
    
        @BeforeTest //this method gets run first
        public void setUpTest() throws IOException {
            DriverController.instance.startAppDriver("12.1","iPhone X", 8102);
            instanceOf(BasePage.class).appFullyLaunched();
        }
    
        @AfterTest // tearDown of AppDriver method happens at very end
        public void tearDownTest() {
            DriverController.instance.stopAppDriver();
        }
    }

Editing the pom.xml

We will now edit the Maven-SureFire plugin in our pom.xml to run each of our TestRunner’s in parallel, in different forks.

Editing the ‘maven-surefire-plugin’
  1. Open up the pom.xml file and edit the ‘maven-surefire-plugin’ so it matches below…
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.14.1</version>
        <configuration>
            <forkCount>2</forkCount>
            <reuseForks>true</reuseForks>
            <includes>
                <include>**/**/*TestRunner.class</include>
            </includes>
        </configuration>
    </plugin>

Again, notice the following that we have included…

  • <forkCount> – This specifies the number of forks we run concurrently.  We have set this to 2 as we currently only have 2 TestRunners for each device session
  •  <reuseForks> – This specifies if we want to reuse forks after one has finished running a TestRunner class
  • <include> – This specifies the location and files to include in our forks.  We have used a glob pattern to specify to include all class files that end with ‘TestRunner’ within any child directory of any parent directory (relative from the project root)

Cleaning up the CucumberHooks

Because we now startup and teardown the appropriate drivers in our TestRunner classes, we only really need the CucumberHooks when running a test locally or debugging. Therefore, we should change our tags in there to reflect this.

  1. Open up the CucumberHooks class and refactor so it’s similar to below…
    package utils.hooks;
    
    import io.cucumber.java.After;
    import io.cucumber.java.Before;
    import pages.BasePage;
    import utils.appium.DriverController;
    import java.io.IOException;
    
    import static pages.Page.instanceOf;
    
    public class CucumberHooks {
    
        @Before("@DebugIphone8")
        public void beforeNexus5xOreo() throws IOException {
            DriverController.instance.startAppDriver("12.1", "iPhone 8", 8101);
            instanceOf(BasePage.class).appFullyLaunched();
        }
    
        @Before("@DebugIphoneX")
        public void beforeNexus6pMarshmallow() throws IOException {
            DriverController.instance.startAppDriver("12.1", "iPhone X", 8102);
            instanceOf(BasePage.class).appFullyLaunched();
        }
        
        @After("@DebugIphone8, @DebugIphoneX")
        public void afterDevices() {
            DriverController.instance.stopAppDriver();
        }
    }

Running our Tests

  1. For simplicity, remove all tags in your Feature files and place both ‘@Nexus5xOreo’ and ‘@Nexus6pMarshmallow’ tags at the top of the ‘NavigationScenarios.feature’ file, like below…
    @Iphone8 @IphoneX
    Feature: Navigation Scenarios
      As a user of Reddit, I can navigate around the Reddit app
    
      Scenario: 01. View the 'Home' tab whilst logged out
        Given the welcome screen has been skipped without logging in
        When I view the Home tab
        Then I see posts and information pertaining to the Home tab
  2. Open up Terminal (or Command line) and change directory (cd) into your project root
  3. Enter
    mvn test

    to run all the tests in parallel 😀