Parallel Test Execution

Introduction

Right now, when we run all of our tests in our framework, they are run sequentially (e.g. one at a time).  This is arguably ok for now as we only have a total of 5 test scenarios (4 base scenarios and 1 proper test).

But let’s imagine we have loads of tests in our framework, and those tests need testing across multiple browsers.  If we ran all these tests sequentially, the time it would take to complete would be relatively long.

This is where parallel test execution comes in. By using multiple threads, we can run our tests concurrently (at the same time) to reduce the total amount of time it takes to execute all the tests in the framework.

In today’s modern age, most computers have multiple processors, which we can use to our advantage to run tests on a few different forks or threads at the same time.  If we want to run on loads of forks or threads without making the computer unusable, we can scale up using something like Selenium Grid, and then scale up even further with a cloud based service like SauceLabs. We will cover both of these in later parts of this blog series.

Let’s get started 😀

Running our tests concurrently

Our current setup

As we have two feature files (and will have more when the test framework grows), we will run our feature files concurrently against one browser (that we can specify in a config.properties file). In a future blog part where I discuss Jenkins, we can schedule different builds to run our tests in parallel against different specified browsers (by changing the property value for each job).

Editing the build.gradle

To run our feature files in parallel, we first need to create TestRunner classes for each of our feature files. This is not very DRY (Do not Repeat Yourself) and would be time-consuming to do manually if we had loads of features. Luckily, this is where the courgette-jvm plugin comes into play. We can use this plugin to automatically run parallel test execution on either the Features level or Scenarios level, specifying the number of concurrent threads we would like to use in a TestRunner class that uses @CourgetteOptions.

build.gradle dependencies

Adding ‘courgette-jvm’

1. In IntelliJ, open up the build.gradle file and edit the ‘dependencies’ section to match below…

   dependencies {
       compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version:'3.13.0'
       compile group: 'org.apache.logging.log4j', name: 'log4j-core', version:'2.11.1'
       compile group: 'org.apache.logging.log4j', name: 'log4j-api', version:'2.11.1'
       compile group: 'com.google.guava', name: 'guava', version:'are'
       compile group: 'com.google.code.gson', name: 'gson', version:'2.8.5'
       compile group: 'io.github.bonigarcia', name: 'webdrivermanager', version:'2.2.4'
       compile group: 'org.json', name: 'json', version:'20180130'
       testCompile group: 'io.github.prashant-ramcharan', name: 'courgette-jvm', version: '3.0.1'
   }

   Notice that we have removed our previous Cucumber and TestNG related dependencies when we added Courgette, due to the fact that Courgette includes these dependencies in its dependency chain. It is important they are removed otherwise you will run into errors due to version mismatches.

   You will also need to add ‘jcenter()’ as a repository so it can find the Courgette-JVM dependency…

   repositories {
       mavenCentral()
       jcenter()
   }

build.gradle tasks

1. Add the following two tasks in your build.gradle file. The second task will allow us to type in 

   gradle runTests

    to run our tests by finding the TestRunner and following the @CourgetteOptions we will specify…

   tasks.withType(Test) {
       systemProperties = System.getProperties()
       systemProperties.remove("java.endorsed.dirs")
   }
   
   task runTests(type: Test) {
       useTestNG() //specify that we use TestNG instead of JUnit
       include '**/TestRunner.class'
       outputs.upToDateWhen { false }
   }

   The whole build.gradle file should look similar to below…

   group = 'com.testifyqagradleframework'
   version = '1.0-SNAPSHOT'
   
   apply plugin: 'java'
   
   compileJava {
       sourceCompatibility = 1.8
       targetCompatibility = 1.8
   }
   
   repositories {
       mavenCentral()
       jcenter()
   }
   
   dependencies {
       compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version:'3.13.0'
       compile group: 'org.apache.logging.log4j', name: 'log4j-core', version:'2.11.1'
       compile group: 'org.apache.logging.log4j', name: 'log4j-api', version:'2.11.1'
       compile group: 'com.google.guava', name: 'guava', version:'25.1-jre'
       compile group: 'com.google.code.gson', name: 'gson', version:'2.8.5'
       compile group: 'io.github.bonigarcia', name: 'webdrivermanager', version:'2.2.4'
       compile group: 'org.json', name: 'json', version:'20180130'
       testCompile group: 'io.github.prashant-ramcharan', name: 'courgette-jvm', version: '3.0.1'
   }
   
   tasks.withType(Test) {
       systemProperties = System.getProperties()
       systemProperties.remove("java.endorsed.dirs")
   }
   
   task runTests(type: Test) {
       useTestNG() // specify that we use TestNG instead of JUnit
       include '**/TestRunner.class'
       outputs.upToDateWhen { false }
   }

Editing the TestRunner

Because we will now be using Courgette-JVM instead of the normal Cucumber-JVM, we need to update our TestRunner so that it uses @CourgetteOptions with @CucumberOptions inside as one of the @CourgetteOptions.

• Open up the ‘TestRunner’ class and edit it so that it matches below…

  import courgette.api.CourgetteOptions;
  import courgette.api.CourgetteRunLevel;
  import courgette.api.testng.TestNGCourgette;
  import cucumber.api.CucumberOptions;
  import org.testng.annotations.Test;
  
  @Test
  @CourgetteOptions(
          threads = 2,
          runLevel = CourgetteRunLevel.FEATURE,
          rerunFailedScenarios = true,
          showTestOutput = true,
          reportTargetDir = "build",
          cucumberOptions = @CucumberOptions(
                  features = "src/test/resources/features",
                  glue = {"utils.hooks", "steps"},
                  tags = {"@Web"},
                  plugin = {
                          "pretty",
                          "json:build/cucumber-report/cucumber.json",
                          "html:build/cucumber-report/cucumber.html"},
                  strict = true
          ))
  
  public class TestRunner extends TestNGCourgette {
  
  }

Notice some of the @CourgetteOption that we have set…

• threads – the number of concurrent threads we will be using
• runLevel – whether we want to run our tests in parallel at the Feature level or the Scenario level
• rerunFailedScenarios – A boolean flag for whether we want to rerun any scenarios that fail or not
• showTestOutput – Whether we want to show the test output
• reportTargetDir – The target directory where our Courgette generated reports will exist

The rest of the @CucumberOptions should already be familiar to us.

Making our WebDrivers thread-safe

Finally, when running Selenium tests in parallel, your Webdriver object should be thread-safe i.e. a single object can be used with multiple threads at the same time without causing problems. Let’s make both our WebDriver’s thread-safe now by putting them into ThreadLocal<>

ChromeWebDriver

1. Open up the ‘ChromeWebDriver’ class and edit it to match below…

   package utils.drivers;
   
   import io.github.bonigarcia.wdm.WebDriverManager;
   import org.openqa.selenium.WebDriver;
   import org.openqa.selenium.chrome.ChromeDriver;
   import org.openqa.selenium.chrome.ChromeDriverService;
   import org.openqa.selenium.chrome.ChromeOptions;
   
   public class ChromeWebDriver {
   
       private static ThreadLocal<WebDriver> driver = new ThreadLocal<>();
   
       private static void setupChromeDriver() {
           WebDriverManager.chromedriver().setup();
       }
   
       public static WebDriver loadChromeDriver(String chromeArgument) {
           setupChromeDriver();
   
           ChromeDriverService driverService = ChromeDriverService.createDefaultService();
   
           ChromeOptions options = new ChromeOptions();
           options.addArguments(chromeArgument);
   
           driver = ThreadLocal.withInitial(() -> new ChromeDriver(driverService, options));
           return driver.get();
       }
   }

FirefoxWebDriver

1. Open up the ‘FirefoxWebDriver’ class and edit it in the same way, like below…

   package utils.drivers;
   
   import io.github.bonigarcia.wdm.WebDriverManager;
   import org.openqa.selenium.WebDriver;
   import org.openqa.selenium.firefox.FirefoxDriver;
   import org.openqa.selenium.firefox.FirefoxOptions;
   
   public class FirefoxWebDriver {
   
       private static ThreadLocal<WebDriver> driver = new ThreadLocal<>();
   
       private static void setupFirefoxDriver() {
           WebDriverManager.firefoxdriver().setup();
       }
   
       public static WebDriver loadFirefoxDriver(String firefoxArgument) {
           setupFirefoxDriver();
   
           FirefoxOptions options = new FirefoxOptions();
           options.addArguments(firefoxArgument);
   
           driver = ThreadLocal.withInitial(() -> new FirefoxDriver(options));
           return driver.get();
       }
   }

Selecting our browser to run on via a properties file

Instead of using Cucumber tags to specify which browser to run our Features/Scenarios on, when running the full test suite we should specify the browser via a parameter we pass in. This allows us to change it more easily if we add our framework to build servers. We can do this via a simple config.properties file!

Adding a ‘config.properties’ file

1. In IntelliJ, right-click on the ‘resource’ directory and select ‘New’ -> ‘File’
   • Name the file ‘config.properties’ and click OK
2. Open up the ‘config.properties’ file and add the following…

   browserName=firefox

    (note that you could change ‘firefox’ to ‘chrome’ instead)

Adding a new Cucumber hook for @Web

We will now add a @Web cucumber hook, which will use simple if statements to read the value of our ‘browserName’ property and launch the correct browser based on that 🙂

1. Open up the ‘CucumberHooks’ class in the ‘utils.hooks’ package and add the following ‘@Before(“@Web”)’ hook, like below…

   @Before("@Web")
   public void beforeWeb() throws IOException {
       Properties browserProps = new Properties();
       browserProps.load(new FileInputStream("src/test/resources/config.properties"));
   
       String browser = browserProps.getProperty("browserName");
   
       if (browser.equalsIgnoreCase("chrome")) {
           DriverController.instance.startChrome("--disable-extensions");
       }
       else if (browser.equalsIgnoreCase("firefox")) {
           DriverController.instance.startFirefox("--disable-extensions");
       }
       else if (browser.equalsIgnoreCase("headlessChrome")) {
           DriverController.instance.startChrome("--headless");
       }
       else if (browser.equalsIgnoreCase("headlessFirefox")) {
           DriverController.instance.startFirefox("--headless");
       }
   }

Running our Tests

1. Add a ‘@Web’ tag at the top of both our feature files (example below)…

   @Web
   Feature: Search Scenarios
     As a user of Google, I want to be able to search for stuff
   
     Scenario: 01. Search and select a result
       Given I am on the search page
       When I search for "Reddit homepage"
       And I view the first result
       Then I see the Reddit homepage

2. Open up a Terminal (or Command prompt window)
3. Change directory (cd) in to the project root
4. Run the tests via the command 

   gradle runTests

    and rejoice! 😀

Liked it? Take a second to support Thomas on Patreon!