Parallel Test Execution

Introduction

Right now, when we run all of our tests in our framework, they are run sequentially (e.g. one at a time).  This is arguably ok for now as we only have a total of 5 test scenarios (4 base scenarios and 1 proper test).

But let’s imagine we have loads of tests in our framework, and those tests need testing across multiple browsers.  If we ran all these tests sequentially, the time it would take to complete would be relatively long.

This is where parallel test execution comes in. By using multiple threads, we can run our tests concurrently (at the same time) to reduce the total amount of time it takes to execute all the tests in the framework.

In today’s modern age, most computers have multiple processors, which we can use to our advantage to run tests on a few different forks or threads at the same time.  If we want to run on loads of forks or threads without making the computer unusable, we can scale up using something like Selenium Grid, and then scale up even further with a cloud based service like SauceLabs. We will cover both of these in later parts of this blog series.

Let’s get started 😀

Running our tests concurrently

Our current setup

As we have two feature files (and will have more when the test framework grows), we will run our feature files concurrently against one browser (that we can specify in a config.properties file). In a future blog part where I discuss Jenkins, we can schedule different builds to run our tests in parallel against different specified browsers (by changing the property value for each job).

Currently the way our framework is setup, is that in our pom.xml file (Maven) or gradle.build file (Gradle), we specify the testng.xml file that contains all of our test classes.

We will be wanting to run our tests in parallel on the Cucumber level when running locally, rather than the TestNG level, due to the simple fact that it’s a lot quicker and easier to get it setup.

Editing the pom.xml

To run our feature files in parallel, we first need to create TestRunner classes for each of our feature files. This is not very DRY (Do not Repeat Yourself) and would be time-consuming to do manually if we had loads of features. Luckily, this is where the cucumber-jvm-parallel-plugin comes into play. We can use the plugin to automatically generate TestRunner classes for all of our Feature files on the fly.

Maven Plugins

Adding the ‘cucumber-jvm-parallel-plugin’

1. Open up the pom.xml file and add the following plugin within the <plugins> tags (which is within the <build> tags)…

   <plugin>
       <groupId>com.github.temyers</groupId>
       <artifactId>cucumber-jvm-parallel-plugin</artifactId>
       <version>5.0.0</version>
       <executions>
           <execution>
               <id>generateRunners</id>
               <phase>generate-test-sources</phase>
               <goals>
                   <goal>generateRunners</goal>
               </goals>
               <configuration>
                   <glue>
                       <package>utils.hooks</package>
                       <package>steps</package>
                   </glue>
                   <featuresDirectory>src/test/resources/features/</featuresDirectory>
                   <cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
                   <useTestNG>true</useTestNG>
                   <namingScheme>pattern</namingScheme>
                   <namingPattern>TestRunner{c}</namingPattern>
                   <plugins>
                       <plugin>
                           <name>json</name>
                       </plugin>
                   </plugins>
                   <tags>~@ignore</tags>
               </configuration>
           </execution>
       </executions>
   </plugin>

Notice here some of the things we specify…

• <glue> – In here we specify the packages which contain our Cucumber hooks and our step definitions (similar to how we specified it in our original TestRunner class)
• <featuresDirectory> – The directory where all of our Feature files are located
• <cucumberOutputDir> – the directory where our report files are generated to
• <useTestNG> – A boolean flag, which only needs to be included when we use TestNG instead of JUnit (defaults to JUnit if not included)
• <namingScheme> – The type of scheme we use for our TestRunner naming conventions
•  <namingPattern> – Only needed if we include <namingScheme>, specifies the naming pattern of our TestRunner files (in our example we name them ‘TestRunner’ with a number appended at the end)
• <tags> – Whether we run Features/Scenarios for specified tags or if we ignore all tags or specified tags etc.More information can be found at https://github.com/temyers/cucumber-jvm-parallel-plugin

Editing the ‘maven-surefire-plugin’

1. Open up the pom.xml file and edit the ‘maven-surefire-plugin’ so it matches below…

   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-surefire-plugin</artifactId>
       <version>2.14.1</version>
       <configuration>
           <forkCount>2</forkCount>
           <reuseForks>true</reuseForks>
           <includes>
               <include>**/**/TestRunner*.class</include>
           </includes>
       </configuration>
   </plugin>

Again, notice the following that we have included…

• <forkCount> – This specifies the number of forks we run concurrently.  We have set this to 2 as we currently only have 2 Feature files
•  <reuseForks> – This specifies if we want to reuse forks after one has finished running a Feature file
• <include> – This specifies the location and files to include in our forks.  We have used a glob pattern to specify to include all class files that begin with ‘TestRunner’ within any child directory of any parent directory (relative from the project root)

Selecting our browser to run on via a properties file

Instead of using Cucumber tags to specify which browser to run our Features/Scenarios on, when running the full test suite we should specify the browser via a parameter we pass in. This allows us to change it more easily if we add our framework to build servers. We can do this via a simple config.properties file!

Adding a ‘config.properties’ file

1. In IntelliJ, right-click on the ‘resource’ directory and select ‘New’ -> ‘File’
   • Name the file ‘config.properties’ and click OK
2. Open up the ‘config.properties’ file and add the following…

   browserName=firefox

    (note that you could change ‘firefox’ to ‘chrome’ instead)

Adding a new Cucumber hook for @Web

We will now add a @Web cucumber hook, which will use simple if statements to read the value of our ‘browserName’ property and launch the correct browser based on that 🙂

1. Open up the ‘CucumberHooks’ class in the ‘utils.hooks’ package and add the following ‘@Before(“@Web”)’ hook, like below…

   @Before("@Web")
   public void beforeWeb() throws IOException {
       Properties browserProps = new Properties();
       browserProps.load(new FileInputStream("src/test/resources/config.properties"));
   
       String browser = browserProps.getProperty("browserName");
   
       if (browser.equalsIgnoreCase("chrome")) {
           DriverController.instance.startChrome("--disable-extensions");
       }
       else if (browser.equalsIgnoreCase("firefox")) {
           DriverController.instance.startFirefox("--disable-extensions");
       }
       else if (browser.equalsIgnoreCase("headlessChrome")) {
           DriverController.instance.startChrome("--headless");
       }
       else if (browser.equalsIgnoreCase("headlessFirefox")) {
           DriverController.instance.startFirefox("--headless");
       }
   }

Running our Tests

1. Add a ‘@Web’ tag at the top of both our feature files (example below)…

   @Web
   Feature: Search Scenarios
     As a user of Google, I want to be able to search for stuff
   
     Scenario: 01. Search and select a result
       Given I am on the search page
       When I search for "Reddit homepage"
       And I view the first result
       Then I see the Reddit homepage

2. Open up Terminal (or Command line) and change directory (cd) into your project root
3. Enter 

   mvn test

    to run all the tests in parallel

   • You could also make one Feature file run with the ‘@Firefox’ tag and the other with the ‘@Chrome’ tag if you want to see both browsers run concurrently 😀

Liked it? Take a second to support Thomas on Patreon!