Scaling up with Docker and Grid

Introduction

In previous sections, we have learnt how to run tests in parallel on the Cucumber level locally.  This is all well and good when you have a small number of tests, but let’s imagine we have a large number of tests / feature files that we want to run in parallel, this is where we would need to start thinking about strategies on how to scale up our parallel test execution.

A popular solution for this is to create a Selenium Grid within Docker containers to run tests in parallel across different browsers (nodes) in the grid.

Let’s get started 😀

Parallelisation via TestNG Test Suites

When using TestNG, each test suite can be represented by a xml file. If you’ve been following this blog series from the beginning, you should already have a testng.xml file. Let’s now extend upon this. We will have a separate XML Test Suite file for each of our feature files.

Test Runners

We will now make a test runner class for each feature file we have.

BaseTestRunner

  1. Right-click the ‘src/test/java’ directory and select ‘New’ -> ‘Java Class’
    • Name the class ‘BaseTestRunner’ and click OK
  2. Add the following in the BaseTestRunner class…
    import io.cucumber.testng.AbstractTestNGCucumberTests;
    import io.cucumber.testng.CucumberOptions;
    
    @CucumberOptions(
            features = "src/test/resources/features/BaseScenarios.feature",
            glue = {"utils.hooks", "steps"},
            tags = {"~@Ignore"},
            plugin = {"html:target/cucumber-reports/cucumber-pretty",
                    "pretty",
                    "html:target/cucumber-reports/cucumber-pretty",
                    "json:target/cucumber-reports/CucumberTestReport.json",
                    "rerun:target/cucumber-reports/rerun.txt"
            })
    public class BaseTestRunner extends AbstractTestNGCucumberTests {
    }

SearchTestRunner

  1. Right-click the ‘src/test/java’ directory and select ‘New’ -> ‘Java Class’
    • Name the class ‘SearchTestRunner’ and click OK
  2. Add the following in the SearchTestRunner class…
    import io.cucumber.testng.AbstractTestNGCucumberTests;
    import io.cucumber.testng.CucumberOptions;
    
    @CucumberOptions(
            features = "src/test/resources/features/SearchScenarios.feature",
            glue = {"utils.hooks", "steps"},
            tags = {"~@Ignore"},
            plugin = {"html:target/cucumber-reports/cucumber-pretty",
                    "pretty",
                    "html:target/cucumber-reports/cucumber-pretty",
                    "json:target/cucumber-reports/CucumberTestReport.json",
                    "rerun:target/cucumber-reports/rerun.txt"
            })
    public class SearchTestRunner extends AbstractTestNGCucumberTests {
    }

TestNG XML Test Suites

base-suite.xml

  1. In IntelliJ, right-click the src/test directory and select ‘New’ -> ‘File’
    • Name the file ‘base-suite.xml’ and click OK
  2. Add the following in the ‘base-suite.xml’ file to make it call the ‘BaseTestRunner’ class…
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
    <suite name="Product Test Suite" verbose="1" parallel="tests" thread-count="1" configfailurepolicy="continue">
        <test name="Product Acceptance Tests" annotations="JDK" preserve-order="true">
            <classes>
                <class name="BaseTestRunner"/>
            </classes>
        </test>
    </suite>

search-suite.xml

  1. In IntelliJ, right-click the src/test directory and select ‘New’ -> ‘File’
    • Name the file ‘search-suite.xml’ and click OK
  2. Add the following in the ‘search-suite.xml’ file to make it call the ‘SearchTestRunner’ class…
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
    <suite name="Product Test Suite" verbose="1" parallel="tests" thread-count="1" configfailurepolicy="continue">
        <test name="Product Acceptance Tests" annotations="JDK" preserve-order="true">
            <classes>
                <class name="SearchTestRunner"/>
            </classes>
        </test>
    </suite>

Docker

Setting up Docker

Docker is a set of platform-as-a-service products that use OS-level virtualisation to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels.

  1. Go to https://www.docker.com/products/docker-desktop and click the button to download Docker Desktop
  2. If this is your first time using Docker, you will be asked to login, click “Signup” to create a new account and follow the simple instructions on their site
  3. Once you’ve created your account, login and download Docker Desktop for your appropriate Operating System (e.g. Mac and Windows)
  4. Follow the normal process to install Docker Desktop on your Operating System
  5. Run Docker Desktop for the first time and work your way through the first time setup flow, which will be similar to below…
    • Click ‘Next’ on the “Welcome to Docker Desktop!” dialog
    • Click ‘OK’ to give privileged access to Docker Desktop (entering your password if necessary)
  6. Now wait for Docker to finish starting up and give confirmation that it is now up and running
  7. Once Docker is up and running, you should see the icon (similar to a whale) in your taskbar, right-click the icon and login to your Docker account

Docker-Compose

Docker-Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure all of your application’s services. Then, with a single command (docker-compose up), you create and start all the services from your configuration. It is just as easy to tear it all down too (docker-compose down).

Creating the Docker-Compose file

  1. With the project open in IntelliJ, right-click the project root in the Project explorer view and select ‘New’ -> ‘File’
    • Name the file ‘docker-compose.yaml’ and click OK
  2. Add the following in the docker-compose.yaml file (comments have been included to help explain what is going on)…
    version: "3.1" #version of docker-compose reference to use
    services: #begin listing our services we want to use for our Grid
      hub:
        image: selenium/hub #get the selenium grid image
        ports:
          - "4444:4444" #set the ports of the hub to 4444:4444
      chrome:
        image: selenium/node-chrome #get the chrome node image
        links:
          - hub #chrome node image requires the hub to work
        environment:
          - HUB_PORT_4444_TCP_ADDR=hub #set the hub port for the chrome node
          - HUB_PORT_4444_TCP_PORT=4444
      firefox:
        image: selenium/node-firefox #get the firefox node image
        links:
          - hub #firefox node image requires the hub to work
        environment:
          - HUB_PORT_4444_TCP_ADDR=hub #set the hub port for the firefox node
          - HUB_PORT_4444_TCP_PORT=4444
      base-tests:
        image: username/selenium-docker #change username to your docker account's username
        container_name: base-tests
        depends_on: #this test image depends on the grid being set up
          - firefox
          - chrome
        environment:
          - TEST_SUITE=base-suite.xml #run the base-suite.xml test suite
          - HUB_URL=hub
          - LOCAL=false
      search-tests:
        image: username/selenium-docker #change username to your docker account's username
        container_name: search-tests
        depends_on: #this test image depends on the grid being set up
          - firefox
          - chrome
        environment:
          - TEST_SUITE=search-suite.xml #run the search-suite.xml test suite
          - HUB_URL=hub
          - LOCAL=false

Adding the RemoteWebDrivers

Creating the ‘hubUrl’ constant

Before we create our remote web drivers that our dockerised grid will use, we need to specify the url of the hub, we will do this by adding a constant which reads from the $HUB_URL environment variable that we have in our docker-compose.yaml file

  1. Open up the ‘Settings’ class in the utils.selenium package and add the following constant…
    public static String hubUrl = System.getenv("HUB_URL");

    The whole thing should look similar to below…

    package utils.selenium;
    
    public class Settings {
    
        public static String baseUrl = "https://start.duckduckgo.com/";
        public static String weHighlightedColour = "arguments[0].style.border='5px solid blue'";
        public static String wdHighlightedColour = "arguments[0].style.border='5px solid green";
        public static String hubUrl = System.getenv("HUB_URL");
    }

Creating the ChromeRemoteWebDriver class

  1. Right-click on the utils.drivers class and select ‘New’ -> ‘Java Class’
    • Name the class ‘ChromeRemoteWebDriver’ and click OK
  2.  Make the class look like below…
    package utils.drivers;
    
    import org.openqa.selenium.chrome.ChromeOptions;
    import org.openqa.selenium.remote.RemoteWebDriver;
    
    import java.net.URL;
    
    import static utils.selenium.Settings.hubUrl;
    
    public class ChromeRemoteWebDriver {
    
        public static RemoteWebDriver loadChromeDriver(String chromeArgument) throws Exception{
            ChromeOptions options = new ChromeOptions();
            options.addArguments(chromeArgument);
            return new RemoteWebDriver(new URL("http://"+hubUrl+":4444/wd/hub"), options);
        }
    }

Creating the FirefoxRemoteWebDriver class

  1. Right-click on the utils.drivers class and select ‘New’ -> ‘Java Class’
    • Name the class ‘ChromeRemoteWebDriver’ and click OK
  2.  Make the class look like below…
    package utils.drivers;
    
    import org.openqa.selenium.firefox.FirefoxOptions;
    import org.openqa.selenium.remote.RemoteWebDriver;
    
    import java.net.URL;
    
    import static utils.selenium.Settings.hubUrl;
    
    public class FirefoxRemoteWebDriver {
    
        public static RemoteWebDriver loadFirefoxDriver(String firefoxArgument) throws Exception {
            FirefoxOptions options = new FirefoxOptions();
            options.addArguments(firefoxArgument);
            return new RemoteWebDriver(new URL("http://"+hubUrl+":4444/wd/hub"), options);
        }
    }

How to be able to still run tests locally

Because we now have some environment variables such as “LOCAL”, if we try to run a feature file locally (e.g. using the @Web) tag, it will fail with a NullPointerException due to it not being able to find the “LOCAL” environment variable. To fix this, we should add a dependency called DotEnv to our pom.xml file, which will allow us to list environment variables in a .env file and have it read from there 🙂

  1. Add the following dependency within the dependencies block of the pom.xml file …
    <dependency>
        <groupId>io.github.cdimascio</groupId>
        <artifactId>java-dotenv</artifactId>
        <version>5.1.3</version>
    </dependency>
  2. Next within IntelliJ, right-click on the project root in the Project explorer and select ‘New’ -> ‘File’
    1. Name the file ‘.env’ and press <Return>
  3. In the ‘.env’ file, add the following environment variable…
    LOCAL=true

Refactoring start() methods in DriverController class

In out Docker-Compose file, you may notice we added an environment variable called ‘LOCAL’ which we set to have a value of false, this is so our DriverController can load the RemoteWebDriver if LOCAL=false, else it will load the normal local WebDriver instance.

Refactoring the startChrome() method

  1. At the top of the DriverController class, add an instance variable underneath the existing WebDriver one that will use DotEnv to load all the environment variables listed in your ‘.env’ file…
    Dotenv dotenv = Dotenv.load();

    Also make sure the correct import has been added…

    import io.github.cdimascio.dotenv.Dotenv;
  2. Next, open up the ‘DriverController’ class in the ‘utils.selenium’ package and refactor the startChrome() method to look like below…
    public void startChrome(String arg) throws Exception {
        if(instance.webDriver != null) return;
        if (System.getenv("LOCAL")==null) {
            String local = dotenv.get("LOCAL");
            assert local != null;
            if (local.equals("true")) {
                instance.webDriver = ChromeWebDriver.loadChromeDriver(arg);
            }
        }
    
        else if(System.getenv("LOCAL").trim().equals("false")) {
            instance.webDriver = ChromeRemoteWebDriver.loadChromeDriver(arg);
        }
    }

Refactoring the startFirefox() method

  1. Open up the ‘DriverController’ class in the ‘utils.selenium’ package and refactor the startChrome() method to look like below…
    public void startFirefox(String arg) throws Exception {
        if(instance.webDriver != null) return;
        if (System.getenv("LOCAL")==null) {
            String local = dotenv.get("LOCAL");
            assert local != null;
            if (local.equals("true")) {
                instance.webDriver = FirefoxWebDriver.loadFirefoxDriver(arg);
            }
        }
    
        else if(System.getenv("LOCAL").trim().equals("false")) {
            instance.webDriver = FirefoxRemoteWebDriver.loadFirefoxDriver(arg);
        }
    }

    The whole class should look like below with the new imports etc. added…

    package utils.selenium;
    
    import io.github.cdimascio.dotenv.Dotenv;
    import org.apache.logging.log4j.LogManager;
    import org.apache.logging.log4j.Logger;
    import org.openqa.selenium.WebDriver;
    import utils.drivers.ChromeRemoteWebDriver;
    import utils.drivers.ChromeWebDriver;
    import utils.drivers.FirefoxRemoteWebDriver;
    import utils.drivers.FirefoxWebDriver;
    
    public class DriverController {
    
        public static DriverController instance = new DriverController();
    
        WebDriver webDriver;
        Dotenv dotenv = Dotenv.load();
    
        private static Logger log = LogManager.getLogger(DriverController.class.getName());
    
        public void startChrome(String arg) throws Exception {
            if(instance.webDriver != null) return;
            if (System.getenv("LOCAL")==null) {
                String local = dotenv.get("LOCAL");
                assert local != null;
                if (local.equals("true")) {
                    instance.webDriver = ChromeWebDriver.loadChromeDriver(arg);
                }
            }
    
            else if(System.getenv("LOCAL").trim().equals("false")) {
                instance.webDriver = ChromeRemoteWebDriver.loadChromeDriver(arg);
            }
        }
    
        public void startFirefox(String arg) throws Exception {
            if(instance.webDriver != null) return;
            if (System.getenv("LOCAL")==null) {
                String local = dotenv.get("LOCAL");
                assert local != null;
                if (local.equals("true")) {
                    instance.webDriver = FirefoxWebDriver.loadFirefoxDriver(arg);
                }
            }
    
            else if(System.getenv("LOCAL").trim().equals("false")) {
                instance.webDriver = FirefoxRemoteWebDriver.loadFirefoxDriver(arg);
            }
        }
    
        public void stopWebDriver() {
            if (instance.webDriver == null) return;
    
            try
            {
                instance.webDriver.quit();
            }
            catch (Exception e)
            {
                log.error(e + "::WebDriver stop error");
            }
    
            instance.webDriver = null;
            log.debug(":: WebDriver stopped");
        }
    }
  2. Next, go in to your ‘CucumberHooks’ class in the ‘utils.hooks’ package and make sure all the hook methods throw the necessary exceptions, like below…
    package utils.hooks;
    
    import cucumber.api.java.After;
    import cucumber.api.java.Before;
    import utils.selenium.DriverController;
    
    import java.io.FileInputStream;
    import java.io.FileNotFoundException;
    import java.io.IOException;
    import java.util.Properties;
    
    public class CucumberHooks {
    
        @Before("@Web")
        public void beforeWeb() throws Exception {
            Properties browserProps = new Properties();
            browserProps.load(new FileInputStream("src/test/resources/config.properties"));
    
            String browser = browserProps.getProperty("browserName");
    
            if (browser.equalsIgnoreCase("chrome")) {
                DriverController.instance.startChrome("--disable-extensions");
            }
            else if (browser.equalsIgnoreCase("firefox")) {
                DriverController.instance.startFirefox("--disable-extensions");
            }
        }
    
        @Before("@Chrome")
        public void beforeChrome() throws Exception {
            DriverController.instance.startChrome("--disable-extensions");
        }
    
        @Before("@Firefox")
        public void beforeFirefox() throws Exception {
            DriverController.instance.startFirefox("--disable-extensions");
        }
    
        @Before("@HeadlessChrome")
        public void beforeChromeHeadless() throws Exception {
            DriverController.instance.startChrome("--headless");
        }
    
        @Before("@HeadlessFirefox")
        public void beforeHeadlessFirefox() throws Exception {
            DriverController.instance.startFirefox("--headless");
        }
    
        @After
        public void stopWebDriver() {
            DriverController.instance.stopWebDriver();
        }
    }

Dockerfile

Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build, users can create an automated build that executes several command-line instructions in succession.

To make it easier when creating the Dockerfile, you can use pre-existing images shared with the community on Docker Hub.

Docker Hub

Docker Hub is a service provided by Docker for finding and sharing container images with your team. It provides the following major features: Repositories: Push and pull container images. … Builds: Automatically build container images from GitHub and BitBucket and push them to Docker Hub.

Creating the Dockerfile

  1. With the project open in IntelliJ, right-click the project root in the Project explorer view and select ‘New’ -> ‘File’
    • Name the file ‘Dockerfile’ and click OK
  2. Add the following into the Dockerfile (comments have been included above each line to help explain what we are doing)…
    #Initialise a new build stage and sets the Base Image for subsequent instructions.
    #As such, a valid Dockerfile must start with a FROM instruction. The image can be any valid image.
    #Here we are getting an image running Linux Alpine OS which has Java JDK 8 on it
    FROM openjdk:8-jdk-alpine
    
    #Add a bash shell to the base image defined above
    RUN apk add --no-cache bash
    
    #create a directory called 'app'
    RUN mkdir /app
    
    #The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile.
    #Here we are setting the app directory we created above as the working directory
    WORKDIR /app
    
    #Copy all files and folders from our project root in to our 'app' working directory
    COPY . /app
    
    #Run the shadowJar gradle task using the bash shell we added previously, in order to package our JAR file
    RUN ./gradlew shadowJar
    
    #An ENTRYPOINT allows you to configure a container that will run as an executable.
    #Here we are executing our JAR file via TestNG with a given xml test suite passed in from our $TEST_SUITE environment variable
    ENTRYPOINT ["/bin/sh", "-c", "java -cp 'build/libs/seleniumgradle-test-1.0-SNAPSHOT-tests.jar' org.testng.TestNG $TEST_SUITE"]
    

Building and Pushing your Docker Image

  1. Open up a Terminal / Command Prompt and change directory (cd) in to your project root…
    cd path/To/ProjectRoot
  2. Once in the project root, let’s build your image (changing <username> to your Docker account’s username)…
    docker build -t <username>/selenium-docker .
  3. Once your image is built, push the image to your Docker hub repository (again changing <username> to your Docker account’s username and also entering Docker login credentials if asked for)…
    docker push <username>/selenium-docker

Running our tests in Docker

Finally put the @Chrome tag above one feature file and the @Firefox one above the other for testing purposes.

We can now run our tests in parallel in the dockerised grid by starting up our containers with the following command…

docker-compose up

We can just as easily tear it down with the command below…

docker-compose down

Congratulations, in the next section we will start looking at how we can run all this in CI/CD with Jenkins 😀

Liked it? Take a second to support Thomas on Patreon!

Previous Article

Next Article

One Reply to “Part 9. Scaling up with Docker and Grid (Maven)”

  1. Hei Tom, I’m having a issue when building the Docker .
    RUN ./gradlew shadowJar gives me the error ” /bin/sh: ./gradlew: not found” which I think is normal since I made my repo using maven and not gradle..I tried searching google for what I should place but could you help me out ?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.