How did I run tests at least 10x faster with Playwright?

In this article, I will explain how you can speed up your tests by at least 10 times with the Playwright.

Let’s start with this first. What is a Playwright?

The playwright is a Node. js library that lets you script and automate browsers using the same API, like Chrome, Firefox, and Safari.

If you wonder why you should use Playwright, you can take a look at this link. Now let’s take a look at how we can speed up tests with Playwright with an example.

Parallelism

One of the ways we can speed up testing is, of course, to run in parallel. First, let’s open a file named example.spec.ts put 20 test cases in it, and write the following code in each test case that will wait 30 seconds for static.

await new Promise(resolve => setTimeout(resolve, 30000));

This number will help us to understand the gain we get by running parallel. The final version of our file will be as below.

test.describe("20 example tests", () => {
test('Test - 1', async ({ page }) => {
  await new Promise(resolve => setTimeout(resolve, 30000));
});

test('Test - 2', async ({ page }) => {
  await new Promise(resolve => setTimeout(resolve, 30000));
});

test('Test - 3', async ({ page }) => {
  await new Promise(resolve => setTimeout(resolve, 30000));
});
.
.
.
test('Test - 19', async ({ page }) => {
  await new Promise(resolve => setTimeout(resolve, 30000));
});

test('Test - 20', async ({ page }) => {
  await new Promise(resolve => setTimeout(resolve, 30000));
});
});

If we run these tests one by one ( yes, you guessed it right! ) it will take 10 minutes.

20 tests ran one by one takes 10 mins

10 minutes is too much for 20 test cases that will take 30 seconds each. Let’s make this a little shorter. The playwright uses worker processes that run simultaneously to run parallel. Let’s run these tests in parallel with 5 workers and see the difference.

20 tests ran with 5 workers take 2 mins

Yes, we successfully ran 20 tests in 2 minutes using 5 workers. We used 5 workers as an example here, but you can increase/decrease this until you reach the most appropriate number for your tests on your system.

So could we run faster? Will 5x acceleration be enough for us? Of course not, we need something faster. So how do we do that? Let’s take a look at Playwright’s definition of sharding.

Sharding

What if we used more than one machine, divided an equal number of test cases on each machine, and ran them in parallel? Sounds good. For this, Playwright offers us the Sharding feature.

20 tests ran with the shard

On the same machine, we divided the test cases into 5 equal parts and ran them in 4 parallel runs. If we ran these commands on 5 different machines, all test cases would have run after 30 seconds. What this means is that whole test cases that run in turns in 10 minutes can now run in 30 seconds. That’s incredible, right?

npx playwright test - shard=index/total-shard

Here you can divide your resources into as many machines as you have. I gave 5 as an example.

Gitlab Integration

Let’s create a pipeline in Gitlab and try it on 5 machines. Create a .gitlab-ci.yaml file and let’s get started.

stages:
  - test

tests:
    stage: test
    image: mcr.microsoft.com/playwright:v1.38.0-focal
    parallel: 
      matrix:
        - SHARD_INDEX: [1,2,3,4,5]
          SHARD_TOTAL: 5   
    script:
        - npm ci
        - npx playwright test --shard=$SHARD_INDEX/$SHARD_TOTAL

We use the “ npm ci “ command to install our dependencies.

Then we make the tests run by running the “npx playwright test — shard=$SHARD_INDEX/$SHARD_TOTAL” command. Let’s see what awaits us.

Pipeline

total 5 jobs

Details of the first job

As you can see here, the pipeline was completed in approximately 1 minute with 5 jobs running in parallel. In fact, the tests ran in 30 seconds and the rest of the time was for the environment to stand up, as you can see in the screenshots. In fact, it went from 10 minutes to 30 seconds.

By using Playwright workers and shards, you can run a large number of test cases in a short time. See you in my next article!