Published on

Cypress vs Playwright – Clash of the Titans: CI/CD integration (5)

Authors

In this series of articles, I am going to dive deep into the capabilities of two popular e2e testing frameworks: Cypress and Playwright. The comparison will cover technical specifications, documentation, setup, features, CI/CD integration, etc. The goal is to uncover the main differences and pros/cons of each framework and potentially help anyone who wants to make a decision between these e2e testing frameworks.

Cypress CI/CD integration

The most customizable way to run Cypress tests on a CI/CD pipeline is to install it on your own Linux machine (Debian/Ubuntu or CentOS). The list of (system) dependencies that need to be installed can be found on the Cypress website: https://docs.cypress.io/guides/continuous-integration/introduction#Dependencies. Once this is done, Node.js needs to be installed (package.json will do the rest), and the desired browsers need to be installed. However, there if you want to simplify the process, Cypress provides three types of Docker images (https://hub.docker.com/u/cypress):

  • cypress/base – image with operating system dependencies
  • cypress/browsers – image with operating system dependencies and browsers
  • cypress/included – image with operating system dependencies, browsers and Cypress

For even more environment-specific guides, Cypress documentation provides examples for a variety of CI/CD tools (https://docs.cypress.io/guides/continuous-integration/ci-provider-examples). Besides examples, there are in-depth guides for the following providers:

  • CircleCI
  • GitHub Actions
  • BitBucket
  • GitLab CI
  • AWS CodeBuild

In the following section, we are going to see an example of GitHub actions integration for Cypress Real World App tests from previous posts of this series.

Cypress GitHub Actions integration

The most basic workflow for running Cypress tests would look something like this:

name: Cypress Tests
on: [push]
  jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
      uses: actions/checkout@v2
      - name: Cypress run
      uses: cypress-io/github-action@v4.2.0

The workflow will do the following:

  • the job will be triggered on every push to the repository
  • the latest version of Ubuntu OS will be the running machine operating system
  • default (Electron) browser will be used
  • the following steps will be taken:
  • code will be checkout to the test machine
  • tests will run
  • dependencies will be installed via npm ci
  • check and verify Cypress installation
  • run tests

This workflow can be further extended to the following:

name: Cypress Tests 2
on:
  [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
        # ...specific container can be used:
         container: cypress/browsers:node12.13.0-chrome78-ff70
        steps:
          - name: Checkout
            uses: actions/checkout@v2
          - name: Cypress run
            uses: cypress-io/github-action@v4.2.0
            with:
              command: npm run cy:run
            # ...or
            # browser: chrome
            # spec: cypress/e2e/real-world-app/\*
            # ...env variables can be added as well
            env:
              CYPRESS_host: localhost
              CYPRESS_port: 3000

The following additions were made to the previous .yml:

  • tests will run inside the container, and the image is specified
  • browser and spec files are specified in two ways:
  • using the command from the package.json that is already configured
  • by setting browser and spec fields
  • necessary environment variables defined

The last step would be handling the test artifacts: screenshots, videos, reports, etc. Artifacts should be copied and served on a dedicated machine to enable quick access and test troubleshooting.

Test parallelization in Cypress

Cypress Dashboard

First citizen test parallelization support in Cypress is provided through the paid service called Cypress Dashboard (https://dashboard.cypress.io).

Cypress Dashboard
Cypress Dashboard

To enable test parallelization via Cypress Dashboard, the following steps need to be taken:

  • have a Cypress dashboard account and organization with the dedicated project
  • add the projectID to the Cypress configuration file
  • run tests with the dedicated key, and –record and –parallel flags across several machines

During parallel runs, Cypress is using the load-balancing strategy, which means it automatically distributes the load across CI machined in order to minimize test execution time. Besides parallel test execution, Cypress is providing a lot of useful insights for a current test run, like distribution visualization, saved time, recommendations, etc.

Sorry Cypress

Sorry Cypress is a third-party tool for the parallel execution of Cypress tests. It comes in two versions:

  • https://docs.sorry-cypress.dev – free, self-hosted service that is used to run Cypress tests in parallel on your own infrastructure
  • https://currents.dev – alternative to Cypress Dashboard, with lower price and limited capabilities (comparing to Cypress Dashboard)

If you are interested in Sorry Cypress, I recommend reading my article that covers this topic in more depth: “Sorry Cypress” – a free (partial) alternative to Cypress Dashboard

Self-made

Since Cypress provides pre-installed Docker images, there is always an option to create your own test parallelization solution. From a high-level perspective, this would require the following logic:

  • implementation of a test distribution mechanism – this mechanism would distribute tests based on the defined logic
  • implementation of results/artifacts aggregation – this should enable a smooth troubleshooting experience without the need to dig through several sources of test results

Playwright CI/CD integration

Users who decide to install Playwright and dependencies on their own can use the npx playwright install --with-deps command to install Playwright together with all necessary dependencies and all supported browsers. For a more fine-grained installation process, a user can do separate installations of the Playwright library, test runner, browsers, etc.

Cypress team also provides the Docker images that can be used for CI integration. There are Docker images on the docker hub for every supported programming language, but users are also encouraged to modify and used dockerfiles as well.

When it comes to other CI providers, Playwright provides shallow information on how can be integrated into the following CI tools:

  • GitHub Actions
  • Azure Pipelines
  • CircleCI
  • Jenkins
  • Bitbucket Pipelines
  • GitLab CI

In the following section, we are going to see an example of GitHub actions integration for Cypress Real World App tests from previous posts of this series.

Playwright GitHub Actions integration

The Playwright provides good support for GitHub actions integration, both in terms of the documentation and predefined workflows that can be created during setup. The default playwright.yml workstream that can be created via the init command looks like this:

name: Playwright Tests
on:
  push:
    branches: [main, master]
  pull_request:
    branches: [main, master]
  jobs:
    test:
      timeout-minutes: 60
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v2
        - uses: actions/setup-node@v2
          with:
            node-version: '14.x'
        - name: Install dependencies
          run: npm ci
        - name: Install Playwright Browsers
          run: npx playwright install --with-deps
        - name: Run Playwright tests
          run: npx playwright test
        - uses: actions/upload-artifact@v2
      if: always()
      with:
      name: playwright-report
      path: playwright-report/
      retention-days: 30

Let’s strip it off a bit to get a minimal example for the cypress-vs-playwright repository:

name: Playwright Tests
on:
  workflow_dispatch:
jobs:
test:
runs-on: self-hosted
steps:
  - uses: actions/checkout@v2
  - uses: actions/setup-node@v2
    with:
      node-version: '16.x'
  - name: Install dependencies
    run: npm ci
  - name: Install Playwright Browsers
    run: npx playwright install --with-deps
  - name: Run Playwright tests
    run: npx playwright test real-world-app/
  - uses: actions/upload-artifact@v2
if: always()
with:
  name: playwright-report
  path: playwright-report/
  • action will be triggered manually (workflow_dispatch)
  • it is running on the self-hosted runner
  • setup step is checking out the code and installing Node.js
  • installing project dependencies and Playwright
  • running tests (from the real-world-app folder)
  • upload artifacts once tests are finished

Note: For a functional trace file analyzer opened from an HTML report, report HTML needs to be served, not just opened in the browser.

Test parallelization in Playwright

Unlike Cypress, the Playwright is providing a native test parallelization capability. For that reason, there are no paid service providers dedicated to test parallelization, and setup and configuration are in the user’s hands. Playwright is, by default, configured to run all files and tests within a single file in parallel. Also, it runs 4 worker processes, which means that 4 tests are executed simultaneously. These parameters can be changed in two places:

  • in the Playwright config file:
playwright.config.ts
import type { PlaywrightTestConfig } from '@playwright/test';
const config: PlaywrightTestConfig = {
// limit the number of workers on CI and locally
workers: process.env.CI ? 2 : 4,
// run tests within file in parallel
fullyParallel: true,
};
export default config;
  • in the command line (number of workers only): npx playwright test --workers=2
  • in the test file (fully parallel mode only):
example.spec.ts
import { test, Page } from '@playwright/test';
// enable fully parallel mode
test.describe.configure({ mode: 'parallel' });
test('some test', async () => {
await page.goto('https://something.com');
});

To further improve the test execution, Playwright enables users to split the execution across several test runners via sharding. This is done via command line argument:

npx playwright test --shard=1/4
npx playwright test --shard=2/4
npx playwright test --shard=3/4
npx playwright test --shard=4/4

This way, four CI machines will split the work from a single test suite.

By default, test files are executed alphabetically. Execution order can be customized via test list file – the file that will only contain describe blocks with test specs in the desired order:

test.list.ts
import { test } from '@playwright/test';
import test1 from './test1.spec.ts';
import test2 from './test2.spec.ts';
import test3 from './test3.spec.ts';
test.describe(test2);
test.describe(test1);
test.describe(test3);

Now, when test parallelization is disabled, the test list file should be specified as testMatch parameter in the Playwright config file.