🏃🏻How I run Integration Tests for my WP plugin using Lando and InstaWP

I am using a couple of really great tools to execute integration tests for my WordPress plugin:

• Lando: a Docker-based local tool to create development environments using any language and technology
• InstaWP: a sandboxing service that allows to spin a WordPress site on-demand

In this blog post I’ll share how I’m leveraging these tools, as to have the same testsuite and project configuration work against both of them (Lando during local development, InstaWP before merging the PR) without customizing the tests for each environment.

Why InstaWP and Lando

InstaWP offers an API to programmatically launch the new site, install the required plugins, and then destroy the instance, and we can use templates to have the WordPress site pre-loaded with data, and with a specific configuration of PHP and WordPress. It allows us to test our themes and plugins against an actual WordPress site, to be conveniently invoked from GitHub Actions (or any other Continuous Integration tool) before merging a Pull Request.

Preparing a new InstaWP instance with my plugin installed and activated takes around 3 minutes (since my plugin weighs 8.4 mb, and its downloading and installation slows down the process), and only then I can start executing the integration tests.

Hence, while InstaWP is ideal to collaborate on the repo, as to make sure that a team member’s code works as expected, I wouldn’t want to wait this time while I’m just working on my own.

For this reason, I also execute the integration tests against a local webserver provided via Lando. Building a Lando server with my plugin’s requirements will take over 5 minutes but, once created, I can start the same instance again in just a few seconds.

I quite like Lando because I can commit my plugin’s required configuration in the repo (defined via a yaml file), so anyone can clone the repo, execute a command, and have the same development environment ready.

The plugin being tested

My WordPress plugin is the GraphQL API for WordPress, and it is open source: leoloso/PoP.

As a side note: I am just days away (🤞🏼) from releasing the new version 0.9, after 16 months of work on over 1000 PRs from 14700 commits 🙀. If you’d like to know more about this upcoming release and be notified when it’s finally out, be welcome to watch the project in GitHub or subscribe to the newsletter (no spam, only announcements).

What must be tested

There are 2 different things that I need to test:

• The plugin’s source code
• The generated WordPress .zip file

This is because the code in these 2 sets is different:

• Files not needed for production are removed from the .zip plugin, such as the JS source code for the WP editor blocks (shipping their build folder is already enough)
• Composer dependencies must be installed and shipped within the .zip plugin.
• These dependencies are those for PROD only, so we must make sure no code under tests is being referenced.

And in my plugin’s case, there are a few additional differences:

• The source code is coded using PHP 8.1
• The code in the .zip file is transpiled down to PHP 7.1 and scoped

Putting it all together, I run my integration tests in three different combinations:

1. While developing the plugin: Using the source code, against a local webserver provided by Lando which runs PHP 8.1
2. Once I think the newly-developed code is ready: Using the generated .zip plugin (by GitHub Actions), against a different local webserver provided by Lando which runs PHP 7.1
3. Before merging the PR in GitHub Actions: Using the generated .zip plugin, against several InstaWP instances, each of them with a different configuration of PHP+WP

Importantly, all 3 combinations must receive the same inputs, and produce the same outputs, and must use the same configuration files to prepare their environments. A single test suite must work everywhere, without customizations or hacks.

In truth, I could skip the first step, as the generated plugin is all that really matters. However the first step allows me to find the bugs immediately, while developing the code. By the time I reach steps 2 and 3, I have a high confidence that all code will work (that’s why I don’t need to enable XDebug for them, as I’ll explain later on).

1st: Running Integration Tests on the PHP source code (on my development computer)

This is the stack I’m using:

• Lando
• XDebug
• Guzzle
• PHPUnit
• WP REST API
• WP-CLI
• The WordPress import & export tool
• Composer

I’ll explain why and how I’m using these, and point to the appropriate source files on the repo.

Lando

The Lando websever is hosted under webservers/graphql-api-for-wp, and it has this configuration:

name: graphql-api
recipe: wordpress
config:
  webroot: wordpress
  php: '8.1'
  config:
    php: ../shared/config/php.ini
  xdebug: true
env_file:
  - defaults.env
  - defaults.local.env
services:
  appserver:
    overrides:
      environment:
        XDEBUG_MODE: ''
      volumes:
        - >-
          ../../layers/GraphQLAPIForWP/plugins/graphql-api-for-wp:/app/wordpress/wp-content/plugins/graphql-api
        - >-
          ../../layers/API/packages/api-clients:/app/wordpress/wp-content/plugins/graphql-api/vendor/pop-api/api-clients
        - >-
          ../../layers/API/packages/api-endpoints-for-wp:/app/wordpress/wp-content/plugins/graphql-api/vendor/pop-api/api-endpoints-for-wp
        - >-
          ../../layers/API/packages/api-endpoints:/app/wordpress/wp-content/plugins/graphql-api/vendor/pop-api/api-endpoints
        # Many more mapping entries...
        # ...Code language: PHP (php)

The noteworthy elements here are:

• The local webserver will be available under https://graphql-api.lndo.site
• The common PHP configuration across all Lando webservers, under shared/config/php.ini, is defined once and referenced by all of them
• XDebug is enabled, but inactive by default; it is executed only when passing environment variable XDEBUG_TRIGGER=1 (eg: executing command XDEBUG_TRIGGER=1 vendor/bin/phpunit )
• Two files define environment variables, but while defaults.env is committed to the repo, defaults.local.env is .gitignored, so the latter contains my personal access tokens.
• The plugin code is mapped to its source code via services > appserver > overrides > volumes, so that changes to the source files are reflected immediately in the application in the webserver.

The last item is a deal breaker for me, because the plugin’s code is distributed into a multitude of independent packages, which are managed via Composer. When running composer install to install the plugin, all these packages would be normally copied under the vendor/ folder, breaking the connection between source code and code deployed to the webserver. Thanks to volume overrides, Lando will read the source files instead. (I used other tools, including Local, DevKinsta and wp-env, and none of them would allow me to map the Composer packages.)

Guzzle, PHPUnit and the WP REST API

Guzzle is a PHP library for executing HTTP requests. PHPUnit is the most popular library for executing unit tests. I use these 2 libraries to execute my integration tests, like this:

1. Execute a PHPUnit test, that uses Guzzle to send an HTTP request to the Lando webserver
2. Have the PHPUnit test analyze if the response is the expected one

In addition, the PHPUnit test can invoke WP REST API endpoints on the webserver (once again, via Guzzle) before and after running the tests, as to change some configuration on the plugin, or enable or disable some module, and assert that those modifications work as expected.

The integration tests are placed under an Integration folder, so to run my integration tests I just execute:

vendor/bin/phpunit --filter=Integration

Guzzle supports logging the user in and keeping the state throughout the tests. I can then assert that the response is different for users with the admin or contributor roles. This is accomplished by using a “cookie jar”, and sending a first HTTP request to log the user in, before executing the tests (source code):

class WithUserLoggedInTest extends TestCase
{
  public static function setUpBeforeClass(): void
  {
    parent::setUpBeforeClass();
    static::setUpWebserverRequestTests();
  }

  protected static function setUpWebserverRequestTests(): void
  {
    $webserverDomain = getenv('INTEGRATION_TESTS_WEBSERVER_DOMAIN');
    $this->cookieJar = CookieJar::fromArray([], $webserverDomain);
    $this->client = new Client(['cookies' => true]);
    
    // Log the user into WordPress, and store the cookies under `$this->cookieJar`
    $response = $this->client->request(
      'POST',
      'https://' . $webserverDomain . '/wp-login.php',
      [
        'cookies' => $this->cookieJar,
        // Pass the user credentials
        'form_params' => [
          'log' => getenv('INTEGRATION_TESTS_AUTHENTICATED_ADMIN_USER_USERNAME'),
          'pwd' => getenv('INTEGRATION_TESTS_AUTHENTICATED_ADMIN_USER_PASSWORD'),
        ],
      ]
    );

    // Make sure the user was authenticated
    if (!static::validateUserAuthenticationSuccess()) {
      throw new RuntimeException('Authentication of the admin user did not succeed');
    }
  }

  protected static function validateUserAuthenticationSuccess(): bool
  {
    foreach ($this->cookieJar->getIterator() as $cookie) {
      if (str_starts_with($cookie->getName(), 'wordpress_logged_in_')) {
        return true;
      }
    }
    return false;
  }

  // ...
}Code language: PHP (php)

Please notice that the webserver domain “graphql-api.lndo.site” is not hardcoded, but is instead retrieved via the environment variable INTEGRATION_TESTS_WEBSERVER_DOMAIN (and same for the username and password). These env vars are defined in file phpunit.xml.dist with the config for the Lando development webserver (source file):

<?xml version="1.0" encoding="UTF-8"?>
<phpunit>
  <php>
    <env name="INTEGRATION_TESTS_WEBSERVER_DOMAIN" value="graphql-api.lndo.site"/>
    <env name="INTEGRATION_TESTS_AUTHENTICATED_ADMIN_USER_USERNAME" value="admin"/>
    <env name="INTEGRATION_TESTS_AUTHENTICATED_ADMIN_USER_PASSWORD" value="admin"/>
  </php>
</phpunit>Code language: HTML, XML (xml)

But now, I can also execute the integration tests against any of the other webservers without having to modify any config file. To switch to InstaWP, I can execute the integration tests by doing:

$ INTEGRATION_TESTS_WEBSERVER_DOMAIN=${INSTAWP_INSTANCE_URL} \
  INTEGRATION_TESTS_AUTHENTICATED_ADMIN_USER_USERNAME=${INSTAWP_ADMIN_USERNAME} \
  INTEGRATION_TESTS_AUTHENTICATED_ADMIN_USER_PASSWORD=${INSTAWP_ADMIN_PASSWORD} \
  vendor/bin/phpunit --filter=Integration

This stack satisfies my requirements because, as my plugin provides a GraphQL server, interacting with the webserver via HTTP requests can already demonstrate if the plugin works as expected.

For instance, I send a GraphQL query to the single endpoint:

{
  post(by: {id: 1}) {
    title
  }
}

And then I assert that the response matches the expectation:

{
  "data": {
    "post": {
      "title": "Hello world!"
    }
  }
}Code language: JSON / JSON with Comments (json)

These are some of the use cases I’m currently testing:

• A request/response cycle for an API (example test, executing this GraphQL query and matching it against this response)
• Enabling/Disabling the single endpoint or custom endpoint (example test)
• Enabling/Disabling a module via the WP REST API, and then re-analyzing the response from the API (example test)
• Configuring a module via the WP REST API (example endpoint), and then re-analyzing the response from the API (example test)
• Enabling/Disabling a client (such as the GraphiQL client) and checking if it returns a 200 or 404 status code (example test)
• Checking that admin and non-admin users have different access permissions to the API (example test)
• Executing the query via GET or POST (example test)
• Passing URL parameters to persisted queries (example test)

This stack is not suitable for everything that can be tested. For instance, my plugin displays a module’s documentation in a modal window in the wp-admin, but I’m not testing that this modal window is indeed opened after the user clicks on the corresponding link.

If I ever decided to test this (or other similar concerns), then I’d consider introducing CodeCeption, which is better for executing and evaluating user interactions (this guide on testing WooCommerce provides some good examples), and I’d also check out if Pest offers advantages over PHPUnit (as suggested in WordPress integration tests with Pest).

WP-CLI, the WordPress import & export tool and Composer

Using WP-CLI is pretty much mandatory, as it satisfies several desired objectives, including:

• The automation of seeding data into the WordPress site
• Using a fixed set of data, always the same one for all environments

I am testing that the execution of a GraphQL query matches some expected response, and this response will depend directly on the data stored in the WordPress site. For instance, when I execute this query (source file):

{
  posts(pagination: { limit: 3 }) {
    id
    title
  }
}

The response from the API, whether executed against any of the local Lando webservers or the InstaWP instance, must be (source file):

{
  "data": {
    "posts": [
      {
        "id": 1,
        "title": "Hello world!"
      },
      {
        "id": 28,
        "title": "HTTP caching improves performance"
      },
      {
        "id": 25,
        "title": "Public or Private API mode, for extra security"
      }
    ]
  }
}Code language: JSON / JSON with Comments (json)

To manage this dataset and update it concerning new requirements, I am using the WordPress export tool to generate the file graphql-api-data.xml, which contains the set of data that I want to test. This method is very practical as it allows me to use the WordPress editor to create the data (which for my plugin mainly comes from a handful of CPTs), and the resulting WXR data file will be committed to the repo.

Then I use WP-CLI to import this file into the Lando webserver (source code):

wp import /app/assets/graphql-api-data.xml --authors=create --path=/app/wordpressCode language: JavaScript (javascript)

To trigger the execution of this command, I could have Lando execute it automatically (under services > appserver > run in the Lando configuration file), but I prefer to satisfy it instead as a Composer script, stored in composer.json. This has the advantage that I can then invoke the script at will, and that it acts as documentation of all scripts I can execute.

For instance, script “build-server” builds the Lando webserver, and then invokes script “install-site” which executes script setup.sh with the above WP-CLI command and several other things:

{
  "scripts": {
    "build-server": [
      "lando init --source remote --remote-url https://wordpress.org/latest.tar.gz --recipe wordpress --webroot wordpress --name graphql-api-for-prod",
      "lando start",
      "@install-site"
    ],
    "install-site": "lando composer install-site-within-container",
    "install-site-within-container": "/bin/sh /app/setup/setup.sh"
  }
}Code language: JSON / JSON with Comments (json)

If I run the integration test and cancel it before it is completed, it may alter the database in a way that breaks the upcoming test runs (for instance, executing the mutation updatePost needs to be reverted immediately after). When this happens, since the local Lando webserver is a permanent single instance (as opposed to InstaWP, which is created and destroyed per test run), I need to regenerate the data into the original state. I could rebuild the webserver, but that takes a bit of time. Instead, I can simply reset the database via the “reset-db” script, and have it call “install-site” to re-install the WordPress site:

{
  "scripts": {
    "reset-db": [
      "lando wp db reset --yes --path=wordpress",
      "@install-site"
    ]
  }
}Code language: JSON / JSON with Comments (json)

Finally, I can avoid figuring out each time how to execute the integration tests by creating a dedicated script integration-test in Composer (source code), and then just run it as:

composer integration-test

2nd: Using Lando to run Integration Tests on the generated .zip WP plugin (on my development computer)

The Lando websever to run integration tests using the actual WordPress plugin is hosted under webservers/graphql-api-for-wp-for-prod, and its configuration is way simpler than the previous one:

name: graphql-api-for-prod
recipe: wordpress
config:
  webroot: wordpress
  php: '7.1'
  ssl: true
env_file:
  - defaults.local.envCode language: JavaScript (javascript)

In this case, the domain is graphql-api-for-prod.lndo.site and the PHP version is 7.1, and there’s no need to provide a mapping to the source code (because the objective is to actually test the code in the generated .zip plugin) or enable XDebug (as any debugging that is needed would already happen on step 1).

To install the plugin, I open a PR in GitHub and wait for it to build the artifact, download it and install it on the site. (I also have a more automated way, but only in a few months I can tell you about it 😉)

To run the integration tests against this webserver, I must only provide the new domain via an env var:

$ INTEGRATION_TESTS_WEBSERVER_DOMAIN=graphql-api-for-prod.lndo.site \
  vendor/bin/phpunit --filter='Integration'Code language: JavaScript (javascript)

To make it convenient, I also have a Composer script (source code):

composer prod-integration-test

The stack is the same one as before, but with one noteworthy addition:

• Monorepo (via the Monorepo Builder)

(To be clear, the monorepo is also used in the previous situation, but its use becomes more justified in this case.)

Using a monorepo is extremely useful to host the code because I’m actually building not 1 but 2 plugins (as can be seen in a GitHub Actions run):

• graphql-api.zip
• graphql-api-testing.zip

In the previous section I explained that I test the plugin for different configurations of its modules, with a new configuration being applied by invoking some dedicated WP REST API endpoint.

This REST API endpoint is needed only while testing the plugin, and I certainly don’t want to ship it for production, as it could create security hazards. So this code won’t be present in graphql-api.zip.

That’s why I also create a second plugin graphql-api-testing.zip, containing all the utilities needed to test the plugin (including the WP REST API endpoints and other helpers), which is installed in the local Lando webserver, and also in the InstaWP instance.

The code for these 2 plugins could be managed via 2 different repos, but that would be nightmarish, as their code is so tightly coupled. The monorepo makes it a breeze to manage all code in a single place yet be able to chunk it into independent plugins or packages for production.

3rd: Using InstaWP and GitHub Actions to run Integration Tests on the generated .zip WP plugin (before merging the PR)

The stack is the same as before, but replacing Lando with InstaWP, and with these additions:

• GitHub Actions
• nightly.link

WP-CLI and InstaWP

Concerning the seeding of data into the WordPress site, InstaWP offers access to SSH, and consequently to WP-CLI, only at the Pro tier. As I’m still on the free tier, I don’t have access to WP-CLI, and so I’m editing my InstaWP templates (which are snapshots of WP sites containing all data and server configuration) manually:

• Using the WordPress import tool to seed the data from graphql-api-data.xml
• Updating the permalinks in the wp-admin
• Adding needed consts to wp-config.php using the Code editor
• Updating the registration date of the users (needed for my tests) directly in the database (😱)

This is not ideal, as it limits my ability to automate the whole process, and it breaks my requirement that the same configuration files must be used to set-up both the Lando and InstaWP environments. However I only have a handful of templates, so that creating them manually just once, and then updating them only every now and then, is not soooooo painful, and I still use the configuration files in guiding me on what changes must be applied manually, so they are still valuable.

(I do plan to upgrade to one of InstaWP’s paid plans. But other than having access to WP-CLI, I am unsure if I need all the features from the Pro tier, and so the cheaper Personal tier may be more justifiable for my needs. I still need to find out.)

GitHub Actions

The WordPress .zip plugin is generated as an artifact by GitHub Actions when opening a PR, via workflow generate_plugins.yml. (I described how this workflow works in 😎 Using GitHub actions to release a WordPress plugin.)

Once this workflow is completed, workflow integration_tests.yml is triggered, and will perform these actions:

• Obtain the URLs of the generated artifacts (for graphql-api.zip and graphql-api-testing.zip) from GitHub Actions
• Generate a URL to access the artifacts via nightly.link (explained in next section)
• Launch a matrix of GitHub Action runners
• [On each runner] Launch an InstaWP instance, spinning it from a pre-defined template that uses some combination of WP and PHP
• Install the generated artifacts in the WordPress site (via param ARTIFACT_URL)
• Wait a bit of time, to make sure the instance is ready (I’m currently guessing how much time to wait)
• Retrieve the site URL and admin credentials from the Lando instance, and set them as environment variables
• Execute the integration tests
• Destroy the InstaWP instance

This is the (slightly shortened) workflow:

name: Integration tests
on:
  workflow_run:
    workflows: [Generate plugins]
    types:
      - completed

jobs:
  provide_data:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    name: Retrieve the GitHub Action artifact URLs to install in InstaWP
    runs-on: ubuntu-latest
    steps:  
      - name: Retrieve artifact URLs from GitHub workflow
        uses: actions/github-script@v6
        id: artifact-url
        with:
          script: |
            const allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
              owner: context.repo.owner,
              repo: context.repo.repo,
              run_id: context.payload.workflow_run.id,
            });
            // Use Nightly Link as it allows InstaWP to access the artifacts, i.e. without having to be logged-in to GitHub
            // @see https://nightly.link
            // Allow installing additional plugins, set via the monorepo configuration
            const artifactURLs = allArtifacts.data.artifacts.map((artifact) => {
              return artifact.url.replace('https://api.github.com/repos', 'https://nightly.link') + '.zip'
            }).concat(${{ steps.input_data.outputs.additional_integration_test_plugins }});
            return artifactURLs.join(',');
          result-encoding: string

      - id: output_data
        run: |
          echo "instawp_config_entries=$(vendor/bin/monorepo-builder instawp-config-entries-json --config=config/monorepo-builder/instawp-config-entries-json.php)" >> $GITHUB_OUTPUT

    outputs:
      artifact_url: ${{ steps.artifact-url.outputs.result }}
      instawp_config_entries: ${{ steps.output_data.outputs.instawp_config_entries }}

  process:
    name: Launch InstaWP site from template "${{ matrix.instaWPConfig.templateSlug }}" and execute integration tests against it
    needs: provide_data
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        instaWPConfig: ${{ fromJson(needs.provide_data.outputs.instawp_config_entries) }}
    steps:
      - name: Create InstaWP instance
        uses: instawp/wordpress-testing-automation@main
        id: create-instawp
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          INSTAWP_TOKEN: ${{ secrets.INSTAWP_TOKEN }}
          INSTAWP_TEMPLATE_SLUG: ${{ matrix.instaWPConfig.templateSlug }}
          REPO_ID: ${{ matrix.instaWPConfig.repoID }}
          INSTAWP_ACTION: create-site-template
          ARTIFACT_URL: ${{ needs.provide_data.outputs.artifact_url }}

      - name: Extract InstaWP domain
        id: extract-instawp-domain        
        run: |
          instawp_domain="$(echo "${{ steps.create-instawp.outputs.instawp_url }}" | sed -e s#https://##)"
          echo "instawp-domain=$(echo $instawp_domain)" >> $GITHUB_OUTPUT

      - name: Sleep a bit to make sure InstaWP is ready
        run: sleep 120s
        shell: bash

      - name: Run tests
        run: |
          INTEGRATION_TESTS_WEBSERVER_DOMAIN=${{ steps.extract-instawp-domain.outputs.instawp-domain }} \
          INTEGRATION_TESTS_AUTHENTICATED_ADMIN_USER_USERNAME=${{ steps.create-instawp.outputs.iwp_wp_username }} \
          INTEGRATION_TESTS_AUTHENTICATED_ADMIN_USER_PASSWORD=${{ steps.create-instawp.outputs.iwp_wp_password }} \
          vendor/bin/phpunit --filter=Integration

      - name: Destroy InstaWP instance
        uses: instawp/wordpress-testing-automation@main
        id: destroy-instawp
        if: ${{ always() }}
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          INSTAWP_TOKEN: ${{ secrets.INSTAWP_TOKEN }}
          INSTAWP_TEMPLATE_SLUG: ${{ matrix.instaWPConfig.templateSlug }}
          REPO_ID: ${{ matrix.instaWPConfig.repoID }}
          INSTAWP_ACTION: destroy-siteCode language: JavaScript (javascript)

nightly.link

GitHub Actions only accepts authorized requests to download artifacts (even when the artifact is public), so the InstaWP service would need to be logged-in to GitHub to retrieve the artifacts passed under ARTIFACT_URL.

To avoid that, access to the artifacts is routed through nightly.link, a service that represents you as an authenticated user to grant access to the artifact, and the actual visitor does not need to be logged-in to GitHub anymore.

That’s all, folks

TL;DR from this blog post: I have a suite of integration tests that I can execute locally during development thanks to Lando, and before merging the PR on GitHub thanks to InstaWP, testing both the source code and the generated WordPress plugin, and I have shared how I did it.

I’m actually quite proud of being able to show this badge of honor in my repo:

This blog post went into quite a bit of detail, but there’s still plenty of other stuff going on, including:

• Setting custom headers on the response to validate some property (eg: the endpoint accessed by the GraphiQL client)
• Enabling the graphql-api-testing.zip plugin only when the website is for development
• Using configuration files instead of hardcoding data for GitHub Actions (all of those vendor/bin/monorepo-builder calls in my workflows)
• How the WP REST API endpoints modifying the plugin configuration work
• Why the response to test is provided via .json files (so the whole response must match), instead of through the PHPUnit .php files (which would allow me to test just a certain condition, such as the key errors appearing or not)

If anyone is interested in any of this, ping me on any channel (Post Status Slack, WordPress Core Slack), and I can provide some more info. (In any case, all the code is in the repo, so feel free to explore it and ask me if you don’t understand some part of the code.)

Also, what about unit tests? Yes, I am also running unit tests for my plugin, but this topic demands another article all by itself. Is this something of your interest? Let me know, and I may write a blog post about it.