Testing

This is the Vespa Cloud Testing reference. Refer to automated deployments for context

The system test framework in com.yahoo.vespa:tenant-cd-api contains tools for runtime-dependent authentication against the Vespa deployment to test, and for endpoint discovery.

System tests

A system test tests the endpoints of a Vespa instance:

In short, it is a test that validates the functionality of a vespa application using its interfaces. Refer to an example system test in FeedAndSearchSystemTest.java

Endpoint

When running Vespa self-hosted like in the sample application, one can access endpoints directly from test code. When running on Vespa Cloud, endpoints are securer and credentials must be provided. Use the Endpoint class to implement this, reading credentials from the environment.

To test using local endpoints, e.g., in a docker container on the development machine, use

-Dvespa.test.config=/path/to/test-config.json

This file defines the endpoints for each of the clusters in services.xml, like:

{
  "localEndpoints": {
    "default": "http://localhost:8080/",
    "container": "http://localhost:8081/"
  }
}

Make sure the TestRuntime is able to start. As it will init an SSL context, make sure to remove config when running locally, in order to use a default context. Remove properties from pom.xml and IDE debug configuration:

<!--
    <dataPlaneKeyFile>data-plane-private-key.pem</dataPlaneKeyFile>
    <dataPlaneCertificateFile>data-plane-public-cert.pem</dataPlaneCertificateFile>
-->

Test it works:

$ mvn test -Dtest.categories=system -Dvespa.test.config=/path/to/test-config.json

@SystemTest

System tests are identified by the @SystemTest annotation. Run system tests by setting test.categories to "system":

$ mvn test -Dtest.categories=system

Developers can set these parameter in the IDE run configuration to debug system tests:

-Dtest.categories=system
-Dtenant=my_tenant
-Dapplication=my_app
-Dinstance=my_instance
-DapiKeyFile=/path/to/myname.mytenant.pem
-DdataPlaneCertificateFile=data-plane-public-cert.pem
-DdataPlaneKeyFile=data-plane-private-key.pem

Find an example in the mvn test. This runs a system test against an instance in the dev environment on Vespa Cloud.

Test using Vespa Cloud

The default behavior of mvn package vespa:deploy is to deploy to the dev environment, and the default behavior of mvn test -Dtest.categories=system is to run system tests against this dev deployment. The tenant and application properties (can be set in pom.xml), together with the instance property which defaults to the current user's username, determines the deployment to create or test. Read more.

System tests in Vespa Cloud pipeline

During automated tests, a fresh deployment is made to the test environment. When tests are run, the endpoints from the test deployment are used. The test deployment is empty when the test execution begins, documents must be fed as part of the system test. The same application package and Vespa runtime combination as to be deployed in production is used, however the size of each test cluster is reduced to 1 node.

When a Vespa application is built — mvn package — all Java JUnit5 tests with the @SystemTest annotation, and all their dependencies, are stored in a separate test code artifact, which is submitted to the Vespa cloud together with the application package.

Staging tests

Just like tests with the @SystemTest annotation, tests with the @StagingTest and @StagingSetup annotations are also included in the test artifact. These are run in the automated staging test job, also against a fresh deployment. The goal of a staging test, however, is not to ensure the new deployment satisfies its functional specifications, like in the system test; rather, it is to ensure the upgradeof an existing production cluster is safe, and compatible with the behaviour expected by existing clients.

import ai.vespa.hosted.cd.StagingSetup;
import org.junit.jupiter.api.Test;

@StagingSetup
class StagingSetupTest {

    @Test
    void feedAndSearch() throws IOException {
        // Feed the static staging test documents; staging clusters are always empty when setup is run.
        // Verify documents are searchable and rendered as expected, prior to upgrade.
    }

}

import ai.vespa.hosted.cd.StagingTest;
import org.junit.jupiter.api.Test;

@StagingTest
public class MiminalStagingTest {
    @Test
    public void testSearchAndFeeding() throws Exception {
        // Test code and assertions here
    }
}

A staging test may, for instance, test an upgrade from application package X to X+1, and from platform version Y to Y+1. The staging test then consists of the following steps:

1. Deploy the initial pair X, Y to the staging environment.
2. Populate the deployment with data, making it reasonably similar to a production deployment. This is done by the @StagingSetup-annotated code, which typically feeds a set of static documents.
3. Upgrade the deployment to the target pair X+1, Y+1.
4. Verify the deployment works as expected after the upgrade. This is done by the @StagingTest-annotated code.

Because the staging tests are there to verify continued service during an upgrade, it is important to hold off changes in the staging tests until new application changes are completely rolled out, and all clients updated. With a significant change, the workflow is to

1. update the application code and the system tests,
2. deploy the change,
3. update all clients, and, possibly, the documents of the application, and then
4. update the staging tests to expect the new functionality, and, possibly, its setup phase to use the new documents.

Staging tests can also be run against a dev deployment, or against a local Vespa deployment, just like system tests. Specify -Dtest.categories=staging-setup for the setup code, and -Dtest.categories=staging for the actual tests. To deploy to a certain platform version, use, e.g., mvn vespa:deploy -DvespaVersion=1.2.3.

The sizes of clusters in staging are by default reduced to 10% of the size specified in services.xml, or at least 2 nodes.

Production tests

Tests with the @ProductionTest annotation will be executed in a production test step, see deployment.xml for details. Unlike the system and staging test, the production test will for security reasons not have access to the endpoint. The production test should e.g. verify the metrics of the production deployment and not the functionality itself. The release pipeline will stop if the tests fail, but the completed regions will remain on the version that failed the tests.

  import ai.vespa.hosted.cd.ProductionTest;
  import org.junit.jupiter.api.Test;

  @ProductionTest
  class VerifyProductionTest {

      @Test
      void verifyProductionMetrics() {
          // Consult external metrics service etc.
      }
  }

The production test can be configured to run at a time later than the actual deployment, this can be expressed in deployment.xml. Example deployment.xml running production tests for a zone 3 hours after deployment has completed:

  <deployment version="1.0">
    <instance id="default">
      <prod>
        <steps>
          <region active="true">aws-us-west-2d</region>
          <delay hours="3" />
          <test>aws-us-west-2d</test>
        </steps>
      </prod>
    </instance>
  </deployment>