This is the Vespa Cloud Testing reference. Refer to automated deployments for context
The system test framework in
com.yahoo.vespa:tenant-cd-api
contains tools for runtime-dependent authentication against the Vespa deployment to test, and for endpoint discovery.
A system test tests the endpoints of a Vespa instance:
In short, it is a test that validates the functionality of a vespa application using its interfaces. Refer to an example system test in FeedAndSearchSystemTest.java
When running Vespa self-hosted like in the sample application, one can access endpoints directly from test code. When running on Vespa Cloud, endpoints are securer and credentials must be provided. Use the Endpoint class to implement this, reading credentials from the environment.
To test using local endpoints, e.g., in a docker container on the development machine, use
-Dvespa.test.config=/path/to/test-config.json
This file defines the endpoints for each of the clusters in services.xml, like:
{ "localEndpoints": { "default": "http://localhost:8080/", "container": "http://localhost:8081/" } }
Make sure the TestRuntime is able to start. As it will init an SSL context, make sure to remove config when running locally, in order to use a default context. Remove properties from pom.xml and IDE debug configuration:
<!-- <dataPlaneKeyFile>data-plane-private-key.pem</dataPlaneKeyFile> <dataPlaneCertificateFile>data-plane-public-cert.pem</dataPlaneCertificateFile> -->
Test it works:
$ mvn test -Dtest.categories=system -Dvespa.test.config=/path/to/test-config.json
System tests are identified by the
@SystemTest annotation.
Run system tests by setting test.categories
to "system":
$ mvn test -Dtest.categories=system
Developers can set these parameter in the IDE run configuration to debug system tests:
-Dtest.categories=system -Dtenant=mytenant -DapiKeyFile=/path/to/myname.mytenant.pem -DdataPlaneCertificateFile=data-plane-public-cert.pem -DdataPlaneKeyFile=data-plane-private-key.pem
Find an example in the mvn test. This runs a system test against an instance in the dev environment on Vespa Cloud.
The default behavior of mvn package vespa:deploy
is to deploy to the
dev environment,
and the default behavior of mvn test -Dtest.categories=system
is to run system tests against this dev deployment.
The tenant and application properties (can be set in pom.xml),
together with the instance property which defaults to the current user's username,
determines the deployment to create or test.
Read more.
During automated tests, a fresh deployment is made to the test environment. When tests are run, the endpoints from the test deployment are used. The test deployment is empty when the test execution begins, documents must be fed as part of the system test. The same application package and Vespa runtime combination as to be deployed in production is used, however the size of each test cluster is reduced to 1 node.
When a Vespa application is built — mvn package
—
all Java JUnit5 tests with the @SystemTest
annotation, and all their dependencies,
are stored in a separate test code artifact, which is submitted to the Vespa cloud together with the application package.
Just like tests with the @SystemTest
annotation, tests with the @StagingTest
and
@StagingSetup
annotations are also included in the test artifact.
These are run in the automated staging test job, also against a fresh deployment.
The goal of a staging test, however, is not to ensure the new deployment satisfies its functional specifications, like in the system test;
rather, it is to ensure the upgradeof an existing production cluster is safe,
and compatible with the behaviour expected by existing clients.
import ai.vespa.hosted.cd.StagingSetup; import org.junit.jupiter.api.Test; @StagingSetup class StagingSetupTest { @Test void feedAndSearch() throws IOException { // Feed the static staging test documents; staging clusters are always empty when setup is run. // Verify documents are searchable and rendered as expected, prior to upgrade. } }
import ai.vespa.hosted.cd.StagingTest; import org.junit.jupiter.api.Test; @StagingTest public class MiminalStagingTest { @Test public void testSearchAndFeeding() throws Exception { // Test code and assertions here } }
A staging test may, for instance, test an upgrade from application package X
to X+1
,
and from platform version Y
to Y+1
.
The staging test then consists of the following steps:
X, Y
to the staging environment.@StagingSetup
-annotated code, which typically feeds a set of static documents.X+1, Y+1
.@StagingTest
-annotated code.Because the staging tests are there to verify continued service during an upgrade, it is important to hold off changes in the staging tests until new application changes are completely rolled out, and all clients updated. With a significant change, the workflow is to
Staging tests can also be run against a dev deployment, or against a local Vespa deployment, just like system tests.
Specify -Dtest.categories=staging-setup
for the setup code,
and -Dtest.categories=staging
for the actual tests.
To deploy to a certain platform version, use, e.g., mvn vespa:deploy -DvespaVersion=1.2.3
.
The sizes of clusters in staging are by default reduced to 10% of the size specified in services.xml, or at least 2 nodes.
Tests with the @ProductionTest
annotation will be executed in a production test step, see deployment.xml for details.
Unlike the system and staging test, the production test will for security reasons not have access to the endpoint. The production test should e.g.
verify the metrics of the production deployment and not the functionality itself. The release pipeline will stop if the
tests fail, but the completed regions will remain on the version that failed the tests.
import ai.vespa.hosted.cd.ProductionTest; import org.junit.jupiter.api.Test; @ProductionTest class VerifyProductionTest { @Test void verifyProductionMetrics() { // Consult external metrics service etc. } }
The production test can be configured to run at a time later than the actual deployment, this can be expressed in deployment.xml. Example deployment.xml running production tests for a zone 3 hours after deployment has completed:
<deployment version="1.0"> <instance id="default"> <prod> <steps> <region active="true">aws-us-west-2d</region> <delay hours="3" /> <test>aws-us-west-2d</test> </steps> </prod> </instance> </deployment>