Testing
This is the Vespa Cloud Testing reference. Refer to Automated Deployments for context
The system test framework in
com.yahoo.vespa:tenant-cd-api
contains tools for runtime-dependent authentication against the Vespa deployment to test, and for endpoint discovery.
System tests
A system test tests the endpoints of a Vespa instance:
In short, it is a test that validates the functionality of a vespa application using its interfaces. Refer to an example system test in FeedAndSearchSystemTest.java
Endpoint
When running Vespa self-hosted like in the sample application, one can access endpoints directly from test code. When running on Vespa Cloud, endpoints are secure, credentials must be provided. Use the Endpoint class to implement this, reading credentials from the environment.
To test using local endpoints, e.g., in a docker container on the development machine, use
-Dvespa.test.config=/path/to/test-config.jsonThis file defines the endpoints for each of the clusters in services.xml, like:
{ "localEndpoints": { "default": "http://localhost:8080/", "container": "http://localhost:8081/" } }Make sure the TestRuntime is able to start. As it will init an SSL context, make sure to remove config when running locally, in order to use a default context. Remove properties from pom.xml and IDE debug configuration:
<!-- <dataPlaneKeyFile>data-plane-private-key.pem</dataPlaneKeyFile> <dataPlaneCertificateFile>data-plane-public-cert.pem</dataPlaneCertificateFile> -->Test it works:
$ mvn test -Dtest.categories=system -Dvespa.test.config=/path/to/test-config.json
@SystemTest
System tests are identified by the @SystemTest annotation. Run system tests by settingtest.categories
to "system":
$ mvn test -Dtest.categories=systemDevelopers can hence set these parameter in the IDE run configuration to debug system tests:
-Dtest.categories=system -Dtenant=mytenant -DapiKeyFile=/path/to/myname.mytenant.pem -DdataPlaneCertificateFile=data-plane-public-cert.pem -DdataPlaneKeyFile=data-plane-private-key.pemFind an example in the mvn test step 12. This runs a system test against an instance in the dev environment on Vespa Cloud.
Test using Vespa Cloud
The default behavior of mvn package vespa:deploy
is to deploy to the
dev environment,
and the default behavior of mvn test -Dtest.categories=system
is to run system tests against this dev deployment.
The tenant and application properties (can be set in pom.xml),
together with the instance property which defaults to the current user's username,
determines the deployment to create or test.
Read more.
System tests in Vespa Cloud pipeline
During automated tests, a fresh deployment is made to the test environment. When tests are run, the endpoints from the test deployment are used. The test deployment is empty when the test execution begins, documents must be fed as part of the system test. The same application package and Vespa runtime combination as to be deployed in production is used, however the size of each test cluster is reduced to 1 node.
When a Vespa application is built — mvn package
—
all Java JUnit5 tests with the @SystemTest
annotation, and all their dependencies,
are stored in a separate test code artifact, which is submitted to the Vespa cloud together with the application package.
Staging tests
Just like tests with the @SystemTest
annotation, tests with the @StagingTest
and
@StagingSetup
annotations are also included in the test artifact.
These are run in the automated staging test job, also against a fresh deployment.
The goal of a staging test, however, is not to ensure the new deployment satisfies its functional specifications, like in the system test;
rather, it is to ensure the upgradeof an existing production cluster is safe,
and compatible with the behaviour expected by existing clients.
import ai.vespa.hosted.cd.StagingSetup; import org.junit.jupiter.api.Test; @StagingSetup class StagingSetupTest { @Test void feedAndSearch() throws IOException { // Feed the static staging test documents; staging clusters are always empty when setup is run. // Verify documents are searchable and rendered as expected, prior to upgrade. } }
import ai.vespa.hosted.cd.StagingTest; import org.junit.jupiter.api.Test; @StagingTest public class MiminalStagingTest { @Test public void testSearchAndFeeding() throws Exception { // Test code and assertions here } }
A staging test may, for instance, test an upgrade from application package X
to X+1
,
and from platform version Y
to Y+1
.
The staging test then consists of the following steps:
- Deploy the initial pair
X, Y
to the staging environment. - Populate the deployment with data, making it reasonably similar to a production deployment.
This is done by the
@StagingSetup
-annotated code, which typically feeds a set of static documents. - Upgrade the deployment to the target pair
X+1, Y+1
. - Verify the deployment works as expected after the upgrade.
This is done by the
@StagingTest
-annotated code.
- update the application code and the system tests,
- deploy the change,
- update all clients, and, possibly, the documents of the application, and then
- update the staging tests to expect the new functionality, and, possibly, its setup phase to use the new documents.
-Dtest.categories=staging-setup
for the setup code,
and -Dtest.categories=staging
for the actual tests.
To deploy to a certain platform version, use, e.g., mvn vespa:deploy -DvespaVersion=1.2.3
.
The sizes of clusters in staging are by default reduced to 10% of the size specified in services.xml, or at least 2 nodes.
Production tests
Tests with the @ProductionTest
annotation will be executed in a production test step, see deployment.xml for details.
Unlike the system and staging test, the production test will for security reasons not have access to the endpoint. The production test should e.g.
verify the metrics of the production deployment and not the functionality itself. The release pipeline will stop if the
tests fail, but the completed regions will remain on the version that failed the tests.
import ai.vespa.hosted.cd.ProductionTest; import org.junit.jupiter.api.Test; @ProductionTest class VerifyProductionTest { @Test void verifyProductionMetrics() { // Consult external metrics service etc. } }The production test can be configured to run at a time later than the actual deployment, this can be expressed in deployment.xml. Example deployment.xml running production tests for a zone 3 hours after deployment has completed:
<deployment version="1.0"> <instance id="default"> <prod> <steps> <region active="true">aws-us-west-2d</region> <delay hours="3" /> <test>aws-us-west-2d</test> </steps> </prod> </instance> </deployment>