Automated Deployments

The Vespa cloud automates deployments to production using a safe procedure where changes are first tested, then rolled out to production zones, with optional verification tests and delays. The deployment process for an application is specified in deployment.xml, placed alongside services.xml in the application source.

You can see status, access logs, and trigger and pause jobs in the console.

The deployment process, including tests is reused when the Vespa Cloud upgrades your application. You can control at what times versions and application revisions are allowed to deploy in deployment.xml.

Continuous Deployment

Getting to Production details the steps needed to prepare and submit a production deployment. These steps should be executed by a build job triggered regularly or on changes to your application package source repo to get continuous deployment.

Refer to this example using GitHub actions.

Notice the trick of passing sourceUrl to point to the source revision. This is displayed in the console and makes it possible to keep track of what exactly is being deployed.

Deployment keys

Deployment jobs use the application API key. You can create new Application API keys in the console and store them as a secret in the repository, see the GitHub actions example.

Some services like Travis CI do not accept multi-line values for Environment Variables in Settings. A workaround is to use the output of

$ openssl base64 -A -a < mykey.pem && echo

in a variable, say VESPA_MYAPP_API_KEY, in Travis Settings. VESPA_MYAPP_API_KEY is exported in the Travis environment, example output:

Setting environment variables from repository settings
$ export VESPA_MYAPP_API_KEY=[secure]

Then, before deploying/submitting to Vespa Cloud, regenerate the key value:

MY_API_KEY=`echo $VESPA_MYAPP_API_KEY | openssl base64 -A -a -d`

and use ${MY_API_KEY} in the deploy/submit command.

vespa:compileVersion

Vespa is backwards compatible on major versions (and major versions rarely change). This means that code compiled with an older version of Vespa APIs can always be deployed to Vespa Cloud on same major version. However, if the application package is compiled with a newer API version, then deployed to an older version currently used in production, it may fail.

To avoid this small possibility, vespa:compileVersion returns the lowest version currently running the application in production. This version is then set in vespa.compile.version when building the application package.

Deployment verification

Vespa applications may contain any number of tests that verify them as paert of deployments to production:

  • System tests verifies application functionality and are run on an empty, downscaled application instance

  • Staging tests verifies that switching to a new application or Vespa version works and involves setting up an instance of the application revision and Vespa version in production before the change, writing data, switching to the new revision and version and then verifying the system.

  • Production tests verifies that a production deployment in a zone did not have adverse effects, typically by checking metrics after a delay. If these tests fail the change will not roll out to further production zones or instances.

System and staging tests are mandatory and always run. Production tests can be configured in deployment.xml.

Vespa applications are compiled against one version of the Vespa Java artifacts, and then deployed to nodes in the cloud where the runtime Vespa version is controlled by the system. This runtime, or platform, version is also continuously updated, independently of application updates. This leads to a number of possible combinations of application packages and platform versions for each application.

Instead of a simple pipeline, Vespa deployments are orchestrated such that any deployment of an application package X to a production cluster with platform version Y is preceded by system and staging tests using the same version pair; and likewise for any upgrade of the platform to version Y of a production cluster running an application package X. System and staging tests therefore guard against both unfortunate changes in the application, and in the Vespa platform.

In addition to tests the Vespa Cloud has an additional safety mechanism: Potentially destructive application changes, such as removing fields, are disallowed by default to avoid accidentally damaging production. Such changes require a validation override as part of the application to ensure it is really intended.

Status and logs of ongoing tests can be found under Deployment in the application view in the console.

Developing tests

System, staging and production tests are JUnit tests. To develop the tests you can deploy the application to dev and run tests from the command line against it:

$ mvn test \
  -Dtest.categories=system \
  -DdataPlaneKeyFile=data-plane-private-key.pem -DdataPlaneCertificateFile=data-plane-public-cert.pem \
  -DapiKey="$API_KEY"

The apiKey is used to fetch the dev instance's endpoints. The data plane key and certificate pair is used by ai.vespa.hosted.cd.Endpoint to access the application endpoint. More details can be found in testing and Vespa Cloud API.

To run tests against a deployment running in Docker on localhost (instead of using dev), configure endpoint location:

{
    "localEndpoints": {
        "container": "http://localhost:8080/"
    }
}

See system tests for details. Refer to album-recommendation-selfhosted for how to create the application package.

Production deployments

Production deployments run sequentially by default, but can be configured to run in parallel. Inside each zone, Vespa itself orchestrates the deployment, such that the change is applied without disruption to read or write traffic against the application. A production deployment in a zone is complete when the change has completed on all nodes. Most changes are applied to running nodes, which makes this a fast process. If restarts are needed, these will happen automatically and safely as part of the deployment. When this is necessary deployments will take longer to complete.

Feature switches and bucket tests

With continuous deployment, it is not practical to hold off releasing a feature until it is done, test it manually until convinced it works and then release it to production. What to do instead? The answer is feature switches: release new features to production as they are developed, but include logic which keeps them deactivated until they are ready, or until they have been verified in production with a subset of users.

Bucket tests is the practice of systematically testing new features or behavior for a controlled subset of users. This is common practice when releasing new science models, as they are difficult to verify in test, but can also be used for other features.

To test new behavior in Vespa, use a combination of search chains and rank profiles, controlled by query profiles, where one query profiles correspond to one bucket. These features support inheritance to make it easy to express variation without repetition.

Some times a new feature require incompatible changes to a data field. To be able to CD such changes, it is necessary to create a new field containing the new version of the data. This costs extra resources but less than the alternative: standing up a new system copy with the new data. New fields can be added and populated while the system is live.

One way to reduce the need for incompatible changes can be decreased by making the semantics of the fields more precise. E.g., if a field is defined as the "quality" of a document, where a higher number means higher quality, a new algorithm which produces a different range and distribution will typically be an incompatible change. However, if the field is defined more precisely as the average time spent on the document once it is clicked, then a new algorithm which produces better estimates of this value will not be an incompatible change. Using precise semantics also have the advantage of making it easier to understand if the use of the data and its statistical properties are reasonable.

Integration testing

Another challenge with continuous deployment is integration testing across multiple services: Another service depends on this Vespa application for its own integration testing. There are two ways to provide this: Either create an additional application instance for testing or use test data in the production instance. Using test data in production requires that some thought is given to separating this data from the real data in queries. A separate instance gives complete isolation, but with some additional overhead, and may not produce quite as realistic testing of queries, as those will run only over the test data in the separate instance.

Deleting an application

To delete an application, remove all production instances first.

WARNING! Following these steps will remove production instances and all data within them. Data will be unrecoverable.

  1. Remove all instances in deployment.xml

    <deployment version="1.0" />
    
  2. Add or modify validation-overrides.xml, allowing Vespa Cloud to remove production deployments:

    <validation-overrides>
        <allow until="2021-03-01" comment="Removing all deployments">deployment-removal</allow>
    </validation-overrides>
    
  3. Trigger a build and submit of the application package.

  4. Delete the application in the console.

  5. Remove your CI job that builds and submits application changes.