Security Guide

Vespa Cloud has several security mechanisms it is important for developers to understand. Vespa Cloud has two different interaction paths, Data Plane and Control Plane. Communication with the Vespa application goes through the Data Plane, while the Control Plane is used to manage Vespa tenants and applications.

The Control Plane and the Data Plane has different security mechanisms, described in this guide.

Data Plane

Data plane is protected using mutual TLS. This means the client needs to present a client TLS certificate that the Vespa application trusts. The application knows which certificate to trust because the certificate is included in the application package.

Configuration

Certificates can be created using the Vespa CLI:

$ vespa auth cert --application <tenant>.<app>.<instance>

$ vespa auth cert --application scoober.albums.default
Success: Certificate written to security/clients.pem
Success: Certificate written to $HOME/.vespa/scoober.albums.default/data-plane-public-cert.pem
Success: Private key written to $HOME/.vespa/scoober.albums.default/data-plane-private-key.pem

The certificates can be created regardless of the application existence in Vespa Cloud. One can use this command to generate security/clients.pem for an application package:

$ cp $HOME/.vespa/scoober.albums.default/data-plane-public-cert.pem security/clients.pem

Certificates can also be created using OpenSSL:

$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout key.pem -out security/clients.pem

The certificate is placed inside the application package in security/clients.pem. Make sure clients.pem is placed correctly if the certificate is created with OpenSSL, while the Vespa CLI will handle this automatically.

security/clients.pem files can contain multiple PEM encoded certificates by concatenating them. This allows you to have multiple clients with separate private keys and to rotate certificates from the old certificate to a new one without downtime.

Requests

Once the application is configured and deployed with a certificate in the application package, requests can be sent to the application. Again, the Vespa CLI can help to use the correct certificate.

$ vespa curl --application <tenant>.<app>.<instance> /ApplicationStatus

$ curl --key $HOME/.vespa/scoober.albums.default/data-plane-private-key.pem \
       --cert $HOME/.vespa/scoober.albums.default/data-plane-public-key.pem \
       "https://albums.scoober.prod-aws-us-eastc.z.vespa-app.cloud/"

Using a browser

In Vespa guides, curl is used in examples, like:

$ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem $ENDPOINT

To use a browser, install key/cert pair - this procedure is tested on macOS Catalina / Chrome 80.0.3987.132, assuming Certificate Common Name is “cloud.vespa.example” (as in the guides):

1. Install key/cert pair:

   $ security import data-plane-private-key.pem -k ~/Library/Keychains/login.keychain
   $ security import data-plane-public-cert.pem -k ~/Library/Keychains/login.keychain
   

2. In Keychain Access: With login keychain -> My Certificates category -> cloud.vespa.example, right-click and "New Identity Preference", then add the endpoint url, like https://me.my-app.mytenant1.aws-us-east-1c.dev.z.vespa-app.cloud.
3. Open the same URL in Chrome, choose the example.com certificate and allow Chrome to read the private key.

Using Postman

Many developers prefer interactive tools like Postman. The Vespa blog has an article on how to use Postman with Vespa.

Using Cloudflare Workers

See Using Cloudflare Workers with Vespa Cloud.

Data plane access control permissions

To support different permissions for clients, it is possible to limit the permissions of a client. Only read or write permissions are supported.

Request mapping

The request actions are mapped from HTTP method. The default mapping rule is:

• GET → read
• PUT, POST, DELETE → write

For /search/ this is replaced by:

• GET, POST → read

Example

Create 3 different certificates, for three different use cases:

• Serving - read
• Ingest - write
• Full access - read, write

$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout key.pem -out security/serve.pem
$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout key.pem -out security/ingest.pem
$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout key.pem -out security/full_access.pem

Notes:

• Files must be placed in the security folder inside the application package
• Certificates must be unique
• Certificate chains are currently not supported
• Files must be written using PEM encoding

Reference the certificate files from services xml using the clients element:

<container version='1.0'>
  ...
  <clients>
    <client id="serve" permissions="read">
      <certificate file="security/serve.pem"/>
    </client>
    <client id="ingest" permissions="write">
      <certificate file="security/ingest.pem"/>
    </client>
    <client id="full_access" permissions="read,write">
      <certificate file="security/full_access.pem"/>
    </client>
  </clients>
  ...
</container>

Custom request mapping

The default mapping can be changed by overriding requestHandlerSpec():

/**
 * Example overriding acl mapping of POST requests to read
 */
public class CustomAclHandler extends ThreadedHttpRequestHandler {

    private final static RequestHandlerSpec REQUEST_HANDLER_SPEC =
            RequestHandlerSpec.builder().withAclMapping(
                    HttpMethodAclMapping.standard()
                            .override(Method.POST, AclMapping.Action.READ)
                            .build())
                    .build();

    @Override
    public RequestHandlerSpec requestHandlerSpec() {
        return REQUEST_HANDLER_SPEC;
    }

Control Plane

The control plane is used to manage the Vespa applications.

There are two different ways for access the Control Plane, using vespa auth login to log in as a regular user and using Application Keys. vespa auth login is intended for developers deploying manually to dev, while Application Keys are intended for deploying applications to production, typically by a continuous build tool. See more about these two methods below.

Managing users

Tenant administrators manage user access through the Vespa Console.

Users have two different privilege levels

• Admin: Can administrate the tenants metadata and the users of the tenant.
• Developer: Can administrate the applications deployed in the tenant.

User access to Control Plane

Outside using the Vespa Console, communicating with the Control Plane is easiest with the Vespa CLI.

$ vespa auth login
Your Device Confirmation code is: ****-****

If you prefer, you can open the URL directly for verification
Your Verification URL: https://vespa.auth0.com/activate?user_code=****-****

Press Enter to open the browser to log in or ^C to quit...

Waiting for login to complete in browser ... done

Successfully logged in.

After logging in with the Vespa CLI, the CLI can be used to deploy applications. Users are logged in with the same privilege as the user described in the Vespa Console.

Application Key

If programmatic access to the Control Plane is needed, for example from a CI/CD system like GitHub Actions, the Application Key can be used.

Configuration

The Application Key can be generated in the Console from the Deployment Screen. The key is generated in the browser but the private key appears as a download in the browser. The public key can be downloaded separately from Deployment Screen. The private key is never persisted in Vespa Cloud, so it is important that the private key is kept securely. If lost, the private key is unrecoverable.

The Application Key can also be generated using the Vespa CLI.

$ vespa auth api-key -a <tenant>.<app>.<instance>

$ vespa auth api-key -a scoober.albums.default
Success: API private key written to $HOME/.vespa/scoober.api-key.pem

This is your public key:
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE5fQUq12J/IlQQdE8pWC5596S7x9f
HpPcyxCX2dXBS4aqKxnfN5HEyTkLCNGCo9HQljgLziqW1VFzshAdm3hHQg==
-----END PUBLIC KEY-----

Its fingerprint is:
91:1f:de:e3:9f:d3:21:28:1b:1b:05:40:52:72:81:4f

To use this key in Vespa Cloud click 'Add custom key' at
https://console.vespa-cloud.com/tenant/scoober/keys
and paste the entire public key including the BEGIN and END lines.

Using the application key

The Application Key can be used from the Vespa CLI to run requests again the Control Plane. Action like deploying applications to Vespa Cloud.

$ vespa deploy -z dev.aws-us-east-1c