Vespa Cloud has several security mechanisms it is important for developers to understand. Vespa Cloud has two different interaction paths, Data Plane and Control Plane. Communication with the Vespa application goes through the Data Plane, while the Control Plane is used to manage Vespa tenants and applications.
The Control Plane and the Data Plane has different security mechanisms, described in this guide.
Data plane is protected using mutual TLS. This means the client needs to present a client TLS certificate that the Vespa application trusts. The application knows which certificate to trust because the certificate is included in the application package.
security/clients.pem
or src/main/application/security/clients.pem
are required files.
Find deployable examples in CI-CD.
Certificates can be created using the Vespa CLI:
$ vespa auth cert --application <tenant>.<app>.<instance>
$ vespa auth cert --application scoober.albums.default
Success: Certificate written to security/clients.pem
Success: Certificate written to $HOME/.vespa/scoober.albums.default/data-plane-public-cert.pem
Success: Private key written to $HOME/.vespa/scoober.albums.default/data-plane-private-key.pem
The certificates can be created regardless of the application existence in Vespa
Cloud. One can use this command to generate security/clients.pem
for an application package:
$ cp $HOME/.vespa/scoober.albums.default/data-plane-public-cert.pem security/clients.pem
Certificates can also be created using OpenSSL:
$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout key.pem -out security/clients.pem
The certificate is placed inside the application package in
security/clients.pem. Make sure
clients.pem
is placed correctly if the certificate is created with OpenSSL,
while the Vespa CLI will handle this automatically.
security/clients.pem
files can contain multiple PEM encoded certificates by
concatenating them. This allows you to have multiple clients with separate
private keys and to rotate certificates from the old certificate to a new one
without downtime.
Once the application is configured and deployed with a certificate in the application package, requests can be sent to the application. Again, the Vespa CLI can help to use the correct certificate.
$ vespa curl --application <tenant>.<app>.<instance> /ApplicationStatus
$ curl --key $HOME/.vespa/scoober.albums.default/data-plane-private-key.pem \
--cert $HOME/.vespa/scoober.albums.default/data-plane-public-key.pem \
"https://albums.scoober.prod-aws-us-eastc.z.vespa-app.cloud/"
In Vespa guides, curl is used in examples, like:
$ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem $ENDPOINT
To use a browser, install key/cert pair - this procedure is tested on macOS Catalina / Chrome 80.0.3987.132, assuming Certificate Common Name is “cloud.vespa.example” (as in the guides):
$ security import data-plane-private-key.pem -k ~/Library/Keychains/login.keychain $ security import data-plane-public-cert.pem -k ~/Library/Keychains/login.keychain
Many developers prefer interactive tools like Postman. The Vespa blog has an article on how to use Postman with Vespa.
See Using Cloudflare Workers with Vespa Cloud.
To support different permissions for clients, it is possible to limit the permissions of a client. Only read
or write
permissions are supported.
The request actions are mapped from HTTP method. The default mapping rule is:
read
write
For /search/
this is replaced by:
read
Create 3 different certificates, for three different use cases:
read
write
read, write
$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout key.pem -out security/serve.pem
$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout key.pem -out security/ingest.pem
$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout key.pem -out security/full_access.pem
Notes:
Reference the certificate files from services xml using the clients
element:
<container version='1.0'>
...
<clients>
<client id="serve" permissions="read">
<certificate file="security/serve.pem"/>
</client>
<client id="ingest" permissions="write">
<certificate file="security/ingest.pem"/>
</client>
<client id="full_access" permissions="read,write">
<certificate file="security/full_access.pem"/>
</client>
</clients>
...
</container>
The default mapping can be changed by overriding requestHandlerSpec()
:
/**
* Example overriding acl mapping of POST requests to read
*/
public class CustomAclHandler extends ThreadedHttpRequestHandler {
private final static RequestHandlerSpec REQUEST_HANDLER_SPEC =
RequestHandlerSpec.builder().withAclMapping(
HttpMethodAclMapping.standard()
.override(Method.POST, AclMapping.Action.READ)
.build())
.build();
@Override
public RequestHandlerSpec requestHandlerSpec() {
return REQUEST_HANDLER_SPEC;
}
The control plane is used to manage the Vespa applications.
There are two different ways for access the Control Plane, using
vespa auth login
to log in as a regular user and using Application Keys.
vespa auth login
is intended for developers deploying manually to dev, while
Application Keys are intended for deploying applications to production,
typically by a continuous build tool. See more about these two methods below.
Tenant administrators manage user access through the Vespa Console.
Users have two different privilege levels
Outside using the Vespa Console, communicating with the Control Plane is easiest with the Vespa CLI.
$ vespa auth login
Your Device Confirmation code is: ****-****
If you prefer, you can open the URL directly for verification
Your Verification URL: https://vespa.auth0.com/activate?user_code=****-****
Press Enter to open the browser to log in or ^C to quit...
Waiting for login to complete in browser ... done
Successfully logged in.
After logging in with the Vespa CLI, the CLI can be used to deploy applications. Users are logged in with the same privilege as the user described in the Vespa Console.
If programmatic access to the Control Plane is needed, for example from a CI/CD system like GitHub Actions, the Application Key can be used.
The Application Key can be generated in the Console from the Deployment Screen. The key is generated in the browser but the private key appears as a download in the browser. The public key can be downloaded separately from Deployment Screen. The private key is never persisted in Vespa Cloud, so it is important that the private key is kept securely. If lost, the private key is unrecoverable.
The Application Key can also be generated using the Vespa CLI.
$ vespa auth api-key -a <tenant>.<app>.<instance>
$ vespa auth api-key -a scoober.albums.default
Success: API private key written to $HOME/.vespa/scoober.api-key.pem
This is your public key:
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE5fQUq12J/IlQQdE8pWC5596S7x9f
HpPcyxCX2dXBS4aqKxnfN5HEyTkLCNGCo9HQljgLziqW1VFzshAdm3hHQg==
-----END PUBLIC KEY-----
Its fingerprint is:
91:1f:de:e3:9f:d3:21:28:1b:1b:05:40:52:72:81:4f
To use this key in Vespa Cloud click 'Add custom key' at
https://console.vespa-cloud.com/tenant/scoober/keys
and paste the entire public key including the BEGIN and END lines.
The Application Key can be used from the Vespa CLI to run requests again the Control Plane. Action like deploying applications to Vespa Cloud.
$ vespa deploy -z dev.aws-us-east-1c