album-recommendation-searcher
This sample app adds custom java plugin code, system and staging tests, and is an introduction into integrating with a CI/CD pipeline. It extends the Quick Start with:
- A Searcher
component for query and result processing.
- See album-recommendation-docproc for a Document Processing example.
- Custom config for the searcher, to demonstrate atomic code and config deployment.
- A System and staging test used in automated-deployments.
Prerequisites: git, Java 11, mvn 3.6.1 and openssl. See Getting Started for troubleshooting. Security notes:
- To deploy, a user API key is required, see step 4.
- To read and write to the instance's endpoint, a certificate and a private key is required, see step 2. Find more details in Data Plane, see "Client certificate".
Dev Environment
Download sample apps:
$ git clone https://github.com/vespa-engine/sample-apps.git && \ cd sample-apps/vespa-cloud/album-recommendation-searcher
Get a X.509 certificate and private key. Create a self-signed certificate / private key:
$ openssl req -x509 -nodes -days 14 -newkey rsa:4096 \ -subj "/CN=cloud.vespa.example" \ -keyout data-plane-private-key.pem -out data-plane-public-cert.pem
Add certificate to application package (it must be copied as clients.pem):
$ mkdir -p src/main/application/security && \ cp data-plane-public-cert.pem src/main/application/security/clients.pem
At console.vespa.ai, choose tenant and click Keys to generate and save the user API key. The key is saved to
$HOME/Downloads/USER.TENANTNAME.pem
. Then click "Create application" - use any name for the application.Update
tenant
andapplication
inpom.xml
— use the values entered in the console in 4.Build the app, deploy your own instance (
my-instance
here) to thedev
environment and wait for it to start — updateapiKeyFile
to be the file downloaded above:$ mvn clean package vespa:deploy -DapiKeyFile=$HOME/Downloads/USER.TENANTNAME.pem -Dinstance=my-instance
Alternatively, put the key in an environment variable:
$ API_KEY=`echo $VESPA_TEAM_API_KEY | openssl base64 -A -a -d` $ mvn clean package vespa:deploy -DapiKey="$API_KEY" -Dinstance=my-instance
Now is a good time to read automated-deployments, as first time deployments takes a few minutes. The endpoint URL is printed in the Install application section when the deployment is successful - store this (change root to actual name) for later steps and test it (the massive JSON output is expected). Also see using a browser:
$ ENDPOINT=https://root.album-rec-searcher.vespa-team.aws-us-east-1c.dev.public.vespa.oath.cloud $ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem $ENDPOINT
Feed documents:
$ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem \ -H "Content-Type:application/json" --data-binary @src/test/resources/A-Head-Full-of-Dreams.json \ $ENDPOINT/document/v1/mynamespace/music/docid/1 $ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem \ -H "Content-Type:application/json" --data-binary @src/test/resources/Love-Is-Here-To-Stay.json \ $ENDPOINT/document/v1/mynamespace/music/docid/2 $ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem \ -H "Content-Type:application/json" --data-binary @src/test/resources/Hardwired...To-Self-Destruct.json \ $ENDPOINT/document/v1/mynamespace/music/docid/3
Dump all document using visit:
$ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem \ "$ENDPOINT/document/v1/mynamespace/music/docid?wantedDocumentCount=100"
Recommend albums, send user profile in query:
$ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem \ "$ENDPOINT/search/?ranking=rank_albums&yql=select%20%2A%20from%20sources%20%2A%20where%20sddocname%20contains%20%22music%22%3B&ranking.features.query(user_profile)=%7B%7Bcat%3Apop%7D%3A0.8%2C%7Bcat%3Arock%7D%3A0.2%2C%7Bcat%3Ajazz%7D%3A0.1%7D"
Limit to albums with the term "to" in title:$ curl --cert ./data-plane-public-cert.pem --key ./data-plane-private-key.pem \ "$ENDPOINT/search/?ranking=rank_albums&yql=select%20%2A%20from%20sources%20%2A%20where%20album%20contains%20%22to%22%3B&ranking.features.query(user_profile)=%7B%7Bcat%3Apop%7D%3A0.8%2C%7Bcat%3Arock%7D%3A0.2%2C%7Bcat%3Ajazz%7D%3A0.1%7D"
Prod Environment
Vespa Cloud has support for running system and staging tests, used in automated-deployments. These tests are run as JUnit tests. To develop such tests, deploy the application to dev (like above) and run tests like FeedAndSearchSystemTest:
$ mvn test \ -Dtest.categories=system \ -DdataPlaneKeyFile=data-plane-private-key.pem -DdataPlaneCertificateFile=data-plane-public-cert.pem \ -DapiKey="$API_KEY"
TheapiKey
is used to fetch the dev instance's endpoints. The data plane key and certificate pair is used by ai.vespa.hosted.cd.Endpoint to access the application endpoint. Find more details in testing and Vespa Cloud API.To run tests against a deployment running in Docker on localhost (instead of using dev), configure endpoint location:
{ "localEndpoints": { "container": "http://localhost:8080/" } }
See system tests for details. Refer to album-recommendation-selfhosted for how to create the application package.When system and staging tests are ready, deploy to production:
$ mvn vespa:compileVersion -DapiKey="$API_KEY" $ mvn -Dvespa.compile.version="$(cat target/vespa.compile.version)" \ -DapiKey="$API_KEY" \ -DsourceUrl="https://github.com/vespa-engine/sample-apps/tree/$(git rev-parse HEAD)/vespa-cloud/album-recommendation-searcher" \ package vespa:submit