Monitoring
Metrics can be pulled in a native Vespa format or Prometheus. Also see vespa.ai monitoring
Vespa Metrics
Metrics are available at $ENDPOINT/metrics/v2/values:
$ curl -s --cert data-plane-public-cert.pem --key data-plane-private-key.pem \
'https://vespacloud-docsearch.vespa-team.aws-us-east-1c.public.vespa.oath.cloud/metrics/v2/values'
...
{
"nodes": [
{
"hostname": "h577a.prod.aws-us-east-1c.vespa-external.aws.oath.cloud",
"role": "content/documentation/0/7",
"node": {
"timestamp": 1601997367,
"metrics": [
{
"values": {
Example: use jq to extract relevant metrics:
$ curl -s --cert data-plane-public-cert.pem --key data-plane-private-key.pem \
'https://vespacloud-docsearch.vespa-team.aws-us-east-1c.public.vespa.oath.cloud/metrics/v2/values' | \
jq -r -c '.nodes[] | .hostname as $h | select(.services[].metrics[].values."content.proton.documentdb.documents.total.last") |'\
'.services[].metrics[] | select(.values."content.proton.documentdb.documents.total.last") |'\
'[$h, .dimensions.documenttype, .values."content.proton.documentdb.documents.total.last"] | @tsv'
h576a.prod.aws-us-east-1c.vespa-external.aws.oath.cloud doc 230
h577a.prod.aws-us-east-1c.vespa-external.aws.oath.cloud doc 230
Prometheus and Grafana
Prometheus metrics are found at $ENDPOINT/prometheus/v1/values:
$ curl -s --cert data-plane-public-cert.pem --key data-plane-private-key.pem \
'https://vespacloud-docsearch.vespa-team.aws-us-east-1c.public.vespa.oath.cloud/prometheus/v1/values'
...
# HELP content_proton_documentdb_documents_total_last
# TYPE content_proton_documentdb_documents_total_last untyped
content_proton_documentdb_documents_total_last{documenttype="doc",zone="prod.aws-us-east-1c",applicationId="vespa-team.vespacloud-docsearch.default",serviceId="searchnode",clusterId="content/documentation",hostname="h577a.prod.aws-us-east-1c.vespa-external.aws.oath.cloud",vespa_service="vespa_searchnode",} 230.0 1601991719000
content_proton_documentdb_documents_total_last{documenttype="doc",zone="prod.aws-us-east-1c",applicationId="vespa-team.vespacloud-docsearch.default",serviceId="searchnode",clusterId="content/documentation",hostname="h576a.prod.aws-us-east-1c.vespa-external.aws.oath.cloud",vespa_service="vespa_searchnode",} 230.0 1601991719000
...
Quick setup:
Prometheus |
Prometheus is configured using |
---|---|
Grafana Dashboard |
Use the provisioning folder as a baseline for further configuration.
In the provisioning folder there are a few different files that all help for configuring Grafana locally.
These work as good examples of default configurations,
but the most important is the file named Click the + button on the side and go to import. Upload the file to the Grafana instance. This should automatically load in the dashboard for usage. For now, it will not display any data as no data sources are configured yet. |
Grafana Data Source |
The Prometheus data source has to be added to the Grafana instance for the visualisation. Click the cog on the left and then "Data Sources". Click "Add data source" and choose Prometheus from the list. Add the URL for the Prometheus instance with appropriate bindings for connecting. The configuration for the bindings will depend on how the Prometheus instance is hosted. Once the configuration details have been entered, click Save & Test at the bottom and ensure that Grafana says "Data source is working". |
To verify the data flow, navigate back to the Vespa Metrics dashboard by clicking the dashboard symbol on the left (4 blocks) and clicking manage and then click Vespa Metrics. Data should now appear in the Grafana dashboard. If no data shows up, edit one of the data sets and ensure that it has the right data source selected. The name of the data source the dashboard is expecting might be different from what your data source is named. If there is still no data appearing, it either means that the Vespa instance is not being used or that some part of the configuration is wrong.
AWS Cloudwatch
Refer to the metrics-emitter.