BCP Test

A business continuity plan (BCP) is a critical part of any organization's risk management strategy.

BCP is automated for Vespa Cloud applications that are deployed to more than one zone, and are using the global endpoint to serve query requests. See deployment.xml for details on how to configure the BCP automation. The automation ensures that the application will continue to operating during an unplanned disruption in the service in a zone.

In case of a production emergency, a zone can manually be set OUT using the routing control to prevent it from receiving requests.

Test the BCP capability

Application owners are encouraged to test their BCP setup regularly. Use this to verify that the application is able to perform with peak traffic routed to n -1 zones.

Monitoring of the traffic shifts during the BCP is done using the monitoring dashboard available in the Cloud Console. All the zones can be displayed in the same graph, and the traffic can be observed in real time. To monitor all zones, replace the request parameter zone in the URL with regexp(prod).

The process is as follows:

1. Have an application deployed in the zones to test, see deployment.xml
2. Apply query load locally in each zone against the global endpoint
3. Use the routing control to set the routing status for all zones, but one, OUT
4. Observe that all the query requests are routed to the single zone that still has routing status IN for 30 minutes
5. Use the routing control to set the routing status for all zones IN
6. Observe that all the query requests are routed locally for 30 minutes
7. Use the routing control to set the routing status OUT for the one zone that remained IN in step 3
8. Observe that the query requests are routed from the zone that was set OUT to another zone for 30 minutes
9. Use the routing control to set the routing status for the zone IN
10. Observe that all query requests are routed locally for 30 minutes

Yahoo Inc BCP Certification for 2024

The BCP certification for 2024 was monitored by the Yahoo Inc BCP team, completed on 2024-02-08. The certification was successful. See the bcp_test_2024.pdf for details.