Graceful shutdown

A server that is shutting down stops accepting new sessions and statements (503 server_draining, with Retry-After), reports not_serving on health so load balancers take it out of rotation, lets running statements finish within its grace period, and then cancels what remains with cancel_reason: server_shutdown (§11.1).

These commands run against a server of their own with a 30-second grace period; SERVER_PID is its process id.

WARP=http://127.0.0.1:31337
TOKEN=$(curl -s $WARP/warp/v1/auth/token -H 'Content-Type: application/json' \
  -d '{"grant_type": "password", "username": "joe", "password": "s3:cret"}' | jq -r .access_token)
AUTH="Authorization: Bearer $TOKEN"
SESSION=$(curl -s $WARP/warp/v1/sessions -H "$AUTH" -H 'Content-Type: application/json' -d '{}' | jq -r .session_id)
curl -s $WARP/warp/v1/health

{"status":"serving"}

A long query is running when the operator stops the server:

STATEMENT=$(curl -s $WARP/warp/v1/sessions/$SESSION/statements -H "$AUTH" -H 'Content-Type: application/json' \
  -H 'Prefer: respond-async' -d '{"sql": "SELECT sum(i) AS total FROM range(6000000000) t(i)"}' | jq -r .statement_id)
kill -TERM "$SERVER_PID"
sleep 0.5
curl -s -i $WARP/warp/v1/health | tr -d '\r' | grep -E '^(HTTP|\{)'

HTTP/1.1 503 Service Unavailable
{"status":"not_serving","reason":"draining"}

New work is refused, with how long to wait before trying again (an integer number of seconds, §2.6):

curl -s -i $WARP/warp/v1/sessions/$SESSION/statements -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"sql": "SELECT 1"}' | tr -d '\r' | grep -E '^(HTTP|Retry-After|\{)' | sed 's/,"request_id".*/,…}/'

HTTP/1.1 503 Service Unavailable
Retry-After: …
{"type":"urn:warp-sql:error:server_draining","title":"Server shutting down","status":503,"code":"server_draining","detail":"the server is shutting down",…}

The statement that was running finishes, and its result can still be read:

curl -s $WARP/warp/v1/sessions/$SESSION/statements/$STATEMENT -H "$AUTH" -H 'Prefer: wait=30' | jq -r .state
curl -s $WARP/warp/v1/sessions/$SESSION/statements/$STATEMENT/result -H "$AUTH" -o total.arrows
python3 -c "import pyarrow.ipc as ipc; print(ipc.open_stream('total.arrows').read_all().to_pylist())"

succeeded
[{'total': Decimal('17999999997000000000')}]

Once nothing is left to run, or the grace period ends, the server stops. A client that gets server_draining retries after Retry-After, on another instance if there is one; health’s starting and check_failed reasons are also 503 not_serving.