Retention and resumption

A retained result can be read again, and from any batch, until it expires. An unretained result can be read once, from batch 0, while its session lives (§6.5). A dropped download of a retained result therefore resumes where it stopped instead of running the query again.

Warp-Retain-Result: true or false on the submit asks; without it the server decides (it should retain for respond-async submits and not for streamed ones). A server may decline to retain; it never retains a result it was asked not to. The statement document says what it chose (retained) and until when (expires_at).

These commands run against a server that keeps results for 3 seconds:

WARP=http://127.0.0.1:31337
TOKEN=$(curl -s $WARP/warp/v1/auth/token -H 'Content-Type: application/json' \
  -d '{"grant_type": "password", "username": "joe", "password": "s3:cret"}' | jq -r .access_token)
AUTH="Authorization: Bearer $TOKEN"
SESSION=$(curl -s $WARP/warp/v1/sessions -H "$AUTH" -H 'Content-Type: application/json' -d '{}' | jq -r .session_id)
STATEMENT=$(curl -s $WARP/warp/v1/sessions/$SESSION/statements -H "$AUTH" -H 'Content-Type: application/json' \
  -H 'Prefer: wait=30' -H 'Warp-Retain-Result: true' \
  -d '{"sql": "SELECT i FROM range(1000000) t(i)"}' | tee doc.json | jq -r .statement_id)
jq -c '.result | {retained, batches: .partitions[0].batches, rows: .partitions[0].rows}' doc.json

{"retained":true,"batches":8,"rows":1000000}

Resume from a batch

from_batch (counting from 0) starts the stream at that batch of the partition; the stream still begins with the schema. A client that read batches 0 to 4 before its connection broke asks for the rest:

curl -s "$WARP/warp/v1/sessions/$SESSION/statements/$STATEMENT/result?from_batch=5" -H "$AUTH" -o rest.arrows
python3 -c "
import pyarrow.ipc as ipc
rest = ipc.open_stream('rest.arrows').read_all()
print(rest.num_rows, 'rows, starting at', rest.column('i')[0])"

… rows, starting at …

Read it again, then let it expire

A retained result can be read whole as often as needed until expires_at. A read already under way when the result expires is delivered in full; a read that starts after it gets 410 result_expired:

curl -s -o /dev/null -w '%{http_code}\n' $WARP/warp/v1/sessions/$SESSION/statements/$STATEMENT/result -H "$AUTH"
sleep 4
curl -s $WARP/warp/v1/sessions/$SESSION/statements/$STATEMENT/result -H "$AUTH" | jq -c '{status, code}'

200
{"status":410,"code":"result_expired"}

Unretained results

An unretained result is read once, from batch 0: a second read, or from_batch greater than 0, gets 409 result_not_resumable. If the client disconnects while reading one, the server should cancel the statement, unless it changes data: a write is never undone because its reader went away.

ONCE=$(curl -s $WARP/warp/v1/sessions/$SESSION/statements -H "$AUTH" -H 'Content-Type: application/json' \
  -H 'Prefer: wait=30' -H 'Warp-Retain-Result: false' -d '{"sql": "SELECT 1 AS one"}' | jq -r .statement_id)
curl -s -o /dev/null -w '%{http_code}\n' $WARP/warp/v1/sessions/$SESSION/statements/$ONCE/result -H "$AUTH"
curl -s $WARP/warp/v1/sessions/$SESSION/statements/$ONCE/result -H "$AUTH" | jq -c '{status, code}'

200
{"status":409,"code":"result_not_resumable"}

Retained results are a resource: a server may limit them per principal (429 result_quota_exceeded), and must protect results it stores outside memory as it protects the source data, deleting them when they expire or their session ends (§9.3).