Conformance

Black-box tests of a Warp SQL server: they speak the protocol over HTTP with warp.py, a few plain calls with nothing hidden, so a failure points at the protocol rather than at a driver. The suite fails, never skips, when it has no server to talk to.

uv venv .venv && uv pip install --python .venv/bin/python -e .
WARP_SQL_TEST_SERVER_BINARY=../cpp/build/warp-duckdb .venv/bin/python -m pytest -q

Pointing it at a server

Variable	Default	Meaning
WARP_SQL_TEST_SERVER_BINARY		A server binary to start (taking --host, --port)
WARP_SQL_TEST_SERVER_PASSWORD_ENV	WARP_DUCKDB_PASSWORD	The variable it reads its administrator’s password from
WARP_SQL_TEST_SERVER_USERNAME	warp	That administrator’s name
WARP_SQL_TEST_SERVER_LOG	a temporary file	Where the started server’s output goes
WARP_SQL_TEST_URL, WARP_SQL_TEST_USERNAME, WARP_SQL_TEST_PASSWORD		A running server instead (then deselect -m "not mtls and not own_server")

Tests marked mtls and own_server start servers of their own, with flags that differ between servers; the defaults are warp-duckdb’s:

Variable	Default	Hardcharger’s
WARP_SQL_TEST_SERVER_MTLS_FLAG	--tls-client-ca	--mtls-ca-cert
WARP_SQL_TEST_SERVER_CORS_FLAG	--cors-origin	--cors-origins
WARP_SQL_TEST_SERVER_IDLE_FLAG	--session-idle-timeout-seconds	--session-idle-timeout
WARP_SQL_TEST_SERVER_TOKEN_LIFETIME_FLAG	--token-lifetime-seconds
WARP_SQL_TEST_SERVER_DRAIN_FLAG	--drain-seconds	--shutdown-grace-period-seconds
WARP_SQL_TEST_SERVER_HEALTH_QUERY_FLAG	--health-check-query	--health-check-query
WARP_SQL_TEST_SERVER_RETENTION_FLAG	--result-retention-seconds	--result-retention-seconds
WARP_SQL_TEST_SERVER_OAUTH_ARGS, WARP_SQL_TEST_SERVER_OAUTH_SECRET_ENV	see test_oauth_browser.py	see there

test_authorization.py needs a server with SQL privileges and the WARP_SQL_TEST_AUTHZ_* variables its docstring describes; on a server without privileges (warp-duckdb) it skips, saying so.

What each file covers

247 test functions (some run several cases), in the suite’s tests/:

File	Tests	Covers
test_authorization.py	2	§9.4 (metadata follows authorization) and §7.2 (a prepared statement is authorized at every execution, not once at prepare time).
test_control.py	8	§6.7–§6.9: idempotent submission, cancellation, timeouts.
test_discovery_auth.py	7	§3 (discovery, health) and §4 (authentication).
test_extensibility.py	4	§2.3, §2.4, §2.5 and §12: what keeps version 1 open to additions. Servers ignore request fields they don’t know, ids are long random strings, and Warp-Query-Tag labels a statement without changing it.
test_http.py	11	§2 and §9.5: HTTP conventions, limits, browser rules.
test_ingest.py	18	§8: bulk ingest.
test_lifecycle.py	4	§11.1 (graceful shutdown), §3.2 (health that answers when its query hangs) and §6.5 (a read under way when its result expires).
test_metadata.py	11	§10: metadata with Arrow Flight SQL’s result schemas.
test_mtls.py	36	§2.1 and §4.4: mutual TLS that carries identity. The client_certificate grant, certificate-bound tokens, revocation, identity match.
test_oauth_browser.py	18	§4.3 and §9.7: browser sign-in (capability oauth_browser).
test_openapi.py	15	§3.3 and Appendix B: the OpenAPI document, the endpoint that serves it, and the check (openapi_check.py) that holds every response of the suite to it.
test_partitioned_ingest.py	14	§8.3: partitioned ingest, its parts uploaded at once and committed atomically.
test_partitions.py	14	§6.10: partitioned results, read concurrently over separate connections.
test_prepare_describe.py	7	§7: preparing a query describes its result without running it.
test_prepared.py	16	§7: prepared statements, with JSON and Arrow bind parameters.
test_principals.py	2	§9.2 and §9.3: every handle is bound to its principal, and §4.4: a forwarded certificate header is no certificate.
test_results.py	18	§6.3–§6.6 and §8.2: async results, retention, resumption, incomplete streams, compression.
test_sessions.py	24	§5: sessions, options, transactions, and handles bound to their principal.
test_statements.py	18	§6: statements, the two response paths, and bind parameters.

Every response against the OpenAPI document

openapi_check.py holds every response the suite receives to spec/openapi.yaml: conftest.py wraps httpx.Client.send, so each request of every test (fixtures included) is checked, and a response that disagrees fails the test that made it, saying what was wrong. For each response it finds the operation by method and path and checks:

• the status is documented for the operation (exactly, or by its 4XX/5XX range);
• the content type is one the response documents;
• the required headers are there, and every documented header’s value fits its schema;
• a JSON body validates against the response’s schema (JSON Schema 2020-12, which OpenAPI 3.1 schemas are), and a problem document’s status and type agree with the response;
• an Arrow body begins like an Arrow IPC stream (the tests read the rest);
• a path or method outside the document gets a 404 or 405 problem document.

The end of a run says how many of the document’s operations had responses checked (all 38 against warp-duckdb). test_openapi.py also checks that the document is valid OpenAPI 3.1, that it and the spec’s Appendix B list the same 38 endpoints (and that every endpoint a section names is in Appendix B), which endpoints need no token, the served document (GET /warp/v1/openapi.json, spec §3.3: unauthenticated, cacheable, the published document apart from servers and x- fields), and that the check itself rejects a real response broken on purpose (a renamed or missing field, an unknown enumeration value, an undocumented status, content type or missing header, a problem document that disagrees with its status).

WARP_SQL_TEST_OPENAPI names another copy of the document to check against. A test that must see a response the document doesn’t describe takes @pytest.mark.openapi_unchecked (none does).

MUSTs without a test

Every MUST of the spec that a server can be tested for has a test, except these:

Spec	MUST	Why not yet
§2.1	Every operation works over HTTP/1.1 through an ordinary reverse proxy	Needs a proxy in the harness
§2.3	Ids carry at least 128 bits of randomness	A black box can’t measure entropy: test_ids_are_long_random_strings refuses ids too short or regular to hold it, no more
§2.6	Retry-After is delta-seconds on 429	Tested on 503 (draining); no portable way to make a server rate-limit
§4.4	An identity shown in error details or logs is escaped	Refusing such identities is tested; their escaping in a server’s logs is not visible to the suite
§4.4	Tokens a principal derives from a bound token keep the binding	The protocol has no way to derive one (an engine’s scoped tokens are its own SQL); Hardcharger tests its own (test_mtls_identity.py)
§9.3	Retained results stored outside memory are protected like the source data and deleted on expiry or session end	Storage is outside the protocol
§9.5	X-Content-Type-Options: nosniff on every response	Checked on a sample of responses, not all
§9.6	No TLS 1.2 renegotiation to change the certificate	Python’s ssl can’t start a renegotiation

These MUSTs bind clients, not servers, so a server suite can’t test them: §3.1 (use only listed features), §4.1 (tokens are opaque), §5.6 (a new token after session_ended for the token reasons), §6.1 (check Content-Type), §6.2 (a failed or cancelled document is an error), §6.6 (an incomplete stream is a failed result), §6.10 (read every partition to its end, all at once, and never guess a missing Warp-Result-Order), §8.2 (read buffers left uncompressed), §8.3 (no reliance on row order after a partitioned ingest) and §12 (ignore what is unknown, accept unknown enumeration values). The drivers’ own suites test them.

§2.1’s “say so in its documentation” (a base path other than /warp) and §4.4’s “end TLS at the server” are about documentation and deployment.