Repository navigation
feat(qwp): add paging, row counts, timings and error positions to the query client - #129
glasstiger wants to merge 9 commits into
Conversation
QwpQueryClient and the Query facade can now use the query-details flags of QUERY_REQUEST, which the server advertises with the CAP_QUERY_DETAILS bit of SERVER_INFO (questdb PR #8356): - QwpQueryOptions sets a row window (skipRows, maxRows) and asks for the row count of the whole result set (countRows), execution statistics (stats) and the error position (errorPosition). It also carries resetSymbolDict. QwpQueryClient.execute() and Query.options() take it; the client reads it once, when the query starts, so a failover replay sends the same options. - The handler receives the answers through new default overloads: onEnd and onExecDone with a QwpQueryDetails (the row count, the compile, execute, count and client-wait times, and the JIT flag), and onError with the position. Each delegates to the next simpler overload, so existing handlers keep working. The client now calls the detailed overloads for every terminal event, client-side failures included, so one override sees them all. - QueryException.getPosition() carries the position to callers of the facade's Completion.await(). A query without options is byte-identical on the wire. Against a server without CAP_QUERY_DETAILS, the client leaves out the count, the statistics and the position, and fails a query that sets a row window before sending it, since that server would stream every row. After a failover reconnect the client checks the new server again. The I/O thread decodes the trailers the request asked for: row_count and the statistics block on RESULT_END, the statistics block on EXEC_DONE, and the position on QUERY_ERROR. A RESULT_END or EXEC_DONE without a requested trailer is a terminal transport error, like any other truncated terminator. A QUERY_ERROR without the position is the baseline frame the server sends for a request whose flags it could not read; it reports no position. RESULT_END and EXEC_DONE now share one bounds-checked varint reader, which also rejects data bits above bit 63 on a tenth byte, as the result batch decoder already does.
The query-details change routes every failure that execute() detects itself to the position-aware onError overload, but the tests written for it reached none of those paths. QwpQueryClientFailurePathsTest now drives each one with a handler that fails on any other overload: - a failover reconnect that fails, and the reconnect of the next query that fails after it; - a stop request between two attempts, from onFailoverReset; - a stop request while the failover reconnect waits for SERVER_INFO; - a stop request right after a transport failure; - a close() between two attempts, from onFailoverReset. A stop request can reach the point right after a transport failure only through a race with another thread, so QwpQueryClient gains setAfterTransportFailureHookForTest(), a barrier that runs there, like setBeforeCloseHookForTest(). TestWebSocketServer gains a hook that runs before SERVER_INFO, and its rejected egress upgrades now drop the connection, as its documentation says, instead of leaving the client to wait out its auth timeout. The default branch of the event switch in executeOnce() is unreachable, as the I/O thread emits only the kinds it handles; an assertion now says so. A comment explains the null guard on serverInfo, which only a connect() racing the query from another thread can trip.
QwpEgressQueryDetailsClientTest sends the new query_flags through the real java-questdb-client, with QwpQueryOptions on both QwpQueryClient.execute() and the facade's Query.options(). It checks the error positions that onError() and QueryException receive, the row window, the full row count, the timings and the JIT flag against /exec on the same server and against SQL LIMIT, and asserts that the client's wire constants match the server's. The released client 1.3.12 cannot send these flags, so the submodule moves to the head of questdb/java-questdb-client#129 and the poms pin questdb.client.version to 1.3.13-SNAPSHOT. CI then builds the client from the submodule with the local-client profile. The Javadoc of QwpEgressQueryDetailsWireTest no longer claims that the pinned client cannot send the flags.
The javadoc of QwpColumnBatchHandler.onError(long, byte, String, int) and of QwpEgressMsgKind.NO_ERROR_POSITION said that a cancellation reports no error position. Only a CANCEL this client sends does: the server reports position 0 for a query timeout and for a cancellation on the server, such as CANCEL QUERY, as the /exec HTTP endpoint does. The javadoc now says so.
The query details took SERVER_INFO capability 0x08 and query flag 0x02, which per-query timeouts already use in released clients: c-questdb-client 7.2.0 and later, and the Python client 6.0.0 and later, read capability 0x08 as "the server honours a per-query timeout" and then send flag 0x02 followed by timeout_ms. A server with the query details would have read that flag as an error-position request, ignored the timeout and answered with a QUERY_ERROR those clients cannot decode. CAP_QUERY_DETAILS moves to 0x10, and the flags to 0x04 (ERROR_POSITION), 0x08 (STATS), 0x10 (COUNT_ROWS), 0x20 (SKIP_ROWS) and 0x40 (MAX_ROWS), as on the server. CAP_QUERY_TIMEOUT (0x08) and QUERY_FLAG_TIMEOUT (0x02) now name the reserved bits, with the values the timeout change gives them, and the docs say a flag's field goes after query_flags in ascending order of the flag bits, so timeout_ms comes ahead of skip_rows and max_rows. New tests pin the allocation: every capability and flag is a bit of its own and none takes the timeout bits. A server that advertises per-query timeouts without the query details gets none of the details, and a row window fails before it is sent.
QwpQueryClient reported the failures execute() detects itself through eleven separate handler.onError() calls. Five of them -- an exhausted failover (two sites), an interrupted backoff, and a rejected or unavailable credential during a failover reconnect -- had no test that would notice a call to the 3-arg overload. A regression there would hide the error from a handler that overrides only onError(long, byte, String, int), as QwpColumnBatchHandler tells users to do. deliverClientError() now makes that call for every client-detected failure, deliverFinal() included, so any test that reaches one of them pins all of them. QwpQueryClientFailurePathsTest adds the exhausted failover, the rejected credential and the unavailable credential; QwpEgressMockServer gains setAuthorizationValidator() for the rejected credential. QuestDBQueryOptionsTest now checks that options() on a closed handle throws and leaves the next borrower's query alone. No test covered the stale-lease guard of the new Query.options() before. Test helper cleanup: - RecordingQueryHandler drops the unread batchCount and hosts the assertNoDetails(QwpQueryDetails) that QueryEventTest and QwpEgressTerminatorDecodeTest each carried a copy of. - QwpEgressFrames.QueryRequest drops the unread initialCredit. - testEachDetailOptionSendsItsFlag becomes testEachBooleanOptionSendsItsFlag: resetSymbolDict is not a query-details option. - The QueryException test moves from QwpQueryOptionsTest to its own QueryExceptionTest. - QwpEgressMockServer drops a connection on its read thread. The comment that sent the close to another thread claimed close() joins the read thread, which it skips when called on that thread.
Query.options() wrote into the handle's options, which the worker thread read only when it started the query, after submit() had returned. A caller that set up the next page before await() changed the query already in flight: it streamed the next window, or every row after options(null). submit() now copies the options into submittedOptions, which execute() reads, so a change made while a query is in flight applies to the next submit() only. The Query javadoc says so, and tells callers not to change the SQL text, binds or handler before await() returns, as the running query reads them. The comment in QueryImpl.runOn() no longer claims that nothing can call sql() while a query runs. QueryClientPool gains setBusyWorkerHookForTesting(), which installs the existing busyWorkerTestHook on its workers. The new QuestDBQueryOptionsTest test uses it to park the worker between two queries, so the caller always changes the options before the worker reads them. RecordingQueryHandler failed a call to a simpler overload by throwing, which the facade's worker thread swallows once WrappingHandler has signalled success. A WrappingHandler that also called a legacy terminal overload passed QuestDBQueryOptionsTest. The handler now records such a call too, and the facade tests assert the record on the test thread after await(); assertOnlyError() and assertNoDetails() check it as well.
execute() reports an interrupt during the failover backoff through deliverClientError(), but no test reached that call: with it reverted to the 3-arg onError(), the whole suite still passed, so a regression would hide the error from a handler that overrides only onError(long, byte, String, int). QwpQueryClientFailurePathsTest now drives that exit. Its hook closes the client and then interrupts the caller, so the pending interrupt ends the backoff's sleep rather than the join() of the failed connection's I/O thread in the failover cleanup. A seeded random makes that sleep 13.9 s, which the interrupt cuts short. The test fails with the call site reverted. QuestDBQueryOptionsTest.testErrorPositionReachesQueryException said its last block checks a failure after one with a position, but the failure before that block carried none. The comment now sits on the block it describes. The last block asks for the position and checks that a failure the client detects itself reports none, with STATUS_INTERNAL_ERROR.
execute() reports a failover deadline that runs out while it tears down the failed connection through deliverClientError(), but no test reached that call: with it reverted to the 3-arg onError(), the whole suite still passed, so a regression would hide the error from a handler that overrides only onError(long, byte, String, int). The window lies between the attempt check, which lets the query fail over, and the backoff, which finds the deadline gone. QwpQueryClient gains setAfterFailoverCleanupHookForTest(), a barrier that runs there, like setAfterTransportFailureHookForTest(). QwpQueryClientFailurePathsTest now drives that exit: its hook waits out a 2 s failover_max_duration_ms, and the test checks that the query ends with no position and that the next query reconnects, as this exit leaves the client disconnected rather than latched. The test fails with the call site reverted.
|
Reviewing PR #129 at level 3 (full review, with an independent check of every candidate issue). Base is CriticalNone. ModerateNone. MinorNew test methods break the alphabetical member order
Where:
Fix: run IntelliJ's Rearrange Code on both files. Coverage mapThe test gate passes. No coverage gaps were admitted.
Summary
|
Each QueryEvent owns a QwpQueryDetails, and the I/O thread built every RESULT_END and EXEC_DONE event with new QueryEvent(). Every query, with or without options, so allocated a details object it did not need: the I/O thread allocated 144 bytes per query, up from 80 before the query details. The I/O thread now takes those events from its event pool, as it does for batches; the user thread already hands every event back to the pool after the handler. decodeResultEnd() and decodeExecDone() decode into the event they get. An event borrowed for a frame that fails to decode stays out of the pool, as only the user thread returns events to it. A query that ends with an EXEC_DONE now allocates 32 bytes on the I/O thread. testDetailsAreReusedAcrossQueries checks that once the pool has gone round, no query brings a QwpQueryDetails the handler has not seen, and that an instance which carried details reports none for a query that asked for none. QueryEventTest and QwpColumnBatchHandlerTest now sort their tests by name, and the class javadoc of QwpColumnBatchHandlerTest also covers the delegation of the detailed overloads it tests.
[PR Coverage check]😍 pass : 284 / 286 (99.30%) file detail
|
|
Reviewing PR #129 at level 3, from base CriticalNone. ModerateNone. MinorNone. The test-method ordering flagged in the earlier review is fixed at head: every changed test class is now sorted. Coverage mapThe test gate passes, with no admitted coverage gaps.
Summary
|
|
Reviewing PR #129 at level 3. The base is CriticalNone. ModerateNone. MinorNone. Coverage mapThe test gate passes, with no coverage gaps found.
Summary
|
|
Reviewing PR #129 at level 3 (full review): base Verdict: approve. I found no Critical, Moderate or Minor issues, and the test gate passes. CriticalNone. ModerateNone. MinorNone. Coverage mapThe test gate passes with no coverage gaps worth reporting.
Summary
|
PR #129 adds server-side paging and
/exec-style query metadata to the Java QWP query client.New functionality
A new
QwpQueryOptionsclass lets callers request:skipRowsandmaxRowsselect a window of results.maxRows(0)returns only the schema.countRows(true)counts rows beyond the requested page, separately from the number actually streamed. This can require the server to read the remaining results.errorPosition(true)provides the offending character’s position separately from the error message.The options also expose the existing SYMBOL-dictionary reset functionality.
API integration
QwpQueryClient.execute(...)overloads andQuery.options(...)on the higher-level facade.QwpQueryDetailsto newonEnd(...)andonExecDone(...)callback overloads, and a position argument toonError(...).QueryException.getPosition()exposes the position whenCompletion.await()fails; unavailable positions use-1.submit(), so preparing the next page cannot alter an in-flight query. Failover retries retain the original options.Compatibility and implementation
CAP_QUERY_DETAILScapability, including after failover. Unsupported metadata requests run without metadata; unsupported paging fails before sending, rather than unexpectedly returning every row.The PR also adds a README example and extensive tests covering encoding, decoding, compatibility, callback delegation, option snapshots, failover/shutdown failures and pooled-state reuse.
Server counterpart: questdb/questdb#8356.