Repository navigation
obs: one trace from the orchestrator through the coordinator, with spans for the waits - #1746
Open
nikhilunni wants to merge 7 commits into
Open
nikhilunni wants to merge 7 commits into
nikhilunni wants to merge 7 commits into
Conversation
The coordinator's app-gRPC server opened no span of its own and read no inbound traceparent, so every orchestrator call started a new trace and the coordinator's work could not be tied to the request that caused it. engram_telemetry::grpc_server_span opens the server span for one request and makes it a child of the caller's W3C traceparent when the request carries one. The app-gRPC server installs it for every service through Server::trace_fn, so each handler's spans nest under the caller's trace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
A session op (resume, evict, deliver, create, destroy, teleport) runs on the op executor, away from the request that enqueued it, so the time a prompt or a resume waited in the op log was not visible in a trace. Each op attempt now runs in a `session.op` span named op.<kind>, with op.queued_ms from enqueue to this attempt. The deliver op also records, for each outbox row it takes, the row's prompt_id and how long the row waited since SendPrompt enqueued it, so the delivery joins the orchestrator's trace by prompt_id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The orchestrator had no live tracing, so a request's time in it, and the link from it to the coordinator's spans, were not visible. telemetry/trace.ts keeps the current span in an AsyncLocalStorage and exports finished spans as OTLP/HTTP JSON to OTEL_EXPORTER_OTLP_ENDPOINT (off when unset), reusing the encoding in otlp.ts. Every inbound request outside preview hosts runs in a server span that continues an inbound traceparent. Every control-plane RPC gets a client span and carries its traceparent to the coordinator. The SendPrompt pre-hook (GetSession and the spec projection's prepare) runs in its own span. Tests prove that the context survives an await chain and node:http body events under Bun, and that the outbound header carries the client span. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
Where a message or a tool call waits inside the orchestrator was not measured. These spans name each wait: - listener.scan and listener.acquire: the listener manager's lease scan (every 5 s) and each lease it takes. - listener.consume: one consumer handling one session event, with event.age_ms, the time since the coordinator stamped the event. - listener.wake: the wake a resuming session RPC sends after it returns. - tool_exec.start, tool.execute, tool.submit: starting the durable tool workflow, then its two steps. The gap between start and execute is the workflow dispatch delay. - automation.step and automation.recv: each durable engine step, and each wait for the next inbox message. The registered DBOS workflow bodies do not change: their source sets the DBOS application version, and a change would strand in-flight workflows during a rolling deploy. The spans wrap the plain functions they call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The value sets OTEL_EXPORTER_OTLP_ENDPOINT on the orchestrator, an OTLP/HTTP collector base URL. Empty, the default, leaves tracing off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
An eviction's trace ended at `host.snapshot_begin`; the finalize that publishes the disk and memory, uploads the snapshot blobs, writes the checkpoint record, and destroys the VM ran with no span. The finalize record now keeps the W3C traceparent of the request that began the eviction (`#[serde(default)]`, so a record from the previous build loads with none and the previous build ignores the field). Each finalize attempt opens `host.evict_finalize` as a child of it, with child spans for the disk leg, the memory leg (dirty ranges and bytes, the re-chunked manifest's chunk count, or the full memory size), the blob upload, the checkpoint record write, and the VM destroy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The eviction-final checkpoint lands in Postgres, and the session turns idle, when a host heartbeat advertises it. That work had no span. `checkpoint.reconcile` covers each advert (kind, session, snapshot, recoverable, inserted, settled), with `checkpoint.settle_idle` around the evicting-to-idle settle. The advert carries no trace context, so the span joins the host's finalize spans by snapshot_id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A message to a session crosses the orchestrator, the coordinator, the host and the harness. Before this change, only the coordinator-to-host part was one trace. The orchestrator had no live tracing, and the coordinator's app-gRPC server ignored an inbound
traceparent. This change connects the hops, and gives the waits on the way their own spans.Coordinator
engram_telemetry::grpc_server_spanopens one server span per app-gRPC request, a child of the caller's W3Ctraceparent. The server installs it for every service throughServer::trace_fn. This is one seam, not a change to each method.session.op(namedop.<kind>): one span per session-op attempt (resume, evict, deliver, create, destroy, teleport), withop.queued_msfrom enqueue to the attempt. For each outbox row the deliver op takes, it records an event with theprompt_idand how long the row waited sinceSendPrompt.Orchestrator (
orchestrator/src/telemetry/trace.ts, no new dependency)AsyncLocalStorage. Spans are exported as OTLP/HTTP JSON, reusing the encoding inotlp.ts, toOTEL_EXPORTER_OTLP_ENDPOINT. While that is unset, tracing is off and costs one branch.<Service>/<Method>for/rpc,HTTP <method>otherwise) continues an inboundtraceparent. Preview hosts are skipped; they carry a session app's own traffic.traceparentheader (telemetry/connect.ts, onmakeTransport).send_prompt.prepare,listener.scan,listener.acquire,listener.consume(withevent.age_ms),listener.wake,tool_exec.start,tool.execute,tool.submit,automation.step,automation.recv.Eviction finalize (host agent). An eviction's trace used to end at
host.snapshot_begin. The finalize that runs after it had no span.traceparentof the request that began the eviction. It is#[serde(default)], so a record from the previous build loads with none, and the previous build ignores the field.host.evict_finalize(snapshot, session and sandbox ids,stage,attempt,outcome) as a child of that trace, with child spans:host.evict_finalize.disk(pending_disks)host.evict_finalize.memory(capturediff or full,dirty_ranges,dirty_bytes,manifest_chunks,memory_bytes)host.evict_finalize.blobs(state_bytes,aux_bundles)host.evict_finalize.checkpoint_recordhost.evict_finalize.destroycheckpoint.reconcile, one span per checkpoint advert a heartbeat reconciles (kind, ids,recoverable,inserted,settled), withcheckpoint.settle_idlearound the evicting-to-idle settle. The advert carries no trace context, so this span joins the finalize bysnapshot_id.Chart:
orchestrator.otel.endpointsetsOTEL_EXPORTER_OTLP_ENDPOINT. Empty, the default, leaves tracing off.Why not the OTel JS SDK
The live surface is small: a span stack, a W3C header, and an OTLP/HTTP POST. The repo already encodes OTLP JSON by hand for the session-telemetry exporter. The SDK would add a set of packages for that. A test proves that
AsyncLocalStoragecontext survivesnode:httpbody events and timers under Bun, which was the open question.What is not joined yet
SendPromptreturns after it enqueues. Delivery runs later on the op executor, so the deliver op's span is a separate trace. It joins the request byprompt_idandsession_id, not as a child. A real parent link would mean storing thetraceparenton the outbox row or the op, which is a schema change for a follow-up.listener.acquiredoes not carry how long the row waited for the scan:listDesiredreturns ids only. The scan interval bounds that wait at 5 s.listener.consumespan is the root of its own trace: events arrive on a stream, not in a request.checkpoint.reconcileis not a child of the eviction: carrying atraceparenton the heartbeat's checkpoint advert is a wire change.Validation
bun run typecheck.bun test src/telemetry(14 pass), and the listener, tools, workflow, control-plane, automation-engine and server suites (433 pass).cargo nextest run -p engram-telemetry:a_grpc_server_span_is_a_child_of_the_callers_traceparent,a_grpc_server_span_without_a_traceparent_starts_a_trace.just check. With the eviction spans: fmt, clippy, hakari, andcargo nextest run --workspace --no-fail-fast(2,834 passed). One fail-fast run hit the known 3 s-deadline flake inengram-harness-codex.eviction_finalize::tests::a_record_without_a_traceparent_loads_and_a_new_one_keeps_it.helm lint, andhelm templatewith and withoutorchestrator.otel.endpoint.site:pnpm build.🤖 Generated with Claude Code
https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF