Skip to content

obs: one trace from the orchestrator through the coordinator, with spans for the waits - #1746

Open
nikhilunni wants to merge 7 commits into
mainfrom
obs/orchestrator-tracing
Open

nikhilunni wants to merge 7 commits into
mainfrom
obs/orchestrator-tracing

Conversation

@nikhilunni

@nikhilunni nikhilunni commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A message to a session crosses the orchestrator, the coordinator, the host and the harness. Before this change, only the coordinator-to-host part was one trace. The orchestrator had no live tracing, and the coordinator's app-gRPC server ignored an inbound traceparent. This change connects the hops, and gives the waits on the way their own spans.

Coordinator

  • engram_telemetry::grpc_server_span opens one server span per app-gRPC request, a child of the caller's W3C traceparent. The server installs it for every service through Server::trace_fn. This is one seam, not a change to each method.
  • session.op (named op.<kind>): one span per session-op attempt (resume, evict, deliver, create, destroy, teleport), with op.queued_ms from enqueue to the attempt. For each outbox row the deliver op takes, it records an event with the prompt_id and how long the row waited since SendPrompt.

Orchestrator (orchestrator/src/telemetry/trace.ts, no new dependency)

  • The current span is kept in an AsyncLocalStorage. Spans are exported as OTLP/HTTP JSON, reusing the encoding in otlp.ts, to OTEL_EXPORTER_OTLP_ENDPOINT. While that is unset, tracing is off and costs one branch.
  • A server span per inbound request (<Service>/<Method> for /rpc, HTTP <method> otherwise) continues an inbound traceparent. Preview hosts are skipped; they carry a session app's own traffic.
  • A client span per control-plane RPC, with the traceparent header (telemetry/connect.ts, on makeTransport).
  • Spans for the waits: send_prompt.prepare, listener.scan, listener.acquire, listener.consume (with event.age_ms), listener.wake, tool_exec.start, tool.execute, tool.submit, automation.step, automation.recv.
  • The registered DBOS workflow bodies do not change. Their source sets the DBOS application version, so a change would strand in-flight workflows during a rolling deploy. The spans wrap the plain functions those bodies call.

Eviction finalize (host agent). An eviction's trace used to end at host.snapshot_begin. The finalize that runs after it had no span.

  • The finalize record keeps the traceparent of the request that began the eviction. It is #[serde(default)], so a record from the previous build loads with none, and the previous build ignores the field.
  • Each finalize attempt opens host.evict_finalize (snapshot, session and sandbox ids, stage, attempt, outcome) as a child of that trace, with child spans:
    • host.evict_finalize.disk (pending_disks)
    • host.evict_finalize.memory (capture diff or full, dirty_ranges, dirty_bytes, manifest_chunks, memory_bytes)
    • host.evict_finalize.blobs (state_bytes, aux_bundles)
    • host.evict_finalize.checkpoint_record
    • host.evict_finalize.destroy
  • The coordinator gets checkpoint.reconcile, one span per checkpoint advert a heartbeat reconciles (kind, ids, recoverable, inserted, settled), with checkpoint.settle_idle around the evicting-to-idle settle. The advert carries no trace context, so this span joins the finalize by snapshot_id.

Chart: orchestrator.otel.endpoint sets OTEL_EXPORTER_OTLP_ENDPOINT. Empty, the default, leaves tracing off.

Why not the OTel JS SDK

The live surface is small: a span stack, a W3C header, and an OTLP/HTTP POST. The repo already encodes OTLP JSON by hand for the session-telemetry exporter. The SDK would add a set of packages for that. A test proves that AsyncLocalStorage context survives node:http body events and timers under Bun, which was the open question.

What is not joined yet

  • SendPrompt returns after it enqueues. Delivery runs later on the op executor, so the deliver op's span is a separate trace. It joins the request by prompt_id and session_id, not as a child. A real parent link would mean storing the traceparent on the outbox row or the op, which is a schema change for a follow-up.
  • listener.acquire does not carry how long the row waited for the scan: listDesired returns ids only. The scan interval bounds that wait at 5 s.
  • Each listener.consume span is the root of its own trace: events arrive on a stream, not in a request.
  • checkpoint.reconcile is not a child of the eviction: carrying a traceparent on the heartbeat's checkpoint advert is a wire change.

Validation

  • bun run typecheck. bun test src/telemetry (14 pass), and the listener, tools, workflow, control-plane, automation-engine and server suites (433 pass).
  • cargo nextest run -p engram-telemetry: a_grpc_server_span_is_a_child_of_the_callers_traceparent, a_grpc_server_span_without_a_traceparent_starts_a_trace.
  • just check. With the eviction spans: fmt, clippy, hakari, and cargo nextest run --workspace --no-fail-fast (2,834 passed). One fail-fast run hit the known 3 s-deadline flake in engram-harness-codex.
  • eviction_finalize::tests::a_record_without_a_traceparent_loads_and_a_new_one_keeps_it.
  • helm lint, and helm template with and without orchestrator.otel.endpoint.
  • site: pnpm build.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF

nikhilunni and others added 7 commits November 5, 2026 09:12
The coordinator's app-gRPC server opened no span of its own and read no
inbound traceparent, so every orchestrator call started a new trace and
the coordinator's work could not be tied to the request that caused it.

engram_telemetry::grpc_server_span opens the server span for one request
and makes it a child of the caller's W3C traceparent when the request
carries one. The app-gRPC server installs it for every service through
Server::trace_fn, so each handler's spans nest under the caller's trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
A session op (resume, evict, deliver, create, destroy, teleport) runs on
the op executor, away from the request that enqueued it, so the time a
prompt or a resume waited in the op log was not visible in a trace.

Each op attempt now runs in a `session.op` span named op.<kind>, with
op.queued_ms from enqueue to this attempt. The deliver op also records,
for each outbox row it takes, the row's prompt_id and how long the row
waited since SendPrompt enqueued it, so the delivery joins the
orchestrator's trace by prompt_id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The orchestrator had no live tracing, so a request's time in it, and the
link from it to the coordinator's spans, were not visible.

telemetry/trace.ts keeps the current span in an AsyncLocalStorage and
exports finished spans as OTLP/HTTP JSON to OTEL_EXPORTER_OTLP_ENDPOINT
(off when unset), reusing the encoding in otlp.ts. Every inbound request
outside preview hosts runs in a server span that continues an inbound
traceparent. Every control-plane RPC gets a client span and carries its
traceparent to the coordinator. The SendPrompt pre-hook (GetSession and
the spec projection's prepare) runs in its own span. Tests prove that the
context survives an await chain and node:http body events under Bun, and
that the outbound header carries the client span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
Where a message or a tool call waits inside the orchestrator was not
measured. These spans name each wait:

- listener.scan and listener.acquire: the listener manager's lease scan
  (every 5 s) and each lease it takes.
- listener.consume: one consumer handling one session event, with
  event.age_ms, the time since the coordinator stamped the event.
- listener.wake: the wake a resuming session RPC sends after it returns.
- tool_exec.start, tool.execute, tool.submit: starting the durable tool
  workflow, then its two steps. The gap between start and execute is the
  workflow dispatch delay.
- automation.step and automation.recv: each durable engine step, and each
  wait for the next inbox message.

The registered DBOS workflow bodies do not change: their source sets the
DBOS application version, and a change would strand in-flight workflows
during a rolling deploy. The spans wrap the plain functions they call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The value sets OTEL_EXPORTER_OTLP_ENDPOINT on the orchestrator, an
OTLP/HTTP collector base URL. Empty, the default, leaves tracing off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
An eviction's trace ended at `host.snapshot_begin`; the finalize that
publishes the disk and memory, uploads the snapshot blobs, writes the
checkpoint record, and destroys the VM ran with no span.

The finalize record now keeps the W3C traceparent of the request that
began the eviction (`#[serde(default)]`, so a record from the previous
build loads with none and the previous build ignores the field). Each
finalize attempt opens `host.evict_finalize` as a child of it, with
child spans for the disk leg, the memory leg (dirty ranges and bytes,
the re-chunked manifest's chunk count, or the full memory size), the
blob upload, the checkpoint record write, and the VM destroy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The eviction-final checkpoint lands in Postgres, and the session turns
idle, when a host heartbeat advertises it. That work had no span.
`checkpoint.reconcile` covers each advert (kind, session, snapshot,
recoverable, inserted, settled), with `checkpoint.settle_idle` around the
evicting-to-idle settle. The advert carries no trace context, so the span
joins the host's finalize spans by snapshot_id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant