HTTP and messaging tracing

HTTP tracing records the relationship between a request and the internal HTTP calls it makes. It is available in the template, accounting, billing, and leases API processes. Export is disabled until explicitly enabled.

Billing's configured Accounting and Leases clients and Leases' Billing client use the shared transport. Other service clients still need to adopt it. Outbox headers and the shared Service Bus transport also carry trace context through publication and consumption. Database spans are not implemented yet. Optional workflow metadata is available; see workflow specifications and metadata.

Publisher adapters pass persisted trace metadata in all existing service publishers. Listener runners initialize the optional exporter in template, broker, communications, documents, foundry, intelligence, payments, residentlogistics, revenue, and workspace. The Billing and Revenue consumer callbacks also receive per-message context, but Revenue's API still needs exporter startup to record its own spans. Other APIs outside the original four do not gain exporter startup from this change.

The architecture decision is docs/decisions/0070-http-tracing-and-durable-trace-context.md↗.

Use the standard HTTP path

httpapi.PlatformRouter installs incoming tracing. For a router assembled manually, use httpapi.Middleware. The order is request ID, tracing, request logging, panic recovery, then request metadata. Logging sees the active span, and tracing sees the 500 response produced by recovery.

Create internal HTTP clients at startup with httpapi.NewClient(timeout) and pass them to generated clients through WithHTTPClient. See generated service clients. The client preserves HTTP behavior and adds trace context and the current request ID. It does not add authentication or retries. Use it for internal services, not arbitrary external URLs.

Handlers pass the context they receive to client methods and database calls. Use slog.InfoContext(ctx, ...) and the corresponding context-aware methods for logs. Ordinary slog.Info(...) calls have no request context and cannot be correlated.

The shared logger adds trace_id and span_id as hexadecimal strings. It also adds dd.trace_id and dd.span_id as decimal strings for Datadog log correlation. The Datadog trace field is the lower 64 bits of the OpenTelemetry trace ID. These fields remain at the root of the JSON record even when a logger uses WithGroup.

Server span names contain the matched route pattern. Client URL attributes contain only the scheme and destination host. The platform does not capture request bodies, arbitrary headers, query values, or variable path values in spans. This does not change the existing request-log format.

Start and stop the exporter

The serving process calls this after initializing logging:

stopTracing := tracing.Start("your-service")
defer stopTracing()

Import common/tracing. Shut down the HTTP server and wait for message workers before returning, so their spans finish before the deferred exporter shutdown. Listener processes also call Start and wait for consumers to stop before returning. Do not call Start in migration or OpenAPI export mode. The template API shows this ordering: services/template/cmd/api/main.go↗.

Configuration comes from the process environment. Its source is common/tracing/tracing.go↗ and the OpenTelemetry SDK/exporter.

Trace context propagation uses W3C traceparent and tracestate. Baggage is not propagated. A request ID is still a separate field; it is not converted into a trace ID.

An unavailable receiver does not prevent serving requests. The exporter has a bounded queue and may drop spans. Shutdown waits at most five seconds. Export failures produce a warning without printing the configured endpoint.

Verify locally

These tests need no running services or external account:

go test -race -count=1 ./common/tracing ./common/httpapi ./common/logging

TestTracingAcrossHTTPServices starts two local HTTP servers using the platform middleware and client. It checks the incoming parent, both server spans, the client span, request ID propagation, log correlation, URL filtering, and exclusion of baggage. Other tests cover malformed context, independent requests, recovered panics, and probes.

TestStartExportsToLocalReceiverAndFlushes receives actual OTLP protobuf requests on a local HTTP server. It checks service identity and that shutdown exports the finished span. This verifies the exporter separately from in-memory span recording.

View local traces in Jaeger

Jaeger receives traces from the backend and displays the calls, their durations, and their HTTP status codes. It uses the same instrumentation as production and needs no Datadog account.

Start Jaeger

With Docker running, start a local receiver and viewer:

docker run -d --rm \
  --name flow-tracing-preview \
  -p 127.0.0.1:16686:16686 \
  -p 127.0.0.1:4318:4318 \
  cr.jaegertracing.io/jaegertracing/jaeger:2.21.0

Port 16686 serves the viewer; port 4318 receives OTLP HTTP traces. Both are bound to localhost. Skip this command if the container is already running. This version was used to verify local HTTP tracing. See the Jaeger setup documentation.

Enable tracing in the backend

From the repository root, set these variables and start the APIs in the same shell:

export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces
export LOG_FORMAT=json

./localenv up accounting-api billing-api leases-api

localenv passes the shell environment to its processes. The APIs run directly on the host, so localhost:4318 reaches Jaeger's published port. LOG_FORMAT=json is optional; it makes the trace fields in logs easier to inspect.

If the local stack is already running, stop it with ./localenv down before starting it with these settings. Exporting variables does not update a running process. These settings apply to this shell and processes started from it; repeat them when starting from another terminal.

Send a request and inspect its trace

Once Accounting is ready, run this in another terminal:

curl -i \
  -H 'X-Request-Id: jaeger-local-check' \
  http://localhost:8080/

Open the local Jaeger viewer, select accounting in the Service dropdown, and click Find Traces. Export happens in batches, so allow several seconds and refresh the page if the service has not appeared yet.

This request produces one GET / server span. An operation that calls another instrumented service produces a tree containing the incoming server span, the outgoing HTTP client span, and the receiving service's server span. Click a span to inspect its duration and attributes. HTTP 500 responses are marked as errors.

Find jaeger-local-check in the Accounting request logs to see its trace_id and span_id. Paste the trace ID into Jaeger's trace lookup to open that same request. Logs from downstream services share the trace ID and have their own span IDs.

The Service dropdown lists services whose traces Jaeger has received. Starting a service alone does not register it. The current tracing startup is installed in the template, accounting, billing, and leases APIs; other services need to adopt it before they can export. Health probes and documentation UI requests are excluded from tracing.

If no trace appears, check that:

• The API was started with the tracing variables in its environment.
• The request reached an instrumented API and was not a health probe.
• Jaeger is running and the exporter endpoint includes /v1/traces on port 4318.
• The API logs do not report tracing export failed.
• Sampling has not been reduced through OTEL_TRACES_SAMPLER or an incoming unsampled trace context.

Stop local tracing

docker stop flow-tracing-preview

This removes the temporary container and its in-memory traces. Run the startup command again for a new session. Jaeger runs separately from localenv, so ./localenv down does not stop it.

To disable export for subsequent API starts in the same shell:

export OTEL_TRACES_EXPORTER=none

Restart the APIs for this setting to take effect. Stop or disable their exporter when stopping Jaeger to avoid export failure warnings.

Enable Datadog later

Enable OTLP ingestion on the Datadog Agent on each application host, make its HTTP receiver reachable from the application, then enable export in the pilot API processes. OTLP ingestion is separate from Datadog's native trace intake; use the OTLP HTTP endpoint, not port 8126. No Datadog connection is needed for the local tests. See Datadog OTLP ingestion.

Check service/environment/version tags, sampling, and log correlation in the deployed environment. The local tests establish application behavior; they do not verify the cluster's Agent configuration. Traces can be sampled or expire, so use durable domain records when determining whether a business action happened.

Outbox and Service Bus tracing

The outbox header helper saves the active trace context inside the same transaction as the domain event. The optional fields are siblings of the existing metadata:

{
  "actor_fid": "...",
  "request_id": "...",
  "traceparent": "...",
  "tracestate": "..."
}

The publisher reads the trace metadata saved in the row and passes it in transport.Message.TraceParent and TraceState. The shared transport restores that parent, creates a send <topic> span for each publication attempt, and injects its context into Service Bus application properties. It does not rewrite the stored event envelope. Each span covers an application publication attempt, including any internal retries the Azure SDK performs.

A consumer prefers valid broker trace context, then falls back to the event headers. It creates a process <queue> span and passes its context to the handler. The processing span includes the handler and message settlement. Handler, abandon, and acknowledgement failures mark the span as failed. Existing retry, lock renewal, and deduplication behavior is unchanged.

Consumer callbacks now accept func(context.Context, transport.ReceivedMessage) error. Use that callback context for logs, HTTP calls, and domain operations, rather than capturing the background consumer loop's context. Events written with envelope.MarshalHeadersFromContext then save the current consumer span as their parent. The original request ID is restored from the envelope, with the broker correlation ID as a fallback. This does not restore actor permissions.

Shared transport logs include trace and span IDs when the process uses the shared logger. Publisher summaries without an event context remain uncorrelated. Trace attributes include destination, message ID, operation, and delivery count; event bodies and arbitrary message properties are not added to spans.

Each message carries its own context, even when several messages are sent in one batch. A batch must not assign one request's parent to unrelated messages. Each retry gets a new span; the event identity remains unchanged. Missing or malformed trace metadata must not prevent processing. Old producers and consumers remain compatible because the fields are optional and the envelope evolves additively.

A request can finish before its outbox row is published. Jaeger may show a gap between the HTTP span ending and the publication span starting. This is expected; it reflects the delay before publication. The gap is not an exact outbox age measurement, because the event may have been committed before the HTTP span ended.

An ordinary delayed delivery can continue the original trace without keeping its parent span open. A trace backend may display late spans separately. Replay tools and intentionally scheduled future work should start new traces linked to their origin when context is available; those integrations are outside this change. Ordinary outbox retries retain the originating trace. No span remains open for days waiting for work.

Test messaging locally

Fast tests need no broker or database:

go test -race -count=1 ./common/envelope ./common/transport

They cover serialized context, HTTP-to-message parent relationships, request ID restoration, malformed and missing metadata, sampling, cancellation, retry attempts, log correlation, and handler and settlement failures.

With the normal local Postgres and Azure Service Bus emulator running, opt into integration tests:

SERVICEBUS_INTEGRATION=1 go test -race -count=1 -v \
  ./common/transport \
  ./services/leases/internal/handlers \
  ./services/billing/internal/consumer \
  -run 'TestServiceBusTracing|TestLeaseHTTPOutboxServiceBusTracing|TestBillingServiceBusFollowOnTracing'

These tests create disposable topics, forwarding subscriptions, queues, and test databases and remove them afterward. They do not consume application queues. They use the Azure emulator on localhost ports 5300 (management) and 5672 (AMQP), and the same Postgres configuration as other repository database tests. A running emulator with management API support is required. The broker tests fail if opted in and the emulator is unavailable; database tests follow the repository's normal skip behavior when local Postgres is unavailable. Check the verbose output for skips.

The tests cover:

• A real Lease creation handler commits an event in a subprocess, which exits. The real Leases publisher then reads and publishes the committed event, and a shared consumer receives it through a disposable forwarding subscription. The test changes only the destination topic.
• A lease activation fixture travels through Service Bus to the real Billing handler, which commits a charge and a follow-on outbox event whose parent is the consumer span. This test starts from a fixture, not a running Leases API.
• Batch publication preserves distinct parents and reports partial failure correctly.
• A deliberately failed handler is redelivered with a new processing span, and the successful retry acknowledges the message.

To view these tests in Jaeger, start it using the commands above and export the same OTEL_TRACES_EXPORTER and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT variables before running the command. The domain tests print their trace IDs. Find them in Jaeger under leases-outbox-test and billing-consumer-test. The batch tests use messaging-batch-test and messaging-batch-failure-test. The retry test uses messaging-redelivery-test.

For normal service use, enable export in both the producing API and the consuming process. Perform an operation that emits an event to that consumer's queue. For example, lease activation produces lease.activated, which Billing consumes. Inspect the leases and billing spans in the same trace. An ignored or deduplicated message can have a successful processing span without changing business state; check domain records when verifying the business result.

Workflow identity

Optional workflow metadata identifies a business instance, specification version, and logical action independently of trace IDs. It travels through the shared HTTP, outbox, and messaging paths and appears as workflow.* fields on spans and contextual logs. No business workflow is tagged yet.

See workflow specifications and metadata for the format, authoring checks, propagation rules, and removal. Tracing does not depend on a workflow association or a runtime specification lookup.