0017 — Structured logging and request correlation
One slog logger per process, JSON on stderr when deployed, and a request id that follows an event to its consumers.
Decided. One slog logger per process, writing JSON to stderr in every
environment except local, which gets a human-readable console handler. Every line
carries serviceDomain, env, and startupPath. Every line belonging to a request
carries requestId, and that id travels onto published events through the outbox
headers column — so one search returns the API call, the publish, and every consumer
that handled the resulting event.
This settles the logging half of the observability question. 0070 later chooses OpenTelemetry for tracing and Datadog for trace storage. Metrics and a PII redaction hook are outside this logging decision.
Fields
Log attributes are camelCase; wire and database fields stay snake_case. The log says
requestId, the outbox row says request_id, beside the actor_fid already there.
| Field | On | Value |
|---|---|---|
serviceDomain | every line | template, or template-domain1 when the line belongs to a domain |
env | every line | APP_ENV, defaulting to local |
startupPath | every line | repo-relative entrypoint, e.g. services/template/cmd/api |
requestId | every request-scoped line | echoed from X-Request-Id, or generated |
command | routed API requests | <domain>.<action>, e.g. domain1.create |
serviceDomain is a single field rather than a service field and a domain field
because it is an identity, not two facets: it names what emitted the line.
Alternatives
- A logging library (zerolog, zap, tint).
slogis stdlib and already at every call site. Only the console formatting is worth writing, and it carries no dependency risk. - JSON everywhere, including local. Cheaper, but the inner loop is where logs are read by a person rather than queried, and a wall of JSON is not read.
- A correlation id generated per hop. Self-consistent per service and useless as a
chain. An inbound
X-Request-Idis echoed instead. - Broker properties as the carrier for events. The id would only exist from publish onward. Committing it with the outbox row means it survives the producing process exiting before the publisher drains.
- APM tracing. Heavier, and it does not naturally survive an outbox delay of arbitrary length. Deferred rather than rejected.
Why stderr
Stdout is the OpenAPI channel: make generate runs cmd/api -openapi and redirects it
into a committed openapi.yaml. A log line on stdout corrupts that file and fails the
drift check with an error pointing nowhere near logging. A correctness constraint, not
a preference, and every service inherits it.
Datadog is a premise here, not a conclusion. The decision is structured JSON with
stable key names; the only vendor-specific part is renaming slog's time key to
timestamp, which Datadog's default JSON preprocessing recognizes. Change the vendor
and that one mapping changes. This decision added no Datadog configuration;
0021 later did,
for reading logs back in the operations console. That is a read path only — how a service
emits a line still depends on nothing but the keys above.
Consequences
- A consumer that writes its own rows in reaction to an event must copy the inbound
request_idinto its own outbox headers, or the chain stops there. No consumer does this yet — the template listeners only log — so it is a rule for the first one that does. Transport.Publishtakes aMessagestruct, and the broker correlation id is set fromRequestID. That copy exists for visibility in Service Bus Explorer; consumers read the envelope body.- Request logging wraps panic recovery rather than the reverse. The other order reports no status for exactly the requests worth investigating.
- Listener binaries no longer log each message as one JSON document. The identifying
fields are attributes; the full message sits behind
LOG_LEVEL=debug. - Correlation is best-effort throughout: a missing or unreadable id never fails a request, a publish, or a delivery.
See the template service architecture page for the concrete middleware chain and getting started for the environment variables.