0021 — Datadog log search in the DevOps console

Decided. The operations console reads retained logs from Datadog, through a proxy in devops-service. GET /v1/logs/config, POST /v1/logs/search, and POST /v1/logs/facets sit on top of services/devops/internal/datadog/↗, which calls the Datadog Logs API. devops-app renders them at /devops/logs and links out to Datadog's own Log Explorer for anything beyond reading.

This is the platform's first vendor integration, and the first read path in devops-service that is not scoped by region and subscription. Both are deliberate.

0017 chose structured JSON on stderr and left the backend vendor open, noting there was no Datadog configuration in the repository. There is now — but only on the read side. Nothing about how a service emits a line depends on this decision, and removing it would not change a single service.

flowchart LR
  browser["devops-app<br/>/devops/logs"] -->|"POST /v1/logs/search"| svc["devops-service"]
  svc -->|"env:dev AND (operator query)"| dd["Datadog Logs API"]
  browser -->|"Open in Datadog"| explorer["Datadog Log Explorer"]

The browser never holds a Datadog credential. It reaches Datadog directly only by following a link the operator clicks, which authenticates as that person's own Datadog session.

Alternatives

• Query the clusters directly, as the pod log pane already does. Rejected: kubectl logs reads one container's current buffer. It cannot answer "which service logged this request id", it returns nothing for a pod that has been replaced, and a rollout erases the evidence of what caused it. Retention and cross-service search are the whole point.
• Index logs ourselves — Loki, OpenSearch, anything with a query language. Rejected outright. The logs are already being shipped to Datadog and paid for. A second index would be a storage system to run, back up, and secure, in exchange for a query language the team knows less well.
• Call Datadog from the browser. Impossible, not merely unwise. The Logs API requires an Application key alongside the API key, and an Application key delivered to a browser is an Application key published. This is the reason the proxy exists at all.
• Embed the Datadog UI in an iframe. Rejected: it needs a separate Datadog session, cannot be scoped to an environment, and gives up the facets and layout that make the screen worth having over a bookmark.

Why the environment filter is server-side

Every outbound query is rewritten as env:<PipelineEnv> (<operator query>), where PipelineEnv is the dev or prod value the registry already carries. The operator's query is parenthesised so an OR inside it cannot widen the result past the environment.

Consequences

• The endpoints are top-level, not under /v1/environments/{region}/{subscription}. Datadog indexes every cluster in one place, so there is no per-subscription resource to address. Every other call in DevOpsClient is scoped, and it now carries a note saying this one is not.
• Two new secrets, and they are not per-subscription. DEVOPS_DD_API_KEY and DEVOPS_DD_APP_KEY belong in the encrypted per-environment secrets; DEVOPS_DD_SITE is non-secret configuration. The Application key needs the logs_read_data scope and nothing else. See devops-service operations.
• Absent keys are not an error. The client reports itself unconfigured, the endpoints answer 501, and the screen says log search is not configured. A deployment without the keys behaves exactly as it did before, which is what makes this safe to land ahead of the secrets.
• The facet list is ours, and it is config. Datadog exposes no API for an organization's configured facets, so services/devops/registry/logfacets.yaml names them: the tags the log agent attaches, plus @serviceDomain from 0017. It sits beside scopes.yaml because it is the same kind of thing — hand-maintained, mounted, and read at startup — and it is a file rather than Go so that adding a facet is a config change. The console gets the list from /v1/logs/config.
• A field Datadog will not aggregate on costs that one row, not the rail. Asking to count a field nobody has made a facet for is an ordinary misconfiguration; it is logged with the field name and dropped from the response. Only fields with something to click reach the console, because a group that cannot be clicked reads as a fault in the tool. Every field failing is a bad key rather than a bad list, and that is still an error.
• Facet selections are structured, not text. The console sends which values are ticked per field rather than pre-rendering them into the query string, because how a field's values combine is the service's call to make: values within a field are ORed and fields are ANDed, which is Datadog's own model. Kubernetes tag facets were dropped: the Containers view already navigates by cluster, namespace, and pod.
• The rail lists values; it does not count them. Each field was first counted against a query with its own selection left out, so that ticking one value did not empty the group it came from. That worked and cost too much: one aggregation per field on every click, to show the same values back, with the rail blank while it ran. Which values a field can take does not depend on the query anyway, so they are now read once at startup over the preceding fortnight and served from memory. The response carries no counts — a count from that reading would not describe the search the operator is running — only the order they imply, commonest first. Ticking a value reloads the table while the rail holds still, and a group shows its first five values with the rest behind an expander.
• Windows are resolved to absolute timestamps before the first request and reused for every page. Datadog's cursor pagination over a relative range slides the window under the cursor and drops rows between pages.
• Read-only, and it stays that way. Nothing here writes to Datadog. Monitors, dashboards, and alerts remain Datadog's own UI; the "Open in Datadog" link is the handoff.