Operations

Intelligence is in DEPLOYED_SERVICES, so its paths are part of the merged platform OpenAPI document.

Releases

One image, four releases. Each release picks which binary to run with container.command (services/intelligence/.azure/azure-pipelines.yml↗):

The install step takes release names from the charts list in services/intelligence/.azure/azure-pipelines.yml↗. Migrations run as a Helm job on intelligence-api only. The pipeline deletes that Job before each upgrade so a failed migrate cannot block the next deploy — Kubernetes will not let Helm change a Job's pod template.

The image build downloads Go modules before compiling the four binaries. Its Dockerfile retries that download twice after a transient proxy failure, with short backoffs. A persistent download failure still stops the build before an image is pushed or a release is upgraded.

The batch worker stays at one copy because each of its jobs reads most of the knowledge base. Two copies would double the database load to produce the same rows. Nothing would break — the queue's lease already stops a second pod taking the same job — so this is about cost, not correctness.

For the enrichment worker, turn up ENRICHMENT_CONCURRENCY before adding replicas. Those jobs spend almost all their time waiting for a model to answer, so a pod running one job at a time is a pod doing nothing.

Configuration

Values that are not secret are committed in .azure/env/config.<env>.yaml. The rest is encrypted with SOPS in a file next to it. No secrets file has been committed yet, so a first deploy will fail at the decrypt step until one exists. services/intelligence/.azure/env/readme.md↗ lists what it needs.

This service writes no blobs today. The reserved documents catalog entry is named here:

All four releases read the same ConfigMap and the same secret. That is not tidiness. They have to agree on the embedding model, because vectors written by one model and searched with another do not produce an error. They produce confident nonsense.

SERVICE_TOKEN is the same value workspace holds, and each service uses it to call the other: workspace calls here for a project's knowledge sections, and this service calls workspace for the text it works from.

Changing the embedding model

EMBEDDING_MODEL and EMBEDDING_DIMENSIONS are not ordinary settings. vector(N) is a column type, so the width is fixed by the first migration and cannot be changed in place.

The reindex flow (services/intelligence/.azure/kestra/reindex.yaml↗) has no schedule. A person runs it. It is committed so that running it leaves a record, rather than being a curl command somebody typed once. Expect hours, not minutes.

Spend

enrichment_budget caps how many tokens each kind of job may use per day. Tokens are reserved before the call and reconciled after, so going over the cap is refused up front instead of discovered on the invoice. A worker that hits a cap records the reason and puts the job off until the next UTC midnight. That is a defer, not a failure: the attempt count stays unused, so a spent cap cannot dead-letter the work.

Leaving it unset means no cap, which is right locally and wrong once deployed. The dev and prod caps are in config.<env>.yaml. Raising a cap updates today's row on the next reserve; it does not wait for midnight.

MAX_EXTRACT_TOKENS, MAX_VERIFY_TOKENS, MAX_CONTRADICTION_TOKENS, and MAX_PROSPECT_TOKENS are the per-call reply caps. They are not the daily budget. The Chat Completions body is chosen from the model family before the first request: GPT-5 and the o-series send max_completion_tokens, omit temperature, and may send reasoning_effort; older chat models send max_tokens. There is no probe request. Extract and prospect default to 128000. Verify and contradiction default to 4096: those replies are a score or a boolean plus a short reason, and a 128000 bound made each judgment take long enough that a few citations exhausted the enrichment worker's lease (the worker cancels 10 seconds before the lease, which is why the logs showed context deadline exceeded at 290 seconds).

EXTRACTION_REASONING_EFFORT is how hard a reasoning model thinks before it answers (none, low, medium, high, and the higher rungs a given model accepts). It is sent as reasoning_effort. Unset is low. If the field is omitted entirely, GPT-5.6's own default is medium.

Each call is charged to the job kind it runs under. A model-calling job kind with no configured cap falls back to a small default rather than to no cap, so the symptom of forgetting one is a stage that stops early in the day rather than an unbounded bill — and the fix is to add the variable, not to raise a limit somewhere else. BUDGET_EXTRACT_TOKENS, BUDGET_VERIFY_TOKENS, BUDGET_CONTRADICTION_TOKENS, BUDGET_PROSPECT_TOKENS, and BUDGET_QUESTION_TOKENS are the ones that matter today; the mapping from a job kind to its variable is in internal/config. BUDGET_QUESTION_TOKENS covers both proposing questions and investigating one approved question. Each kind has its own counter.

Scheduled work

Every flow posts to an endpoint that queues the work and returns. The batch worker does the actual work (ADR 0026). If Kestra restarts halfway through, nothing is lost, because the jobs are already rows in a table.

A reviewer can enqueue the same work from /intelligence/admin without holding the scheduler secret. Those routes are /v1/admin/jobs and /v1/admin/flows. They put the same rows on the queue.

The Queue table on that page is one row per kind. Pending is waiting. Claimed is in a worker. Wait is how long the oldest due job has sat. Typical is a successful run in the last day. A long wait with nothing claimed means no worker is taking that kind. A long wait with claimed jobs means the workers are slower than arrivals. Typical much longer than usual is the job itself. The same numbers are GET /v1/admin/jobs/health. /v1/admin/jobs/depth is still the pending and claimed counts Kestra records. Successful runs also log worker: job done with the kind and duration.

A harness tool named list_queue_errors reads that dead-letter list and the kinds that have a dead job. Agent Harness V1 offers it when a person grants it. Coding Harness V1 always offers it, including on a coding agent that was just started and has no other grants, so that run can read the error, change the repository, and open a pull request. requeue_queue_jobs and expire_queue_job put a dead job back or retire it. Only a person can grant them, and only to an Agent Harness V1 agent: a coding run ends before its fix is deployed.

The call is made with the service token on behalf of the agent's own mailbox (<slug>@ai.flow.life, or <slug>@dev-ai.flow.life in dev), not its owner. Intelligence lets that mailbox read, requeue, and expire dead jobs because the domain is in INTELLIGENCE_QUEUE_OPERATORS. Running a job from /v1/admin/jobs and promoting claims still need a reviewer.

What each kind does, and how they feed each other, is on Job kinds.

The order inside the nightly run matters, even though the calls themselves are fast. Clustering has to finish before the summaries read the links it writes, and the context packs are rebuilt last so they contain the night's work rather than yesterday's.

The digest only runs on weekdays. One sent on Saturday gets read on Monday next to Monday's, and each one then gets half the attention.

Re-deriving a project after extraction changes

Improving the prompt or the model does not improve what is already in the corpus, and re-reading the evidence does not replace it either. Extraction merges a new claim into an existing one only when the text is identical or the meaning is very close (see Duplicate claims). A better prompt usually writes text different enough to pass both checks, inserts it alongside the old claims, and leaves two readings of the same evidence standing. Nothing in the pipeline removes the first one, because from its point of view nothing about it is wrong.

Three steps, in this order, from /intelligence/admin under One project:

Clearing last would throw away the corrected claims, which is why the order is not a suggestion.

To clear every project rather than one, Every project on the same page queues clear_all_derived, which walks each project that has derived knowledge and applies the same guards. It is queued rather than immediate: one project is a few bounded transactions and a corpus is not. Re-derive and rebuild are still per project after it drains, because which projects to re-read is a judgement rather than a default. services/intelligence/.scripts/rederive-projects.sh↗ still runs all three steps across every project from a shell, and makes no requests at all without -apply.

Repairing project edges in the graph

Extraction writes an edge from each insight to the projects its evidence came from, but only for insights written since it started doing so. Anything older has a citation naming a project and no edge saying so, which shows up as a project that looks connected to nothing when it is opened in the graph.

Repair project edges under Every project queues link_projects, which walks the corpus and writes the missing edges. Safe to run more than once: edges are reinforced rather than duplicated, and the nightly decay brings an over-reinforced weight back down.

Org import

orgimport loads an org chart: each person's formal title, their manager, and optionally their work email. It is a stopgap until the reporting structure comes from identity.

DATABASE_URL=postgres://... go run ./services/intelligence/cmd/orgimport -csv org.csv -dry-run
DATABASE_URL=postgres://... go run ./services/intelligence/cmd/orgimport -csv org.csv

The CSV has Name, Title, and Manager columns. Email is optional. Any other column, such as Notes, is ignored. A file with only the first three columns still imports. Manager is a name or a work email.

• Titles and managers are replaced, not added to. If a person has a different manager in the new file, the old manager row is closed at the time of the import and a new one is opened. The same goes for a title. The reports_to edge in the graph follows. Earlier values stay in person_title and person_reporting.
• Running the same file again changes nothing.
• An Email makes a link. The email becomes an accepted email identity link for that person, which the people lookup reports as login. If an earlier import created the person from a name alone, the email is attached to that person rather than creating a second one.
• Someone left out of a later file is kept. They are listed as absent in the report. Use the depart endpoint for a real departure. A manager cell left blank clears a manager that an earlier import set.
• A name that matches more than one person is not guessed. The row is reported as ambiguous, nothing is written for it, and a review item is opened in the identity review queue with the candidates. A reviewer decides through /v1/spine/identity-reviews. Fix the data (give each person an email) and import again.
• A dry run is exact. It does the whole import inside a transaction and rolls it back, so the report lists the titles it would set or change, the manager changes, and the ambiguous rows. It needs DATABASE_URL for that. Without one it only lists what it read.

Duplicate claims

A duplicate splits one claim's citations and source count across two rows, so neither row looks as well supported as the claim is. Two checks stop extraction from writing one, and a manual job merges the ones already in the corpus.

Identical text. At most one candidate or published insight may hold a given title and body; a unique index on the content hash enforces it. Extracting the same text again, under any practice, adds the new citations to that insight and adds the new practice to its practice list. The insight's primary practice does not change. When the text belongs to an insight that was merged away, the new citations go to the insight that replaced it, found through superseded_by_insight_id. The API, packet, table-analysis, and formal-model paths that create insights use this check too.

Close meaning. Extraction alone also compares new text with live insights by meaning. It embeds the title and body and searches company insights. A neighbour at a cosine distance of 0.10 or less (a similarity of about 0.90) is the same claim when all of these hold:

• it has the same kind, because a fact and a decision with the same words say different things
• its evidence shares a project with the new claim's evidence, or one of the two names no project, so two teams with no project in common stay separate and cluster_links can still report the duplicated effort
• no contradiction is recorded between them

Practice is not compared. The new claim's citations and practice go onto the existing insight, which keeps its own wording, search vectors, and primary practice. When the comparison cannot run, for example because the embedding model is unavailable, the claim is inserted as before. Text that already has a row in any status skips this check, so a rejected sentence extracted again stays rejected.

Cleaning up. Under Repairs and backfills on /intelligence/admin, Find duplicate claims (dry run) queues dedupe_insights in dry_run mode. It compares every published insight with its neighbours using the stored vectors, so it makes no model calls, and it writes nothing. The worker log has one dedupe: duplicate pair line per pair with its distance, and one dedupe: group line per group naming the insight it would keep. Read those before merging; 0.10 has not been tuned against labelled pairs, and the logged distances are what should move it.

Merge duplicate claims runs the same job in apply mode. In each group it keeps, in order of preference, the insight a person or a system of record vouched for, then a published one over a candidate, then the one with more independent sources, then more citations, then the oldest. Every other member is checked against that survivor again, and one that differs in kind, shares no project, or has a recorded contradiction is left alone and logged as dedupe: kept apart. The rest have their citations copied and their practices added to the survivor. They are marked superseded with a pointer to it and removed from search, and the survivor is queued for verify_grounding. Each group is its own transaction.

Migration 0085_insight_dedupe.sql merged rows that already shared identical text across practices when it ran, using the same survivor order. It could not recompute source counts or write events, so run Merge duplicate claims once after deploying it: apply mode recomputes every insight that a merged row points at before it looks for new groups.

Bus topology

services/intelligence/.azure/bus/topology.yaml↗:

attention publishes and has no subscribers. Briefs are read over HTTP, and the topic exists so the outbox can drain.

retrieval has no entry, and no outbox. Nothing outside this service cares that a chunk was embedded. A topic with no subscribers is still infrastructure somebody has to create, monitor, and drain, for no reader.

Event payloads carry ids, states, and scores — never a title or a body. That is partly a PII decision (ADR 0005) and partly a correctness one. A claim's text can be withdrawn by a redaction, and text copied onto an event is a copy nobody can call back.

Coming in, this service reads workspace-conversation-queue-intelligence, workspace-evidence-queue-intelligence, workspace-project-queue-intelligence, and workspace-fact-queue-intelligence. The fact queue is the one that never waits: recording or rewriting a fact queues an observation at interactive priority and marks the company facts section out of date, and superseding or archiving one withdraws the derived claims straight away. Those are declared in workspace's topology, because the service that publishes an event owns the topology for it.

Failure modes worth knowing