Job kinds
Every queue kind on the Admin page, how work moves from one to the next, and which worker runs it.
The name of each job is the queue kind on /intelligence/admin. That page lists every kind and
what it does. Two workers split the kinds. The fast worker (intelligence-enrichment) handles
jobs people are waiting on. The scheduled worker (intelligence-batch) handles whole-company jobs
that run at night or when someone starts them. There is only one of those on purpose.
A solid arrow means the first job queues the second. A dashed arrow means the second job reads what the first wrote, or is scheduled after it on purpose.
rebuild_packs and reclaim show up on the Admin page but are not rows in the queue table: the
first fans out compile_pack over stale projects, and the second runs inline because a dead worker
cannot be repaired by adding a row to its own queue.
The longer write-up of each kind is docs/intelligence/jobs.md.
Diagram source
flowchart TB
WS[workspace event]
WS --> OBS[observe]
WS -->|speaker assigned| REL[relink_authorship]
WS -->|meeting summary ready| PCM[promote_commitments]
WS -->|unfiled conversation| SUG[suggest_filing]
WS -->|company fact| CCF[compile_company_facts]
REL -.-> RAS[reassess_epistemics]
OBS --> EXT[extract]
EXT --> VG[verify_grounding]
EXT -->|published claim| DC[detect_contradiction]
EXT -.->|marks the project briefing stale| CP[compile_pack]
CCF --> CP
EXT -.-> RES[resolve_entities]
EXT -.-> RAS[reassess_epistemics]
EXT -.-> DEC[decay_relationships]
EXT -.-> LP[link_projects]
EXT -->|a decider or influence was read| IA[infer_authority]
EXT -->|a decider or influence was read| IPP[infer_person_practices]
subgraph nightly [Nightly corpus pass, in this order]
CL[cluster_links]
RC[rollup_concentration]
RB[rollup_bottleneck]
DEC
RAS
RF[rollup_followthrough]
DIR[directory_snapshot]
IA
IPP
COMMIT[resolve_commitments]
SF[ingest_portfolio_snowflake] --> TER[compile_portfolio_terrain]
TER --> HAR[run_portfolio_harness]
HAR --> MEMO[compile_portfolio_memo]
MEMO --> BRIEF[compile_portfolio_weekly_brief]
end
CL -.->|packs rebuilt last| CP
REI[reindex] -->|one job per claim| EMB[embed]
EXT -.-> EXP[expire_candidates]
EXP -.-> RD[reviewer_digest]
DC -.-> PRO[prospect]
CAD[clear_all_derived] -.->|operator re-evaluates| OBSLive extraction writes search vectors in the same run as extract. The embed job is the later
rebuild: reindex walks the corpus and queues one per claim so the fast worker does the model
calls.
directory_snapshot and rollup_followthrough still run so a missing job does not look like a
broken schedule. Both currently skip: there is no directory source, and the follow-through counts
live in workspace.
Fast worker
These kinds are claimed by intelligence-enrichment.
| Kind | Does |
|---|---|
observe | Notices new text and records that we saw it. Queues extract. Writes no claims. On a project task it also drops the support of a reading an edit or a reopen replaced (Projects). |
extract | Reads that text and writes Insight: One claim the knowledge layer holds: a fact, decision, open question, or journal entry, derived from conversations and answerable across projects. It is never a source of truth — every insight points back at the evidence it came from., each with a quote. Calls the language model. |
verify_grounding | Checks whether each quote actually supports the claim. Pulls the claim back if none do. |
embed | Rebuilds the search numbers for one claim after the text or the model changes. |
resolve_entities | Re-runs mention detection on one claim so names can match people, teams, and tools added later. |
detect_contradiction | Compares a published claim with nearby claims. Marks both sides if they cannot both be true. |
reassess_epistemics | Lowers the weight of claims nobody has repeated in a long time, and withdraws a claim whose checked quotes do not support it. Does not delete the row. |
compile_pack | Rebuilds one project's Context pack: What an agent is given before it answers: one project's own state plus the company knowledge that bears on it, served by workspace from a single endpoint. It degrades to the project sections alone when intelligence is unreachable, and says so, so an empty knowledge section is never mistaken for nothing being known.. |
compile_company_facts | Rebuilds the one Company fact: A statement about the business that a platform admin has decided is settled — a legal entity name, what a product is, which market the company sells into. It is a record in workspace's fact domain, written as markdown or uploaded as a file, readable by anyone and writable only by a platform admin. Intelligence derives claims from it the ordinary way and then confirms them instead of queueing them for review, and those claims go into every project's context pack. Distinct from an insight whose kind happens to be `fact`, which is any claim the pipeline extracted from a conversation and which still goes to a reviewer. list every project's briefing reads. |
suggest_filing | Suggests which project an unfiled conversation belongs to. Does not move it. |
relink_authorship | After a human names a Zoom or Plaud speaker, rewrites who already-extracted claims belong to. Also moves claims from a record a model wrote, such as a Flow Tools PRD, out of the human tier. Does not extract again. |
promote_commitments | When a Zoom, phone, or Plaud meeting summary is ready, writes each next step as a commitment claim quoting the transcript lines it cites. Expires a commitment the newer summary no longer lists when that meeting was its only support. No model call. |
Scheduled worker
These kinds are claimed by intelligence-batch. Never more than one copy.
| Kind | Does |
|---|---|
cluster_links | Links claims about the same work, and flags duplicated effort. Nightly, first. |
prospect | Proposes a pattern inside one practice that none of its recent claims state alone. Goes to review, not publish. Sundays. |
rollup_concentration | Counts, per practice, who the published claims quote. Recent quotes count more. Does not read the links cluster_links wrote. |
rollup_bottleneck | Counts people that open questions pile up on and that those questions are not getting closed. |
rollup_followthrough | Meant to show work that started and did not finish. Skips; the counts live in workspace. |
decay_relationships | Fades graph edges that have gone quiet, and removes one once it is weak enough to be noise. Does not delete people or claims. |
expire_candidates | Closes Candidate: A proposed insight waiting on a decision. Most are resolved without a person: well-grounded corroborated claims promote automatically, poorly grounded ones auto-reject, and anything unreviewed past its deadline expires rather than accumulating. that sat too long. |
reviewer_digest | Tells each reviewer that items are waiting. Writes a notice; workspace delivers it. |
directory_snapshot | Meant to refresh who works here. Skips; no directory source is wired up. |
infer_authority | Rebuilds every observed authority claim from the signals in the graph. See Authority and practices. Nightly, and a minute after an extraction that read a decider or an influence statement. No model call. |
infer_person_practices | Re-infers which practices each person works in from their activity. Keeps a tag a reviewer confirmed or removed, and confirming can add a tag inference never produced. Same schedule as infer_authority. No model call. |
backfill_offices | Copies the officer appointments that older extractions saved as grant proposals into the offices table, creating the company entities they name. Manual. Safe to run twice. No model call. |
resolve_commitments | Ages open commitments past their deadline and marks the ones that were asked again. Nightly. |
ingest_portfolio_snowflake | Copies occupancy, units, vacant rent, spend, prospect counts, email lists, and hospitality POS counts from Snowflake onto named metrics. Skips when warehouse credentials are not set. |
compile_portfolio_terrain | Rebuilds named metrics, peer baselines, entity rollups, attention share, and the decision log. Counts and titles; no model call. Warehouse numbers win when ingest has run. |
run_portfolio_harness | Surveys commercial deviations and baseline-generated questions on that terrain, checks insight memory, kills data artifacts and seasonality, investigates a diverse slice, verifies with drilldown, and writes shadow findings that join a metric to a published claim, the operating book, audience or email counts, or two commercial metrics. Slack silence is not a source. Mechanical; no model call. Findings are not published. |
compile_portfolio_memo | Writes one executive memo from harness briefs that survived falsify. Mechanical numbers; a configured model may pick among survivor slugs and polish the prose. Empty if nothing survived. |
compile_portfolio_weekly_brief | Records reviewer precision for the ISO week. Copies at most five accepted findings into a weekly brief only after three completed weeks each have precision of 80% or higher and at least five reviews. Otherwise it skips and writes nothing. |
reindex | Walks every claim and queues an embed job for each. Manual, after an embedding-model change. |
link_projects | Fills in missing project edges on older claims. Safe to run twice. |
clear_all_derived | Removes claims the computer wrote, across every project, so they can be written again. Leaves claims a person confirmed. |
question_generate | Proposes company questions from recent evidence in the practices already on file. Every conversational question that clears the checks is shown. The nightly job reads that list. The Admin page can leave the subject blank for the same list, or type a phrase to limit that one run. Skips a repeat of the inquiry catalogue, a published decision, an open question, or a discovery packet. May propose none. A person approves one before it is investigated. An open assigned question is also sent as a Slack direct message that names the channel, meeting, or thread it was drawn from. A reply from that person, in the thread or as a message in the direct message, marks it already answered. |
question_investigate | Investigates one approved question through the existing discovery sweep. Queued when a person approves it. Not a button on the admin page. |
attention_brief | Drafts one daily brief for each active attention profile from recent company questions, discovery investigations, and watchlist project records. A reviewer approves the wording, then a separate action sends the email. May select nothing. |
dedupe_insights | Finds published claims that say the same thing in different words. A dry run, the default, only logs them; apply merges each group into one claim. Manual. See Duplicate claims. |
NOI-focused discovery investigation
discovery_investigate runs discovery-research-v14-relationship-paths from the
Admin Discover action and the batch worker. Overlapping-work and culture
questions are investigated first, alternating between those two lenses. The
remaining attempts are interleaved 40% insight-led, 40% original-evidence-led
and 20% other catalogue questions. Empty lanes lend their capacity. Original
sources are reopened using the configured reading principal; extracted insights
and their typed neighbors are navigation, not independent evidence. The total
run token cap still applies to every call.
An admin run can choose one category: profit, overlapping work, or culture. That run searches only those questions and skips identifier pairing. Its queue continues only on a later run of the same category. The scheduled sweep leaves the category unset and still covers every category. A queue from one category is not drained by a run of another. A pair whose records are already quoted together on a card in review or published is left, and the question goes on to the next discussions the search returned. From those discussions, the sweep also opens records cited by a linked claim. That link only chooses what to read. The claim text is not evidence. Half of a question's pair slots stay available for those linked discussions. A v13 queue is not continued: this version scans again.
The batch worker stops a sweep after 30 minutes so a dead worker can be replaced and so one sweep does not hold the batch pool. Four investigations run at the same time inside that window. Each investigation still waits on its own model calls. That stop is a checkpoint. Pairs not opened are stored as record ids on the run. The next sweep with the same researcher version rereads those originals and continues the queue. It does not scan the corpus again until the queue is empty or the researcher version changes. A changed version starts a new scan. Up to four unfinished candidates from the preceding run can still be refreshed, as described below.
The production scan reserves up to 100 Slack and 100 Zoom seed records (at most a quarter of the total record cap each) using caller-authorized workspace listings. This prevents a bulk document import from taking every recent-source slot. Each source listing examines at most its first 200 records, with topic rotation before hydration; it is not exhaustive department coverage. Listing failures are explicit run failures. Originals still pass the normal access and source-purpose checks.
Within each lane, candidate attempts rotate through business-topic hints from source title, channel and practice metadata. When those are unknown, two specific operational phrases in bounded source text can supply a navigation topic; this does not establish a department. Known cross-topic pairs go first; unknown attribution stays explicit. These hints guide navigation, not proof of departmental independence. Seed searches also rotate through these topic groups. Seed selection reserves a Slack and a Zoom record when available. Each evidence search also reserves its first relevant Slack and Zoom hit from the last 30 days (or the shorter trigger window). Workspace applies connector, date and access filters before ranking and limiting these searches; empty lanes lend their slots. The run warnings report the initial topic distribution and candidate exclusions. This can diversify the available pool, but cannot recover departments absent from the bounded initial scan.
Candidates must have evidence in the trigger window. Older evidence remains available as context. A non-document supporting citation dated in that window is a current-status label, not a publication gate. A recent unrelated reply cannot make old cited passages current. Historical and document-only cards are saved as leads marked current status unknown, because a document timestamp or an older event does not establish a live operating condition. Generic diligence questions without an observed practice or condition are rejected even when worded as hypotheses. Obvious revision-only document seed pairs are omitted. The investigator and verifier reject ordinary version changes as findings unless independent operational evidence establishes a remaining consequence or simultaneous incompatible use.
Each investigation reserves context for challenge searches, deduplicates preloaded records and can replace context with more relevant originals. Up to three research rounds follow missing mechanism links and competing explanations. Evidence selection uses short candidate labels mapped to authorized originals. Valid selections survive alongside invalid ones; an entirely invalid response gets one repair call. Valid empty selections are respected. Malformed research plans and follow-up plans also get one repair call, within the same run budget. An empty evidence search retains existing context without another ranking call. Source-purpose classifications are explicitly unverified retrieval hints. Citation span IDs resolve to exact displayed source text in code. Original source access, independent origins and claim verification remain mandatory.
A qualified opportunity does not need three originals or NOI keywords in every link. Its citations must establish the conditions it asserts; inference, missing links and proposed tests must be explicit. An incomplete but source-grounded mechanism is saved as a research lead in the review queue, with its citations and uncertainty. One independent origin is enough for a lead. A finding still needs two. Budget-interrupted investigations retain their last research checkpoint as incomplete. On the next sweep, up to four unfinished candidates from the preceding run are refreshed through original-source access checks. Interrupted work can resume twice; a research lead resumes only when a retained source changes. This bounded continuation is not yet a corpus-wide persistent frontier or a source-change event subscription.
A researcher-version change does not replace the next run with a diagnostic replay. Technical failures from the previous report stay eligible for a separate replay. The scheduled run records that they exist and continues as a fresh sweep. A failed record search or utterance search does not discard the list that succeeded, and a single request timeout does not stop the sweep while the run context is still alive. Valid no-finding outcomes are not retried. Every selected pair receives the run's date window before research and citation checks, including pairs restored for replay. The v9 replay path omitted that window and could label current citations as lacking operational support. v11 saves those labeled leads instead of withholding them. The shared investigator no longer treats net operating income as the question; that reading is the profit lens, which remains the catalogue default. v12 adds overlapping-work and culture inquiries. An overlap card may ask for a coordination check and may not treat duplication as established. A culture card is about one person, or about people the sources show doing the same work. It can record someone lifting the work, or a drop from that person's own earlier words, including frustration, stuck work, and a stated way of working the behavior contradicts. Speaking time alone is not the reading, and the card is not a judgment of the person. There is no card that joins separate people's separate work. v13 opens those two lenses before insight leads, related evidence, and the earlier profit questions, and keeps the unopened queue for the next sweep. v14 also opens discussions reached through a linked claim, alongside the keyword search, and does not continue a v13 queue.
Run cards show the research version, lane attempts, missing inquiry data and
research steps. The displayed window applies to recent triggers; historical
research can retrieve older evidence. Complete means the bounded sweep finished,
not that all company records were covered. The Admin action requires deployment
of both the API/worker and the UI; apply migration 0075_discovery_research_leads
before the new worker writes leads or incomplete outcomes.
The recovery package does not yet provide a full business-episode index, semantic original-passage retrieval, automatic property/entity joins, measurement adapters or realized-NOI attribution. Those later milestones require coverage checks and reviewed business episodes. More packets alone is not a release criterion.
The Discovery screen's Insights count includes readable evidence-led packets and numerical cycle findings. Its investigation list pages through all readable packets, with 25, 50, or 100 per page. Status filtering resets to the first page. An unchanged source revision does not produce a second packet when the job runs again.
Discovery execution and recovery
Repeated Discover requests return the existing claimed job. The queue serializes discovery claims and retires pending duplicates left by older deployments. A database session lock prevents concurrent investigation execution across replicas. Other batch job kinds can still run concurrently.
Identifier matching keeps at most 160 compact candidate references per business-topic combination before hydrating the retained pairs. It does not materialize the full quadratic candidate list. The initial scan holds at most 64 MiB of conversation text. Document identity stays in that scan, and the document body is read when the document is selected for a pair. Each loaded original retains at most 64 KiB of text. Oversized originals are explicitly marked partial, and search hits start the excerpt at the matching utterance. Exact source spans remain citable, but omitted text cannot establish absence or unresolved status. Run warnings count partial reads and report when the conversation text budget is reached. This does not provide exhaustive transcript coverage or a full-document content hash.
Deploy the workspace API with the new source and since search filters before
the intelligence API and batch worker. Keep migration 0076 applied for the
existing run/job recovery linkage; this change adds no migration.
Run cards are diagnostic attempts linked to queue jobs, not a count of live workers. Source-read and between-pair checkpoints update the existing card. Findings are published only by the final transaction. An admitted candidate awaiting that commit is checkpointed as incomplete without a packet link, so a killed attempt cannot advertise an unsaved finding. Up to four incomplete candidates can be reopened by the existing continuation path; source enumeration is repeated on retry.
The worker logs job start, run start, source progress, pair/model work and completion. An orderly deadline saves finished work as a partial run and does not automatically restart with a fresh token allowance. Unhandled failures count toward the retry limit. Reclaiming a crashed worker also enforces that limit (five attempts by default), then leaves a dead job for inspection. The minute-by-minute reclaimer closes run cards whose linked job is no longer claimed or whose lease expired. Legacy cards without a job link are closed after 35 minutes without an update. Historical cards are retained; they are never deleted to hide a crash.
Deploy migration 0076_discovery_run_job.sql before the updated API and workers.
The new reclaimer repairs abandoned legacy cards automatically. Check the Admin queue
for one claimed discovery job, advancing checkpoint logs, and stable pod restart count.
Evaluate discovery quality
Export a readable run's report JSON and assemble a frozen, authorized set of
reviewed reference episodes. Run the offline scorer:
go run ./services/intelligence/cmd/discoveryquality < reviewed-run.jsonThe input has report and cases. Each case supplies a unique id, the zero-based
candidate index in report.considered (or -1 when retrieval missed the episode),
required_records containing the original record UUIDs, and should_promote.
Optional useful, novel, and critical_error are human labels; omit them until
reviewed. Never infer these labels from whether the system saved a packet.
The scorer measures record recovery within one candidate bundle, false promotion, reviewed usefulness/novelty, critical errors, unreviewed candidates and tokens. Record recovery is only a coarse retrieval measure, not passage recall or proof that a claim is supported. Compare versions on the same corpus, cases and token allowance. Include different-property traps, copied sources, later corrections and cases with insufficient evidence. No production quality score is established by the synthetic regression tests.