Operations

Workspace is in DEPLOYED_SERVICES, so its paths are part of the merged platform OpenAPI document.

Releases

One image, twelve releases. Each release picks which binary to run with container.command (services/workspace/.azure/azure-pipelines.yml↗):

The MCP gateway used to be a release here. It now builds and deploys from its own pipeline, because it fronts intelligence as much as workspace and has no schema to own: see MCP Gateway.

The API also drains the outbox. Each poll opens a transaction and takes a transaction-scoped advisory lock for that transaction, so only one replica publishes no matter how many are running. The rows for one topic go out in a single Service Bus send. A send that does not finish within 10 seconds fails, those rows stay unpublished, and the next poll tries them again. Migrations run as a Helm job before each install and upgrade. The install step takes release names from the charts list in services/workspace/.azure/azure-pipelines.yml↗ — a copied template-api name would overwrite the template service and run workspace migrations against its database. Helm 4 applies server-side. The upgrade passes --force-conflicts so a live kubectl patch or kubectl set image cannot block the next deploy.

The old pods keep serving until the new ones are ready, so they run against the migrated database for the length of the rollout. Migration 0047 makes a task's type, status, and rank required, and the server before it does not write them: a task added through an old pod during that release's rollout is refused, and one it resolves keeps an open status. A project an old pod creates then gets General, the column's default, but no saved views: the old server does not make them, and migration 0048 gave views only to the projects that existed when it ran. The app shows an unsaved List view until someone saves one. Creating a project from a template with starter tasks fails outright on an old pod, since those tasks are refused. Deploy that release when nobody is adding or closing tasks, or creating projects.

Configuration

Values that are not secret are committed in .azure/env/config.<env>.yaml. Everything else is encrypted with SOPS in a file next to it. No secrets file has been committed yet, so a first deploy will fail at the decrypt step until one exists. services/workspace/.azure/env/readme.md↗ lists what it needs to contain and why each item is a secret rather than ordinary config.

Project files go to the documents catalog entry named here. Documents owns the Azure account and signs the SAS; these two keys are how workspace picks the destination.

The value must be an account listed in documents' DOCUMENTS_STORAGE_ACCOUNTS (the Azure account name). Documents owns the connection string and signs the SAS. See documents operations.

Zoom meeting audio is separate from those project files. When a meeting ends, zoomcapture uploads one mixed WAV. If the bot's pod was replaced partway through, the replacement uploads its part under a -part-<unix time> name instead of overwriting the first. The launcher copies the three settings below onto that sidecar from this service's config, so the next meeting uses whatever is deployed here. RECORDING_STORAGE_KEY, in the encrypted secrets file, has to be a key for that same account.

Meeting summaries are written by workspaceMeetingSummarizer. These keys are committed config; SUMMARY_API_KEY is in the encrypted secrets file. With no summary endpoint or key set, the summarizer uses TRANSCRIPTION_BASE_URL and TRANSCRIPTION_API_KEY. The API reads COMMUNICATIONS_URL and INTELLIGENCE_APP_URL too, for the share route.

Zoom bot Jobs

workspace-zoomlauncher starts one Kubernetes Job per meeting, with the bot and the zoomcapture sidecar in one pod. The bot records without asking the host and stays until the meeting ends (docs/decisions/0083-the-zoom-bot-stays-until-the-meeting-ends.md↗). It joins as a licensed Zoom user so the account's auto-approve recording setting applies. Two config keys control that:

Both are empty in dev and prod until the Zoom account setup in docs/zoom-auto-join.md↗ is done. That setup also adds the user:read:token:admin and meeting:read:local_recording_token:admin scopes to the Server-to-Server app.

Each line of speech is transcribed with a short prompt naming the meeting and the people in it, from the bot's participant list ("Meeting: Leasing sync. People in this meeting: Ilan Stern, Sara Garner."). The model reads it as earlier speech, so names come out spelled the way the people in the room spell them, rather than as whatever they sound like. Bots and participants the bot has not identified are left out, and the prompt is cut at 600 characters. zoomcapture probe takes the same prompt from TRANSCRIPTION_HINT.

How a bot Job behaves:

• The bot's exit code decides what happens next. 0 means the meeting ended. 1 starts a replacement pod, up to two. 2 means a new pod would fail the same way (bad credentials, a wrong passcode, removed by the host, an SDK version Zoom no longer accepts) and fails the Job. The bot's log says which, with Zoom's end reason.
• The pod is annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "false", and an eviction does not count against its retries.
• A later occurrence of a meeting that already has a bot is created suspended. The launcher unsuspends it when Zoom reports the earlier occurrence ended, or within 30 seconds of the earlier Job finishing. This needs the patch verb on Jobs in the launcher's Role.

Zoom raises its minimum Meeting SDK version every quarter. The next enforcement is 7 November 2026; a bot below the minimum logs CLIENT_INCOMPATIBLE and exits 2. The SDK version is the zoombotImage tag in services/workspace/.azure/k8s/values/zoomlauncher.prod.yaml↗.

Two secrets are worth calling out here, because another service holds the same values:

• SERVICE_TOKEN is the same string in workspace and intelligence, and each uses it to call the other. SERVICE_TOKEN_PREVIOUS is also accepted, so during a rotation the two services can be redeployed one at a time instead of together.
• SCHEDULER_TOKEN is how the scheduler proves who it is. If it is unset, the /internal/jobs/* routes are not registered.

Bus topology

The service that publishes an event owns the topology for it (ADR 0016). So intelligence's subscription to workspace's events is declared here, in services/workspace/.azure/bus/topology.yaml↗:

workspace-conversation-queue-intelligence holds its message lock for five minutes instead of the one-minute default. Its consumer may still be waiting on a model call when the default expires. When that happens the broker hands the message to someone else, the work starts a second time, and neither attempt finishes.

The evidence queue carries three events that its consumer handles straight away rather than putting on a queue: evidence.redaction_recorded, evidence.deleted, and evidence.knowledge_changed when it is being turned off. Those mean "stop using this now", so they hold the lock while the database work runs. Delete is stronger than a redaction or opt-out: citations to the record drop, and an insight with no remaining support is removed. evidence.speaker_attributed is queued as relink_authorship — it rewrites who already-extracted claims belong to, and does not extract again. evidence.summary_ready for a Zoom, phone, or Plaud meeting is queued as promote_commitments, which turns the summary's next steps into commitment claims. Other sources are ignored.

The fact queue carries the four events a curated fact publishes — fact.recorded, fact.updated, fact.superseded, fact.archived. The first two mean derive from this text, and the last two mean withdraw what was derived from it, which intelligence runs immediately for the same reason it runs a redaction immediately.

audit is on every workspace domain. workspaceAuditListener writes one human-readable row per event into the workspace audit table. Review is GET /v1/audit, platform admins only. Intelligence is on conversation, evidence, project, and fact text, not on the fact that a file was attached.

In the other direction, workspace-knowledgelistener reads intelligence-knowledge-queue-workspace, which is declared in intelligence's topology.

Event payloads carry ids only, never the text of a turn. That is what keeps them on the core topics instead of needing the separate PII ones (ADR 0005).

Scheduled work

services/workspace/.azure/kestra/expire-sessions.yaml↗ runs every fifteen minutes. services/workspace/.azure/kestra/unblock-reminders.yaml↗ runs once a day:

services/workspace/.azure/kestra/person-context-refresh.yaml↗ runs every half hour. Those runs only read sources that changed since the last run. A nightly run reads the whole 90 day window again and retires assignments that no longer name the person. The job refreshes everyone in the people roster and anyone workspace already knows. It skips people who turned personalization off. If intelligence is down, it still refreshes the people workspace knows and reports that the roster could not be read.

The grant sweep runs even if the session sweep failed. An expired grant is an open door. It should not stay open because an unrelated task timed out.

services/workspace/.azure/kestra/meeting-summaries.yaml↗ calls POST /internal/jobs/meeting-summaries on workspace-meetingsummarizer every two minutes, using the Kestra key workspace-meetingsummarizer-url. That key has to be set before the flow can run. One call writes up to five summaries and stops after four minutes. A call where every summary failed answers 502, so Kestra shows the failure.

services/workspace/.azure/kestra/slack-sync.yaml↗ hits workspace-slacksync, not the API, because a run is paced for Slack's rate limit and would sit in front of user traffic. services/workspace/.azure/kestra/question-slack.yaml↗ hits workspace-questionslack once a day and posts each open assigned company question as a direct message. The message names the channel, meeting, or thread the question was drawn from. Slack posts replies to POST /slack/events on that release. A reply in the question's thread answers that question. A message that is not in a thread answers the latest open question in the direct message. Subscribe the app to the message.im event. It needs chat:write, im:write, and users:read.email (it finds each person by their Flow address) in addition to the read scopes the sync already uses. SLACK_SIGNING_SECRET and INTELLIGENCE_SCHEDULER_TOKEN are secrets on that release.

workspace-questionslack also sends goal check-in reminders. services/workspace/.azure/kestra/goal-checkin-slack.yaml↗, in the flow.cron.workspace namespace, calls POST /internal/jobs/goal-checkin-slack on it at 13:00 UTC on weekdays. That is 09:00 in New York in summer and 08:00 in winter, so a run always falls on the New York day it reminds about. It reads the same Kestra keys as the question flow, workspace-questionslack-url and workspace-scheduler-token, and the release uses the same Slack token and scopes. Each run messages the lead of every due or overdue goal on an active project, once per goal and window, and answers {sent, skipped, failed}. A window is the state and the check-in day, so a goal whose check-in is missed gets at most two messages that week: one on the check-in day and one on the next weekday. A goal less than a week old is not due yet. A lead who is not a member of the project, has no Slack account, or was already messaged for that window is skipped. A window is recorded only after its message is sent, so a failed send is tried again on the next run. Runs are paced for Slack, about two seconds a message, so the job allows itself fifteen minutes and the flow waits as long. Two runs at once are not prevented, so a manual run during the scheduled one can message a lead twice. Without the flow nobody is reminded, but goals still show as due and overdue in Flow, because that is worked out whenever a goal is read.

The pipeline does not deploy Kestra flows. Upload services/workspace/.azure/kestra/goal-checkin-slack.yaml↗ by hand once the workspace-questionslack image with the route is out in that environment; before that the job answers 404. In dev the release uses dev's SLACK_TOKEN; if that token is for the company workspace, goals in the dev database message real people. What due and overdue mean is on Tasks, statuses, and goals.

On workspace-slacksync, SLACK_CHANNELS is a comma-separated allowlist of channel ids. When it is empty, SLACK_DISCOVERY decides what a run reads:

Under public the bot joins any public channel it is not already in, because a bot token gets not_in_channel from conversations.history otherwise. That needs the channels:join scope, and it gives up the consent signal that joined relies on: the app ends up in channels nobody asked it into, and each join posts a visible "has joined the channel" message. SLACK_MAX_JOINS_PER_RUN caps how many it joins per run, so a large workspace fills in over successive runs; the run's response reports pendingJoins until the backlog is empty. Private channels are unaffected either way — Slack has no API that adds an app to one. Direct messages are never imported. A private channel is still recorded, but it does not feed the knowledge layer unless SLACK_CONTRIBUTES_TO_KNOWLEDGE is set to true.

A run is bounded by SLACK_SYNC_BUDGET and gets through a few dozen channels, so it records where it stopped in evidence_slack_sync_cursor and the next run resumes after that channel. When the cursor cannot be read, the run falls back to a rotation derived from the clock. That fallback advances one channel per interval while a run covers around twenty, so on its own a full pass would take as many intervals as there are channels.

The response reports revisit, an estimate of how long a full pass takes at the current rate, and sets stale when that is longer than SLACK_SYNC_WINDOW. Stale means threads that move between two reads of the same channel are never picked up: Since is always now minus the window and there is no per-channel watermark. The fix is a wider window, a bigger budget, or fewer channels — not a faster schedule, because the limit is Slack's per-app rate limit rather than the trigger.

cmd/slackbackfill reads history the scheduled sync never got to. It sweeps newest-first across every channel the app is in — the last week everywhere, then the week before that, and so on — rather than finishing one channel at a time, because a run takes hours and the order decides what the corpus has while it is still going. -slices sets the boundaries and -days the outer bound.

It is safe to stop and safe to repeat: records are found by external_ref and lines by client_line_id, so a re-read writes only what is missing, and progress is a cursor row under its own name so a restart resumes at the slice and channel it stopped on. -restart starts over, -dry-run reports the plan and its cost. It joins nothing — that is the scheduled sync's job, and doing it from a hand-run tool would announce the app in a few hundred channels as a side effect of a backfill.

Slack's pacing is per app, so running this while the scheduled sync is working splits one budget rather than doubling it. Reckon on one call per thread found, which is the term that decides the runtime.

services/workspace/.azure/kestra/azuredevops-sync.yaml↗ hits workspace-azuredevopssync, not the API, for the same reason. AZURE_DEVOPS_REPOS is a comma-separated org/project/repo allowlist. When it is empty, each run is idle rather than scanning every repository the token can see. Private repos default to not feeding the knowledge layer unless AZURE_DEVOPS_CONTRIBUTES_TO_KNOWLEDGE is set to true. Drafts are skipped. cmd/azureimport is the hand-run backfill for one repo. Azure DevOps list-pulls has no last-updated sort, so a run covers every open PR plus completed and abandoned PRs that closed inside the window. A description edited on an old closed PR is picked up by the backfill, not the scheduled window.

services/workspace/.azure/kestra/github-sync.yaml↗ hits workspace-githubsync, not the API, for the same reason. GITHUB_REPOS is a comma-separated owner/name allowlist. When it is empty, each run is idle rather than scanning every repository the token can see. Private repos default to not feeding the knowledge layer unless GITHUB_CONTRIBUTES_TO_KNOWLEDGE is set to true. Drafts are skipped. cmd/githubimport is the hand-run backfill for one repo. Prod allowlists life-in-flow/flow with knowledge contribution on.

Flow Tools

workspace-flowtoolssync (services/workspace/cmd/workspaceFlowToolsSync↗, local port 8148) reads every repository in the FLOW_TOOLS_GITHUB_ORG organization, which defaults to life-in-flow-business. It is triggered with POST /internal/jobs/flow-tools-sync and the scheduler token. It uses its own token, FLOW_TOOLS_GITHUB_TOKEN, which needs read access to contents and pull requests. Archived repositories and those named in FLOW_TOOLS_SKIP_REPOS are skipped unless FLOW_TOOLS_INCLUDE_ARCHIVED is true.

Each run does two things per repository:

1. Pull requests are written as github evidence, the same way the GitHub sync writes them, with the flow-tools label. When a description is empty or shorter than FLOW_TOOLS_MIN_DESCRIPTION characters (default 80), a model writes one from the commits and changed files. The author's text stays first, followed by a line saying the description was generated by Flow Tools. A pull request whose head commit has not changed is not sent to the model again.
2. One note under the flow_tools source, titled <owner>/<repo>: product requirements (Flow Tools), describes what the repository is for. Its body starts with the repository URL, which links it to that repository's pull requests. It is rewritten only when the default branch has moved and a pull request has merged since the last summary.

The model is the one intelligence extracts with, read from the same EXTRACTION_PROVIDER, EXTRACTION_BASE_URL, EXTRACTION_API_KEY, EXTRACTION_MODEL, and EXTRACTION_REASONING_EFFORT settings. The broker and openai providers are supported. FLOW_TOOLS_CONTRIBUTES_TO_KNOWLEDGE has no default and must be set to true or false, because the repositories are private. FLOW_TOOLS_SYNC_WINDOW, FLOW_TOOLS_SYNC_BUDGET, and FLOW_TOOLS_SYNC_MAX_PAGES bound one run the same way the GitHub sync's settings do.

There is no Helm release or Kestra schedule for it yet. The first backfill is run by hand:

go run ./services/workspace/cmd/githubimport -org life-in-flow-business \
  -token-env FLOW_TOOLS_GITHUB_TOKEN -label flow-tools -describe -prd -dry-run

Drop -dry-run to write. See docs/decisions/0094-flow-tools-github-ingestion.md↗.

services/workspace/.azure/kestra/azuredevops-sync.yaml↗ hits workspace-azuredevopssync, not the API, for the same reason. AZURE_DEVOPS_REPOS is a comma-separated org/project/repo allowlist. When it is empty, each run is idle rather than scanning every repository the token can see. Private repos default to not feeding the knowledge layer unless AZURE_DEVOPS_CONTRIBUTES_TO_KNOWLEDGE is set to true. Drafts are skipped. Open pull requests are always read; completed and abandoned ones are read when they closed inside the sync window. cmd/azureimport is the hand-run backfill for one repo. Prod allowlists lifeinflow/Flow/flow-technology with knowledge contribution on.

Recordings from a Plaud device arrive through the companion app in frontend/plaud-app↗. The app pulls audio off the recorder over Bluetooth and uploads it with the file and meeting calls: POST /v1/files/upload-url, the bytes straight to storage, POST /v1/files, then POST /v1/meetings with source=plaud and external_ref=plaud:<serial>:<sessionId>. While the recorder is connected and recording, the app also streams audio to the phone and uploads it in one-minute blocks. That upload asks for purpose=live_recording on the upload URL, which stays valid for six hours. If the stream misses audio, the app exports the finished file instead. A recording made on the phone's own microphone uses source=phone and external_ref=phone:<install id>:<uuidv7>, with no device_serial, and records in the same one-minute mp3 slices through a native capture rather than one file at the end. A transcript with no lines deletes the meeting, because the app creates it before anything has been said.

A recording can outlast the six hours its write URL is good for. The app asks for a new one with POST /v1/files/upload-url/renew half an hour before the old one expires, and once more if storage refuses a block as expired. The blob name stays the same, so the blocks already written are still there — storage keeps uncommitted blocks for seven days — and the recording continues rather than being closed and restarted. Workspace records each live upload it mints a URL for and refuses to renew one for a different principal, or one that already has a document recorded against it.

Those recordings are transcribed while they are being made, not after. The app opens the meeting with live: true and posts each minute of audio to POST /v1/meetings/{evidenceId}/audio-slices; the API transcribes it on arrival and the words land on the record as lines. When the recording stops, the app uploads the finished file and says how many slices it sent. If every one of them was read, those lines are the transcript and the file is stored without being read again. If any is missing, the document listener transcribes the whole file and its lines replace the live ones — two transcripts of one recording cannot be merged, because the provider segments a whole file differently from a minute of it, so one has to win whole.

Both paths use the same provider, chosen the same way. Azure fast transcription (AZURE_SPEECH_ENDPOINT and AZURE_SPEECH_KEY) is preferred because it returns a speaker number per phrase; otherwise TRANSCRIPTION_BASE_URL with DOCUMENT_TRANSCRIPTION_MODEL. With neither set the slice endpoint answers 503 and an audio upload is skipped, which leaves the recording stored and untranscribed.

POST /v1/plaud/user-token and POST /v1/plaud/devices mint a token for the signed-in person and register a serial. Both take the Plaud user id from the caller. They answer 503 when PLAUD_CLIENT_ID and PLAUD_CLIENT_SECRET are unset.

services/workspace/.azure/kestra/plaud-sync.yaml↗ still hits workspace-plaudsync, which lists files with GET /files/. Plaud's documented API has no such call, so that worker does not receive recordings. See

and docs/decisions/0066-plaud-live-upload-and-phone-microphone.md↗, amended by docs/decisions/0067-live-transcription-for-plaud-and-phone.md↗.

Gmail

workspace-gmailsync runs services/workspace/cmd/gmailimport↗ continuously. A pass finishes, the process waits 15 minutes, and the next pass resumes from each mailbox's history cursor. The one-shot job is the same binary without -interval. services/workspace/cmd/gmailsurvey↗ counts messages in every non-suspended mailbox and writes nothing. services/workspace/cmd/gmailimport↗ uploads each message body, and each attachment the document extractor can read, to the documents service. The document listener turns those files into evidence the same way it does a Drive file. The evidence source is email, not document. The author is the From address. The account the job runs as stays the audit identity. Spam, trash, chats, and drafts are left out. Mail on an address list is not read. GMAIL_IMPORT_BLOCKED_ADDRESSES (or -block) is the general list. Four more lists cover people by location: GMAIL_IMPORT_BLOCKED_MENA, GMAIL_IMPORT_BLOCKED_EU_UK, GMAIL_IMPORT_BLOCKED_NY_CT_DE, and GMAIL_IMPORT_BLOCKED_CALIFORNIA. GMAIL_IMPORT_BLOCKED_DOMAINS drops mail to or from those domains, including subdomains. GMAIL_IMPORT_ALLOWED_DOMAINS is the exception that is not dropped as an external party. None of these lists are committed, because they name people and firms. An empty list blocks nobody, and the run says which lists are empty. GMAIL_IMPORT_ALLOWED_ADDRESSES (or -only) limits a run to those mailboxes. Empty means every mailbox the other lists do not skip. A mailbox on that list and on a block or location list is still skipped. The header prints the count, or that the allow list is empty.

A message is also dropped when it is about legal privilege, pay, health, harassment, a government ID, a resident application, wire instructions, or a secret, including an attachment whose name or extracted text is. One drop anywhere drops the whole message. Financial results, board mail, fundraising, and investor mail are stored, and so is mail on GMAIL_IMPORT_UNQUERYABLE_DOMAINS, but company knowledge is forced off for that message even when -contributes is set. A term sheet that ascribes a valuation is dropped instead. The check runs in the connector and does not send the text out to decide. The survey does not read bodies, so its count does not apply that check. It does skip mailboxes on the address, location, and blocked domain lists. -dry-run does not read bodies either; the number is an upper bound. The default window is 60 days. A mailbox whose previous run finished resumes from Gmail's history instead of listing that window again. That read is every message added since the saved cursor, including one Gmail recorded as arriving in the inbox or in sent mail. The cursor saved at the end is the history id on that response. A run that stops early does not move it. A dropped message is not marked ingested, and a finished mailbox does not read it again.

The same message in many inboxes is one document, keyed by its Message-ID header. -contributes defaults to off, so the bytes are stored and nothing is published as company knowledge. Turning it on publishes whatever the window still contains after the drops. Messages that are stored but not queried stay out of company knowledge anyway.

is the scope, amended by docs/decisions/0099-gmail-exclusions-drop-or-stay-out-of-knowledge.md↗. The service account is the one Drive already uses. The Workspace admin console has to grant it gmail.readonly and admin.directory.user.readonly as well. GMAIL_IMPORT_DIRECTORY_USER is the admin the listing impersonates. The pod is services/workspace/.azure/k8s/values/gmailsync.prod.yaml↗. The one-shot job is services/workspace/.azure/k8s/jobs/gmailimport.yaml↗. docs/decisions/0102-gmail-and-drive-keep-running.md↗ is why this is a pod rather than a timer inside the API.

Google Drive

workspace-drivesync runs services/workspace/cmd/driveimport↗ the same way, over every file the impersonated account can see. A file already stored is one state read. -contributes stays off. The one-shot job is the same binary without -interval.

is the pod.

Repairs run by hand

services/workspace/cmd/documentbackfill↗ files document evidence into the projects their documents were already uploaded to.

It exists for one bug. docsync used to build its evidence record from the documents.created event without reading the event's references, so a file uploaded to a project — which carries a project reference written by workspace's own file domain — became an unfiled record. Claims derived from those records have no project to be attributed to, which is how document material reached other projects' context packs with no stated origin.

Ingestion now keeps the reference, but only for records it creates fresh, so that a redelivered event cannot refile something a person deliberately unlinked. Existing records need this command:

documentbackfill -dry-run    # what would be filed
documentbackfill             # file it

It needs DATABASE_URL and DOCUMENTS_URL. It reads references from GET /v1/documents?ref_type=project rather than the single-document route, because that route mints a read SAS and would make a metadata repair depend on blob access. Re-running is safe: it finds records by the absence of a link, so a filed record is not revisited and a document with no project references is left alone.

Filing does not re-derive. Citations already extracted keep pointing where they did, so pick up the new attribution by re-deriving the affected projects from the intelligence admin page.

Failure modes worth knowing