Operations

Foundry is not in DEPLOYED_SERVICES yet, so its paths stay out of the merged platform OpenAPI document. The Azure pipeline deploys the Helm releases below.

Local

COMMUNICATIONS_URL, INTELLIGENCE_URL, DOCUMENTS_URL, and BROKER_URL default to the local ports. The harness reads the first two from the environment. Skill upload-url calls documents. The UI talks to broker for the catalog; foundry does not call broker on save.

Cluster

FOUNDRY_RUNTIME=cluster (or a pod with KUBERNETES_SERVICE_HOST) creates:

• Deployment foundry-agent-{agentId} for persistent start
• Job foundry-agent-{agentId}-{generation} for ephemeral spawn

The API release (agentRuntime: true) creates namespace foundry-agents, a ServiceAccount for foundry-api in default, a Role in foundry-agents for apps/deployments, apps/replicasets, and batch/jobs, and ServiceAccount foundry-harness in foundry-agents with no token. Image is Helm harnessImage (FOUNDRY_HARNESS_IMAGE), Debian 13 foundry-harness in that environment's ACR. Harness pods use the agent node pool from agentNodeSelector / agentTolerations. A NetworkPolicy in foundry-agents on app: foundry-harness denies all ingress and allowlists DNS plus those five APIs on 8080 in default. A second policy, foundry-harness-coding, also lets coding pods reach public TCP 22 and 443; see Architecture for the ranges it excludes and when policies are enforced. A reconcile loop marks leaked ephemeral members expired when ttl_at has passed and deletes the Job. It takes any member still active, including one that has been stopped, so resting a member does not postpone its TTL. A finished ephemeral run expires itself before the process exits. Restart of that agent starts a new Job, moves ttl_at four hours ahead, and runs the instructions on the row from the start. If Start or Stop fails, the API returns the kube error. kubectl logs on the harness pod should show one JSON object per line: harness started immediately, then heartbeat lines every 30 seconds, plus evaluating prompt / evaluated: every 15 seconds. A failed broker call logs evaluate failed with provider, model, and the broker message (including the upstream provider text). Foundry-api request failures put the cause on the same request log line as error (Cedar deny includes fid and action). Dev and production do not print plain-text log lines. The same events are stored on GET /v1/agents/{id}/logs (heartbeat N). Cluster pods inherit APP_ENV from foundry-api so the console format is not env: local.

Coding harness SSH keys

The coding pod mounts Kubernetes secret foundry-harness-ssh from the foundry-agents namespace at /etc/harness-ssh. The foundry-api SOPS file holds the API environment variables. The SSH keys live only in this Kubernetes secret. The pipeline does not create it. A missing secret does not stop the pod from starting. Clone fails until the secret exists.

The secret holds two private keys. Every coding pod uses the same pair. The remote host selects the file:

id_ed25519 is the GitHub key. azure_devops is the Azure DevOps RSA key. known_hosts must already contain a line for github.com and a line for ssh.dev.azure.com. Add a missing GitHub line with ssh-keyscan before creating the secret:

ssh-keyscan -t ed25519 github.com >> ~/.ssh/known_hosts

The dev and prod clusters use Azure AD. kubectl calls kubelogin for that login. Install Azure's binary, not the Homebrew formula that is also named kubelogin (that one installs kubectl-oidc_login):

brew install Azure/kubelogin/kubelogin

If the next error is from Azure login rather than a missing executable, run az login and retry.

Create the secret once in each cluster. kubectl must already be pointed at that cluster.

kubectl create secret generic foundry-harness-ssh \
  --namespace foundry-agents \
  --from-file=id_ed25519=$HOME/.ssh/id_ed25519 \
  --from-file=azure_devops=$HOME/.ssh/azure_devops \
  --from-file=known_hosts=$HOME/.ssh/known_hosts

The SSH key is who may clone and push. The commit email is the agent mailbox, [email protected] in prod and [email protected] in dev. The harness sets user.name to the agent name and user.email to that mailbox on every git command.

Opening a pull request needs a token on the foundry-api process. The coding pod receives GITHUB_TOKEN or AZURE_DEVOPS_TOKEN only when that variable is already set there. GitHub uses the token against api.github.com. Azure DevOps uses it against dev.azure.com. The remote host picks which one. Both keys go in foundry's secrets.<env>.yaml. When the token for the remote is missing, the pod writes a warn log line as it starts, because the agent would otherwise push its branch and fail only at open_pull_request. An Azure DevOps token that is expired or wrong comes back as HTTP 203, and the error says the host rejected the token.

open_pull_request refuses while there are uncommitted changes and pushes the branch itself when origin does not have the latest commit. When a pull request from the same branch into the same target is already open, for example after a Job restart, it returns that pull request instead of failing.

One run can open several pull requests. The tools do not decide which branches a change goes to or in what order; the task and the repository's instructions do.

• open_pull_request with base. Opens the pull request against that branch instead of the target. When the current branch does not start from the latest origin/<base>, the pull request still opens, but the result carries a warning: the pull request includes every commit the branch has that base does not.
• cherry_pick_branch with base. Fetches that branch, because the pod clones only one. It then makes <work branch>-<base> from it, cherry-picks the work branch's commits in order, and checks the new branch out. A conflict aborts, removes the new branch, checks the work branch out again, and names the files. A restarted run that finds the branch already holding the same patches reuses it.
• Branches used as a base. Neither tool will push one.
• The run record. It keeps every link in pull_request_urls.

The run record also keeps what the run changed, so two runs of the same task can be compared. When a task starts, the pod records the commit the work branch is on. When the run is recorded, it measures the work branch against that commit with git diff --numstat. The result is stored on agent_run and returned as change on the run and on last_run:

• lines_added, lines_removed, files_changed. Committed changes only. A file git sees as renamed counts once, with the lines that changed. The per-file list, files, holds at most 300 entries; the totals count every file.
• uncommitted_files. Files the run left changed or untracked in the working tree. A pull request does not carry them, so a number above zero usually means the run stopped before it committed.
• checks. Every verification command the run ran, in order, with its exit code. current is false when a file was edited after the command ran, so the result no longer describes the final change. The list keeps the last 100.

The diff is measured on the work branch even when cherry_pick_branch left another branch checked out. A persistent agent measures each task from where that task started, not from where the branch began. When a plan waits for approval, the plan and the build are recorded as two runs; the plan's run shows no lines changed. change is absent on runs with no repository and on runs recorded before this was added. The agent detail in intelligence-app shows the change, the last check, and up to 10 files.

For this repository, the self-healing agent prompt (docs/agents/self-healing-agent.md) gives the order. Open the main pull request from the work branch. Next, run cherry_pick_branch with base set to develop, then rerun the checks with LINT_BASE=origin/develop. Last, open the [dev] pull request with base set to develop.

A coding agent that calls ask_question writes the question to its logs and waits up to 30 minutes. On the agent Logs tab, type the answer and send it. That posts POST /v1/agents/{id}/answer. The harness reads the answer from the inbox and does not start a new task from it. When a plan is waiting, the same tab shows the plan location and a Build button, which sends the answer "build". A written reply revises the plan. Approval on the repository section chooses "Wait for a person" or "Approve its own plan". A pod restart drops a plan that was still waiting. Who waits, and the pull request write-up rule, are in Architecture.

The task and the finish summary are saved to a workspace conversation for the agent's owner. The next task reads those turns back. The agent detail shows how full the context window is. The number is an estimate. The window comes from the model's max_context_tokens in the broker catalog. FLOW_CONTEXT_TOKENS pins a different size and wins over the catalog; 128000 is the fallback when neither answers.

FLOW_REASONING_EFFORT is sent only to models whose catalog row lists that level. When it is dropped the agent log carries one info line naming the model and the levels it does accept. An empty completion from the provider is retried twice before the run fails, and the failure message carries the provider's response body so the cause is on the log.

Coding pods may open public TCP 22 and 443. That is git SSH, the Go module proxy, and the npm registry. The policy does not apply to Agent Harness V1.

To run harnessCodingV1 on a workstation, point the same two files at the local paths. The pod paths above are the defaults.

export HARNESS_SSH_KEY_GITHUB=$HOME/.ssh/id_ed25519
export HARNESS_SSH_KEY_AZURE=$HOME/.ssh/azure_devops
export HARNESS_KNOWN_HOSTS=$HOME/.ssh/known_hosts

Service Bus topology

services/foundry/.azure/bus/topology.yaml↗ declares domains agent and skill, each with subscribers listener and audit. foundry-api writes the review log from the outbox. foundry-auditlistener writes the same rows from the bus; a duplicate inserts nothing.

Configuration

Values that are not secret are committed in .azure/env/config.<env>.yaml. The rest is encrypted with SOPS in a file next to it. Each release receives its own ConfigMap built from those shared files, including the gateway. The gateway uses the caller's token rather than Foundry's service token. services/foundry/.azure/env/readme.md↗ lists every key, which file it belongs in, and why. AGENT_EMAIL_DOMAIN is config (dev-ai.flow.life / ai.flow.life). Twilio and Mailgun stay on communications. Model keys stay on broker.

Two images, several releases

Dev API ingress: foundry-api.dev.apps-and-services.com (services/foundry/.azure/k8s/values/api.dev.yaml↗); prod is foundry-api.apps-and-services.com. The harness is not a Helm release; cluster runtime creates per-agent Deployments and Jobs from FOUNDRY_HARNESS_IMAGE. The pipeline builds a second image, foundry-harness, from Dockerfile.harness and --set harnessImage on upgrade.

The Foundry MCP gateway uses the same Foundry image, with its own deployment and ingress: foundry-mcp.dev.apps-and-services.com in dev and foundry-mcp.apps-and-services.com in prod. Its values set neither migrations nor agent runtime RBAC. It runs on goserve1, separately from agent pods. The gateway has not been deployed yet; connecting the Flow Intelligence Desktop App to it is covered in services/foundry/cmd/mcp/README.md↗.

Pipeline

services/foundry/.azure/azure-pipelines.yml↗ triggers on develop / main changes under services/foundry. The Helm install step applies every entry in that file's charts list. Stages match template's: decrypt SOPS secrets, build the API image and the Debian 13 harness image from the repository root, busctl plan (exit 2 additive, exit 3 destructive and gated by a manual approval), apply, then helm upgrade --install per chart with harnessImage pinned to that build.

All four releases get the same Foundry image tag on each pipeline run, even for a gateway-only change. They have separate Deployments and can restart and scale independently, but the pipeline releases them together, in order: API, agent listener, audit listener, gateway. This is not an atomic upgrade across releases; a later failure does not roll back earlier successful upgrades.

For the first rollout of the Foundry tools, apply the catalog migration through the API release and run the new Agent Harness V1 binary for agents using them. The harness reloads its catalog during its normal 15-second evaluation cycle, but an existing process cannot load new Go executors. Restart existing builder agents after deploying the new harness image; a catalog refresh alone is not enough.

Migrations

Goose SQL in services/foundry/migrations/↗. cmd/api -migrate is a separate invocation, not startup. The migrate Job is named foundry-api and is a Helm pre-upgrade hook. Kubernetes will not rewrite a Job's pod template, so the pipeline deletes that Job before helm upgrade. A leftover Job from a failed hook is why a prod upgrade can fail with spec.template: field is immutable.