0026 — Scheduled work is a scheduler calling an endpoint

Decided. Work that runs on a timer is triggered by Kestra, our workflow scheduler, calling POST /internal/jobs/{jobType}.

For slow work, the endpoint puts a job on a queue and returns 202 with a job id. It does not do the work. A worker that is already draining that queue picks it up. The scheduler can then poll GET /internal/jobs/{jobId} to find out whether it finished, which is how it knows when to start a task that depends on it.

Callers prove who they are with a shared secret, handled by common/schedulerauth.

This applies to every service, not just one. It does not change the outbox publisher, which loops continuously behind a pg_try_advisory_lock rather than running on a schedule.

When the endpoint does the work itself

The queue exists for work with no fixed size: rebuilding the whole search index, clearing an extraction backlog, anything that reads more rows the longer the platform has been running.

Some jobs are not like that. For those, the queue adds a way to fail without removing one — a queue row, a worker, and a poll, all to close forty rows in ten milliseconds.

A scheduled endpoint may do the work inline when all four of these are true:

• the amount of work is capped by a number written in the code, not by how much data exists;
• a run finishes comfortably within the ingress timeout at that cap;
• running it twice is harmless, so a retried trigger repeats rather than breaks something;
• a backlog can drain over several runs instead of needing one long one.

An endpoint like that returns 200 with a count of what it did, rather than 202 with a job id. Workspace's idle-session sweep and documents' orphan cleanup both work this way. Each one caps how much it does per run and is safe to run twice.

The rule being protected is that the endpoint answers quickly. Queueing is how you manage that when the work is slow. It is not a ritual to perform when the work is already fast.

Alternatives

A timer inside the service, guarded by an advisory lock. No new dependency, and we already use this pattern in the publisher.

Rejected because it gives no history of past runs, no alert when a run quietly did not happen, and no way to re-run or backfill without a deploy. It also cannot express that one job must run after another, and our rollups read what the clustering job writes.

A Kubernetes CronJob. Real scheduling, and each run is a fresh process that needs its own database pool, its own configuration, and its own image start. It still cannot express ordering between jobs.

Endpoints that just do the work and wait. The simplest thing to write, and wrong for the slow cases: a full reindex runs for minutes, hits the ingress timeout, and the scheduler's retry then starts a second copy.

Why

• One thing runs the work. A unit of work is a queue row whether a bus event or the scheduler created it. That means one retry policy, one dead-letter queue, one spend cap, and one place to look when something did not happen.
• Ordering is a real requirement. The rollup jobs read what the clustering job writes, so they have to run after it.
• Queue-and-poll survives retries. A duplicate trigger collapses into the existing job through a dedupe key.

Consequences

• Services expose /internal/jobs/... behind common/schedulerauth. It reads the secret from Authorization: Bearer and compares it, in constant time, against two configured values: the current token and the previous one. Accepting both means rotating the token is two separate deploys instead of one coordinated switch. DEV_AUTH=1 skips the check, in local and dev only.
• A leaked token cannot do much. These endpoints accept no content, return only a job id or a count, collapse repeats through dedupe keys, and are limited by the daily spend cap.
• The workflow files are committed, at services/<name>/.azure/kestra/*.yaml, next to the bus topology and for the same reason: a schedule that exists only in a scheduler's web UI cannot be found from the code it runs.
• Batch work and event-driven work get separate worker pools, so a twenty-minute reindex cannot starve the jobs somebody is waiting on.
• The background sweeps this replaces — workspace's pending files, documents' orphaned blobs — become endpoints the scheduler calls rather than timers inside the service. Both are capped per run, so both take the inline form above.

Source: docs/decisions/0026-scheduled-work-via-enqueue-endpoints.md↗.