Tenant-aware job queues

Multi-Tenant Queues & Scheduling

When to Use

Use when designing or debugging background jobs, message queues, event subscribers, webhook dispatchers, or scheduled/cron workflows in a multi-tenant system.

1. Mandatory Tenant Envelope on Every Queued Job

Background workers run outside the HTTP request lifecycle. A job without an explicit tenant envelope is a data-leak or crash waiting to happen:

• Required Job Payload Schema:

  interface TenantJobEnvelope<T> {
    tenantId: string;           // Mandatory workspace/tenant identifier
    actorId: string | null;     // User or service principal who triggered the job
    actorType: 'user' | 'api_key' | 'system';
    enqueuedAt: string;         // ISO timestamp
    correlationId: string;      // Trace/request ID
    payload: T;
  }
  

• Reject job enqueue calls for tenant-scoped job types if tenantId is missing.

2. Worker Context Initialization, Cleanup & State Hygiene

Long-lived worker processes execute jobs for Tenant A, then Tenant B, then Tenant C sequentially or concurrently in the same OS process:

• Per-Job Context Scope: Wrap every job execution in a fresh AsyncLocalStorage.run(jobTenantContext, ...) (or contextvars / DB transaction SET LOCAL app.current_tenant_id) and ensure context exits cleanly in try / finally.
• Zero Worker-Global Mutable State: Never cache tenant data, auth clients, or temp files on module-level variables inside a worker without keying by tenantId and cleaning up after the job finishes.

3. Time-of-Check vs. Time-of-Execution (TOCTOU) Re-Verification

A job enqueued today (or scheduled for 24 hours later) may execute after:

• The workspace was suspended or deleted.
• The workspace's subscription lost entitlement to the feature.
• The user who scheduled the action was removed from the workspace or had their role downgraded.

At the start of worker execution, re-verify:

1. The workspace exists and is active (if suspended or deleted, abort/cancel the job cleanly without retrying).
2. For user-attributed delayed mutations (exports, scheduled sends, bulk edits), verify whether the initiating user or token still holds permission according to the domain policy.

4. Fan-Out Scheduling Pattern for Cron Jobs

Never run a single monolithic cron query that loads all rows across all 5,000 workspaces into one giant in-memory array (SELECT * FROM items WHERE due_at < NOW()):

• One corrupt record or slow external API in Workspace #42 will crash or block the cron job for all other 4,999 workspaces!
• Dispatcher $\rightarrow$ Per-Tenant Job Fan-Out:
  1. The scheduler queries only active workspace_ids that have due work (SELECT DISTINCT workspace_id ...).
  2. It enqueues an isolated per-workspace job (process-tenant-digest, { tenantId }) with deterministic idempotency key (digest:${tenantId}:${dateWindow}).
  3. Each workspace job runs in its own isolated tenant context, transaction, and error boundary.

5. Noisy-Neighbor Fairness & Per-Tenant Concurrency Limits

Prevent a single tenant enqueueing 50,000 jobs from starving all other tenants:

• Enforce per-tenant rate limits or per-tenant active concurrency caps in the queue consumer, or separate fast interactive queues from bulk batch queues.