feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains

[d-cs]

Summary

Adds drainBatchSize to MollifierDrainer (default 1 — preserves existing behaviour) and wires TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE through the webapp (default 50). Each tick the drainer now pops up to drainBatchSize from each chosen env, then dispatches every popped entry through the shared concurrency-bounded pLimit. Per-org/per-env fairness is unchanged — only the in-env pop count grows.

Pre-existing behaviour was one pop per env per tick. For a single-env burst that single-flighted the drain at the per-tick floor of pop + engine.trigger ≈ 50–60 ms. With buffer entries piling up under a real-world tenant burst that's tens of minutes of tail latency to fully materialise — even though PG itself could comfortably sustain the writes.

Why this matters — heavy-tail illustration

Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire 100, 1 fires 1000. Gate at THRESHOLD=10/s, HOLD_MS=500. First 10 of each burst hit PG directly; the rest buffer.

Customers	Triggers each	PG direct	Buffered each	Total buffered
94 small	20	10	10	940
5 medium	100	10	90	450
1 heavy	1000	10	990	990

With DRAIN_BATCH_SIZE=50, DRAIN_CONCURRENCY=50, ~50 ms engine.trigger:

Tick	Pops	Dispatch waves	Wall-clock
1	94×10 + 5×50 + 1×50 = 1 240	25 × 50 ms	~1 300 ms (94 smalls done)
2	5×40 + 1×50 = 250	5 × 50 ms	~300 ms (5 mediums done)
3–20	heavy alone, 50/tick	1 × 50 ms	~100 ms each

Customer class	Buffered fully drained
94 small	~1.3 s
5 medium	~1.6 s
1 heavy	~3.4 s

Without batching (one pop per env per tick — current behaviour):

Customer class	Buffered fully drained
94 small	~500 ms
5 medium	~4.5 s
1 heavy	~49 s

So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster) without changing PG load characteristics. Smalls go up slightly in this scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch queue — that's the trade we accept for the heavy tail; the worst-case small wait is still inside one tick. PG load is identical either way (50 concurrent inserts at a time, capped by DRAIN_CONCURRENCY).

What changed

packages/redis-worker

• New drainBatchSize option (default 1 — full backward compat).
• runOnce() refactored to pop per-env batches in parallel, then dispatch all popped entries through the existing global pLimit. Mid-batch pop failure aborts only that env's batch and counts as one failure (same semantic as the old per-env path).
• Removed the now-unused processOneFromEnv helper.

apps/webapp

• TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE env var (default 50, matching DRAIN_CONCURRENCY).
• Wired into mollifierDrainer.server.ts.

Test cloud config (separate cloud PR): TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50" on the worker service. Production rollout deferred until we've watched it on test cloud.

Test plan

• All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7 new tests under MollifierDrainer.drainBatchSize:
  • pops up to drainBatchSize across ticks
  • global concurrency cap still holds when batch > concurrency
  • mid-batch pop failure isolation
  • multi-env batch fan-out in one tick
  • hierarchical org fairness preserved at drainBatchSize > 1 (load-bearing — guards against future regressions to per-env-instead-of-per-org rotation)
  • mixed success/failure accounting in a batched tick
  • bounded pops on empty queue (no Lua spam past drainBatchSize)
• All pre-existing tests still pass unchanged at default drainBatchSize=1 → backward-compat locked.
• pnpm run build --filter @trigger.dev/redis-worker clean.
• pnpm run typecheck --filter webapp clean.
• redisTest block (real Redis via testcontainers) — couldn't run locally on this branch due to testcontainers runtime discovery; will validate in CI.
• Test-cloud smoke after cloud PR lands: fire burst 50 against a flagged env and confirm the 50th entry's drain time drops from ~2.5 s to <200 ms.

Notes

• Per-tick memory bound: maxOrgsPerTick × drainBatchSize entries can sit in the JS pLimit queue between pop and dispatch. At defaults that's 500 × 50 = 25 000 × ~5 KB snapshot ≈ ~125 MB worst case per worker — well within headroom.
• The pre-batch model's strict per-env throughput cap of 1/tick is documented as the fairness baseline elsewhere. Org-level fairness is what callers actually rely on; this change does not weaken that.

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: d38b449

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 32 packages

Name	Type
@trigger.dev/redis-worker	Patch
@internal/run-engine	Patch
@internal/schedule-engine	Patch
@trigger.dev/build	Patch
@trigger.dev/core	Patch
@trigger.dev/plugins	Patch
@trigger.dev/python	Patch
@trigger.dev/react-hooks	Patch
@trigger.dev/rsc	Patch
@trigger.dev/schema-to-json	Patch
@trigger.dev/sdk	Patch
@trigger.dev/database	Patch
@trigger.dev/otlp-importer	Patch
@trigger.dev/rbac	Patch
trigger.dev	Patch
references-ai-chat	Patch
d3-chat	Patch
references-d3-openai-agents	Patch
@internal/cache	Patch
@internal/clickhouse	Patch
@internal/llm-model-catalog	Patch
@internal/redis	Patch
@internal/replication	Patch
@internal/testcontainers	Patch
@internal/tracing	Patch
@internal/tsql	Patch
@internal/zod-worker	Patch
references-nextjs-realtime	Patch
references-realtime-hooks-test	Patch
references-realtime-streams	Patch
@internal/sdk-compat-tests	Patch
references-telemetry	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

This PR adds a per-environment batching option to MollifierDrainer. A new option/env var (TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE, server default 50) is wired into initialization and logging. The drainer gains an optional drainBatchSize option and refactors runOnce to a worker-pool that enforces per-env pop budgets, catches buffer.pop failures per-env, processes each entry with inline try/catch, and returns aggregated drained/failed counts. The per-env helper and p-limit usage are removed. Tests and changeset/server-changes docs were added.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically summarizes the main change: adding batched pop functionality to MollifierDrainer to improve single-environment drain performance.
Description check	✅ Passed	The description provides comprehensive context including a summary, performance impact analysis, test plan coverage, and architectural notes; it aligns well with the template requirements despite some sections being custom content rather than template-prescribed fields.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/mollifier-drain-batch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
trigger	🟢 Ready	View Preview	Jun 1, 2026, 7:24 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.