The agent added a queue consumer that sends a welcome email when a new user event lands. To avoid reprocessing, it tracked the last processed event id in an environment constant, started the consumer locally, watched one event go through, and committed. A fresh local run stayed green. On the first redeploy, the environment constant reset to its default, the consumer read from the start of the queue, and every previously welcomed user received the email again.

The agent confused deployment configuration with processing state. The id it had reached is where this consumer instance stopped; that position changes with every event and must survive restarts, deploys, and instance replacement. An env value and a code constant are rebuilt from the manifest on every boot, so they cannot remember where the stream left off. The local harness ran the consumer once and never restarted it, which is exactly when the loss appears.

Why the offset belongs outside the code

Position is runtime state, not configuration. An env var sets how the app starts; the high-water mark records how far this consumer has already advanced. These move on different schedules, and conflating them means every deploy rewinds the stream.

A local single-run test never restarts the consumer. The suite feeds one event, checks the side effect, and exits; it never drops the process and asks it to resume. Replay only shows up after a fresh boot.

A restarted consumer treats old events as new. Without a durable position, "start here" defaults to the beginning, and an at-least-once queue hands over everything that was already delivered.

Keep the checkpoint durable and delivery idempotent

Persist the offset in durable state. A consumer table or a managed offset store survives redeploys; commit the position alongside the event, not in the deployment manifest.

Advance the checkpoint after the side effect lands. Mark the event done only once the email or notification is sent; crash before that, and the event replays exactly once more rather than vanishing.

Give every event a stable delivery key. Dedupe sent notifications by event id so a replay suppresses the duplicate instead of mailing twice; the checkpoint and the dedup log cover different failure windows.

Test the restart path, not just the happy path. Feed two events, restart the consumer, and assert neither old event is resent before the new one runs. That case fails on an in-config offset.

The offset looks like a small detail next to the consumer logic, but it is the state that outlives the deploy. Where has an in-memory or in-config position rewound on you?

This is part of a larger workflow; the AI coding agent guide collects the related decisions and keeps them in sync.