Concurrent moderation without duplicates
How ModPilot uses Kafka consumer groups and Redis to hand every moderator a unique item from the pending-review queue, with no race conditions.
Two moderators open the pending queue at the same second. Both read from the database, both get the same post, both review it. That is the failure the naive approach walks straight into. At best you waste a reviewer’s time. At worst you ship two conflicting decisions on one piece of content.
Here is how ModPilot makes sure every moderator gets something different.
Why reading from the database breaks down
A queue built on SELECT ... WHERE status = 'pending' has no coordination between readers. Without row locking and careful transactions, concurrent moderators race for the same rows. You can lock row by row, but then read latency is tied to write contention, and it buckles under a spike.
Treating “needs review” as an event
Instead of polling a table, ModPilot treats a manual-review decision as an event.
When the AI returns manual_review, two things happen. The result is written to MongoDB as the audit record, and the item is published to a per-client Kafka topic, moderation-pending-{client_id}.
A background consumer subscribes to that topic through a consumer group, stores each message in Redis, and lets entries expire on a TTL so the cache stays clean.
When a moderator opens the queue, the API reads from Redis, not MongoDB:
GET /api/v1/moderation/pendingKafka consumer groups hand each message to exactly one consumer, and Redis serves it once. So every moderator gets a unique post. No locks, no retries, no two people staring at the same listing.
Closing the loop
When a moderator approves or rejects an item:
POST /api/v1/moderation/:post_id/approveModPilot updates MongoDB, publishes the result to moderation-manual-results-{client_id}, deletes the item from Redis so it can’t be served again, and commits the Kafka offset.
Where this earns its keep
MongoDB stays the system of record, so nothing is lost if Redis gets flushed. Kafka holds onto messages, Redis cleans itself up on its TTL, and the whole path falls back to MongoDB when a client hasn’t set up Kafka and Redis yet. The part that matters most is the simplest one: one message reaches one moderator, and that comes from the consumer group rather than from hope.
The result is a queue that scales with the size of your moderation team instead of fighting it.