Killing cron: moving a dispatch pipeline to SQS + Lambda

Why not Kafka?

The QC dispatch pipeline at Spyne assigns incoming images to human reviewers. The original design polled a database table every few seconds, a cron job that checked for unassigned work and handed it out.

The problem wasn't volume. At 150–250K images a day, the throughput was fine. The problem was latency. A reviewer finishing one image had to wait up to 4 seconds for the next assignment, because the cron only ran every few seconds.

Kafka was the obvious answer, but it was wrong for this workload. We had a single consumer group doing FIFO dispatch with no fan-out, no replay, no stream processing. Standing up a Kafka cluster (or even MSK) for what amounts to a job queue would have been operationally expensive for no architectural benefit.

SQS with Lambda gave us the event-driven trigger we needed without the operational overhead of a broker.

The result

• p95 assignment latency: 4 seconds → 250 milliseconds
• Infrastructure cost: Reduced (no always-on consumer)
• Operational complexity: One Lambda function vs. a Kafka consumer service