Designing pause and resume into a multi-tenant pipeline

Two scopes

The QC pipeline processes images for multiple car dealerships simultaneously. When a dealership asks to pause processing, maybe because they uploaded wrong images or need to update specs, we need to halt work at the right scope.

There are two levels: tenant-level (pause everything for a dealership) and SKU-level (pause a single car listing). The challenge is doing this without losing work that's already in flight.

Recomputing SLA timers

The trickiest part was SLA timers. Each image has a deadline for QC completion. When you pause a SKU, the clock should stop. When you resume, it should continue from where it left off, not restart.

We store the elapsed SLA time at pause, then recompute the deadline on resume by adding the remaining time to the current timestamp.