Design and Implement a Distributed Job Scheduler

Problem Statement

Design and implement a distributed job scheduler service that:

1. Accepts job submissions with a scheduled execution time
2. Executes jobs at or near their scheduled time
3. Supports job priorities
4. Handles worker failures gracefully
5. Prevents duplicate execution
6. Can be deployed on cloud infrastructure

This is a live coding round — you need to produce working code, not just a whiteboard design.

Requirements

Functional:

• Submit a job (payload, scheduled_time, priority, max_retries)
• Cancel a pending job
• Query job status (pending, running, completed, failed)
• Support recurring/cron-style jobs (bonus)

Non-Functional:

• At-least-once execution guarantee
• Jobs execute within a few seconds of scheduled time
• Horizontally scalable (more workers = more throughput)
• Fault tolerant (worker crash doesn't lose jobs)

What the Interviewer Expects

1. Architecture: API server + job queue (Redis/Kafka) + worker pool + persistence (Postgres)
2. Job lifecycle: submitted → queued → picked up → running → completed/failed
3. Worker failure handling: heartbeat mechanism. If no heartbeat for N seconds, reassign job to another worker.
4. Idempotency: each job has a unique ID. Workers check before executing. Prevents double-execution on retries.
5. Priority: use a priority queue or sorted set. Higher priority jobs get picked first.
6. Scheduling: a "ticker" process that moves jobs from scheduled state to queue when their time arrives.
7. Deployment: containerized service, discuss how you'd deploy on Kubernetes with auto-scaling workers.

Follow-ups

1. How do you handle job dependencies (job B should only run after job A completes)?
2. How do you prevent starvation of low-priority jobs?
3. What happens if the scheduler process itself crashes? How do you recover?
4. How would you implement rate limiting per tenant in a multi-tenant scheduler?
5. How would you add observability — monitoring, alerting, and debugging failed jobs?