Problem Statement
You are migrating data from a primary database to a secondary datastore. Design a verifier process that raises an alert if replication lag exceeds K seconds.
Events:
- An event is generated every time a piece of data leaves the primary DB
- Another event is generated once that data arrives at the secondary datastore
If a data item from the primary DB has not appeared in the secondary datastore within K seconds, the verifier should raise an alert.
Constraints
- Events can arrive out of order
- Millions of data items being migrated concurrently
- Each data item has a unique ID
- Clock skew between primary and secondary is negligible
- K is configurable (e.g., 30 seconds, 60 seconds)
- False positives should be minimized
Example
K = 10 seconds
Event: data_item_42 LEFT primary at t=100
Event: data_item_43 LEFT primary at t=101
Event: data_item_42 ARRIVED at secondary at t=107 → OK (lag = 7s < 10s)
Event: data_item_43 NOT arrived by t=112 → ALERT (lag > 10s)
What the Interviewer Expects
- Core data structure — HashMap tracking {itemId → departure_time} for items that left primary but haven't arrived at secondary yet.
- Alert mechanism — periodic sweep or timer-based check. For each pending item, if
current_time - departure_time > K, raise alert. - Cleanup — remove items from the map once they arrive at secondary.
- Scalability concerns — what if there are millions of in-flight items? Memory management, partitioning.
- Edge cases — events arriving out of order, duplicate events, items that never arrive.
Follow-ups
- How would you handle events arriving out of order (secondary arrival event before primary departure event)?
- How would you scale this if the migration involves 100M+ items?
- What if you want to track not just alerts but also P50/P95/P99 lag metrics?
- How would you make this fault-tolerant — what if the verifier process crashes and restarts?
- How does this relate to MongoDB's own replication lag monitoring (oplog-based)?