Problem Statement
Design a chat application similar to ChatGPT. Users should be able to start new conversations, continue previous conversations, and receive streamed AI responses.
Requirements
Functional:
- Start a new chat conversation
- Continue a previous conversation (with full context)
- Stream AI responses token-by-token as they're generated
- List and retrieve past conversations
- Handle very long conversations (context window limits)
Non-Functional:
- Low latency to first token
- Scale to millions of concurrent users
- Reliable message persistence
- Handle long-running LLM inference requests
What the Interviewer Expects
- Core entities & data model:
User,Conversation,Message(role: user/assistant, content, timestamp)- Conversations store an ordered list of messages
- Streaming responses — Server-Sent Events (SSE) or WebSockets to stream tokens as the LLM generates them. Discuss why SSE fits one-directional streaming well.
- Context management — sending prior messages to the LLM for continuity. Handling context window limits (truncation, summarization of old messages).
- Architecture — client → API gateway → chat service → LLM inference service (queued, possibly GPU-backed) → stream back.
- Persistence — store messages in a DB (Postgres/Cassandra). Cache active conversations in Redis.
- Scaling LLM inference — request queue, batching, autoscaling GPU workers, handling long inference times without blocking.
Follow-ups
- How do you handle a user who refreshes mid-stream? (resumable streams, storing partial responses)
- How do you manage the context window when a conversation exceeds the LLM's token limit?
- How do you rate-limit expensive LLM calls per user/tier?
- How would you support conversation branching (edit a message and regenerate)?
- How do you ensure message ordering and consistency in a distributed setup?