Design a Multi-User AI Assistant

Question

Design the backend for a production chat assistant used concurrently by many customers. It should stream model output, invoke approved tools, preserve conversation state, and remain useful under model or dependency failures.

## Requirements

- Define APIs for creating conversations, sending messages, streaming events, cancelling work, and retrieving history.
- Isolate tenants and authorize every history or tool access.
- Orchestrate model calls and tool calls with timeouts, retries, idempotency, and bounded loops.
- Enforce rate limits, quotas, content policy, and cost budgets.
- Store message lineage, model/tool versions, and audit data without leaking sensitive prompts.
- Set explicit latency, scale, durability, and availability targets.

Describe the request path, core data model, streaming protocol, and failure states visible to the client.

## Follow-ups

1. Balance latency, quality, and cost through routing or fallbacks.
2. Support long conversations with summarization or retrieval.
3. Add human approval for sensitive tool actions.
4. Evaluate answer quality and investigate regressions safely.