Design an interactive model-backed product feature

Question

## Scenario

A collaborative writing product wants an assistant that can transform a selected passage according to a user instruction. The interaction should feel immediate, stream useful output, preserve user control, and remain dependable when the model is slow, unavailable, or returns unsafe content.

Design the feature from the client interaction through model serving and operational controls.

## Product requirements

- Start visible progress quickly and stream partial output when possible.
- Let the user cancel, retry, compare with the original, and explicitly accept a result.
- Never overwrite the document merely because generation completed.
- Enforce per-user and organizational cost limits.
- Protect document content and honor data-retention settings.
- Handle timeouts, transient failures, rate limits, malformed output, and policy violations.
- Measure usefulness without logging sensitive content unnecessarily.

## Design discussion

Address:

1. Client/server protocol for request creation, streaming, cancellation, and idempotent retry.
2. Prompt construction, context limits, model selection, and response validation.
3. Latency budget and the difference between time to first token and total completion time.
4. Admission control, quotas, caching, batching, and spend protection.
5. Fallback behavior when a preferred model is overloaded or unavailable.
6. Safety, privacy, observability, and staged rollout.
7. State transitions visible to the user and recovery after a network interruption.

## Follow-ups

- The provider becomes unavailable for thirty minutes.
- A user closes the tab halfway through a long generation and returns later.
- Enterprise customers require regional processing and zero retention.
- Product asks for speculative execution against two models to reduce tail latency.