Design a Hosted Model Playground

Question

Design a web application where users enter a prompt, adjust generation parameters, and receive a streamed completion from one of several hosted models.

## Requirements

- Define APIs for generation, streaming, cancellation, model discovery, and saved presets.
- Validate parameter ranges and enforce per-user rate, token, and spend limits.
- Route to model backends with timeouts, retries, load shedding, and clear partial-failure behavior.
- Protect against abuse while preserving prompt privacy and configurable data retention.
- Record enough metadata for latency, quality, and cost analysis without exposing sensitive content.
- Support experiments that compare two models or configurations fairly.

Describe the browser-to-gateway streaming protocol, request state, routing layer, and observability path. State scale, latency, and availability assumptions.

## Follow-ups

1. Add organization workspaces and shared prompt templates.
2. Resume or replay interrupted generations deterministically where possible.
3. Run an A/B test without cross-contaminating caches or users.
4. Handle a model that streams slowly and fails near completion.