Real-time inference gateway

Question

Design a gateway for low-latency model inference across several model versions and accelerator pools. Include admission control, routing, batching, deadlines, retries, overload behavior, observability, and safe rollout.

## Preparation

- Write the user or stakeholder, the constraint, and the decision in one sentence each.
- Choose one measurement or observable fact that can support the answer.
- Name one alternative and the reason it was not selected.

## Review checklist

1. Did I state the concrete constraint before describing the solution?
2. Did I separate my own decision and work from the team's work?
3. Did I name evidence or a measurement rather than relying on confidence?
4. Can I answer every follow-up without changing the original facts?

## Staged follow-ups

1. Route requests when one accelerator pool is degraded.
2. Balance batching efficiency against individual deadlines.
3. Prevent retries from multiplying overload.
4. Canary a model whose output shape changed.