Question
Design a service that runs repeatable evaluations for model versions and blocks promotion when a required suite regresses. Support immutable datasets, parallel workers, partial retries, audit history, and comparison across versions. ## Preparation - Write the user or stakeholder, the constraint, and the decision in one sentence each. - Choose one measurement or observable fact that can support the answer. - Name one alternative and the reason it was not selected. ## Review checklist 1. Did I state the concrete constraint before describing the solution? 2. Did I separate my own decision and work from the team's work? 3. Did I name evidence or a measurement rather than relying on confidence? 4. Can I answer every follow-up without changing the original facts? ## Staged follow-ups 1. Add expensive human review to a small sample. 2. Prevent a changed dataset from masquerading as model progress. 3. Recover a run after half the workers disappear. 4. Support private datasets without exposing examples to every worker.