- Difficulty
- Advanced
- Time
- 70–90 min
Define the immutable evidence chain
Define identities and relationships for dataset snapshots, evaluator versions, model builds, run configurations, outputs, and reviewer decisions. State which objects are immutable and why.
What to demonstrate: Can an old result be reproduced without relying on mutable names?
Accept concurrent runs safely
Design the write path for thousands of workers uploading partial outputs, retries, and final manifests without duplicating or silently mixing runs.
What to demonstrate: Are idempotency, ownership, and completion explicit?
Control sensitive artifacts
Add tenant isolation, purpose-limited access, redaction, retention, and deletion while preserving enough audit evidence to explain a released result.
What to demonstrate: Do privacy controls and reproducibility have an explicit boundary?
Operate and migrate the registry
Plan observability, corruption detection, backfills, schema evolution, and recovery from a bad evaluator release without rewriting historical truth.
What to demonstrate: Can operators detect and contain incorrect evidence?