Preparing this page…
Your current navigation and account state stay available while the next page loads.
Design a reproducible registry connecting datasets, evaluator configurations, model outputs, and reviewed results. This independent, original scenario is designed for Research Engineer, Agents / Evals, Software / Infrastructure preparation. Work through the stages in order, explain each decision, and use the review dimensions to test whether the result remains defensible when constraints change.
Design the write path for thousands of workers uploading partial outputs, retries, and final manifests without duplicating or silently mixing runs.
Are idempotency, ownership, and completion explicit?Add tenant isolation, purpose-limited access, redaction, retention, and deletion while preserving enough audit evidence to explain a released result.
Do privacy controls and reproducibility have an explicit boundary?Plan observability, corruption detection, backfills, schema evolution, and recovery from a bad evaluator release without rewriting historical truth.
Can operators detect and contain incorrect evidence?Use these checks after a first draft. They are review prompts, not six more questions to answer.
Clarifies goals, constraints, and ambiguous requirements.
Protects invariants and handles boundary conditions.
Identifies realistic failures and designs recovery paths.
Makes ownership and transitions explicit.
Provides enough visibility to diagnose and run the system.
Explains what is gained, lost, and intentionally deferred.
Use a different system, project, or situation and check whether the structure still produces a specific, defensible answer.