Question
Design a shared workflow service where many teams submit pipelines composed of dependent jobs. Jobs should advance only after prerequisites succeed, survive worker crashes, and remain isolated across tenants. ## Requirements - Define APIs for submitting, cancelling, retrying, and inspecting runs, plus a pipeline DAG representation. - Separate logical jobs from attempts and make dispatch idempotent. - Use leases and heartbeats so abandoned attempts can be retried safely. - Enforce tenant quotas, secrets isolation, sandboxing, and fair scheduling. - Cache reusable artifacts without allowing cross-tenant data leakage. - Stream status to the UI while preserving a durable event history. - State scale, consistency, and retention assumptions. Describe orchestration state, ready-queue construction, workers, artifact storage, and recovery after coordinator failure. ## Follow-ups 1. Approach exactly-once effects for deployments. 2. Add fan-out matrices and dynamic jobs. 3. Prevent one tenant from monopolizing workers. 4. Invalidate caches safely when dependencies change.