FORGE operations runbook
Use the FORGE operations view to distinguish one bad build from a systemic worker, provider, validation, or storage problem.
1. Find the build
Search by build ID, user, organization, project, or time window when the current Control Center surface supports those fields.
2. Identify the current/failed stage
Check whether the build is:
- queued
- planning
- preparing workspace
- generating
- validating
- repairing
- packaging
- completed/failed/cancelled
3. Check queue and worker health
If many unrelated builds are stuck in QUEUED or the same processing stage, inspect worker/queue health before retrying individual jobs.
4. Check provider/model health
If planning/generation calls fail broadly, correlate with model/provider metrics and routing.
5. Check validation
For build-specific validation failures, inspect validator output and repair attempts. Repeated identical failures often indicate a requirement, dependency, or environment issue rather than temporary infrastructure.
6. Check packaging/storage
A build can generate and validate successfully but fail during artifact packaging or upload. Inspect artifact/storage state separately.
7. Cancel or retry
Cancel when the build should no longer consume resources. Retry/requeue only after identifying whether the failure was temporary and whether source context is still current.
8. Preserve historical truth
Do not mutate a failed build record into “completed.” A retry should preserve the original failure and create a new attempt/resource according to the active job model.
9. Escalate systemic failures
Create/update an incident when many builds are affected by the same queue, worker, provider, validator, or storage problem.
Operators should use build metadata and logs to diagnose jobs. Routine FORGE operations should not require unrestricted browsing of private source contents or artifacts.
