Operate
Practice outages
CLI takes an app down on purpose, then checks the outage is noticed, someone is told, and it comes back.
The worst time to find out your alerts don't work is during a real outage. A practice outage (a "chaos drill") finds that out on a quiet afternoon instead.
CLI stops the app, as if it had crashed, then checks four things:
- The app stops.
- Health checks notice (and how long that took).
- Someone is told: an alert channel, a phone page or an email actually went out.
- It comes back: CLI restarts it and times the recovery.
You get a report: "Passed. Noticed in 3 min, people told, and back in 4 min." Or, if something's missing: "Failed: someone is told. The alert was raised, but nobody is set up to receive it."
How to do it: open an app in a staging environment, go to Settings, find Practice outages, and press Run a drill now. Set Repeat to weekly or monthly to keep checking. On production a drill really takes the app down for a few minutes, so only the owner can run one, after ticking a box that says so.
Starting one
POST /v1/services/:id/drills {"allowProduction": false}
GET /v1/services/:id/drills
GET /v1/drills/:id
POST /v1/drills/:id/cancel
PUT /v1/services/:id/drills/schedule {"everyDays": 7}Admins start drills. On production (a project rather than an environment) the body needs allowProduction: true and the caller must be the owner; schedules (everyDays 7, 14, 30 or null) only run on environments. The app must be running with health alerts on, and one drill runs at a time per app.
What's checked
The drill stops the app's container (driver.stop), then the minutely job checks: an incident opened (within 8 minutes), a service_down event plus at least one way it reached a person (a channel on the app's own project or environment, a successful phone page, or an owner/admin email), and recovery. CLI redeploys the live version and waits for a healthy check (within 10 more minutes). Each step is ok, failed, skipped or waiting, with a detail; the drill ends passed, failed or cancelled with a one-line summary. Cancelling restarts the app straight away.