AwsIncidentsBench · T04 · recorded run replay

Purged Routes

On Friday night a maintenance job tombstoned 371 merchants' routes in routemap-state. Scheduled workflows rebuild a tombstoned route with no cells and the publisher pushes it, so that merchant's card payments fail. Replay a recorded agent run on its own exported world: each tile is one of the 371 routes, and it breaks, is restored and is served again when that run's export says it did.

ET13:12:00
failing29
to restore371
payment failures / min–
workflows–
Loading
Agentcalls 0 of 200
not paged yet
The page comes at 13:11.
route tombstoned or empty route restored still served its old route served an empty route: fails restored and served restored, not as before agent call
space play · ←/→ 1 min · shift 10 min · [ ] steps · drag to orbit

Replayed from the run's own exports, not a simulation: the agent's calls from the aws server's meter, every write to the purged routes and every push to the data plane, the workflows' runs and switches, the restores, the alarms and the status page from the world's export, the colleagues' lines and the grader's rows. AWS service icons are from AWS's Architecture Icons set and are trademarks of Amazon.com, Inc. or its affiliates; AwsIncidentsBench is not affiliated with or endorsed by Amazon Web Services.