Stripe's engineering team has reduced database-related pager alerts by approximately 30% by automating incident recovery through graph-based modeling of its global infrastructure, according to a technical article published by the payments company this month. The new system uses graph search algorithms and state machines to calculate and execute remediation plans without human intervention. This improvement translates to 200 fewer pages annually for on-call engineers, while eliminating roughly 12 days of unhealthy database shard states each year.
Stripe's original remediation system relied on hard-coded, plugin-based logic that proved difficult to scale as infrastructure complexity grew. Over a six-month period, the control plane triggered 124 operator pages for misconfigured shards and another 32 pages for single-node failures complicated by additional health problems. Fragile dependencies, multi-failure scenarios, layout-specific logic, and unhandled intermediate states frequently demanded manual operator intervention. Essential operations including index builds and scheduled maintenance were blocked for an average of one hour per incident during this timeframe.
To address these limitations, Stripe modeled its MongoDB infrastructure as a graph, where nodes represent infrastructure components, edges capture their relationships, and node attributes describe current state. The team initially deployed breadth-first search to identify valid remediation paths, then switched to Dijkstra's algorithm to prioritize lower-cost recovery plans. According to the authors, "Dijkstra's algorithm explores paths to all reachable states rather than only the goal state," enabling partial remediation when complete recovery paths don't exist by returning "the path to the least misconfigured state." The system now dynamically adjusts to different MongoDB shard layouts without requiring layout-specific code changes.
The graph-based approach works by removing fixed remediation sequences and instead using graph traversal to discover recovery paths, allowing identical logic to adapt automatically as infrastructure evolves. Rather than embedding recovery logic in rigid workflows, Stripe models remediation as composable rules with explicit state transitions, enabling the planner to combine operations dynamically. The team plans to extend the framework beyond failure recovery to automate topology changes and blue-green deployments, and to orchestrate planned maintenance alongside reactive healing. Scott MacVicar, head of developer infrastructure at Stripe, notes that operating a global database fleet means hardware degradation and unhealthy shards are daily occurrences, and at scale the challenge isn't simply fixing issues but doing so without burning out on-call engineers.
The team concludes that for organizations managing complex distributed infrastructure, this pattern of state machine modeling, simulation-based planning, and runtime pathfinding offers a compelling alternative to accumulating increasingly specific runbooks, noting that runbooks encode known recovery procedures while a state machine discovers novel ones. Stripe isn't alone in pursuing automated infrastructure operations—Uber recently described its declarative, self-healing Odin platform, while Meta has detailed AI-assisted tooling to accelerate incident response. As cloud architectures grow more distributed and failure modes multiply, the choice between procedural automation and adaptive systems may increasingly determine which engineering teams can scale operations without proportionally scaling headcount.