91̽»¨

Is Chaos Engineering Worth It? A Cost-Benefit Analysis Guide

Originally published December 2021. Updated September 2026 with current downtime data and a new worked example.

On 20 October 2025, a race condition in DynamoDB’s automated DNS management took out a regional endpoint in AWS us-east-1. Slack, Atlassian, Snapchat and thousands of other services went down with it, and the knock-on effects lasted more than 15 hours ().

No unit test would have caught that. But a team that had asked “what happens to us when DynamoDB in us-east-1 stops resolving?” would have known the answer before it happened.

That’s the case for chaos engineering. The harder question, and the one every engineering leader has to answer for their budget holder, is whether it pays for itself. This guide shows how to work that out.

Why the question matters more now

When we first wrote this post in 2021, the main driver was the shift to microservices: more components, more connections, more ways to fail. That hasn’t gone away. Four things have made it sharper:

• Outages cost more. New Relic’s 2025 Observability Forecast puts the median cost of a high-impact outage at $2 million an hour, or $76 million a year per business (). Splunk and Oxford Economics estimate downtime now costs the Global 2000 $600 billion a year, up from $400 billion in 2024 ().
• Dependencies are concentrated. Most systems now lean on a handful of cloud regions, managed databases and SaaS APIs. When one of them degrades, the failure is inherited, not caused.
• More change, less human review. AI-assisted coding and autonomous agents ship changes faster than any team can reason through every failure path. The Splunk report calls out AI agents as a new source of failure modes.
• Regulators now expect proof. The EU’s Digital Operational Resilience Act has applied to financial entities since 17 January 2025 and requires regular resilience testing (). “We think it would hold” is no longer an answer. Read more in our DORA guide.

Automated tests with 80–90% coverage are still valuable. But they almost always assume every dependency is up and responding normally. Chaos engineering tests the conditions they skip: a slow database, a dropped zone, a saturated node, a third-party API returning errors.

The ROI formula

There is a simple way to frame the return. You weigh the cost of the outages you prevent against two costs: running the programme, and any harm the experiments themselves cause.

\text{ROI} = \frac{C_{\text{prevented outages}} – C_{\text{chaos-induced harm}} – C_{\text{chaos effort}}}{C_{\text{chaos effort}} + C_{\text{chaos-induced harm}}}

A worked example

Take a mid-sized SaaS company. Over a year, its chaos programme uncovers weaknesses that would have caused two high-impact outages of 90 minutes each.

ROI = ($900,000 − $20,000 − 200,000)÷(200,000 + $20,000) = 309%.

The break-even point is more telling. At $300k an hour, the programme pays for itself if it prevents about 44 minutes of high-impact downtime a year. Most teams have had more than that in the last quarter.

Filling in your own numbers

The formula is easy. The inputs take some honest work.

1. Cost of downtime per hour. Start with lost revenue per hour for your most critical user journeys. Add SLA credits, the engineering hours spent on incident response (New Relic found engineers spend a third of their week on it), and any regulatory exposure.
2. Outages prevented. Go through your last 12 months of postmortems. For each, ask whether an experiment would have exposed the weakness first: a missing timeout, a retry storm, a single-zone dependency. That gives you a defensible baseline.
3. Chaos-induced harm. Keep it low with a small blast radius, staging-first experiments and clear stop conditions. With good guardrails, this is usually the smallest number in the equation.
4. Chaos effort. Include licences, setup, and the time engineers spend designing, running and following up on experiments. Pre-built templates and automated risk discovery cut this significantly.

The returns the formula doesn’t count

Some of the value is harder to put a price on, but it shows up quickly:

• Faster recovery. Teams that have rehearsed a failure detect and fix it faster when it happens for real.
• Evidence for auditors. Documented, repeatable experiments are exactly what resilience regulations ask for.
• Confidence to ship. Running experiments in CI/CD turns resilience from a one-off exercise into a release check.
• Lower cloud bills. Knowing how your system behaves under failure lets you right-size redundancy instead of over-provisioning. See how to reduce cloud costs with chaos engineering.

Making the results show up

We said it in 2021 and it’s still true: breaking things is a lot of fun, but results need to show up. A chaos programme that can’t show progress will lose out to the next priority.

That means measuring from day one. Track which risks you’ve found, which you’ve fixed, and how your coverage grows over time. Tie experiments to the services and user journeys the business cares about, not to individual pods.

This is where 91̽»¨ helps:

• Reliability Advice automatically discovers your system and flags risks against Kubernetes best practices before you run a single experiment.
• Services groups targets around what you actually operate, so testing and risk assessment map to real business services.
• Advanced Reporting turns experiment results into progress you can show leadership and auditors.
• SteadyBuddy, our AI companion, helps teams design experiments and understand results in plain language.
• The offers 200+ open-source actions, targets and templates, so you’re not starting from a blank page.

The bottom line

Chaos engineering is worth it when it prevents more downtime than it costs, and with outage costs where they are now, that bar is low. Start with your riskiest dependency, run one experiment, and measure what you learn.

or book a demo to see how quickly you can get there.

Sources

• (17 Sep 2025)
• (22 Jul 2026)
• 91̽»¨ product page