Skip to content
← All articlesTesting & QA

Flaky Tests: Causes, Fixes and a Practical Policy

Flaky tests erode trust in CI and slow every release. Learn the common causes, how to find them, and a practical policy for fixing flaky automated tests.

A flaky test is one that passes and fails on the same code without any change. It is one of the most corrosive problems in a delivery pipeline, because it teaches the team to ignore red builds. Once developers start saying “just re-run it”, a real regression can slip through unnoticed.

This guide explains why tests become flaky, how to find them, and a simple policy that keeps an automated suite trustworthy, which matters even more when development and QA are split across time zones.

Why flaky tests are expensive

  • Lost trust. If a failure is probably noise, nobody investigates it promptly.
  • Wasted time. Every re-run delays merges, and engineers context-switch while they wait.
  • Hidden defects. Some “flaky” failures are real race conditions or data bugs that only appear intermittently in production too.
  • Slower releases. Teams add manual checks to compensate for automation they no longer believe.

The most common causes

Most flakiness comes from a small set of patterns.

Cause Typical symptom Usual fix
Timing and async waits Passes locally, fails in CI Wait for a condition or event, not a fixed sleep
Shared or leftover test data Fails only when run after another test Isolate data per test and clean up reliably
Test order dependence Passes alone, fails in the full suite Make each test set up its own state
Unstable environments Random timeouts or connection errors Use stable, dedicated test environments or containers
External services Fails when a third party is slow or down Stub or mock external calls at the boundary
Time and date logic Fails near midnight, month end or across time zones Inject a controllable clock
Brittle UI selectors Fails after small front-end changes Use stable test identifiers rather than layout-based selectors
Resource contention Fails under parallel runs Avoid shared ports, files and accounts

Timing is the biggest offender

Fixed sleeps are the classic trap. A two-second pause is too long on a fast machine and too short on a busy CI runner. Modern tools such as Playwright and Cypress include auto-waiting, but you still need to assert on the right condition: wait for the element to be visible, the network call to finish, or the state to change.

Data and ordering

Tests that depend on records created by earlier tests will eventually fail when the order changes or tests run in parallel. Each test should create what it needs, use unique identifiers, and not assume an empty or pre-seeded database unless that state is explicitly reset.

How to find flaky tests

You cannot fix what you cannot see. A few practical techniques:

  1. Track results over time. Record pass/fail per test across CI runs. A test that fails on some runs of the same commit is flaky by definition.
  2. Re-run failures once, and log it. Automatic retries can keep the pipeline moving, but only if every retry is recorded and reviewed. A retry that is silently swallowed hides the problem.
  3. Run suspects in a loop. Repeat a single test many times, ideally under CPU load and in parallel, to reproduce the failure.
  4. Randomise test order. Many frameworks can shuffle order, which quickly exposes hidden dependencies.
  5. Capture evidence. Keep logs, screenshots, traces and network recordings from failed runs so you can diagnose without re-creating the failure.

A practical flaky test policy

Tooling helps, but policy is what keeps a suite healthy. A workable approach:

  • Treat flakiness as a defect. Raise a ticket with an owner and a due date, the same as a product bug.
  • Quarantine, do not delete. Move a known flaky test to a separate non-blocking job so it stops disrupting merges, but keep it visible. Set a deadline: fix it or remove it.
  • Limit retries. Allow at most one automatic retry, and report retried tests on a dashboard.
  • Set a flake budget. Agree a target, such as no quarantined test older than two weeks, and review it in your regular QA sync.
  • Fix at the right level. If a UI test is flaky because of a slow backend call, consider testing that behaviour at the API or component level instead. A healthy test pyramid has fewer slow end-to-end tests to begin with.

Prevention checklist

Use this when reviewing new automated tests:

  • No fixed sleeps; waits are tied to explicit conditions
  • Test creates and cleans up its own data
  • Test passes when run alone, in any order, and in parallel
  • External services are stubbed or run in a controlled environment
  • Time, randomness and locale are controlled
  • Selectors use stable identifiers
  • Failure output includes enough logs or traces to diagnose

Working with an outsourced QA team

Flaky tests are easy to ignore when the people writing tests and the people shipping code are in different places. Make reliability an explicit part of the engagement: agree the flake policy in your working agreement, share the CI dashboard with everyone, and review the slowest and least reliable tests in a regular retrospective. Our guides on test automation strategy and regression testing cover how to decide what to automate in the first place.

If your pipeline has stopped being trusted, AkMo Technologies can audit your suite, stabilise the unreliable tests and set up reporting so problems stay visible. See our software testing services or get in touch to talk through your setup.

Let’s build what’s next.

Tell us about your platform, your product idea or the systems you want to modernise. Someone from our Chennai engineering team will get back to you personally.