I’ll go first: about 40% of ours is effectively dead weight.
We have ~1,800 automated tests in a Django + React app. A solid chunk of them:
- Haven’t been touched in 18+ months
- Fail intermittently and get ignored rather than fixed
- Test implementation details instead of behavior
I’m a QA lead in Indonesia and this came up in a retrospective last week. The devs were actually the ones who raised it — said they don’t trust the test suite anymore, which is a bad sign.
We’re planning a test audit sprint next quarter. Has anyone done one of these formally? What does your process look like for deciding what to delete vs fix vs rewrite?
Our current flaky test count
We have 73 tests marked as @flaky in our suite that have been in that state for over 6 months. Yes, it’s embarrassing.
We did a formal test audit at my last company (fintech startup, based in the UK). Here’s roughly what we did:
- Pulled test run history from the last 90 days out of our CI system
- Tagged anything with >15% flake rate as “quarantine candidates”
- Ran a coverage diff to see what was actually protected by the remaining tests
- Deleted ruthlessly — killed about 600 tests
The hardest part was getting stakeholder buy-in to delete tests. People see test count as a metric of quality. You have to reframe it: fewer, trustworthy tests beat many unreliable ones.
73 flaky tests for 6 months… I felt that in my soul. We had a similar situation.
The fix that worked for us was treating flaky tests exactly like production bugs — they go into the backlog with severity and priority, get assigned to someone, and have a deadline. No more “we’ll fix it later” energy.
Once the team felt the pain of being paged because of a false failure at 2am, suddenly people cared a lot more about test reliability. 😅