Dealing with flakiness in Cypress tests!
Logically, when an application is put through the same test over and over, the code should produce the same result, either PASS or FAIL.
What is a Flaky test?
It is a test case that showcases inconsistent outcome — one moment you see it pass, next moment it might just fail in spite of running in the same environment with the same test data.
Why is it a BIG problem?
Flakiness directly or indirectly affects the confidence of the application that is being tested. Some of the reasons for flaky tests can be - changes in source code or automation code, issue with framework or an unstable environment. The presence of flaky tests may cause substantial problems to the testers and developers :
(1) they may hide real bugs, other than being hard to reproduce because of their unpredictable behavior
(2) increase in maintenance costs, it may take a lot of time to debug failures that are not connected to any defect.
— — — — — — — — — — — — — — — — — — — — — — — — — —
For me, it’s been an year working on Cypress and all I can say is -
As the test cases increase in number and complexity, level of flakiness might just shoot up if not identified early.
— — — — — — — — — — — — — — — — — — — — — — — — — —
Well, Cypress has a Awesome feature for folks like us. It has helped us in drastically reducing flaky tests.
Introducing our savior — Test retries in Flaky Test Management!
Cypress Dashboard lets us view our flaky tests in every run. It has the failure message along with screenshots of each retry attempt it has made.
How do we go about solving Flaky issues in Cypress?
- Identifying flaky tests in Cypress:
In the Cypress dashboard under your project, in the left-side menu bar, look for Flaky Tests section.
The UI of this section is user-friendly way providing the following facilities:
- Views: Test case and Spec file
- Filters: Filter by Branch, Tag, Run group, Browser, Cypress version etc.
- Flakiness report: Overall flakiness %; High, medium and low severity, Flaky tests and Flaky rate
Using the above, we can filter out recent as well as past data that is as old as 12 months😳.
— — — — — — — — — — — — — — — — — — — — — — — — — —
2. Analyzing these filtered flaky tests:
Every flaky test comes with its individual flakiness report that consists of the the error message, changelog, its failure rate and flaky rate.
The error message will help us understand why the test case has failed and how many retries did it require to PASS. Understanding the root cause of this failure message will help us find the fix for the problem.
We can come across scenarios where we are unable to replicate the flakiness itself. In such cases, it becomes important for us to understand the state of the system when flakiness occurred.
I’ll be covering the most common flaky issues and their quick fixes in my next article.
See you soon 😄 !
