Upgrading our testing framework to Cypress 12

We recently decided it was time to move from Cypress 9 to Cypress 12 on the platform team. Our testing framework hadn’t been scaling as fast as our engineering had over the last couple of months, and we were quickly finding ourselves in an increasingly sticky situation. We wanted to use this opportunity to restructure the test framework.

We had a lot of unresolved technical debt. The old Cypress version we were using had become a bottleneck for other tooling upgrades, and our product teams were creating new test projects using the old framework setup, essentially creating even more technical debt.

We wanted to use new Cypress features – including testing in isolation, improved test setup performance and more debugging possibilities – so we could move faster and have less friction during test automation.

It was time for Cypress 12! We knew that such a large-scale upgrade might possibly give us (and the product teams) a massive collective headache. But we also knew that in the long-run the change would be worth the effort. 

Here’s a brief look at what went on behind the scenes for all those who are about to embark on a similar journey.  

Test framework

We have about 130 spec files across 25 test projects. Each test project is a separate Nx application that defines its own dependencies. This allows us to run tests only for affected parts of our application by code changes. We also have custom generators to quickly bootstrap new test projects with custom configurations.

Our test framework consists of a custom Nx Cypress executor that executes our tests. Imagine this as an orchestration tool that takes care of space setup, resolves project configurations and calls either `cypress open` or `cypress run`, depending on the test environment. The framework also has custom helpers that consist of test data, command overwrites, custom commands, and shared functions for multiple domains. 

Our structure looks something like this: 

Strategy

Each new Cypress version has introduced some challenges for our test framework. Version 10 introduced breaking changes in project configuration logic. Version 11 caused breaking changes for our component tests. Version 12, meanwhile, completely changed the way we handle authentication and execution in Nx Cypress executor. 

Imagine reviewing all of these changes in one pull request 🤯.

After an initial discussion, we decided to migrate the changes version by version  and include some refactoring along the way.

Discovery

We started by reading the Cypres migration guides. These guides are available for each major Cypress version and identify main changes needed for the upgrade. 

Our discovery was based on the Cypress migration guide document and a quick showcase migration on a simple test project. Based on this, we identified tasks that needed to be done for each test project and understood how much of the work was required. 

Our tasks were split into five main areas:

1. Files and test projects to be deleted
2. Refactoring of current test framework
3. Upgrade of test projects
4. Upgrade of project generators
5. Monitoring and alerting

Using this as a guideline, we were able to split our work into smaller tasks that could be easily groomed and estimated.

Migration

We started by deleting all unused or skipped tests of more than four months. In some cases, this involved removing the whole test project setup. Our goal was to migrate only projects that were actively used, that saved us time during the migration.

Next, we continued refactoring the old framework code related to the migration. As part of this work, we revisited current logic and identified parts of the test framework that were obsolete, hard to understand, and slow in execution. 

Finally, we upgraded Cypress. With the exception of Cypress 10 changes, which were difficult to split per project, we handled migration project by project. This made it faster for collecting approvals and easier for others to review code changes. 

Before the merge, we ran each migration multiple times on local and CI environments to make sure there was no regression. We focused on test stability and the CI resources needed for the execution. After the merge, monitoring tools were put in place to observe stability and trigger alerts in case there was a large amount of test failures or a degradation in test performance.

In total, we  removed around 4,500 lines of code (⬆️3,513 lines, ⬇️7,979 lines) during this initiative. 

Does that mean our technical debt is gone? 

No, but this migration helped us to address some of the problematic parts of our code base. and we are closer to a more robust and scalable test framework.

Our game plan for Cypress 13

While we wait for Version 13 to be released,  there are certain steps we would definitely follow again. 

Here are our recommendations for your next Cypress upgrade:

• Use Cypress migration guides to identify changes for your upgrade.
• Start with a simple project migration as part of your discovery. You will have a better idea what code needs to be changed, and you can estimate tasks better.
• Delete unused/skipped tests. If they are skipped for months, are they still useful?
• Refactor some technical debt as part of the migration. Focus on the code connected with the migration change as it will simplify your migration later.
• Do not migrate all the test projects at once. It will be easier for others to review the changes and faster to merge them.
• Test your changes multiple times. Locally but also on CI. This will help to identify potential flakiness and understand if CI resources are sufficient for introduced changes.
• Setup monitoring and alerting. Monitor higher failure rate and slower test performance after the merge. You can then act immediately if any degradation occurs.

 

Fancy joining us on our mission to make products that matter? We’re hiring across the board. Head over to our careers page to see our available positions. We’d love to hear from you!