In most professions, seeing software bugs on the computer first thing in the morning is a bad way to start a Monday. For software engineers, it’s a welcome sign of a successful automated regression test.
Once considered a tedious and expensive process, regression tests have been upgraded with new frameworks and tools that provide scaffolding for practices like end-to-end tests and scale/load testing, rendering them easier to deploy and necessary to many companies’ software development processes. Engineers today generally see regression tests as more than preventative measures: they’re fail-safes that catch issues the naked eye would miss, ensuring applications aren’t released with unforeseen consequences.
Engineers across Seattle use regression tests to check for coding errors, simulate users and performance, and validate performance changes. In many instances, they said that regression testing reminded them that sometimes the smallest alterations to an application’s source code can break surprising, seemingly unrelated functions.
Every young startup must prepare for scale. For engineers, that means preparing for an increase in data traffic and using regression testing to ensure quality performance. At digital freight network Convoy, Engineering Manager Waj Sampathkumar said they use load/scale testing to simulate multiple users accessing the software, resulting in solves like reduced CPU load.
What are some best practices your team follows when handling regression testing?
As our engineers write code, they leverage our local service deployment framework to manually test changes and validate fixes. Changes are accompanied by unit tests. During the code review process, untested code is flagged and the engineers are encouraged to add tests. We also generate code coverage results for the changes, to help engineers test comprehensively.
Our services interact in complex ways, and to safeguard against regressions to our business logic, we write end-to-end tests and integration tests. These tests run as a part of our CI/CD systems and automatically fail builds with regressions. Before deploying to production, we deploy to canaries as an additional layer of protection against regressions to our broader customer base. We also periodically do redline testing to identify load issues and catch regressions to our scale limits. Lastly, as a part of our production incidents post-mortem, we look for possible test gaps and address them as follow-ups.
We have a dedicated team to analyze each stage of the development workflow.”
What are some steps your team has taken to make regression testing more efficient and streamlined?
We have a dedicated team to analyze each stage of the development workflow and identify and roll out the right set of tools for testing. To ensure that engineers can manually test their changes locally, we’ve developed Convoy-in-a-Box (CBox) which deploys locally the services related to the code being developed and provides production-like data to help engineers test effectively.
On the unit test front, we use Jest as our preferred test framework due to its ability to provide test isolation when running in parallel, and also to provide the code coverage statistics. We’ve also invested in a set of tools to help engineers write end-to-end tests. These automatically set up scaffolding for end-to-end tests so that engineers can focus on adding business logic rather than dealing with test setup. We also have built Slack integration to our test runners, so that test regressions are handled in a timely fashion.
Tell us about a time when regression testing worked and caught a major bug in your code. What lessons did you learn from this?
As a fast-growing startup, we need to prepare for ever-increasing traffic. We’ve learned that load/scale testing is a valuable tool to ensure that we continue to handle our incoming traffic and provide a high quality of service to our customers.
During one recent run, we discovered and fixed a few high-cost queries hitting the database of our largest service. Fixing these individually reduced the CPU load from over 75 percent to 20 percent. This brought significant headroom to help scale this service during the unanticipated market shift in February. We now do these types of testing on a regular basis to uncover regressions early, and also use these lessons to build better tools to help avoid such issues.
Whenever regression testing seems unnecessary, Nick Ward, a software engineering manager, said he remembers how interconnected the tech is at social casino company DoubleDown. When adding new features, regression tests help him spot effects on other applications that he wouldn’t have checked otherwise.
What are some best practices your team follows when handling regression testing?
When it comes to regression testing there are a few practices that really help here at DoubleDown. First is the prioritization of tests in our suites. Some tests don’t need to be run with every iteration of a project and prioritization helps others know the importance of a test. Categorization can really help as well. Knowing what changed and what might be affected by the change really narrows down what tests need to be run. Keeping tests targeted and focused makes them easier to understand and run.
What are some steps your team has taken to make regression testing more efficient and streamlined?
We’re in the process of automating wherever we can. This alleviates the need to have testers run through many scenarios and keeps us aware of smaller changes that the naked eye might miss. For our server team, this means we have our test suites running automatically against every service deployed to ensure nothing is broken.
One thing that always shocks me is how interconnected some things are.”
Tell us about a time when regression testing worked and caught a major bug in your code.
One thing that always shocks me is how interconnected some things are. Our applications are complex and often a change to one area will affect another in ways we didn’t think they would. Without our regression suite of tests we would probably miss these effects and release with unforeseen consequences.
At travel group Expedia, Senior Engineering Manager Chris Cantu runs regression tests to make sure new releases will operate on Internet Explorer since it runs differently than more modern browsers. They also test frequently for UX errors to ensure customers are receiving the correct visual experience.
What are some best practices your team follows when handling regression testing?
The Vrbo core experience teams leverage multiple types of testing in our continuous integration pipeline.
These testing strategies include a hundred-percent unit test suite coverage to validate code at the function level; visual difference testing to validate cosmetic changes; end-to-end testing to validate specific functional use cases in the user interface; chaos testing to validate how the application reacts when services or data are not available; GraphQL schema testing to validate changes to our schema do not break the application; and Google Lighthouse testing to validate our accessibility and ensure performance scores do not fall below target thresholds.
What are some steps your team has taken to make regression testing more efficient and streamlined?
We parallelize build steps where it makes sense using tools such as Google Lighthouse, E2E Tests and GraphQL Schema Check.
We leverage Jest for web unit testing which automatically parallelizes unit tests in the test suite. Also, GraphQL Doctor is a great tool to validate all schema changes before they hit production. Finally, we use ChaosBox, which is an internal tool used to validate graceful application degradation when underlying data and asset dependencies drop out.
GraphQL Doctor is a great tool to validate all schema changes before they hit production.”
Tell us about a time when regression testing worked and caught a major bug in your code.
Internet Explorer is a legacy browser which still represents a significant amount of web traffic. Given this fact, we implemented a simple smoke test for Internet Explorer that is run before every release. This smoke test has caught numerous P1 (high priority) bugs given that IE behaves differently than modern browsers.
Beyond regression testing, the Vrbo core experiences team implemented UX alerts. These alerts monitor critical visual components, as opposed to lower-system-level error rates and status codes. For example, search results may sometimes have empty data fields due to our distributed architecture. These partially rendered results would not be caught by status codes or error rates for the web application that renders the page, as the overall orchestration to build and render the page was successful.
By implementing a simple React component, we are able to monitor and alert when users are presented with incomplete results. This emphasis on UX alerts over lower-level system alerts is far less fragile when underlying implementation details change and allows us to be more customer-centric and scale 24/7 support across the organization and multiple continents.
Our engineers enjoy daylight-only support rotations because we provide intuitive alerting that is focused on the actual customer experience and is paired with straightforward incident response runbooks.
