Guide for Catching Regressions with GitHub Actions and CI/CD Monitors

This guide aims to help your team shift testing left, simulate real user behavior, and catch critical issues early as part of CI/CD, prevent regressions from reaching production by automating tests as part of your CI/CD and aborting deployments that contain issues.

Synthetic monitoring is a great way to check important flows in production and make sure everything is working the way it’s supposed to. However, as this poll by Peter Zaitsev highlights, most regressions in productions come from deployments.

I mean, that’s sort of by definition, isn’t it? 🙂

This problem has a solution, though, and it’s not terribly hard to set up technically either. But today, most organizations don’t have this in place. You don’t have to be in that camp. By utilizing Synthetics CI/CD monitors you can catch regressions in preview environments before they reach production. This expands the options you have with Synthetics monitors to not only detect production issues and minimize downtime, but to also prevent them from popping up in the first place. How? By leveraging User Journey scripts, you can automatically test even complex flows that span multiple pages and simulate real user interactions, ensuring that critical parts and paths in your applications remain functional as your code evolves.

While writing User Journey scripts for monitors that integrate into CI/CD pipelines shares many similarities with creating regular Synthetics monitors, there are several unique considerations to keep in mind. This guide will help you adapt your existing monitors for CI/CD use cases and avoid common pitfalls when testing in preview environments.

But before we get into that, let’s first define some of these concepts, like User Journey Scripts, CI/CD, and Shift-left. If you already know all this, you can skip to the best practices section or even the list of relevant tools further down.

What are User Journey Scripts?

Simply put, User Journey Scripts are scripts that simulate a user’s actions in a web application. Think of them as programmable robots that mimic real people: clicking buttons, filling out forms, logging in, navigating pages, and checking that the expected content is there. They’re typically written using browser automation frameworks like Playwright or Puppeteer, and they serve as the foundation for synthetic monitoring.

In a synthetic check, these scripts run on a schedule or in response to a trigger (like a CI/CD pipeline run), and validate that the application behaves and performs as expected. This is incredibly useful for catching regressions or broken user flows before your real users do.

Just for reference, here are our User Journey Script docs. They are a part of Sematext Synthetics, which provides a flexible way to write and manage these scripts, supporting full browser-based testing with assertions, custom code, and performance measurements.

What is CI/CD?

CI/CD stands for Continuous Integration and Continuous Delivery (or Deployment). It’s a DevOps practice that automates how code changes go from development to production. Here’s a quick breakdown:

• CI (Continuous Integration): Developers push code to a shared repo multiple times a day. Each push triggers a build and automated tests to catch issues early.
• CD (Continuous Delivery/Deployment): After CI passes, the code is automatically delivered to staging (delivery) or directly to production (deployment), depending on your pipeline setup.

The goal of CI/CD is to ship code faster and with higher confidence, thanks to early and frequent testing. You can learn a lot more about CI/CD here.

What is Shift-left?

Shift-left is a mindset and practice of moving testing and monitoring earlier (i.e., to the left) in the software development lifecycle. Traditionally, testing happened after code was written, sometimes even after deployment. Shift-left says: test as early as possible. Same thing with monitoring. Typically we monitor production for performance, availability, errors, etc. but neglect to monitor all that prior to shipping to production.

Why it matters:

• Catch bugs early: It’s cheaper and easier to fix issues in dev or staging than in prod. You can think about this from multiple aspects. From the time aspect, you will literally spot issues earlier, so you will save time by iterating on a fix earlier.
• Faster feedback: Developers know immediately if a change broke something. Again, faster iteration, faster cycles, faster velocity. We all want that.
• Better quality: You can simulate real user behavior and performance before your users ever touch the app. It could be very costly to have your users or customers hit regressions – if they have a bad user experience not only will they leave earlier, but they may never come back again!

How it’s different from traditional monitoring:

• Shift-left monitoring: Uses synthetic checks in CI/CD to test the app in staging or some sort of pre-production environment. You have that, right? 🙂
• Traditional monitoring: Happens post-deployment, tracks real user metrics, logs, traces, etc. – in production.

Obviously, both are valuable, but shift-left gives you a head start.

Best Practices for Setting Up User Journey Checks in CI/CD

If you have never put together this sort of setup, here are some of the suggestions based on our own experience at Sematext. Our own CI/CD Monitoring is triggered through GitHub Actions, hence the references to that.

• Keep checks fast
  Synthetic tests should be fast enough to not slow down your pipeline. If your application is non-trivial, you might need to accept that you might not be able to test all of it. Focus on critical paths.
• Optimize test frequency and resource usage
  Since you’re in complete control of how often CI/CD monitors are executed, you should balance the test frequency with your available GitHub Actions minutes by cleverly setting the conditions which invoke the tests. You can also check out our example of implementing a self-hosted runner to offset some of the costs and handle some parts of the testing workflow outside of GitHub.
• Prepare the data you need, don’t assume its existence
  Preview environments often spin up with fresh databases that contain little to no data. This can break scripts that assume certain records exist, such as when you need some specific data to access the pages you’d like to test. To circumvent this issue, design your scripts to create any necessary test data at the beginning, then use conditional checks to verify data exists before interacting with it.
• Clean up after yourself
  Depending on how your preview environments are set up, you might also have to pay attention to cleaning up after the tests. This may not be relevant if the environment you’re testing are transient, but if your setup is such that you have a shared DB for a certain type of change (such as for frontend-only changes), you’ll have to pay attention to cleaning up after your tests are done in order to avoid cluttering the environment with test data. Try to clean up through the UI if that’s something you’d also like to test, and make sure to catch any errors and perform a forced cleanup through API calls if that doesn’t work out.
• Use environment-specific configs
  Preview and production environments often differ in several ways beyond just resources, such as with:
  • SSL certificates – Preview environments may use self-signed certificates. Consider disabling strict SSL validation if you encounter certificate errors.
  • Feature flags – Different features may be enabled/disabled between environments.
  • Third-party integrations – External services might be mocked or use sandbox versions.
  • Authentication – Preview environments might use different auth providers or simplified auth flows.

  Make it easy to run the same checks in staging, QA, or prod with environment variables.

• Include performance assertions
  Don’t test only for correct behavior. Include check load times, responsiveness, and other aspects of your applications.
• Run on every PR or push to main… or?
  Make checks an integral part of your pipeline. That said, you will want to be mindful of when you run tests because running them too frequently could also have negative side-effects, such as increased resource usage, slowing down of some other part of CI/CD, or cost increase. We hit this problem ourselves and quickly learned that even if your Github Action takes 2 seconds to execute, it is billed as a full minute of usage. Guess what happened next? We quickly switched to a self-hosted runner. In these docs we provided a public repo you can use to get this set up for yourself, too.
• Working with dynamic URLs
  Preview environments typically generate unique URLs for each deployment. For this very purpose, Sematext Synthetics CI/CD monitors offer a Dynamic URLs feature. Make sure to set the base Dynamic URL as a variable, and then combine it with whichever paths you’d like to test later on in the script, for the script to remain flexible across different preview environments.
• Managing environment instability
  Preview environments often run with reduced resources compared to production, leading to potential instabilities and slower response times. After implementing the User Journey scripts for your CI/CD monitors, keep an eye on them and check for places where intermittent error rates might be high. Playwright already offers automatic retries with most of its functions, but adding some manual ones to error-prone parts of your scripts could save you a lot of headache. These slowdowns could potentially make your scripts last for longer than the 2 minutes allowed for Sematext Synthetics monitors, so you should take that into consideration while writing them and potentially break up complex flows into multiple monitors. Cold starts might also affect the timing and stability of your runs, so you may have to add some special logic to handle them.
• Write readable scripts
  This is a given, but just to be explicit here – treat your CI/CD Monitors and User Journey Script as code, because that is what they are. Use comments and abstractions to make scripts maintainable, along with all other good practices. For example, keep your User Journey Scripts versioned, in a GitHub repository, don’t just treat them as some ad-hoc scripts. If you end up using Sematext for this, use the GitHub <> Sematext sync functionality.
• Prevent conflicts in parallel execution
  The CI/CD monitors within a CI/CD Group Run will all run in parallel, meaning you should take into consideration how they might affect each other. Thus, ensure that each test is able to fully execute not only on its own, but also alongside the other tests you’ll be running. Data that might be entered from other tests shouldn’t interfere with its results – so look out for possible issues like multiple monitors trying to create things with the same name in places where naming should be unique, or one test potentially referencing a bit of data that a different test might delete. Where possible, try to capture and reuse unique IDs to prevent such issues from arising.




• Alert on failures
  While CI/CD Monitor failures will show up in GitHub Actions console, you should also set things up so that you have to do the least manual checking. Instead of you going to the GitHub Action console, make notifications of any failure come to your team. Integrate with Slack, email, or whatever your team uses to notify on broken journeys.

Tools for Simulating Real User Journeys in CI/CD

There are existing tools that you can use to add the sort of testing/monitoring described in this guide to your CI/CD. First, a short list of recommended solutions to choose from:

Solution	Strongest Features
Sematext Synthetics	Browser-based Playwright scripts, API tests, CI/CD integration, performance metrics, global locations, uptime checks, alerting, GitHub Actions integration and repo syncing
Checkly	Headless browser checks, CI/CD-friendly, built-in Playwright support, alerting, GitHub Actions integration
Datadog Synthetic Monitoring	Browser tests, API tests, CI/CD support, integrates with wider Datadog ecosystem, visual testing
Grafana Synthetic Monitoring (k6)	Scriptable load testing with JavaScript, good for performance benchmarking, can simulate user journeys in k6-browser
QA Wolf	Records tests from browser actions, easy setup, CI integration, cloud-based test runner
Cypress	Great for frontend testing, supports browser automation, dev-friendly syntax, good local + CI support

In the following table you will see the list of tools or frameworks to consider. Use them only if you really prefer the DIY approach. The above list of solutions will be easier to set up, more feature rich, and ultimately give you more value for your money, but if you are a strong proponent of the DIY approach, here’s the list of tools to use.

Tool/Framework	Strongest Features
Playwright	Modern automation, great for headless/full browsers, supports multiple browsers, excellent for CI/CD, strong debugging features.
Puppeteer	Headless Chrome automation, lightweight and easy to use, great for simple user flow tests
TestCafe	Browser testing without WebDriver, good for CI, easy syntax, no browser plugins needed
Nightwatch.js	End-to-end testing framework, CI/CD friendly, supports multiple browsers, integrates with Selenium/Grid

Summary

While writing scripts for CI/CD monitors (or converting regular Synthetics monitors to CI/CD ones) might seem daunting at first, it’s actually a fairly straightforward process most of the time. Having a solid understanding of how your preview environments work and keeping in mind some of the points discussed in this post will let you handle most of the use cases you need without much hassle. With a bit of experimentation, you’ll soon be building reliable tests that catch regressions early without creating false positives.

If you’d like to try out integrating Synthetics testing into your CI/CD pipeline, why not try Sematext Synthetics? We offer a 14-day trial without having to add any credit card info, so you can see for yourself if it works for you without worrying about unwanted charges.