How To Implement Commit Tracing For ArgoCD Setups

How do you use ArgoCD and a GitHub workflow together with an external CI/CD tool (Jenkins in our case) to trace a specific PR/commit hash code in a GitHub repository that’s external to ArgoCD?

The term ‘commit tracing’ comes from Michael Crenshaw – one of the lead developers of Argo. When he talked about propagating commits, in one of his Slack messages he said ‘This feature would be called “commit tracing,” and I want to add it to Argo CD, specifically the source hydrator feature.’

That’s what we’ll cover in this article. We learned the hard way how this can be accomplished and hope this writeup helps others who follow our footsteps. Hint: you need more than just ArgoCD for this ‘commit tracing’.

At Sematext, we’re big fans of dogfooding, so we use various parts of Sematext Cloud to monitor our own services internally – we gather and analyze Logs to get better insights into how things are (not) working, run Synthetics checks against our production environments to spot issues and keep an eye on the various Infrastructure metrics to make sure everything is running smoothly.

So, in the spirit of dogfooding, we wanted to use our nifty Synthetics CI/CD feature to run some Synthetics tests for our preview environments – to catch bugs in PRs instead of on production, but also to test the feature before release and introduce potential improvements or fixes. If you are new to the idea of running synthetic tests as part of your CI/CD pipeline in order to prevent a deployment when these tests fail, you can learn more about that in our Guide for Catching Regressions with GitHub Actions and CI/CD Monitors.

Synthetics tests run against real environments instead of mock data. Thus, the services we wish to test have to be up in order for the tests to run. In other words – we have to know when the environments are ready in order to trigger the tests. And this is where the first challenge appeared: how do you know when the environment is ready for running tests? Hmmmm 🤔

Sematext’s GitHub and CI/CD Setup

Here’s what our setup looks like:

• We have two GitHub repositories
  • One for application code – We’ll call it our code repo
    
  • Another for k8s manifests, yaml files and other infra-related code – Our deployment repo
• Every GitHub commit from our code repo would trigger a GitHub workflow that sent an event to Jenkins
• Jenkins would get the notification, then trigger a CI pipeline for that commit/PR
• Once Jenkins completed the pipeline successfully, it would create a values.yaml file in the deployment repo, which is a Helm object that would, along with default values overrides, contain some metadata regarding the commit/PR which initiated the build
• ArgoCD would create an Application for the commit/PR and deploy the Helm charts using the templates and values from the deployment repo, along with newly created values.yaml file after successful CI pipeline execution.
• The PR environment would be up and running

If you prefer a more schematic version of the above, here it is:

GitHub code commit/PR → event → Jenkins → trigger build → success → create Helm values.yaml → create ArgoCD Application → running Application as Helm chart in PR env

With this setup in mind, things seemed straightforward enough – we’d find the specific event in Argo that signalled a finished deployment, and use that as the trigger for the tests. After digging through the documentation for a bit, we realized that we could use Argo’s notification controller along with some templates and triggers for this purpose, with the condition being that the Application is marked as Healthy and Synced. We figured we’d just send a repository_dispatch event to our code repo, and that we’d invoke the tests as a GitHub Action that way.

This is where we encountered another unanticipated hurdle. When your CI pipelines are handled from outside of GitHub (such as in our case – with Jenkins and ArgoCD) and you send a repository_dispatch event, you have to keep in mind that such an event isn’t attached to any specific GitHub commit. Alright, that can be worked around by manually creating a check-run within GitHub, but for that we need to attach it to a specific commit using its hash. And this is where we hit a snag – Argo worked using the information in our deployment repo, but we had to attach a commit hash from our code repo. There was no direct link between the information Argo had access to, and the information that we needed.

Note that if we had a single repo for code and deployments we wouldn’t have a hard-time with this. However, this separation of code and deployment repos is common and not specific to Sematext. For example, we can see it in ArgoCD itself, as well as Jenkins:

Code repo: https://github.com/argoproj/argo-cd

Deployment repo: https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd

Code repo: https://github.com/jenkinsci/jenkins

Deployment repo: https://github.com/jenkinsci/helm-charts

Failed Experiments with Argo

Read this section if you want to see what we’ve tried. Ultimately, our attempts described in this section did not work out. But we did find a solution, described in the next section.

We remembered how Jenkins passed some information along from the code repo to the deployment repo, by way of the aforementioned values.yaml file for each ArgoCD Application we wanted to create (one per PR environment). We made Jenkins pass the latest commit hash to the values.yaml file with each successful CI pipeline, which bridged the gap somewhat. However, Argo would still need to access that information somehow. The Argo notification controller couldn’t directly access it by checking out the repository, and it wasn’t part of the same k8s namespace as our PR charts, where each PR was deployed in its own k8s namespace (basically, Argo and its pods were in their own namespace).

We then looked into ApplicationSets and their annotations. We already had a couple of different ApplicationSets for the different kinds of preview environments we wanted to deploy in order to optimize infrastructure costs, so we thought we could add some metadata to each Application which would contain the commit hash we wanted using Argo’s Git File Generator. Sure enough, this worked, but with a serious limitation: these annotations were only set once – when the Application was created. This meant that subsequent commits on the same PR wouldn’t have their hashes properly propagated, and the annotation we initially set would stay the same regardless of how many additional changes were added. In other words, each repository_dispatch event we send will always contain the commit hash that was initially set, meaning that every Synthetics test would be linked to that initial commit instead of the one it was meant to run for. Clearly not what we wanted.

The Solution: Commit Tracing with GitHub Action Workflow

After many hours spent experimenting with various approaches, combing through notification controller logs, we realized that the solution hinged on one simple fact – we didn’t have to run everything directly from Argo. Argo couldn’t get the exact commit hash for the PR it was looking for, but since we created one Argo Application for each PR, we always knew the PR number – each Application had its name equal to the number of the PR it was in charge of creating a deployment for. As we mentioned before, we had already implemented propagating the latest commit hash from the code repo to our deployment repo by Jenkins, it’s just that Argo couldn’t access it. However, Argo’s role wasn’t to go through files and find information, it was simply to inform us when an environment was successfully deployed.

In the end, we simply changed our initial plan of how these systems would interact. The original idea was:

1. Argo finishes deployment successfully
2. Argo somehow fetches the appropriate commit hash from the deployment repo
3. Argo sends repository_dispatch event with the appropriate commit hash to the code repo
4. The code repo runs the Synthetics tests

We simply added a middleman that would take care of step 2, which Argo had trouble with. The simplest solution was to use a GitHub Actions Workflow in the deployment repo with a GitHub token that had the read permissions for that repo, which would read the repository and find the commit hash we were looking for, then propagate that information to the code repo. That’s what we ended up doing, so the flow became:

1. Argo finishes deployment successfully
2. Argo sends a repository_dispatch event to the deployment repo, passing the PR number which is the same as {{path.basename}} from Argo’s aforementioned Git File Generator
3. The deployment repo runs a workflow on the specified event, fetching the appropriate commit hash for the PR we’re interested in by looking for that PR’s directory (specifically the values.yaml file inside that directory), and sending a repository_dispatch event to the code repo with the commit hash
4. The code repo runs the Synthetics tests

The takeaway

ArgoCD is a fantastic continuous delivery tool, but be mindful of what its purpose is. Even though its functionality can be extended through annotations, templates and triggers, sometimes it pays to think outside the box. Instead of trying to do everything within Argo, use it as something to trigger other events, which you can then handle by tools with more flexibility (through custom scripting) or better access to the data you may need (without limitations that your specific Argo setup may face).

And since you’re interested in improving your CI/CD pipeline, why not give Sematext Synthetics CI/CD monitoring a go? You can sign up for a free 14-day trial without having to enter any credit card info, so you can see how it can help you prevent regressions from making it to production worry-free!