CI Workflows as Kubernetes First Class Citizens
Back in July I wrote a blog post about our journey with ArgoCD. By leveraging the gitops methodology it made it easy for us to enable developers to deploy applications onto our K8s clusters. Each cluster had a repo for the app of apps where we could add a new application manifest and it would be deployed through ArgoCD. To make it even easier for our developers, who had varying degrees of experience with K8s, we setup a helm chart that had all the components we needed for a basic application. This way almost all the application manifests would look the same with just a few parameters changed like the container image location and tag.
The challenge with this approach was getting the applications into a CI workflow. We already had ArgoCD handling the CD for the applications, but how should they be built/tested/scanned/uploaded? It would erase all the ease of deployment if the developers had to engage another team to setup pipelines for their application or add the application to another system to deploy the CI workflows. Since everything else was already in K8s and using gitops we looked for a solution that was K8s native. Enter Tekton Pipelines.
Tekton Pipelines allowed us to define our CI process directly in K8s through CRDs (Tasks, Pipelines, TaskRuns, PipelineRuns, etc). They conveniently have a catalog of predefined Tasks that allowed us to get many of the pieces for our Pipelines (like cloning from git, scanning with Sonarqube, building and uploading the docker image with kaniko, and syncing with ArgoCD). We created some additional Tasks for additional compliance checks, like running Chef Inspec and uploading to Heimdall and scanning with Dependency Track. We then pieced everything together with Pipelines triggered through Tekton Triggers (which provides additional CRDs for event handling). These core Tasks and Pipelines could be defined once and used by all applications.
The only pieces we needed to include with each application were the Triggers and TriggerBindings to kick off the PipelineRuns. And since they were defined through K8s manifests we could include those as part of our helm chart. We also included a TaskRun when the helm chart was deployed that would run a Task to add the github webhook for the repo to send to the Tekton EventListener defined in the Trigger. So, everything we needed for a developer to add an application to the cluster and have it fully automated through CI/CD was just to create a single application manifest in the app of apps repo.
There were some rough patches with the implementation of course. With Tekton Pipelines being relatively new and still in Beta, a lot of the examples and articles online are outdated. For instance, many of them use PipelineResources, which are deprecated in the Beta release. We deployed the Tekton Dashboard for easy visibility into the CI process, but it doesn’t support any auth mechanisms which poses a major security concern. We also ran into some quirks due to the use of ArgoCD with Tekton. Because a PipelineRun inherits the labels of the Pipeline it is running, it has the label that ArgoCD uses to track which application resources belong to and will erroneously be marked out of sync as part of that application. We were able to get around this by adding an exclusion to ArgoCD for PipelineRuns, but it is not ideal.
By having all applications defined in a uniform way with built-in CI/CD it makes it super easy to enable developers to iterate. We look forward to exploring additional ways we can leverage Tekton Pipelines, ArgoCD, and other cloud native applications to further empower developers and remove roadblocks in the deployment of applications.
