A typical CI/CD pipeline explained

Recently, I was asked on Twitter if I could give some pointers for reading material on (production) CI/CD pipelines – what they look like, and how they work. I decided to shed some light on this by describing a typical CI/CD pipeline (from my perspective), in a series of tweets. This article is an expansion of that thread.

On pipelines, and Continuous Integration

Let me start with a short disclaimer. In my career, I’ve created, seen & used many different CI/CD pipelines. While they are conceptually pretty much the same and have (technical) similarities, pretty much every pipeline is (somewhat) unique in its implementation. This is due to teams making different architectural choices, tool selections, etc. etc. The text below is a reflection of my current thinking on the subject.

A typical CI/CD pipeline is linear and one-directional. It is composed of a few stages, or phases. The first phase (CI, or Continuous Integration) takes a commit from the mainline (usually referred to as trunk or master, depending on your version control system), and runs a few steps to verify that commit.

The actual list of steps depends highly on the language and platform used, but typically consists of (at least) checkout, compilation and running unit tests. Some pipelines also add steps to perform code analysis and run additional tests (such as integration tests).

Continuous Integration Stage

The pipeline should give early feedback and be as fast / short as possible. I typically try to keep my pipelines finish within 15-20 minutes. To achieve this, it makes sense to parallelize steps as much as possible, so that test failures or other problems can be reported quickly.

Building and publishing artifacts

When all these steps are successful, a unique artifact is built, packaged, and published to a repository. This can be a JAR, .tar.gz file, a container image, or whatever is applicable to the chosen language and platform.

Some pipelines deploy the generated artifact to a test environment, to run additional checks. I
Next, the artifact is deployed to staging/acceptance (or whatever you want to call it, the environment should be equivalent to production) and verified.

CD?

Now this is where Continuous Delivery and Continuous Deployment will start to diverge. If you want to do the former, there will be a manual gate, usually in the form of a button or similar. Pressing the button is required to “promote” an artifact to production.

When and what to promote is a choice. Promotion typically happens when a build is considered to be “good enough”. QA has run their tests, the Product Owner has signed off, etc. This means that not all artifacts make it to production!

Continuous Deployment gets rid of the manual gate and fully relies on automatic verification of the acceptance environment to determine whether the pipeline can continue on to production. Production, in turn, is verified in the same way. I wrote a little bit about the difference between the two CD’s here: CD: Continuous Delivery or Continuous Deployment?

Deployment Stages

Automatic verification

Automation in general, and test automation specifically, is a crucial aspect of any CI/CD process and pipeline. Automatic verification is used in all stages, to validate artifacts, deployments, etc.

Automatic tests come in all sorts, shapes and sizes:

The goal of test automation is to catch known problems. Exploratory testing, usability reports, customer feedback, etc. helps us to identify the unknown issues, at which point automated tests can be adjusted to cover whatever’s been found.

Closing & further reading

As an expansion of a series of tweets, I hope this is a useful blog post, detailing my thoughts about CI/CD pipelines and their configuration. Let me know what you think!

For your reading pleasure, here’s some more material that (partially) covers pipelines in the CI/CD context:

Michiel Rook

Michiel Rook

Michiel Rook is an experienced, passionate & pragmatic freelance coach, developer & speaker from the Netherlands. He loves helping teams and companies to develop better software and significantly improve their delivery process. When he’s not thinking about continuous deployment, DevOps or event sourcing he enjoys music, cars, sports and movies.

4 thoughts on “A typical CI/CD pipeline explained

Leave a Reply

Your email address will not be published. Required fields are marked *