Automate Docker compose using Updatecli

7-minute read
Infrastructure
Continuous Deployment • Docker • gitOps • YAML

For quite a long time, I was looking for a tool that could stand between the "CI" and the "CD" of my infrastructure, I wanted a tool that could automatically update my infrastructure as code aka my YAML files.

The rise of Docker containers in production accelerated that need as it was becoming harder and harder to keep running up to date applications.

While some solutions already existed, I wanted an easy and CI agnostic way of defining custom file update strategies. At the same time, I was looking for a Golang side project to practice my dev skill which comes in handy.

Updatecli is a small cli that reads a YAML or Golang template file where we define our update strategy into three stages, a source rule where we retrieve a value, condition rules to define conditions that need to be pass before going further, and finally a target to update. If the target is changed then commits our changes and/or we validate them via a pull request. The project is still very early and it only supports YAML files for targets but I am planning to add more file format in the future depending on my need and my ability to assign enough time to this project.

I propose to give you an overview of how to automate you docker compose files.

Docker Compose

In this scenario, we have to maintain a docker-compose file and we want to be sure that we always use the newest docker images available.

Docker image versions can be identified using two methods, either we use the docker image tag or the docker image digest.

Tag

A docker image tag is useful when we want to have an easy to identify human-readable version like in nginx:1.17 where 1.17 is the tag, unfortunately, a docker image tag can be easily overridden, on purpose or not. Like when using the tag latest. This means that a docker image tag can’t be considered as a source of truth.

Digest

The second method to identify a docker image version is by using its digest like in nginx@sha256:8269a7352a7dad1f8b3dc83284f195bac72027dd50279422d363d49311ab7d9b where 8269a7352a7dad1f8b3dc83284f195bac72027dd50279422d363d49311ab7d9b is the digest. A digest is a hash referencing a specific version of a docker image tag like in this case nginx:1.17. A digest is immutable, we can always rollblack to a specific digest if need. But we must admit that it’s not convenient to identify which version we are referencing to without having to dig into it.

The problem here could be resumed by "Do we want to only use well-defined docker image by using digests but with the cost of losing readability?" or "Are we ok to take the risk of running an unwanted version by using image tags?".

It depends on how important we value readability versus uniqueness, but a rule of thumb is to consider the docker image build strategy, what’s the risk of running an unwanted version, and how big is the chance of having to rollback. If we only build one image by version and that version is regularly updated then preferring readability by using docker image tag seems ok-ish otherwise in any other scenarios It’s usually safer to use docker image digest.

Now that we know what we need to update and why, let see how we can do it using updatecli.

Docker Image Tag

When we want to automate image tag updates, we have to identify the process used to define those tags, and if that process is automated, then you can more than likely also automate version updates. That’s where updatecli can play a role.

In updatecli, you specify this process in the source stage, then if needed you apply some condition and finally, if everything goes well then you update the version in your target file. So let take this example where we want to regularly update the Jenkins version, we know that a weekly release is published every week on a maven repository, then a docker image is built and published on DockerHub. Considering the build strategy, it seems fine to use docker image tag.

Let’s consider this docker compose file:

docker-compose.yaml
version: '3'
services:
  jenkins:
    image: jenkins/jenkins:2.245
    ports:
        - 8080:8080

Now we write an updatecli configuration that contains a source, a conditions, and one target

strategy.yaml
source:
  kind: maven
  postfix: "-jdk11" # will be add to the value returned by the spec definition
  spec:
    owner: "maven"
    url: "repo.jenkins-ci.org"
    repository: "releases"
    groupID: "org.jenkins-ci.main"
    artifactID: "jenkins-war"
conditions:
  docker:
    name: "Docker Image Published on Registry"
    kind: dockerImage
    spec:
      image: "jenkins/jenkins"
targets:
  imageTag:
    name: "docker-compose jenkins service"
    kind: "yaml"
    prefix: "jenkins/jenkins:" # will be add to the value returned by the spec definition
    spec:
      file: "./docker-compose.yaml"
      key: "services.jenkins.image"

Now let’s run updatecli by executing following command:

docker run -i -t -v $(pwd):/updatecli --workdir /updatecli olblak/updatecli:v0.0.16 diff --config ./strategy.yaml

Updatecli will query the maven repository 'releases' located on repo.jenkins-ci.org, to have the latest version. If it finds one, then we add to the value retrieved, the prefix jenkins/jenkins: and the postfix -jdk11 as we are looking specifically for the java 11 version. We validate that docker image effectively exist on DockerHub and then we test if the value in the file 'docker-compose.yaml' for the key services.jenkins.image is correct otherwise we plan to update it.

If we are good with the changes then we apply them by using apply instead of diff by running:

docker run -i -t -v $(pwd):/updatecli --workdir /updatecli olblak/updatecli:v0.0.16 apply --config ./strategy.yaml

Docker Image Digest

As explained earlier, sometimes we prefer to use docker image digest, if we need to rely on a specific version or if we want to easily rollback. While most of the configuration remains the same as in the previous example, this time the source will be DockerHub and we won’t test if an image already exists on DockerHub as we already ask DockerHub for the latest image available 😈 .

docker-compose.yaml
version: '3'
services:
  jenkins:
    image: jenkins/jenkins:2.245
    ports:
        - 8080:8080
strategy.yaml
source:
  kind: dockerDigest
  spec:
    image: "jenkins/jenkins"
    tag: "lts-jdk11"
targets:
  imageTag:
    name: "jenkins/jenkins:lts-jdk11 docker digest"
    kind: yaml
    spec:
      file: "./docker-compose.yaml"
      key: "services.jenkins.image"

We run:

docker run -i -t -v $(pwd):/updatecli -workdir /updatecli olblak/updatecli:v0.0.16 diff --config strategy.yaml

This time updatecli queries DockerHub to retrieve the digest for the docker image jenkins/jenkins:lts-jdk11. If it finds one, then we test if the value in the file 'docker-compose.yaml' for the key services.jenkins.image is correct otherwise we plan to update it.

Again if we are ok with the changes then we apply them by using apply instead of diff.

docker run -i -t -v $(pwd):/updatecli -workdir /updatecli olblak/updatecli:v0.0.16 apply --config stragegy.yaml

Git/GitHub

Now that we have an easy way to update docker image version, we are missing a way to save, review, rollback those changes, and git for this is a tremendous tool. Either we directly commit and push to a git repository or we use the GitHub workflow by pushing to a temporary branch then submit our changes via a pull request which can then be approved.

docker-compose.yaml
version: '3'
services:
  jenkins:
    image: jenkins/jenkins:2.245
    ports:
        - 8080:8080

While the configuration remains quite similar to our previous example, this time we introduce two new elements. Firstly, strategy.yaml becomes strategy.tpl which is a go template. By using go template we can define generic values and reference them from our template or read values from an environment variable like {{ requiredEnv GITHUB_TOKEN }} which is what we want here so we don’t put the GitHub token in the file. The second major change is the 'scm' block which should be quiet obvious and defines where to push commits.

strategy.tpl
source:
  kind: dockerDigest
  spec:
    image: "jenkins/jenkins"
    tag: "lts-jdk11"
targets:
  imageTag:
    name: "jenkins/jenkins:lts-jdk11 docker digest"
    kind: yaml
    spec:
      file: "./docker-compose.yaml"
      key: "services.jenkins.image"
    scm:
      github:
        user: "John"
        email: "john@example.com"
        owner: "jenkins-infra"
        repository: "charts"
        token: "{{ requiredEnv GITHUB_TOKEN }}"
        username: "johnDoe"
        branch: "master"

And now you can use the same command than before

  1. docker run -i -t -v $(pwd):/updatecli -workdir /updatecli olblak/updatecli:v0.0.16 diff --config strategy.tpl

  2. docker run -i -t -v $(pwd):/updatecli -workdir /updatecli olblak/updatecli:v0.0.16 apply --config strategy.tpl

Conclusion

In this scenario, we saw how to automatically update docker-compose file using custom strategies with updatecli. Because Updatecli is a small tool you can also use it from your favorite CI environment whatever is.

Now we can replace our docker-compose file by any other YAML file to automate YAML update.

Feel free to provide any feedback you may have by opening Github issue on the project here

Links