Where the hell did that come from?

We’ve all been there. There is an issue in production. You connect to the cluster and try to determine the version of a container that’s running and then you see it… wtf!!! gcr.io/randomimage:wip27 I know someone who does this all the time but I’ve changed his name to protect the guilty! He knows who he is. For the purposes of this post lets call him “Rames Jawlings”.

To be honest, even if the image came from a proper release process we still have no idea where or how it was built, or by whom.

The OCI spec doesn’t appear to support any way of proving the origin of an image. However, you can add labels to images to try to provide hints about where an image has originated from. These labels are not perfect, and can of course be faked, but they are better than nothing.

Whilst searching for labels / tools to do this, I came across two specifications: the label-schema, and the open containers spec. Both seem to be doing something fairly similar in what they aim to provide. Both specifications define a number of common labels that can be applied to an image to provide some hints on the origin of an image. e.g. for open containers:

org.opencontainers.image.created=$(BUILD_DATE)
org.opencontainers.image.revision=$(GIT_COMMIT_REV)
org.opencontainers.image.source=$(GIT_SCM_URL)
org.opencontainers.image.url=$(SCM_URI)

And for label-schema:

org.label-schema.build-date=$(BUILD_DATE)
org.label-schema.vcs-ref=$(GIT_COMMIT_REV)
org.label-schema.url=$(GIT_SCM_URL)
org.label-schema.vcs-url=$(SCM_URI)

Where the recommended values for those fields should be as follows:

BUILD_DATE=$(date --utc +%Y%m%d-%H:%M:%S)
GIT_COMMIT_REV=$(git rev-parse --short HEAD)
GIT_SCM_URL=$(git config --get remote.origin.url)
SCM_URI=$(echo $GIT_SCM_URL | sed -e 's|git@github.com|https://github.com|')

A complete list of Open Container labels can be found here.

Just to reiterate…there are no guarantees that these labels are correct.

I found trying to query and compare the labels on these images quite time consuming and error prone so I decided to create a little tool called inspect to help out.

https://github.com/garethjevans/inspect

Installing Inspect

At present, inspect can be installed directly from the github releases page, or via brew using:

brew tap garethjevans
brew install inspect

If you’d prefer to run directly from an image it is also available on docker hub as garethjevans/inspect .

Inspecting an image

The idea behind the inspect image command is to display all labels assigned to an image without pulling. You can run it with:

inspect image <image>
e.g.
inspect image jenkinsciinfra/terraform:1.0.0

Diffing two images (or two versions of the same image).

Quite often, when debugging issues, you want to know what has changed between two versions of an image. You can use the inspect diff command to do this. e.g.

inspect diff <image1> <image2>
e.g.
inspect diff jenkinsciinfra/terraform:1.0.0 \   jenkinsciinfra/terraform:1.1.0

This shows a table of the differences, highlighting where differences appear. At the end of the table it also gives you a link to compare the differences on GitHub: https://github.com/jenkins-infra/docker-terraform/compare/ad902ec..441c261

Using the helper functions to build an image.

To assist in getting these labels into an existing image, I’ve created a few helper functions to get you started. The first is inspect labels. This will generate a set of label commands so that it can be used directly in a docker or img build. e.g.

docker build $(inspect labels) ...

If you’d prefer to control these labels from inside the docker image, you can use inspect build-args to generate the build-args commands to pass these values into the image. e.g.

docker build $(inspect build-args) ...

Finally…

This method of using labels is not perfect but it has already been useful in trying to visualise the difference between two images. Feel free to raise some issues/PRs against the repo for bugs/enhancements. I hope you find it useful, I know “Rames Jawlings” has!