DXR is a code search and navigation tool aimed at making sense of large projects. It supports full-text and regex searches as well as structural queries.

Mercurial (923415cae003)

VCS Links

Line Code
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210
.. taskcluster_dockerimages:

Docker Images

TaskCluster Docker images are defined in the source directory under
``taskcluster/docker``. Each directory therein contains the name of an
image used as part of the task graph.


Each folder describes a single docker image.  We have two types of images that can be defined:

1. Task Images (build-on-push)
2. Docker Images (prebuilt)

These images depend on one another, as described in the `FROM
<https://docs.docker.com/v1.8/reference/builder/#from>`_ line at the top of the
Dockerfile in each folder.

Images could either be an image intended for pushing to a docker registry, or
one that is meant either for local testing or being built as an artifact when
pushed to vcs.

Task Images (build-on-push)

Images can be uploaded as a task artifact, :ref:`indexed <task-image-index-namespace>` under
a given namespace, and used in other tasks by referencing the task ID.

Important to note, these images do not require building and pushing to a docker registry, and are
built per push (if necessary) and uploaded as task artifacts.

The decision task that is run per push will :ref:`determine <context-directory-hashing>`
if the image needs to be built based on the hash of the context directory and if the image
exists under the namespace for a given branch.

As an additional convenience, and a precaution to loading images per branch, if an image
has been indexed with a given context hash for mozilla-central, any tasks requiring that image
will use that indexed task.  This is to ensure there are not multiple images built/used
that were built from the same context. In summary, if the image has been built for mozilla-central,
pushes to any branch will use that already built image.

To use within an in-tree task definition, the format is:

.. code-block:: yaml

        type: 'task-image'
        path: 'public/image.tar.zst'
        taskId: '<task_id_for_image_builder>'

.. _context-directory-hashing:

Context Directory Hashing

Decision tasks will calculate the sha256 hash of the contents of the image
directory and will determine if the image already exists for a given branch and hash
or if a new image must be built and indexed.

Note: this is the contents of *only* the context directory, not the
image contents.

The decision task will:

1. Recursively collect the paths of all files within the context directory
2. Sort the filenames alphabetically to ensure the hash is consistently calculated
3. Generate a sha256 hash of the contents of each file
4. All file hashes will then be combined with their path and used to update the
   hash of the context directory

This ensures that the hash is consistently calculated and path changes will result
in different hashes being generated.

.. _task-image-index-namespace:

Task Image Index Namespace

Images that are built on push and uploaded as an artifact of a task will be indexed under the
following namespaces.

* gecko.cache.level-{level}.docker.v2.{name}.hash.{digest}
* gecko.cache.level-{level}.docker.v2.{name}.latest
* gecko.cache.level-{level}.docker.v2.{name}.pushdate.{year}.{month}-{day}-{pushtime}

Not only can images be browsed by the pushdate and context hash, but the 'latest' namespace
is meant to view the latest built image.  This functions similarly to the 'latest' tag
for docker images that are pushed to a registry.

Docker Registry Images (prebuilt)

***Warning: Registry images are only used for ``decision`` and
``image_builder`` images.***

These are images that are intended to be pushed to a docker registry and used
by specifying the docker image name in task definitions.  They are generally
referred to by a ``<repo>@<repodigest>`` string:


.. code-block:: none

    image: taskcluster/decision:0.1.10@sha256:c5451ee6c655b3d97d4baa3b0e29a5115f23e0991d4f7f36d2a8f793076d6854

Such images must always be referred to with both a version and a repo digest.
For the decision image, the repo digest is stored in the ``HASH`` file in the
image directory and used to refer to the image as above.  The version for both
images is in ``VERSION``.

The version file serves to help users identify which image is being used, and makes old
versions easy to discover in the registry.

The file ``taskcluster/docker/REGISTRY`` specifies the image registry to which
the completed image should be uploaded.

Docker Hashes and Digests

There are several hashes involved in this process:

 * Image Hash -- the long version of the image ID; can be seen with
   ``docker images --no-trunc`` or in the ``Id`` field in ``docker inspect``.

 * Repo Digest -- hash of the image manifest; seen when running ``docker
   push`` or ``docker pull``.

 * Context Directory Hash -- see above (not a Docker concept at all)

The use of hashes allows older tasks which were designed to run on an older
version of the image to be executed in Taskcluster while new tasks use the new
version.  Furthermore, this mitigates attacks against the registry as docker
will verify the image hash when loading the image.

(Re)-Building images

Generally, images can be pulled from the Docker registry rather than built
locally, however, for developing new images it's often helpful to hack on them

To build an image, invoke ``mach taskcluster-build-image`` with the name of the
folder (without a trailing slash):

.. code-block:: none

    ./mach taskcluster-build-image <image-name>

This is a wrapper around ``docker build -t $REGISTRY/$FOLDER:$VERSION``.

It's a good idea to bump the ``VERSION`` early in this process, to avoid
``docker push``-ing  over any old tags.

For task images, test your image locally or push to try. This is all that is

Docker Registry Images

Landing docker registry images takes a little more care.

Begin by bumping the ``VERSION``.  Once the new version of the image has been
built and tested locally, push it to the docker registry and make note of the
resulting repo digest.  Put this value in the ``HASH`` file for the
``decision`` image and in ``taskcluster/taskgraph/transforms/docker_image.py``
for the ``image_builder`` image.

The change is now safe to use in Try pushes.

Note that ``image_builder`` change can be tested directly in try pushes without
using a registry, as the in-registry ``image_builder`` image is used to build a
task image which is then used to build other images.  It is referenced by hash
in ``taskcluster/taskgraph/transforms/docker_image.py``.

Special Dockerfile Syntax

Dockerfile syntax has been extended to allow *any* file from the
source checkout to be added to the image build *context*. (Traditionally
you can only ``ADD`` files from the same directory as the Dockerfile.)

Simply add the following syntax as a comment in a Dockerfile::

   # %include <path>


   # %include mach
   # %include testing/mozharness

The argument to ``# %include`` is a relative path from the root level of
the source directory. It can be a file or a directory. If a file, only that
file will be added. If a directory, every file under that directory will be
added (even files that are untracked or ignored by version control).

Files added using ``# %include`` syntax are available inside the build
context under the ``topsrcdir/`` path.

Files are added as they exist on disk. e.g. executable flags should be
preserved. However, the file owner/group is changed to ``root`` and the
``mtime`` of the file is normalized.

Here is an example Dockerfile snippet::

   # %include mach
   ADD topsrcdir/mach /builds/worker/mach