DXR is a code search and navigation tool aimed at making sense of large projects. It supports full-text and regex searches as well as structural queries.

Mercurial (1aeaa33a64f9)

VCS Links

Line Code
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176
.. _mozbuild-files:

===============
moz.build Files
===============

``moz.build`` files are the mechanism by which tree metadata (notably
the build configuration) is defined.

Directories in the tree contain ``moz.build`` files which declare
functionality for their respective part of the tree. This includes
things such as the list of C++ files to compile, where to find tests,
etc.

``moz.build`` files are actually Python scripts. However, their
execution is governed by special rules. This is explained below.

moz.build Python Sandbox
========================

As mentioned above, ``moz.build`` files are Python scripts. However,
they are executed in a special Python *sandbox* that significantly
changes and limits the execution environment. The environment is so
different, it's doubtful most ``moz.build`` files would execute without
error if executed by a vanilla Python interpreter (e.g. ``python
moz.build``.

The following properties make execution of ``moz.build`` files special:

1. The execution environment exposes a limited subset of Python.
2. There is a special set of global symbols and an enforced naming
   convention of symbols.
3. Some symbols are inherited from previously-executed ``moz.build``
   files.

The limited subset of Python is actually an extremely limited subset.
Only a few symbols from ``__builtin__`` are exposed. These include
``True``, ``False``, ``None``, ``sorted``, ``int``, and ``set``. Global
functions like ``import``, ``print``, and ``open`` aren't available.
Without these, ``moz.build`` files can do very little. *This is by design*.

The execution sandbox treats all ``UPPERCASE`` variables specially. Any
``UPPERCASE`` variable must be known to the sandbox before the script
executes. Any attempt to read or write to an unknown ``UPPERCASE``
variable will result in an exception being raised. Furthermore, the
types of all ``UPPERCASE`` variables is strictly enforced. Attempts to
assign an incompatible type to an ``UPPERCASE`` variable will result in
an exception being raised.

The strictness of behavior with ``UPPERCASE`` variables is a very
intentional design decision. By ensuring strict behavior, any operation
involving an ``UPPERCASE`` variable is guaranteed to have well-defined
side-effects. Previously, when the build configuration was defined in
``Makefiles``, assignments to variables that did nothing would go
unnoticed. ``moz.build`` files fix this problem by eliminating the
potential for false promises.

After a ``moz.build`` file has completed execution, only the
``UPPERCASE`` variables are used to retrieve state.

The set of variables and functions available to the Python sandbox is
defined by the :py:mod:`mozbuild.frontend.context` module. The
data structures in this module are consumed by the
:py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct
the sandbox. There are tests to ensure that the set of symbols exposed
to an empty sandbox are all defined in the ``context`` module.
This module also contains documentation for each symbol, so nothing can
sneak into the sandbox without being explicitly defined and documented.

Reading and Traversing moz.build Files
======================================

The process for reading ``moz.build`` files roughly consists of:

1. Start at the root ``moz.build`` (``<topsrcdir>/moz.build``).
2. Evaluate the ``moz.build`` file in a new sandbox.
3. Emit the main *context* and any *sub-contexts* from the executed
   sandbox.
4. Extract a set of ``moz.build`` files to execute next.
5. For each additional ``moz.build`` file, goto #2 and repeat until all
   referenced files have executed.

From the perspective of the consumer, the output of reading is a stream
of :py:class:`mozbuild.frontend.reader.context.Context` instances. Each
``Context`` defines a particular aspect of data. Consumers iterate over
these objects and do something with the data inside. Each object is
essentially a dictionary of all the ``UPPERCASE`` variables populated
during its execution.

.. note::

   Historically, there was only one ``context`` per ``moz.build`` file.
   As the number of things tracked by ``moz.build`` files grew and more
   and more complex processing was desired, it was necessary to split these
   contexts into multiple logical parts. It is now common to emit
   multiple contexts per ``moz.build`` file.

Build System Reading Mode
-------------------------

The traditional mode of evaluation of ``moz.build`` files is what's
called *build system traversal mode.* In this mode, the ``CONFIG``
variable in each ``moz.build`` sandbox is populated from data coming
from ``config.status``, which is produced by ``configure``.

During evaluation, ``moz.build`` files often make decisions conditional
on the state of the build configuration. e.g. *only compile foo.cpp if
feature X is enabled*.

In this mode, traversal of ``moz.build`` files is governed by variables
like ``DIRS`` and ``TEST_DIRS``. For example, to execute a child
directory, ``foo``, you would add ``DIRS += ['foo']`` to a ``moz.build``
file and ``foo/moz.build`` would be evaluated.

.. _mozbuild_fs_reading_mode:

Filesystem Reading Mode
-----------------------

There is an alternative reading mode that doesn't involve the build
system and doesn't use ``DIRS`` variables to control traversal into
child directories. This mode is called *filesystem reading mode*.

In this reading mode, the ``CONFIG`` variable is a dummy, mostly empty
object. Accessing all but a few special variables will return an empty
value. This means that nearly all ``if CONFIG['FOO']:`` branches will
not be taken.

Instead of using content from within the evaluated ``moz.build``
file to drive traversal into subsequent ``moz.build`` files, the set
of files to evaluate is controlled by the thing doing the reading.

A single ``moz.build`` file is not guaranteed to be executable in
isolation. Instead, we must evaluate all *parent* ``moz.build`` files
first. For example, in order to evaluate ``/foo/moz.build``, one must
execute ``/moz.build`` and have its state influence the execution of
``/foo/moz.build``.

Filesystem reading mode is utilized to power the
:ref:`mozbuild_files_metadata` feature.

Technical Details
-----------------

The code for reading ``moz.build`` files lives in
:py:mod:`mozbuild.frontend.reader`. The Python sandboxes evaluation results
(:py:class:`mozbuild.frontend.context.Context`) are passed into
:py:mod:`mozbuild.frontend.emitter`, which converts them to classes defined
in :py:mod:`mozbuild.frontend.data`. Each class in this module defines a
domain-specific component of tree metdata. e.g. there will be separate
classes that represent a JavaScript file vs a compiled C++ file or test
manifests. This means downstream consumers of this data can filter on class
types to only consume what they are interested in.

There is no well-defined mapping between ``moz.build`` file instances
and the number of :py:mod:`mozbuild.frontend.data` classes derived from
each. Depending on the content of the ``moz.build`` file, there may be 1
object derived or 100.

The purpose of the ``emitter`` layer between low-level sandbox execution
and metadata representation is to facilitate a unified normalization and
verification step. There are multiple downstream consumers of the
``moz.build``-derived data and many will perform the same actions. This
logic can be complicated, so we have a component dedicated to it.

:py:class:`mozbuild.frontend.reader.BuildReader`` and
:py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a
stream-based API courtesy of generators. When you hook them up properly,
the :py:mod:`mozbuild.frontend.data` classes are emitted before all
``moz.build`` files have been read. This means that downstream errors
are raised soon after sandbox execution.

Lots of the code for evaluating Python sandboxes is applicable to
non-Mozilla systems. In theory, it could be extracted into a standalone
and generic package. However, until there is a need, there will
likely be some tightly coupled bits.