DXR is a code search and navigation tool aimed at making sense of large projects. It supports full-text and regex searches as well as structural queries.

Mercurial (b6d82b1a6b02)

VCS Links

Line Code
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206
Testing
=======

We verify and test Marionette in a couple of different ways, using
a combination of unit tests and functional tests.  There are three
distinct components that we test:

  - the Marionette **server**, using a combination of xpcshell
    unit tests and functional tests written in Python spread across
    Marionette- and WPT tests;

  - the Python **client** is tested with the same body of functional
    Marionette tests;

  - and the **harness** that backs the Marionette, or `Mn` job on
    try, tests is verified using separate mock-styled unit tests.

All these tests can be run by using [mach].

[mach]: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach

xpcshell unit tests
-------------------

Marionette has a set of [xpcshell] unit tests located in
_testing/marionette/test/unit.  These can be run this way:

	% ./mach test testing/marionette/test/unit

Because tests are run in parallel and xpcshell itself is quite
chatty, it can sometimes be useful to run the tests sequentially:

	% ./mach test --sequential testing/marionette/test/unit/test_error.js

These unit tests run as part of the `X` jobs on Treeherder.

[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests


Marionette functional tests
---------------------------

We also have a set of [functional tests] that make use of the Marionette
Python client.  These start a Firefox process and tests the Marionette
protocol input and output, and will appear as `Mn` on Treeherder.
The following command will run all tests locally:

	% ./mach marionette-test

But you can also run individual tests:

	% ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py

In case you want to run the tests with another binary like [Firefox Nightly]:

	% ./mach marionette-test --binary /path/to/nightly/firefox TEST

When working on Marionette it is often useful to surface the stdout
from Gecko, which can be achived using the `--gecko-log` option.
See <Debugging.html> for usage instructions, but the gist is that
you can redirect all Gecko output to stdout:

    % ./mach marionette-test --gecko-log - TEST

Our functional integration tests pop up Firefox windows sporadically,
and a helpful tip is to suppress the window can be to use Firefox’
[headless mode]:

    % ./mach marionette-test -z TEST

`-z` is an alias for the `--headless` flag and equivalent to setting
the `MOZ_HEADLESS` output variable.  In addition to `MOZ_HEADLESS`
there is also `MOZ_HEADLESS_WIDTH` and `MOZ_HEADLESS_HEIGHT` for
controlling the dimensions of the no-op virtual display.  This is
similar to using Xvfb(1) which you may know from the X windowing system,
but has the additional benefit of also working on macOS and Windows.

[functional tests]: PythonTests.html
[Firefox Nightly]: https://nightly.mozilla.org/


### Android

Prerequisites:

*   You have [built Fennec](https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_for_Android_build).
*   You can run an Android [emulator](https://wiki.mozilla.org/Mobile/Fennec/Android/Testing#Running_tests_on_the_Android_emulator),
    which means you have the AVD you need.

When running tests on Fennec, you can have Marionette runner take care of
starting Fennec and an emulator, as shown below.

	% ./mach marionette-test --emulator --app fennec
    --avd-home /path/to/.mozbuild/android-device/avd
    --emulator-binary /path/to/.mozbuild/android-sdk/emulator/emulator
    --avd=mozemulator-x86

For Fennec tests, if the appropriate `emulator` command is in your `PATH`, you may omit the `--emulator-binary` argument.  See `./mach marionette-test -h`
for additional options.

Alternately, you can start an emulator yourself and have the Marionette runner
start Fennec for you:

    % ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828

To connect to an already-running Fennec in an already running emulator or on a device, you will need to enable Marionette manually by setting the browser preference
`marionette.enabled` set to true in the Fennec profile.

Make sure port 2828 is forwarded:

	% adb forward tcp:2828 tcp:2828

If Fennec is already started:

    % ./mach marionette-test --app='fennec' --address=127.0.0.1:2828

If Fennec is not already started on the emulator/device, add the `--emulator`
option. Marionette Test Runner will take care of forwarding the port and
starting Fennec with the correct prefs. (You may need to run
`adb forward --remove-all` to allow the runner to start.)

    % ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828 --disable-e10s
    --startup-timeout=300

If you need to troubleshoot the Marionette connection, the most basic check is
to start Fennec, make sure the `marionette.enabled` browser preference is
true and port 2828 is forwarded, then see if you get any response from
Marionette when you connect manually:

    % telnet 127.0.0.1:2828

You should see output like `{"applicationType":"gecko","marionetteProtocol":3}`

[headless mode]: https://developer.mozilla.org/en-US/Firefox/Headless_mode
[geckodriver]: /testing/geckodriver/geckodriver


WPT functional tests
--------------------

Marionette is also indirectly tested through [geckodriver] with WPT
(`Wd` on Treeherder).  To run them:

	% ./mach wpt testing/web-platform/tests/webdriver

WPT tests conformance to the [WebDriver] standard and uses
[geckodriver].  Together with the Marionette remote protocol in
Gecko, they make up Mozilla’s WebDriver implementation.

This command supports a `--webdriver-arg '-vv'` argument that
enables more detailed logging, as well as `--jsdebugger` for opening
the Browser Toolbox.

A particularly useful trick is to combine this with the [headless
mode] for Firefox we learned about earlier:

	% MOZ_HEADLESS=1 ./mach wpt --webdriver-arg '-vv' testing/web-platform/tests/webdriver

[WebDriver]: https://w3c.github.io/webdriver/webdriver-spec.html


Harness tests
-------------

The Marionette harness Python package has a set of mock-styled unit
tests that uses the [pytest] framework.  The following command will
run all tests:

	% ./mach python-test testing/marionette

To run a specific test specify the full path to the module:

	% ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py

[pytest]: https://docs.pytest.org/en/latest/


One-click loaners
-----------------

Additionally, for debugging hard-to-reproduce test failures in CI,
one-click loaners from <Taskcluster.html> can be particularly useful.


Out-of-tree testing
-------------------

All the above examples show tests running _in-tree_, with a local
checkout of _central_ and a local build of Firefox.  It is also
possibly to run the Marionette tests _without_ a local build and
with a downloaded test archive from <Taskcluster.html>.

If you want to run tests from a downloaded test archive, you will
need to download the `target.common.tests.tar.gz` artifact attached to
Treeherder [build jobs] `B` for your system.  Extract the archive
and set up the Python Marionette client and harness by executing
the following command in a virtual environment:

	% pip install -r config/marionette_requirements.txt

The tests can then be found under
_marionette/tests/testing/marionette/harness/marionette_harness/tests_
and can be executed with the command `marionette`.  It supports
the same options as described above for `mach`.

[build jobs]: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=build