Using pytest-order with other pytest plugins¶
Relationship with pytest-dependency¶
The pytest-dependency
plugin also manages dependencies between tests (skips tests that depend
on skipped or failed tests), but doesn’t do any ordering. If you
want to execute the tests in a specific order to each other, you can use
pytest-order
. If you want to skip or xfail tests dependent on other
tests you can use pytest-dependency
. If you want to have both behaviors
combined, you can use both plugins together with the
option --order-dependencies. As mentioned before, only
static dependency
markers are considered for ordering.
Usage with other ordering plugins¶
There is a number of other pytest plugins that change the order in which tests
are executed. Most of these plugins only reorder tests if given some command
line options or in the presence of specific markers (as does pytest-order
).
These plugins will not have any effect on pytest-order
if not actively
used. A few plugins always do reordering, most notably pytest-randomly
.
pytest-randomly¶
This plugin executes tests in a random order to avoid unknown test
dependencies. The plugin is effective if installed and not actively disabled.
pytest-order
should still work correctly, because it is executed after
pytest-randomly
(except if you use the option
--indulgent-ordering). The marked tests will be ordered correctly, but
the order of the unordered tests will change unpredictably. For example, if
you run the following tests:
import pytest
@pytest.mark.order(1)
def test_second():
assert True
def test_third():
assert True
def test_fourth():
assert True
@pytest.mark.order(0)
def test_first():
assert True
the output could either be:
test_randomly.py::test_first PASSED
test_randomly.py::test_second PASSED
test_randomly.py::test_third PASSED
test_randomly.py::test_fourth PASSED
or:
test_randomly.py::test_first PASSED
test_randomly.py::test_second PASSED
test_randomly.py::test_fourth PASSED
test_randomly.py::test_third PASSED
The same is true for relative ordering. The tests will be correctly ordered before and after the tests as configured, but all other tests will be in an arbitrary order. If you want the tests to execute in a random order except the ones that you know depend on each other, this can be a good strategy to prevent test dependencies.
If you don’t want the installed pytest-randomly
be effective in a test
run, you can exclude it explicitly on the command line:
python -m pytest -p no:randomly
pytest-reverse¶
pytest-reverse is another plugin by the same author to find test
dependencies by running tests in the reverse order. Other than
pytest-randomly
, it is only effective if the tests are called with the
command line option --reverse
.
As with pytest-randomly
, you can use this plugin combined with
pytest-ordering
to make sure that all known test dependencies are
handled, and possible unknown dependencies are found.
pytest-random-order¶
pytest-random-order is very similar to pytest-randomly
, except that the
tests are only reordered if the option --random-order
is given. Except
from that, what was mentioned for pytest-randomly
is also true for
this plugin.
pytest-ordering¶
As mentioned, pytest-ordering can coexist with pytest-order
due to
the different marker names, but using markers of both plugins in the same test
run is not recommended. One plugin may partially revert the effects of the
other plugin in unpredictable ways, because the order in which they are
executed is not deterministic. The same is true for other plugins that define
the test order and are run last.
pytest-depends¶
pytest-depends has a goal somewhat similar to pytest-dependency with additional ordering, but due to a known issue it always reorders the tests. If you have installed this plugin, the order of the unordered tests will change even without using it actively.
pytest-find-dependencies¶
This is a small plugin by the same author as pytest-order
that tries to
find specific dependencies between tests by running a subset of them repeatedly
in reverse order until the dependencies are found. This plugin would run the
tests as ordered by any ordering plugin in the first run, but reverse the
test order in the second run, so that already ordered tests are not run in
the correct order. You have the possibility to exclude ordered tests
completely by using the --markers-to-ignore
option:
python -m pytest --find-dependencies --markers-to-ignore=order
Usage with pytest-xdist¶
The pytest-xdist plugin
schedules tests unordered, and the order configured by pytest-order
will normally not be preserved. But if we use the --dist=loadfile
option, provided by xdist
, all tests from one file will be run in the
same thread. So, to make the two plugins work together, we have to put
each group of dependent tests in one file, and call pytest with
--dist=loadfile
(this is taken from
this issue).