Configuration¶
There are a few command line options that change the behavior of the
plugin. As with any pytest option, you can add the options to your
pytest.ini
if you want to have them applied to all tests automatically:
[pytest]
; always order tests with dependency markers
addopts = --order-dependencies
--order-scope
¶
By default, tests are ordered per session, e.g. across all modules in the test run. Sometimes, you want to order tests per module or per test class instead. Consider that you have a growing number of test modules that you want to run simultaneously, with tests ordered per module. Per default you would need to make sure that the order numbers increases globally, if you want to run the test modules consecutively and order the test per module.
If you use the option --order-scope=module
, there is no need for this.
You can enumerate your tests starting with 0 or 1 in each module, and the tests
will only be ordered inside each module. Using --order-scope=class
additionally considers test classes–each test class is considered
separately for ordering the tests. If a module has both test classes and
separate test functions, these test functions are handled separately from the
test classes. If a module has no test classes, the effect is the same as
if using --order-scope=module
.
For example consider two test modules:
tests/test_module1.py:
import pytest
@pytest.mark.order(2)
def test2():
pass
@pytest.mark.order(1)
def test1():
pass
tests/test_module2.py:
import pytest
@pytest.mark.order(2)
def test2():
pass
@pytest.mark.order(1)
def test1():
pass
Here is what you get using session and module-scoped sorting:
$ pytest tests -vv
============================= test session starts ==============================
...
tests/test_module1.py:9: test1 PASSED
tests/test_module2.py:9: test1 PASSED
tests/test_module1.py:5: test2 PASSED
tests/test_module2.py:5: test2 PASSED
$ pytest tests -vv --order-scope=module
============================= test session starts ==============================
...
tests/test_module1.py:9: test1 PASSED
tests/test_module1.py:5: test2 PASSED
tests/test_module2.py:9: test1 PASSED
tests/test_module2.py:5: test2 PASSED
--order-scope-level
¶
This is an alternative option to define the order scope. It defines the
directory level which is used as the order scope, counting from the root
directory. The resulting scope is between the session and module
scopes defined via --order-scope
, where --order-scope-level=0
is the
same as session scope, while setting the level to the number of test
directory levels would result in module scope.
Consider the following directory structure:
order_scope_level
feature1
__init__.py
test_a.py
test_b.py
feature2
__init__.py
test_a.py
test_b.py
with the test contents:
test_a.py:
import pytest
@pytest.mark.order(4)
def test_four():
pass
@pytest.mark.order(3)
def test_three():
pass
test_b.py:
import pytest
@pytest.mark.order(2)
def test_two():
pass
@pytest.mark.order(1)
def test_one():
pass
The idea here is to test each feature separately, while ordering the tests across the test modules for each feature.
If we use session scope, we get:
$ pytest -v order_scope_level
============================= test session starts ==============================
...
order_scope_level/feature1/test_a.py::test_one PASSED
order_scope_level/feature2/test_a.py::test_one PASSED
order_scope_level/feature1/test_a.py::test_two PASSED
order_scope_level/feature2/test_a.py::test_two PASSED
order_scope_level/feature1/test_b.py::test_three PASSED
order_scope_level/feature2/test_b.py::test_three PASSED
order_scope_level/feature1/test_b.py::test_four PASSED
order_scope_level/feature2/test_b.py::test_four PASSED
which mixes the features.
Using module scope instead separates the features, but does not order the modules as wanted:
$ pytest -v --order-scope=module order_scope_level
============================= test session starts ==============================
...
order_scope_level/feature1/test_a.py::test_three PASSED
order_scope_level/feature1/test_a.py::test_four PASSED
order_scope_level/feature1/test_b.py::test_one PASSED
order_scope_level/feature1/test_b.py::test_two PASSED
order_scope_level/feature2/test_a.py::test_three PASSED
order_scope_level/feature2/test_a.py::test_four PASSED
order_scope_level/feature2/test_b.py::test_one PASSED
order_scope_level/feature2/test_b.py::test_two PASSED
To get the wanted behavior, we can use --order-scope-level=2
, which keeps
the first two directory levels:
$ pytest tests -v --order-scope-level=2 order_scope_level
============================= test session starts ==============================
...
order_scope_level/feature1/test_b.py::test_one PASSED
order_scope_level/feature1/test_b.py::test_two PASSED
order_scope_level/feature1/test_a.py::test_three PASSED
order_scope_level/feature1/test_a.py::test_four PASSED
order_scope_level/feature2/test_b.py::test_one PASSED
order_scope_level/feature2/test_b.py::test_two PASSED
order_scope_level/feature2/test_a.py::test_three PASSED
order_scope_level/feature2/test_a.py::test_four PASSED
Note that using a level of 0 or 1 would cause the same result as session scope in this example, and any level greater than 2 would emulate module scope.
--order-group-scope
¶
This option is also related to the order scope. It defines the scope inside which tests may be reordered. Consider you have several test modules which you want to order, but you don’t want to mix the tests of several modules because the module setup is costly. In this case you can set the group order scope to “module”, meaning that first the tests are ordered inside each module (the same as with the module order scope), but afterwards the modules themselves are sorted without changing the order inside each module.
Consider these two test modules:
tests/test_module1.py:
import pytest
@pytest.mark.order(2)
def test1():
pass
def test2():
pass
tests/test_module2.py:
import pytest
@pytest.mark.order(1)
def test1():
pass
def test2():
pass
Here is what you get using different scopes:
$ pytest tests -vv
============================= test session starts ==============================
...
tests/test_module2.py:9: test1 PASSED
tests/test_module1.py:9: test1 PASSED
tests/test_module1.py:5: test2 PASSED
tests/test_module2.py:5: test2 PASSED
$ pytest tests -vv --order-scope=module
============================= test session starts ==============================
...
tests/test_module1.py:9: test1 PASSED
tests/test_module1.py:5: test2 PASSED
tests/test_module2.py:9: test1 PASSED
tests/test_module2.py:5: test2 PASSED
$ pytest tests -vv --order-group-scope=module
============================= test session starts ==============================
...
tests/test_module2.py:9: test1 PASSED
tests/test_module2.py:5: test2 PASSED
tests/test_module1.py:9: test1 PASSED
tests/test_module1.py:5: test2 PASSED
The ordering of the module groups is done based on the lowest non-negative order number present in the module (e.g. the order number of the first test). If only negative numbers are present, the highest negative number (e.g. the number of the last test) is used, and these modules will be ordered at the end. Modules without order numbers will be sorted between modules with a non-negative order number and modules with a negative order number, the same way tests are sorted inside a module.
The group order scope defaults to the order scope. In this case the tests are ordered the same way as without the group order scope. The setting takes effect only if the scope is less than the order scope, e.g. there are three possibilities:
order scope “session”, order group scope “module” - this is shown in the example above: first tests in each module are ordered, afterwards the modules
order scope “module”, order group scope “class” - first orders tests inside each class, then the classes inside each module
order scope “session”, order group scope “class” - first orders tests inside each class, then the classes inside each module, and finally the modules relatively to each other
This option will also work with relative markers, and with dependency markers if using the --order-dependencies option.
Here is a similar example using relative markers:
tests/test_module1.py:
import pytest
@pytest.mark.order(after="test_module2.py::test1")
def test1():
pass
def test2():
pass
tests/test_module2.py:
import pytest
def test1():
pass
@pytest.mark.order(before="test1")
def test2():
pass
Here is what you get using different scopes:
$ pytest tests -vv
============================= test session starts ==============================
...
tests/test_module1.py:5: test2 PASSED
tests/test_module2.py:9: test1 PASSED
tests/test_module2.py:5: test2 PASSED
tests/test_module1.py:9: test1 PASSED
$ pytest tests -vv --order-group-scope=module
============================= test session starts ==============================
...
tests/test_module2.py:9: test1 PASSED
tests/test_module2.py:5: test2 PASSED
tests/test_module1.py:9: test1 PASSED
tests/test_module1.py:5: test2 PASSED
You can see that in the second run the second test module is run before the first because of the dependency, but the tests inside each module remain in the same order as before. Note that using module scope as in the example above doesn’t make sense here due to the dependencies between modules.
This will also work with dependency markers if using the --order-dependencies option.
Note
This option will not work together well with the sparse ordering option.
--order-dependencies
¶
This defines the behavior if the pytest-dependency plugin is used.
By default, dependency
marks are only considered if they coexist with an
order
mark. In this case it is checked if the ordering would break the
dependency, and is ignored if this is the case. Consider the following:
import pytest
def test_a():
assert True
@pytest.mark.dependency(depends=["test_a"])
@pytest.mark.order("first")
def test_b():
assert True
In this case, the ordering would break the dependency and is therefore ignored. This behavior is independent of the option. Now consider the following tests:
import pytest
@pytest.mark.dependency(depends=["test_b"])
def test_a():
assert True
@pytest.mark.dependency
def test_b():
assert True
By default, test_a
is not run, because it depends on test_b
, which
is only run after test_b
:
$ pytest tests -vv
============================= test session starts ==============================
...
test_dep.py::test_a SKIPPED
test_dep.py::test_b PASSED
If you use --order-dependencies
, this will change–the tests will now be
reordered according to the dependency and both run:
$ pytest tests -vv --order-dependencies
============================= test session starts ==============================
...
test_dep.py::test_b PASSED
test_dep.py::test_a PASSED
Note that a similar feature may be added to pytest-dependency
-
if this is done, this option will not be needed, but for the time being you
can use both plugins together to get this behavior.
Note that pytest-order
does not replace pytest-dependency
–it just
adds ordering to the existing functionality if needed.
Note
pytest-dependency
also has the possibility to add dependencies at
runtime using pytest_dependency.depends
. These dependencies cannot be
detected at collection time and therefore are not included in ordering.
The same is true for the dynamic compilation of marked parameters.
--indulgent-ordering
¶
You may sometimes find that you want to suggest an ordering of tests, while
allowing it to be overridden for good reason. For example, if you run your test
suite in parallel and have a number of tests which are particularly slow, it
might be desirable to start those tests running first, in order to optimize
your completion time. You can use the pytest-order
plugin to inform pytest
of this.
Now suppose you also want to prioritize tests which failed during the
previous run, by using the --failed-first
option. By default,
pytest-order will override the --failed-first
order, but by adding the
--indulgent-ordering
option, you can ask pytest to run the sort from
pytest-order before the sort from --failed-first
, allowing the failed
tests to be sorted to the front (note that in pytest versions from 6.0 on,
this seems not to be needed anymore, at least in this specific case).
--sparse-ordering
¶
Ordering tests by ordinals where some numbers are missing by default behaves the same as if the the ordinals are consecutive. For example, these tests:
import pytest
@pytest.mark.order(3)
def test_two():
assert True
def test_three():
assert True
def test_four():
assert True
@pytest.mark.order(1)
def test_one():
assert True
are executed in the same order as:
import pytest
@pytest.mark.order(1)
def test_two():
assert True
def test_three():
assert True
def test_four():
assert True
@pytest.mark.order(0)
def test_one():
assert True
e.g. you get:
$ pytest tests -vv
============================= test session starts ==============================
...
test_module.py:13: test_one PASSED
test_module.py:3: test_two PASSED
test_module.py:6: test_three PASSED
test_module.py:9: test_four PASSED
The gaps between numbers, and the fact that the starting number is not 0,
are ignored. This is consistent with the current behavior of
pytest-ordering
.
If you use the --sparse-ordering
option, the behavior will change:
$ pytest tests -vv --sparse-ordering
============================= test session starts ==============================
...
test_module.py:6: test_three PASSED
test_module.py:13: test_one PASSED
test_module.py:9: test_four PASSED
test_module.py:3: test_two PASSED
Now all missing numbers (starting with 0) are filled with unordered tests, as
long as unordered tests are left. In the shown example, test_three
is filled in for the missing number 0, and test_four
is filled in for the
missing number 2. This will also work for tests with negative order numbers
(or the respective names). The missing ordinals are filled with unordered
tests first from the start, then from the end if there are negative numbers,
and the rest will be in between (e.g. between positive and negative numbers),
as it is without this option.