[Pytest-commit] Issue #545: _pytest/mark.py pytest_collection_modifyitems orders results differently across threads (hpk42/pytest)

[Patrick Putnam]

New issue 545: _pytest/mark.py pytest_collection_modifyitems orders results differently across threads
https://bitbucket.org/hpk42/pytest/issue/545/_pytest-markpy

Patrick Putnam:

I had an issue running marked tests in multi-threaded fashion where each thread would seem to get the tests in a different order once the marks had been applied. This would cause report_collection_diff in xdist/dsession.py to report errors like:

INTERNALERROR> AssertionError: Different tests were collected between gw0 and gw2. The difference is:
INTERNALERROR> --- gw0
INTERNALERROR>
INTERNALERROR> +++ gw2
INTERNALERROR>
INTERNALERROR> @@ -1,3 +1,3 @@
INTERNALERROR>
INTERNALERROR> +scripts/Departments.py::TestMain::()::test_select_department_palette
INTERNALERROR>  scripts/Header.py::TestMain::()::test_login
INTERNALERROR>  scripts/Header.py::TestMain::()::test_basic_search
INTERNALERROR> -scripts/Departments.py::TestMain::()::test_select_department_palette

(Same test, just different location).

I managed to fix this with a one-line change in mark.py:

 64     remaining = []
 65     deselected = []
>66     for colitem in sorted(items): # sorted items preserves ordering across threads
 67         if keywordexpr and not matchkeyword(colitem, keywordexpr):
 68             deselected.append(colitem)

I'm not sure if I'm doing something wrong somewhere, or if I've found an actual bug, but I've been running with this fix for a week now and everything seems to be working well. Should I submit this as a pull request, or am I doing something incorrect somewhere, and if so how what's the best way to figure that out?