Recently, I started to experiment "./python -m test [options] -F -j100" to attempt to reproduce some tricky race conditions: -j100 spawns 100 worker processes in parallel and -F stands for --forever (run tests in loop and stop at the first failure).

Interesting, even as someone who has recently worked within asyncio a decent amount recently, I was entirely unaware of this method of producing race conditions in the tests. If continued experimentation shows this to be a reliable means of reproducing various race conditions, it might be worth adding this to https://devguide.python.org/runtests/. Thanks for sharing! On Mon, Oct 21, 2019 at 7:08 AM Victor Stinner <vstinner@python.org> wrote:

Hi,

Right now, there are 14 open issues with "test_asyncio" in the title. Many test_asyncio tests have race conditions. I'm trying to fix them one by one, but it takes time, and then new tests are added with new race condition :-( For example, the following new test is failing randomly on Windows:

"Windows: test_asyncio: test_huge_content_recvinto() fails randomly with ProactorEventLoop" is failing randomly since 6 months: https://bugs.python.org/issue36732

test_asyncio uses more and more functional tests which is a good thing. In the early days of asyncio, most tests mocked more than half of asyncio to really be "unit test". But at the end, the test tested more mocks than asyncio... The problem of functional tests is that it's hard to design them properly to avoid all race conditions, especially when you consider multiplatform (Windows, macOS, Linux, FreeBSD, etc.).

It would help me if someone could try to investigate these issues, provide a reliable way to reproduce them, and propose a fix. (Simply saying that you can reproduce the test and that you would like to work on an issue doesn't really help, sorry.)

Recently, I started to experiment "./python -m test [options] -F -j100" to attempt to reproduce some tricky race conditions: -j100 spawns 100 worker processes in parallel and -F stands for --forever (run tests in loop and stop at the first failure). I was surprised that my Fedora 30 didn't burn in blame. In fact, the GNOME desktop remains responsible even with a system load higher than 100. The Linux kernel (5.2) is impressive! Under such high system load (my laptop has 8 logical CPUs), race conditions are way more likely.

The problem of test_asyncio is that it's made of 2160 tests, see:

./python -m test test_asyncio --list-cases

You may want to only run a single test case (class) or even a single test method: see --match option which can be used multiple times to only run selected test classes or selected test methods. See also --matchfile which is similar but uses a file. Example:

$ ./python -m test test_asyncio --list-cases > cases # edit cases $ ./python -m test test_asyncio --matchfile=cases

test_asyncio is one of the most unstable test: I'm getting more and more buildbot-status emails about test_asyncio... likely because we fixed most of the other race conditions which is a good thing ;-)

Some issues look to be specific to Windows, but it should be possible to reproduce most issues on Linux as Linux. Sometimes, it's just that some specific Windows buildbot workers are slower than other buildbot workers.

Good luck ;-)

Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R7X6NKGE... Code of Conduct: http://python.org/psf/codeofconduct/