> Recently, I started to experiment "./python -m test [options] -F
-j100" to attempt to reproduce some tricky race conditions: -j100
spawns 100 worker processes in parallel and -F stands for --forever
(run tests in loop and stop at the first failure).

Interesting, even as someone who has recently worked within
asyncio a decent amount recently, I was entirely unaware of
this method of producing race conditions in the tests. If continued
experimentation shows this to be a reliable means of reproducing
various race conditions, it might be worth adding this to
https://devguide.python.org/runtests/.

Thanks for sharing!


On Mon, Oct 21, 2019 at 7:08 AM Victor Stinner <vstinner@python.org> wrote:
Hi,

Right now, there are 14 open issues with "test_asyncio" in the title.
Many test_asyncio tests have race conditions. I'm trying to fix them
one by one, but it takes time, and then new tests are added with new
race condition :-( For example, the following new test is failing
randomly on Windows:

"Windows: test_asyncio: test_huge_content_recvinto() fails randomly
with ProactorEventLoop" is failing randomly since 6 months:
https://bugs.python.org/issue36732

test_asyncio uses more and more functional tests which is a good
thing. In the early days of asyncio, most tests mocked more than half
of asyncio to really be "unit test". But at the end, the test tested
more mocks than asyncio... The problem of functional tests is that
it's hard to design them properly to avoid all race conditions,
especially when you consider multiplatform (Windows, macOS, Linux,
FreeBSD, etc.).

It would help me if someone could try to investigate these issues,
provide a reliable way to reproduce them, and propose a fix. (Simply
saying that you can reproduce the test and that you would like to work
on an issue doesn't really help, sorry.)

Recently, I started to experiment "./python -m test [options] -F
-j100" to attempt to reproduce some tricky race conditions: -j100
spawns 100 worker processes in parallel and -F stands for --forever
(run tests in loop and stop at the first failure). I was surprised
that my Fedora 30 didn't burn in blame. In fact, the GNOME desktop
remains responsible even with a system load higher than 100. The Linux
kernel (5.2) is impressive! Under such high system load (my laptop has
8 logical CPUs), race conditions are way more likely.

The problem of test_asyncio is that it's made of 2160 tests, see:

   ./python -m test test_asyncio --list-cases

You may want to only run a single test case (class) or even a single
test method: see --match option which can be used multiple times to
only run selected test classes or selected test methods. See also
--matchfile which is similar but uses a file. Example:

$ ./python -m test test_asyncio --list-cases > cases
# edit cases
$ ./python -m test test_asyncio --matchfile=cases

test_asyncio is one of the most unstable test: I'm getting more and
more buildbot-status emails about test_asyncio... likely because we
fixed most of the other race conditions which is a good thing ;-)

Some issues look to be specific to Windows, but it should be possible
to reproduce most issues on Linux as Linux. Sometimes, it's just that
some specific Windows buildbot workers are slower than other buildbot
workers.

Good luck ;-)

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R7X6NKGEOKWD3PBWIL2LPZWZ6MMRANN5/
Code of Conduct: http://python.org/psf/codeofconduct/