[issue20910] Make sleep configurable in tests

[STINNER Victor]

New submission from STINNER Victor:

Hi,

I'm trying to make Python test suite reliable, but it's hard because the speed of our farm of buildbots varies a lot. The whole test suite takes less than 10 minutes on some buildbots (or event less than 5 minutes), while it took 10 hours on other buildbots.

Some tests use time.sleep() as a basic synchronization primitive, because other synchronization primitive cannot be used. For example, lock_tests.py tests locks and so cannot use lock in its tests. The problem is that slow buildbots require long sleep. It makes the test suite slow, whereas fast buildbots could use short lseep.

Some tests make sure that an operation takes less than N seconds. It's very hard to choose a N value for all buildbots. To make tests pass on all buildbots, N can be configured for the slowest buildbots.

I propose to make "time" configurable in the test suite:

- add support.TEST_SLEEP constant: it can be between 50 ms and 1 sec, or longer for very slow buildbots
- add support.TEST_SHORT_SLEEP constant: shorter than TEST_SLEEP, it is usually used to yield the control to another thread or another process and it can be very short (1 ms)
- add a new check_time_delta() functions to check time deltas
- add a new --test-sleep command line option to regrtest.py (python -m test)

check_time_delta() uses the resolution of the clock to tolerate a small difference. This is important on Windows where the resolution is usually 15.6 ms whereas tests use timings close to 100 ms.

Seen values in tests:

- TEST_SLEEP: sleep between 100 ms and 3 sec
- TEST_SHORT_SLEEP: 1 ms or 10 ms
- check_time_delta(): tolerate between 500 and 800 ms on slow buildbots

I chose TEST_SHORT_SLEEP for time < 100 ms, and TEST_SLEEP for the other sleep.

Some issues related to slow buildbots:

- #11185
- #20101
- #20336

The goal of my changes is to run the test suite as fast as possible, so to reduce the sleep. My plan:

- keep current timings for buildbots, but use very short timings by default
- configure slow buildbots to use longer sleep
- use short sleep for the default buildbot configuration

Buildbot config of my patch:

- TEST_SLEEP: 500 ms
- TEST_SHORT_SLEEP: 10 ms
- check_time_delta().slow_buildbot: 1 sec

Default config of my patch:

- TEST_SLEEP: 50 ms
- TEST_SHORT_SLEEP: 1 ms
- check_time_delta().slow_buildbot: 0 ms

The patch doesn't change timings in tests executed in subprocesses, because test.regrtest only modifies test.support variables in the current process. It can be improved later.

----------
files: test_sleep.patch
keywords: patch
messages: 213395
nosy: haypo
priority: normal
severity: normal
status: open
title: Make sleep configurable in tests
versions: Python 3.4
Added file: http://bugs.python.org/file34398/test_sleep.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20910>
_______________________________________