[issue7946] Convoy effect with I/O bound threads and New GIL

[Nir Aides]

Nir Aides <nir at winpdb.org> added the comment:

> Ouch. CLOCK_THREAD_CPUTIME_ID is not a required part of the standard. Only CLOCK_REALTIME is guaranteed to exist.

Right, however the man page at kernel.org says the following on CLOCK_THREAD_CPUTIME_ID: 
"Sufficiently recent versions of glibc and the Linux kernel support the following clocks"
http://www.kernel.org/doc/man-pages/online/pages/man2/clock_getres.2.html

The same statement shows up as early as 2003:
http://www.tin.org/bin/man.cgi?section=3&topic=clock_gettime

However, if this is indeed a problem on some systems (none Linux?), then a fall back could be attempted for them. 

There could also be a problem on systems where the counter exists but has low resolution 10ms+

What platforms do you think this could be a problem on?

> By the way, it's not obvious cpued tests anything meaningful. I understand the bias you are trying to avoid but constructing artificial test cases is not very useful, because we are playing with heuristics and it's always possible to defeat some expectations. That's why benchmarks should try to model/represent real-world situations.

I came up with cpued.py after reading the patches in an attempt to understand how they behave. In this case one thread is pure Python while the other occasionally releases the GIL, both CPU bound. I don't claim this is a real-world situation. However, it is a case in which bfs.patch behaves as expected.

> I've tried ccbench with your patch and there's a clear regression in latency numbers.

Please specify system and test details so I can try to look into it. On my system ccbench behaves as expected:

$ ~/build/python/bfs/python ccbench.py
== CPython 3.2a0.0 (py3k) ==
== x86_64 Linux on '' ==

--- Throughput ---

Pi calculation (Python)

threads=1: 1252 iterations/s.
threads=2: 1199 ( 95 %)
threads=3: 1178 ( 94 %)
threads=4: 1173 ( 93 %)

regular expression (C)

threads=1: 491 iterations/s.
threads=2: 478 ( 97 %)
threads=3: 472 ( 96 %)
threads=4: 477 ( 97 %)

SHA1 hashing (C)

threads=1: 2239 iterations/s.
threads=2: 3719 ( 166 %)
threads=3: 3772 ( 168 %)
threads=4: 3464 ( 154 %)

--- Latency ---

Background CPU task: Pi calculation (Python)

CPU threads=0: 0 ms. (std dev: 1 ms.)
CPU threads=1: 0 ms. (std dev: 1 ms.)
CPU threads=2: 0 ms. (std dev: 1 ms.)
CPU threads=3: 0 ms. (std dev: 1 ms.)
CPU threads=4: 0 ms. (std dev: 1 ms.)

Background CPU task: regular expression (C)

CPU threads=0: 0 ms. (std dev: 0 ms.)
CPU threads=1: 6 ms. (std dev: 0 ms.)
CPU threads=2: 2 ms. (std dev: 2 ms.)
CPU threads=3: 1 ms. (std dev: 0 ms.)
CPU threads=4: 5 ms. (std dev: 7 ms.)

Background CPU task: SHA1 hashing (C)

CPU threads=0: 0 ms. (std dev: 1 ms.)
CPU threads=1: 0 ms. (std dev: 1 ms.)
CPU threads=2: 0 ms. (std dev: 1 ms.)
CPU threads=3: 1 ms. (std dev: 1 ms.)
CPU threads=4: 1 ms. (std dev: 0 ms.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7946>
_______________________________________