logging from time critical tasks -- QueueListener().stop() takes the whole CPU

Thomas Jollans tjol at tjol.eu
Mon Jul 16 02:33:39 EDT 2018

On 16/07/18 08:24, Gerlando Falauto wrote:
> On Monday, July 16, 2018 at 8:13:46 AM UTC+2, Thomas Jollans wrote:
>> On 16/07/18 07:39, Gerlando Falauto wrote:
>>> On Monday, July 16, 2018 at 6:56:19 AM UTC+2, dieter wrote:
>>>>> ...
>>>>> Why is the main thread taking up so much CPU?
>>>>> I believe at this point listener.stop() should only be waiting for the helper thread to terminate, which I reckon would be implemented by waiting on a semaphore or something (i.e. iowait i.e. 0% CPU).
>>>> Maybe, you look at the source code of "listener.stop"?
>>> I did, forgot to mention that. Culprit is self._thread.join().
>>> Which is where it waits for the internal thread to terminate,
>>> which I would've expected to wait on a lock or semaphore (pthread_join()?)
>>> So there's something I'm totally missing here, which has more to do
>>> with queues and threads in general than it has with logging.
>>> Any ideas?
>> I have no idea what's really going on there, but a quick look through
>> the source reveals that the actual waiting in Thread.join() is done by
>> sem_wait().
>> via
>> https://github.com/python/cpython/blob/13ff24582c99dfb439b1af7295b401415e7eb05b/Python/thread_pthread.h#L304
>> via
>> https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L45
> Hmm... do you think it's possible it's really getting interrupted the whole time, effectively turning the lock into a sort of spinlock?
> Any idea how to confirm that without recompiling the whole code?

Profiling, maybe? Or you might be able to find out if it's being
interrupted by listening for signals yourself (though I'm not sure which

More information about the Python-list mailing list