[Python-ideas] Add an introspection API to Executor

[Guido van Rossum]

Doesn't queue.Queue also have methods qsize(), empty() and full()? We could
easily wrap those. There's always the caveat that the numbers may be out of
date as soon as you print them.


On Mon, Aug 25, 2014 at 6:44 AM, Dan O'Reilly <oreilldf at gmail.com> wrote:

> Adding active/idle/total worker counts for both ThreadPoolExecutor and
> ProcessPoolExecutor is pretty straightforward; I just threw a patch
> together for both in 30 minutes or so. However, I don't think its possible
> to inspect the contents of a ProcessPoolExecutor's queue without actually
> consuming items from it. While it *is* possible with ThreadPoolExecutor, I
> don't think we should expose it - the queue.Queue() implementation
> ThreadPoolExecutor relies on doesn't have a public API for inspecting its
> contents, so ThreadPoolExecutor probably shouldn't expose one, either.
> Identifying which task each worker is processing is possible, but would
> perhaps require more work than its worth, at least for ProcessPoolExecutor.
>
> I do think adding worker count APIs is reasonable, and in-line with a TODO
> item in the ThreadPoolExecutor source:
>
>    # TODO(bquinlan): Should avoid creating new threads if there are more
>    # idle threads than items in the work queue.
>
> So, at the very least there have been plans to internally keep track
> active/idle thread counts. If others agree it's a good idea, I'll open an
> issue on the tracker for this and include my patch (which also addresses
> that TODO item).
>
>
>
> On Sun, Aug 24, 2014 at 5:41 PM, Ram Rachum <ram.rachum at gmail.com> wrote:
>
>> Sometimes I want to take a live executor, like a `ThreadPoolExecutor`,
>> and check up on it. I want to know how many threads there are, how many are
>> handling tasks and which tasks, how many are free, and which tasks are in
>> the queue.
>>
>> I asked on Stack Overflow:
>> http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-futures-threadpoolexecutor
>>
>> There's an answer there, but it uses private variables and it's not part
>> of the API.
>>
>> I suggest it become a part of the API. There should be an API for
>> checking on what the executor is currently doing and answering all the
>> questions I raised above.
>>
>>
>> Thanks,
>> Ram.
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/f6a668d1/attachment-0001.html>