[Python-ideas] Add an introspection API to Executor

[Dan O'Reilly]

Adding active/idle/total worker counts for both ThreadPoolExecutor and
ProcessPoolExecutor is pretty straightforward; I just threw a patch
together for both in 30 minutes or so. However, I don't think its possible
to inspect the contents of a ProcessPoolExecutor's queue without actually
consuming items from it. While it *is* possible with ThreadPoolExecutor, I
don't think we should expose it - the queue.Queue() implementation
ThreadPoolExecutor relies on doesn't have a public API for inspecting its
contents, so ThreadPoolExecutor probably shouldn't expose one, either.
Identifying which task each worker is processing is possible, but would
perhaps require more work than its worth, at least for ProcessPoolExecutor.

I do think adding worker count APIs is reasonable, and in-line with a TODO
item in the ThreadPoolExecutor source:

   # TODO(bquinlan): Should avoid creating new threads if there are more
   # idle threads than items in the work queue.

So, at the very least there have been plans to internally keep track
active/idle thread counts. If others agree it's a good idea, I'll open an
issue on the tracker for this and include my patch (which also addresses
that TODO item).



On Sun, Aug 24, 2014 at 5:41 PM, Ram Rachum <ram.rachum at gmail.com> wrote:

> Sometimes I want to take a live executor, like a `ThreadPoolExecutor`, and
> check up on it. I want to know how many threads there are, how many are
> handling tasks and which tasks, how many are free, and which tasks are in
> the queue.
>
> I asked on Stack Overflow:
> http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-futures-threadpoolexecutor
>
> There's an answer there, but it uses private variables and it's not part
> of the API.
>
> I suggest it become a part of the API. There should be an API for checking
> on what the executor is currently doing and answering all the questions I
> raised above.
>
>
> Thanks,
> Ram.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/33f67d4d/attachment.html>