[Python-ideas] Add an introspection API to Executor

[Andrew Barnert]

I don't think there's any issue with letting people introspect the executor. The problem is that the main thing you get is a queue, and there's a limit to how introspectable a queue can be.

In particular, if you want to iterate the waiting tasks, you have to iterate the queue, and there's no safe way to do that.

Since CPython's queue.Queue happens to be just a deque and a mutex, you could make it iterable at the cost of blocking all producers and consumers (which might be fine for many uses, like debugging or exploratory programming), or provide a snapshot API to return a copy of the deque.

But do you want to make that a requirement on all subclasses of Queue, and all other implementations' queue modules? Does ProirityQueue have to nondestructively iterate a heap in order? Does Jython have to use a mutex and a deque instead of a more efficient (and possibly lock-free) queue from the Java stdlib? What does multiprocessing.Queue do on each implementation?

I don't think the costs are worth the benefit. And I assume that's why the existing queue API doesn't provide an iteration or snapshot mechanism.

But there's an option that might be worth doing:

Provide a queue.IntrospectableQueue type that _is_ defined to have such a mechanism, but to otherwise work like a Queue (except maybe less efficiently). Then provide an optional parameter for the Executor that lets you specify an alternate queue constructor in place of the default. So, when exploring or debugging, you could pass queue.IntrospectableQueue (or multiprocessing.IntrospectableQueue for ProcessPoolExecutor).

Whether the interface is "lock_and_return_iterator" or "snapshot", this would be trivial to implement in CPython, and other Pythons could just copy the CPython version instead of extending their native queue types.

Sent from a random iPhone

On Aug 25, 2014, at 11:54, Guido van Rossum <guido at python.org> wrote:

> It might be worth it to make the implementation somewhat more complicated if it serves a good purpose, for example giving the user of the program insights into how well the executor is performing. Without such insight you may be attempting to tune parameters (like the pool size) without being able to evaluate their effect.
> 
> 
> On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou <antoine at python.org> wrote:
>> Le 25/08/2014 14:16, Ram Rachum a écrit :
>> 
>>> Maybe I'm missing something, but I don't think that's something that
>>> should block implementation.
>>> 
>>> Information not available? Change the executor code to make that
>>> information available.
>> 
>> Not if that would make the implementation much more complicated, or significantly slower.
>> 
>> 
>> Regards
>> 
>> Antoine.
>> 
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/2a2f55ef/attachment-0001.html>