On Feb 22, 2010, at 2:37 PM, Jeffrey Yasskin wrote:

Where's the current version of the PEP?

http://code.google.com/p/pythonfutures/source/browse/branches/feedback/PEP.txt

On Sun, Feb 21, 2010 at 1:47 AM, Brian Quinlan <brian@sweetapp.com> wrote:

On 21 Feb 2010, at 14:41, Jeffrey Yasskin wrote:

* I'd like users to be able to write Executors besides the simple

ThreadPoolExecutor and ProcessPoolExecutor you already have. To enable

that, could you document what the subclassing interface for Executor

looks like? that is, what code do user-written Executors need to

include?

I can do that.

I don't think it should include direct access to

future._state like ThreadPoolExecutor uses, if at all possible.

Would it be reasonable to make Future an ABC, make a _Future that subclasses

it for internal usage and let other Executor subclasses define their own

Futures.

What interface are you proposing for the Future ABC? It'll need to
support wait() and as_completed() from non-library Futures. I wouldn't
mind making the type just a duck-type (it probably wouldn't even need
an ABC), although I'd like to give people trying to implement their
own Executors as much help as possible. I'd assumed that giving Future
some public hooks would be easier than fixing the wait() interface,
but I could be wrong.

See below.

* Could you specify in what circumstances a pure computational

Future-based program may deadlock? (Ideally, that would be "never".)

Your current implementation includes two such deadlocks, for which

I've attached a test.

* Do you want to make calling Executor.shutdown(wait=True) from within

the same Executor 1) detect the problem and raise an exception, 2)

deadlock, 3) unspecified behavior, or 4) wait for all other threads

and then let the current one continue?

What about a note saying that using any futures functions or methods from

inside a scheduled call is likely to lead to deadlock unless care is taken?

Jesse pointed out that one of the first things people try to do when
using concurrency libraries is to try to use them inside themselves.
I've also tried to use a futures library that forbade nested use
('cause I wrote it), and it was a real pain.

You can use the API from within Executor-invoked functions - you just have to be careful.

It should be easy enough to detect that the caller of
Executor.shutdown is one of the Executor's threads or processes, but I
wouldn't mind making the obviously incorrect "wait for my own
completion" deadlock or throw an exception, and it would make sense to
give Executor implementors their choice of which to do.

* This is a nit, but I think that the parameter names for

ThreadPoolExecutor and ProcessPoolExecutor should be the same so

people can parametrize their code on those constructors. Right now

they're "max_threads" and "max_processes", respectively. I might

suggest "max_workers".

I'm not sure that I like that. In general consolidating the constructors for

executors is not going to be possible.

In general, yes, but in this case they're the same, and we should try
to avoid gratuitous differences.

num_threads and num_processes is more explicit than num_workers but I don't really care so I changed it.

* I'd like users to be able to write Executors besides the simple

ThreadPoolExecutor and ProcessPoolExecutor you already have. To enable

that, could you document what the subclassing interface for Executor

looks like? that is, what code do user-written Executors need to

include? I don't think it should include direct access to

future._state like ThreadPoolExecutor uses, if at all possible.

One of the difficulties here is:

1. i don't want to commit to the internal implementation of Futures

Yep, that's why to avoid requiring them to have direct access to the
internal variables.

2. it might be hard to make it clear which methods are public to users and

which methods are public to executor implementors

One way to do it would be to create another type for implementors and
pass it to the Future constructor.

If we change the future interface like so:

class Future(object):

  # Existing public methods

  ...

  # For executors only

  def set_result(self):

    ...

  def set_exception(self):

    ...

  def check_cancel_and_notify(self):

    # returns True if the Future was cancelled and

    # notifies anyone who cares i.e. waiters for

    # wait() and as_completed

Then an executor implementor need only implement:

def submit(self, fn, *args, **kwargs):

With the logic to actual execute fn(*args, **kwargs) and update the returned future, of course.

Thoughts?

* Could you specify in what circumstances a pure computational

Future-based program may deadlock? (Ideally, that would be "never".)

Your current implementation includes two such deadlocks, for which

I've attached a test.

Thanks for the tests but I wasn't planning on changing this behavior. I

don't really like the idea of using the calling thread to perform the wait

because:

1. not all executors will be able to implement that behavior

Why not?

What if my executor sends the data to a remove cluster for execution and running it locally isn't feasible?

Thread pools can implement it,

Do you have a strategy in mind that would let you detect arbitrary deadlocks in threaded futures?

Cheers,

Brian

and process pools make it
impossible to create cycles, so they also can't deadlock.

2. it can only be made to work if no wait time is specified

With a wait time, you have to avoid stealing work, but it's also
guaranteed not to deadlock, so it's fine.