Mailman 3 February 2010 - stdlib-sig

Re: [stdlib-sig] futures - a new package for asynchronous execution
by Jeffrey Yasskin March 5, 2010

March 5, 2010

Where's the current version of the PEP? On Sun, Feb 21, 2010 at 1:47 AM, Brian Quinlan <brian(a)sweetapp.com> wrote: > > On 21 Feb 2010, at 14:41, Jeffrey Yasskin wrote: > >> Several comments: >> >> * I see you using the Executors as context managers, but no mention in >> the specification about what that does. > > I can't see such documentation for built-in Python objects. To be > symmetrical with the built-in file object, i've documented the context > manager behavior as part of the Executor.shutdown method. For locks, it has its own section: http://docs.python.org/library/threading.html#using-locks-conditions-and-se… But I don't care too much about the formatting as long as the PEP specifies it clearly. >> You need to specify it. (Your >> current implementation doesn't wait in __exit__, which I think is the >> opposite of what you agreed with Antoine, but you can fix that after >> we get general agreement on the interface.) > > Fixed. > >> * I'd like users to be able to write Executors besides the simple >> ThreadPoolExecutor and ProcessPoolExecutor you already have. To enable >> that, could you document what the subclassing interface for Executor >> looks like? that is, what code do user-written Executors need to >> include? > > I can do that. > >> I don't think it should include direct access to >> future._state like ThreadPoolExecutor uses, if at all possible. > > Would it be reasonable to make Future an ABC, make a _Future that subclasses > it for internal usage and let other Executor subclasses define their own > Futures. What interface are you proposing for the Future ABC? It'll need to support wait() and as_completed() from non-library Futures. I wouldn't mind making the type just a duck-type (it probably wouldn't even need an ABC), although I'd like to give people trying to implement their own Executors as much help as possible. I'd assumed that giving Future some public hooks would be easier than fixing the wait() interface, but I could be wrong. >> * Could you specify in what circumstances a pure computational >> Future-based program may deadlock? (Ideally, that would be "never".) >> Your current implementation includes two such deadlocks, for which >> I've attached a test. > >> * Do you want to make calling Executor.shutdown(wait=True) from within >> the same Executor 1) detect the problem and raise an exception, 2) >> deadlock, 3) unspecified behavior, or 4) wait for all other threads >> and then let the current one continue? > > What about a note saying that using any futures functions or methods from > inside a scheduled call is likely to lead to deadlock unless care is taken? Jesse pointed out that one of the first things people try to do when using concurrency libraries is to try to use them inside themselves. I've also tried to use a futures library that forbade nested use ('cause I wrote it), and it was a real pain. It should be easy enough to detect that the caller of Executor.shutdown is one of the Executor's threads or processes, but I wouldn't mind making the obviously incorrect "wait for my own completion" deadlock or throw an exception, and it would make sense to give Executor implementors their choice of which to do. >> * This is a nit, but I think that the parameter names for >> ThreadPoolExecutor and ProcessPoolExecutor should be the same so >> people can parametrize their code on those constructors. Right now >> they're "max_threads" and "max_processes", respectively. I might >> suggest "max_workers". > > I'm not sure that I like that. In general consolidating the constructors for > executors is not going to be possible. In general, yes, but in this case they're the same, and we should try to avoid gratuitous differences. >> * You should document the exception that happens when you try to pass >> a ProcessPoolExecutor as an argument to a task executing inside >> another ProcessPoolExecutor, or make it not throw an exception and >> document that. > > The ProcessPoolExecutor limitations are the same as the multiprocessing > limitations. I can provide a note about that and a link to that module's > documentation. And multiprocessing doesn't document that its Pool requires picklability and isn't picklable itself. Saying that the ProcessPoolExecutor is equivalent to a multiprocessing.Pool should be enough for your PEP. >> * If it's intentional, you should probably document that if one >> element of a map() times out, there's no way to come back and wait >> longer to retrieve it or later elements. > > That's not obvious? Maybe. >> * You still mention run_to_futures, run_to_results, and FutureList, >> even though they're no longer proposed. > > Done. > >> >> * wait() should probably return a named_tuple or an object so we don't >> have people writing the unreadable "wait(fs)[0]". > > Done. > >> >> * Instead of "call finishes" in the description of the return_when >> parameter, you might describe the behavior in terms of futures >> becoming done since that's the accessor function you're using. > > Done. > > >> * Is RETURN_IMMEDIATELY just a way to categorize futures into done and >> not? Is that useful over [f for f in fs if f.done()]? > > That was an artifact of the previous implementation; removed. > >> * After shutdown, is RuntimeError the right exception, or should there >> be a more specific exception? > > RunTimeError is what is raised in similar situations by threading e.g. when > starting an already started thread. Ok, works for me. On Sun, Feb 21, 2010 at 5:49 AM, Brian Quinlan <brian(a)sweetapp.com> wrote: > A few extra points. > > On 21 Feb 2010, at 14:41, Jeffrey Yasskin wrote: >> >> * I'd like users to be able to write Executors besides the simple >> ThreadPoolExecutor and ProcessPoolExecutor you already have. To enable >> that, could you document what the subclassing interface for Executor >> looks like? that is, what code do user-written Executors need to >> include? I don't think it should include direct access to >> future._state like ThreadPoolExecutor uses, if at all possible. > > One of the difficulties here is: > 1. i don't want to commit to the internal implementation of Futures Yep, that's why to avoid requiring them to have direct access to the internal variables. > 2. it might be hard to make it clear which methods are public to users and > which methods are public to executor implementors One way to do it would be to create another type for implementors and pass it to the Future constructor. >> * Could you specify in what circumstances a pure computational >> Future-based program may deadlock? (Ideally, that would be "never".) >> Your current implementation includes two such deadlocks, for which >> I've attached a test. > > Thanks for the tests but I wasn't planning on changing this behavior. I > don't really like the idea of using the calling thread to perform the wait > because: > 1. not all executors will be able to implement that behavior Why not? Thread pools can implement it, and process pools make it impossible to create cycles, so they also can't deadlock. > 2. it can only be made to work if no wait time is specified With a wait time, you have to avoid stealing work, but it's also guaranteed not to deadlock, so it's fine.

7 24

Re: [stdlib-sig] futures - a new package for asynchronous execution
by Brian Quinlan Feb. 21, 2010

Feb. 21, 2010

On Nov 8, 2009, at 7:01 PM, Jeffrey Yasskin wrote: > Did you mean to drop the list? Feel free to cc them back in when you > reply. No, that was a brain malfunction. Redirecting the discussion to the list. > On Sat, Nov 7, 2009 at 3:31 PM, Brian Quinlan <brian(a)sweetapp.com> > wrote: >> >> On 8 Nov 2009, at 06:37, Jeffrey Yasskin wrote: >> >>> On Sat, Nov 7, 2009 at 7:32 AM, Jesse Noller <jnoller(a)gmail.com> >>> wrote: >>>> >>>> On Sat, Nov 7, 2009 at 10:21 AM, Antoine Pitrou <solipsis(a)pitrou.net >>>> > >>>> wrote: >>>>> >>>>>> Which API? My comment wasn't aimed at the API of the package - >>>>>> in the >>>>>> time I got to scan it last night nothing jumped out at me as >>>>>> overly >>>>>> offensive API-wise. >>>>> >>>>> Not offensive, but probably too complicated if it's meant to be >>>>> a simple >>>>> helper. Anyway, let's wait for the PEP. >>>> >>>> >>>> The PEP is right here: >>>> >>>> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt >>>> >>>> I'm interested in hearing specific complaints about the API in the >>>> context of what it's trying to *do*. The only thing which jumped >>>> out >>>> at me was the number of methods on FutureList; but then again, each >>>> one of those makes conceptual sense, even if they are verbose - >>>> they're explicit on what's being done. >>> >>> Overall, I like the idea of having futures in the standard library, >>> and I like the idea of pulling common bits of multiprocessing and >>> threading into a concurrent.* package. Here's my >>> stream-of-consciousness review of the PEP. I'll try to ** things >>> that >>> really affect the API. >>> >>> The "Interface" section should start with a conceptual description >>> of >>> what Executor, Future, and FutureList are. Something like "An >>> Executor >>> is an object you can hand tasks to, which will run them for you, >>> usually in another thread or process. A Future represents a task >>> that >>> may or may not have completed yet, and which can be waited for and >>> its >>> value or exception queries. A FutureList is ... <haven't read that >>> far>." >>> >>> ** The Executor interface is pretty redundant, and it's missing the >>> most basic call. Fundamentally, all you need is an >>> Executor.execute(callable) method returning None, >> >> How do you extract the results? > > To implement submit in terms of execute, you write something like: > > def submit(executor, callable): > future = Future() > def worker(): > try: > result = callable() > except: > future.set_exception(sys.exc_info()) > else: > future.set_value(result) > executor.execute(worker) > return future I see. I'm not sure if that abstraction is useful but I get it now. >>> and all the >>> future-oriented methods can be built on top of that. I'd support >>> using >>> Executor.submit(callable) for the simplest method instead, which >>> returns a Future, but there should be some way for implementers to >>> only implement execute() and get submit either for free or with a >>> 1-line definition. (I'm using method names from >>> >>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/ExecutorService.… >>> in case I'm unclear about the semantics here.) run_to_futures, >>> run_to_results, and map should be implementable on top of the Future >>> interface, and shouldn't need to be methods on Executor. I'd >>> recommend >>> they be demoted to helper functions in the concurrent.futures module >>> unless there's a reason they need to be methods, and that reason >>> should be documented in the PEP. >>> >>> ** run_to_futures() shouldn't take a return_when argument. It should >>> be possible to wait for those conditions on any list of Futures. >>> (_not_ just a FutureList) >> >> I packaged up Futures into FutureLists to fix an annoyance that I >> have with >> the Java implementation - you have all of these Future objects but no >> convenient way of operating over them. > > Yep, I totally agree with that annoyance. Note, though, that Java has > the CompletionService to support nearly same use cases as > run_to_futures. CompletionService's use case is handling results as they finish (just like the callbacks do in Deferreds). The FutureList use case is querying e.g. which callables raised, which returned, which are still running? >> I made the FutureList the unit of waiting because: >> 1. I couldn't think of a use case where this wasn't sufficient > > Take your webcrawl example. In a couple years, when Futures are widely > accepted, it's quite possible that urllib.request.urlopen() will > return a Future instead of a file. Then I'd like to request a bunch of > URLs and process each as they come back. With the run_to_futures (or > CompletionService) API, urllib would instead have to take a set of > requests to open at once, which makes its API much harder to design. > With a wait-for-any function, urllib could continue to return a single > Future and let its users combine several results. If we go down this road then we should just switch to Twisted :-) Seriously, the idea is that no one would ever change their API to accommodate futures - they are a way of making a library with no notion of concurrency concurrent. But I am starting to be convinced that individual futures are a good idea because it makes the run/submit method easier to use. > Alternately, say you have an RPC system returning Futures. You've sent > off RPCs A, B, and C. Now you need two separate subsystems D and E to > do something with the results, except that D can continue when either > A or B finishes, but E can continue when either B or C finishes. Can D > and E express just what they need to express, or do they have to deal > with futures they don't really care about? > >> 2. It makes the internal locking semantics a bit easier and faster >> (if you >> can wait on any future then the wait has to acquire a lock for >> every future >> [in a consistent order to prevent deadlocks when other threads are >> doing the >> same thing with an intersecting set of futures], add a result >> listener for >> each and then great some sort of object to aggregate their state) > > Yep. I suspect the extra overhead isn't significant compared to the > cost of scheduling threads. > >> But I am thinking that maybe FutureLists aren't the right >> abstraction. >> >>> The code sample looks like Executor is a context manager. What does >>> its __exit__ do? shutdown()? shutdown&awaitTermination? I prefer >>> waiting in Executor.__exit__, since that makes it easier for users >>> to >>> avoid having tasks run after they've cleaned up data those tasks >>> depend on. But that could be my C++ bias, where we have to be sure >>> to >>> free memory in the right places. Frank, does Java run into any >>> problems with people cleaning things up that an Executor's tasks >>> depend on without awaiting for the Executor first? >>> >>> shutdown should explain why it's important. Specifically, since the >>> Executor controls threads, and those threads hold a reference to the >>> Executor, nothing will get garbage collected without the explicit >>> call. >> >> Actually, the threads hold a weakref to the Executor so they can >> exit (when >> the work queue is empty) if the Executor is collected. Here is the >> code from >> futures/thread.py: >> >> while True: >> try: >> work_item = work_queue.get(block=True, timeout=0.1) >> except queue.Empty: >> executor = executor_reference() >> # Exit if: >> # - The interpreter is shutting down OR >> # - The executor that owns the worker has been collected OR >> # - The executor that owns the worker has been shutdown. >> if _shutdown or executor is None or executor._shutdown: >> return > > Oh, yeah, that sounds like it might work. So why does shutdown exist? It does work - there are tests and everything :-) .shutdown exists for the same reason that .close exists on files: - Python does not guarantee any particular GC strategy - tracebacks and other objects may retain a reference in an unexpected way - sometimes you want to free your resources before the function exits > >>> ** What happens when FutureList.wait(FIRST_COMPLETED) is called >>> twice? >>> Does it return immediately the second time? Does it wait for the >>> second task to finish? I'm inclined to think that FutureList >>> should go >>> away and be replaced by functions that just take lists of Futures. >> >> It waits until a new future is completed. > > That seems confusing, since it's no longer the "FIRST" completed. Maybe "NEXT_COMPLETED" would be better. Cheers, Brian > >>> In general, I think the has_done_futures(), exception_futures(), >>> etc. >>> are fine even though their results may be out of date by the time >>> you >>> inspect them. That's because any individual Future goes >>> monotonically >>> from not-started->running->(exception|value), so users can take >>> advantage of even an out-of-date done_futures() result. However, >>> it's >>> dangerous to have several query functions, since users may think >>> that >>> running_futures() `union` done_futures() `union` cancelled_futures() >>> covers the whole FutureList, but instead a Future can move between >>> two >>> of the sets between two of those calls. Instead, perhaps an atomic >>> partition() function would be better, which returns a collection of >>> sub-lists that cover the whole original set. >>> >>> I would rename result() to get() (or maybe Antoine's suggestion of >>> __call__) to match Java. I'm not sure exception() needs to exist. >>> >>> --- More general points --- >>> >>> ** Java's Futures made a mistake in not supporting work stealing, >>> and >>> this has caused deadlocks at Google. Specifically, in a bounded-size >>> thread or process pool, when a task in the pool can wait for work >>> running in the same pool, you can fill up the pool with tasks that >>> are >>> waiting for tasks that haven't started running yet. To avoid this, >>> Future.get() should be able to steal the task it's waiting on out of >>> the pool's queue and run it immediately. >>> >>> ** I think both the Future-oriented blocking model and the >>> callback-based model Deferreds support are important for different >>> situations. Futures tend to be easier to program with, while >>> Deferreds >>> use fewer threads and can have more predictable latencies. It should >>> be possible to create a Future from a Deferred or a Deferred from a >>> Future without taking up a thread.

6 20

socket.makefile() questions
by tim＠ksu.edu Feb. 1, 2010

Feb. 1, 2010

My apologies if this message is to the wrong group. I have been experimenting with socket.makefile()from Python 3.1.1. I have not had much difficulty reading from the returned file object, but I don't understand the behavior when trying to write (send on the socket). I'm hoping that someone can explain how this is supposed to work. I find that this works for an established connection on socket s: fd = s.makefile('wb', buffering = 0) fd.write("This is a test message\n".encode('ascii')) A mode of 'rwb' also works. The object fd is of type SocketIO. fd = s.makefile('w', buffering = 0) -> ValueError exception fd = s.makefile('w') -> io.BufferedWriter, which does not send data. fd = s.makefile('wb') -> io.TextIOWrapper, which does not send data. The default value of the "buffering" parameter is None, which from my testing has a different result than 0 (zero). So, questions: 1) Why does buffering = None result in a buffered file object? 2) Are there bugs or incomplete work with socket.makefile(), io.BufferedWriter and io.TextIOWrapper in terms of why the latter two objects are returned, but fail to send data? Thank you, -- Tim Bower Assistant Professor Kansas State University at Salina Computer Systems Technology tim(a)ksu.edu

5 5