Mailman 3 December 2012 - Python-ideas

Re: [Python-ideas] Allow accessing return value inside finally clause
by alex23 26 Dec '12

26 Dec '12

On 26 Dec, 08:06, Ram Rachum <r...(a)rachum.com> wrote: > Now of course, you can find other solutions to this problem. You can write > a decorator to do the post-processing phase, or you could divide the whole > thing into 2 functions. But I think that sometimes, the > `finally`-postprocess idiom I propose will be the most succinct one. I initially responded to say "use a decorator", but you're already aware of the common pattern for dealing with this, and yet you'd rather the language change instead? > (Regarding lack of return value and propagating exceptions: This all sounds > solvable to me. Why not let the `finally` clause detect what's going on and > react appropriately? No `return` value? Don't postprocess. Exception > raised? Don't interfere.) People already struggle with understanding the semantics of try/ finally - you yourself demonstrated this in your first post by not being away that the 'return value' may not be set - and you want to make it _more_ magic? You're a programmer. At the end of the day, you're going to have to do _some_ "heavy" lifting by yourself. Assign your return values to an object that performs the post-processing on demand. Create a context manager that does it when it exits. Write a loop to decorate your functions if typing @decorator is so strenuous. Making Python less clear to save yourself some typing isn't a decent trade off.

1 0

Allow accessing return value inside finally clause
by Ram Rachum 26 Dec '12

26 Dec '12

Say I have this function: def f(): try: return whatever() finally: pass # I want to get what `whatever()` returned in here I want to get the return value from inside the `finally` clause. I understand that this is currently not possible. I'd like that to be possible because that would allow post-processing of a function's return value. What do you think? Thanks, Ram.

3 2

Tree as a data structure (Was: Graph class)
by anatoly techtonik 23 Dec '12

23 Dec '12

On Sun, Dec 16, 2012 at 6:41 PM, Guido van Rossum <guido(a)python.org> wrote: > I think of graphs and trees as patterns, not data structures. In my world strings, ints and lists are 1D data types, and tree can be a very important 2D data structure. Even if it is a pattern, this pattern is vital for the transformation of structured data, because it allows to represent any data structure in canonical format. Speaking of tree as a data structure, I assume that it has a very basic definition: 1. tree consists of nodes 2. some nodes are containers for other nodes 3. every node has properties 4. every node has 0 or 1 parent 5. every container has 1+ children 6. tree has a single starting root node 7. no child of a parent can be its ancestor (no cyclic dependencies between elements) List of trees is a forest. Every subtree is a complete tree. To see which tree data type would be really useful in Python distribution (e.g. provides a simple, extendable and intuitive interface), I see only one way - is to scratching some itches relevant to Python and then try to scale it to other problems. The outcome should be the answer - what for native tree type is not suitable? More ideas: [ ] Much experience for working with trees can be brought from XML and DOM manipulation practices (jQuery and friends) [ ] every element in a tree can be accessed by its address specificator as 'root/node[3]/last' [ ] but it is also convenient to access tree data using node names as 'mylib.books[:1]' [ ] and of course, you can run queries over trees [ ] Tree is the base for any "Data Transformation Framework" as it allows to jump from "data type conversion" to "data structure conversion and mapping" [ ] Trees can be converted to other trees and to more complicated structures [ ] Conversion can be symmetrical and non-symmetrical [ ] Conversion can be lossy and lossless [ ] Conversion can be lossless and non-symmetrical at the same time Trees can be used, for example, for realtime migration of issues from one tracker to another. For managing changesets with additional meta information. For presenting package dependencies and working with them. For atomic (transactional) file management. For managing operating system capability information. For logging setup. For debugging structures in Python. For working and converting binary file formats. For the common AST transformation and query interface. For the understanding how 2to3 fixers work. For the common ground of visual representation, comparison and transformation of data structures. That's probably enough of my itches.

3 3

Async context managers and iterators with tulip
by Nick Coghlan 23 Dec '12

23 Dec '12

On Sat, Dec 22, 2012 at 4:17 PM, guido.van.rossum <python-checkins(a)python.org> wrote: > +- We might introduce explicit locks, though these will be a bit of a > + pain to use, as we can't use the ``with lock: block`` syntax > + (because to wait for a lock we'd have to use ``yield from``, which > + the ``with`` statement can't do). Actually, I just realised that the following can work if the async lock is defined appropriately: with yield from async_lock: ... The secret is that async_lock would need to be a coroutine rather than a context manager. *Calling* the coroutine would acquire the lock (potentially registering a callback that is scheduled when the lock is released) and return a context manager that released the lock. The async_lock itself wouldn't be a context manager, so you'd get an immediate error if you left out the "yield from". We'd be heading even further down the path of two-languages-for-the-price-of-one if we did that, though (by which I mean the fact that async code and synchronous code exist in parallel universes - one, more familiar one, where the ability to block is assumed, as is the fact that any operation may give concurrent code the chance to execute, and the universe of Twisted, tulip, et al, where possible suspension points are required to be explicitly marked in the function where they occur). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 8

Tkinter and tulip
by Terry Reedy 23 Dec '12

23 Dec '12

Though not mentioned much in the tulip discussion, tkinter is a third 'T' package with its own event loop. (And by the way, I associate 'tulip' with 'Floriade', with 10s of thousands of tulips in bloom. It was a +++ experience. But I suppose it is too cute for Python ;-) Yesterday, tk/tkinter expert Kevin Walzer asked on python-list how to (easily) read a pipe asynchonously and post the result to a tk text widget. I don't know the answer now, but is my understanding correct that in the future a) there should be a tk loop adapter that could replace the default tulip loop and b) it would then be easy to add i/o events to the tk loop? My personal interest is whether it will some day be possible to re-write IDLE to use tulip so one could edit in an edit pane while the shell pane asynchronously waits for and displays output from a 'long' computation.* It would also be nice if ^C could be made to work better -- which is to say, take effect sooner -- by decoupling key processing from socket reading. I am thinking that IDLE could be both a simple test and showcase for the usefulness of tulip. *I currently put shell and edit windows side-by-side on my wide-screen monitor. I can imagine putting two panes in one window instead. -- Terry Jan Reedy

2 1

PEP 3156 feedback: wait_one vs par vs concurrent.futures.wait
by Nick Coghlan 23 Dec '12

23 Dec '12

I figure python-ideas is still the best place for PEP 3156 feedback - I think it's being revised too heavily for in-depth discussion on python-dev to be a good idea, and I think spinning out a separate list would lose too many people that are interested-but-not-enough-to-subscribe-to-yet-another-mailing-list (including me). The current draft of the PEP suggests the use of par() for the barrier operation (waiting for all futures and coroutines in a collection to be ready), while tentatively suggesting wait_one() as the API for waiting for the first completed operation in a collection. That inconsistency is questionable all by itself, but there's a greater stdlib level inconsistency that I find more concerning The corresponding blocking API in concurrent.futures is the module level "wait" function, which accepts a "return_when" parameter, with the permitted values FIRST_COMPLETED, FIRST_EXCEPTION and ALL_COMPLETED (the default). In the case where everything succeeds, FIRST_EXCEPTION is the same as ALL_COMPLETED. This function also accepts a timeout which allows the operation to finish early if the operations take too long. This flexibility also leads to a difference in the structure of the return type: concurrent.futures.wait always returns a pair of sets, with the first set being those futures which completed, while the second contains those which remaining incomplete at the time the call returned. It seems to me that this "wait" API can be applied directly to the equivalent problems in the async space, and, accordingly, *should* be applied so that the synchronous and asynchronous APIs remain as consistent as possible. The low level equivalent to par() would be: incomplete = <tasks, futures or coroutines> complete, incomplete = yield from tulip.wait(incomplete) assert not incomplete # Without a timeout, everything should complete for f in complete: # Handle the completed operations Limiting the maximum execution time of any task to 10 seconds is straightforward: incomplete = <tasks, futures or coroutines> complete, incomplete = yield from tulip.wait(incomplete, timeout=10) for f in incomplete: f.cancel() # Took too long, kill it for f in complete: # Handle the completed operations The low level equivalent to the wait_one() example would become: incomplete = <tasks, futures or coroutines> while incomplete: complete, incomplete = yield from tulip.wait(incomplete, return_when=FIRST_COMPLETED) for f in complete: # Handle the completed operations par() becomes easy to define as a coroutine: @coroutine def par(fs): complete, incomplete = yield from tulip.wait(fs, return_when=FIRST_EXCEPTION) for f in incomplete: f.cancel() # Something must have failed, so cancel the rest # If something failed, calling f.result() will raise that exception return [f.result() for f in complete] Defining wait_one() is also straightforward (although it isn't clearly superior to just using the underlying API directly): @coroutine def wait_one(fs): complete, incomplete = yield from tulip.wait(fs, return_when=FIRST_COMPLETED) return complete.pop() The async equivalent to "as_completed" under this scheme is far more interesting, as it would be an iterator that produces coroutines: def as_completed(fs): incomplete = fs while incomplete: # Phase 1 of the loop, we yield a coroutine that actually starts operations running @coroutine def _wait_for_some(): nonlocal complete, incomplete complete, incomplete = yield from tulip.wait(fs, return_when=FIRST_COMPLETED) return complete.pop().result() yield _wait_for_some() # Phase 2 of the loop, we pass back the already complete operations while complete: # Note this use case for @coroutine *forcing* objects to behave like a generator, # as well as exploiting the ability to avoid trips around the event loop @coroutine def _next_result(): return complete.pop().result() yield _next_result() # This is almost as easy to use as the synchronous equivalent, the only difference # is the use of "yield from f" instead of the synchronous "f.result()" for f in as_completed(fs): next = yield from f Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 6

``with from`` statement
by Andrew Svetlov 23 Dec '12

23 Dec '12

Crazy idea. Guido van Rossum mentioned after working on PEP 3156 that context managers cannot use yield from statement inside __enter__ and __exit__ magic methods. Explicit call for entering and leaving context (for locking for example) is not convenient. What do you think about with from f(): do_our_work() ``with from ...` construction calls __enter_from__ generator and iterates via ``yield from`` for that. Returned value is our context manager. The same for __exit_from__ — do``yield from`` for that and stop on StopIteration or exception. -- Thanks, Andrew Svetlov

4 6

PEP 3156 - Asynchronous IO Support Rebooted
by Guido van Rossum 22 Dec '12

22 Dec '12

Dear python-dev *and* python-ideas, I am posting PEP 3156 here for early review and discussion. As you can see from the liberally sprinkled TBD entries it is not done, but I am about to disappear on vacation for a few weeks and I am reasonably happy with the state of things so far. (Of course feedback may change this. :-) Also, there has already been some discussion on python-ideas (and even on Twitter) so I don't want python-dev to feel out of the loop -- this *is* a proposal for a new standard library module. (But no, I haven't picked the module name yet. :-) There's an -- also incomplete -- reference implementation at http://code.google.com/p/tulip/ -- unlike the first version of tulip, this version actually has (some) unittests. Let the bikeshedding begin! (Oh, happy holidays too. :-) -- --Guido van Rossum (python.org/~guido)

6 9

Re: [Python-ideas] An async facade?
by Jonathan Slenders 22 Dec '12

22 Dec '12

Hi All, This week I finished some Python syntax on a Pypy fork. It was an experiment I was working on this week. We really needed a cleaner way of writing asynchronous code. So, instead of using the yield keyword and an @async decorator, we implemented the 'await' keyword, similar to c#. So, because I just now subscribed to python-ideas, I cannot reply immediately to the following thread: An async facade? (was Re: [Python-Dev] Socket timeout and completion based sockets) Anyway, like c# does, I implemented the await keyword for Python, and should say that I'm really confident of the usability of the result. Personally, I think this is a very clean solution for Twisted's @defer.inlineCalbacks, Tornado's @gen.engine, and similar functions in other async frameworks. We use it right now in a commercial web environment, where third party users should have to be able to write asynchronous code as easy as possible in a web based IDE. https://bitbucket.org/jonathanslenders/pypy Two interpreter hooks were added: (both accept a callable as parameter.) >>>> sys.setawaithandler(wrapper) >>>> sys.setawaitresultwrapper(result_wrapper) The first will set the I/O scheduler a functions for wrapping others functions which contain 'await' instead of 'yield'. This wrapper function will receive a generator as input. So, 'await' still acts like 'yield' for the interpreter, but the result is automatically wrapped by this function, if the await keyword was found. The second function will wrap the return result of asynchronous functions. So, unlike normal generators with 'yield' keywords, where 'await' has been used, we still can return a result. But this result will be wrapped by this function, so that the generator in the scheduler will be able te recognize the returned result. This: @defer.inlineCallbacks def async_function(deferred_param): a = yield deferred_param b = yield some_call(a) yield defer.returnValue(b) will now become: def async_function(deferred_param): a = await deferred_param b = await some_call(a) return b So, while still being explicit, it requires minimal syntax, and allows distinguishing between when to 'wait' for an asynchronous task, and when to pass the Future object around. I really have no idea whether this has been proposed before, I can only say that we are using it and it works pretty well. Cheers, Jonathan

4 10

Tulip patches
by Geert Jansen 22 Dec '12

22 Dec '12

Hi, [if this is not the right forum to post patches for tulip, please redirect me to the correct one. There doesn't appear to be a mailing list for tulip at the moment. And this list is where most/all of the discussion is taking place.] Please find attached 4 patches: 0001-run-fd-callbacks.patch This patch will run callbacks for readers and writers in the same loop iteration as when the fd got ready. Copying from my previous email, this is to support the following idiom: # handle_read() sets the "ready" flag loop.add_reader(fd, handle_read) while not ready: loop.run_once() The patch currently dispatches callbacks twice in each iteration, once before blocking and once after. I tried to dispatch only once after blocking, but this made the SSL transport test hang. The reason is that the create_transport task is scheduled with call_soon(), and only when the task first runs, a file descriptor is added. So unless you dispatch before blocking, this task will never get started. 0002-call-every-iteration.patch This adds a call_every_iteration() method to the event loop. This callback runs *before* blocking. 0003-fix-typo.patch 0004-remove-wrong-comments.patch Two trivial patches. Regards, Geert

3 7