Mailman 3 November 2012 - Python-ideas

Async API: some code to review
by Guido van Rossum Nov. 7, 2012

Nov. 7, 2012

I am finally ready to show the code I worked on for the past two weeks. This is definitely not ready for anything except as a quick demo, but I learned enough while writing it to feel comfortable with the PEP 380 paradigm. I've set up a Hg repo on code.google.com, and I picked a codename: tulip. View the code here: http://code.google.com/p/tulip/source/browse/ It runs on Linux and OSX; I have no easy access to Windows but I'd be happy to take contributions. Key files in the directory: - … [View More]main.py: the main program for testing, and a rough HTTP client - sockets.py: transports for sockets and SSL, and a buffering layer - scheduling.py: a Task class and related stuff; this is where the PEP 380 scheduler is implemented - polling.py: an event loop and basic polling implementations for: select(), poll(), epoll(), kqueue() Other junk: .hgignore, Makefile, README, p3time.py (benchmark yield from vs. plain functions), longlines.py (stupid style checker) More detailed discussions per file follows; please read the code along with my description (separately they may not make much sense): polling.py: http://code.google.com/p/tulip/source/browse/polling.py I found it remarkably easy to come up with polling implementations using all those different system calls. I ended up mixing in the pollster class with the event loop class, although I'm not sure that's the best design -- perhaps it's better if the event loop just references the pollster as a separate object. The pollster has a very simple API: add_reader(fd, callback, *args), add_writer(<ditto>), remove_reader(fd), remove_writer(fd), and poll(timeout) -> list of events. (fd means file descriptor.) There's also pollable() which just checks if there are any fds registered. My implementation requires fd to be an int, but that could easily be extended to support other types of event sources. I'm not super happy that I have parallel reader/writer APIs, but passing a separate read/write flag didn't come out any more elegant, and I don't foresee other operation types (though I may be wrong). The event list started out as a tuple of (fd, flag, callback, args), where flag is 'r' or 'w' (easily extensible); in practice neither the fd nor the flag are used, and one of the last things I did was to wrap callback and args into a simple object that allows cancelling the callback; the add_*() methods return this object. (This could probably use a little more abstraction.) Note that poll() doesn't call the callbacks -- that's up to the event loop. The event loop has two basic ways to register callbacks: call_soon(callback, *args) causes callback(*args) to be called the next time the event loop runs; call_later(delay, callback, *args) schedules a callback at some time (relative or absolute) in the future. It also inherits add_reader() and add_writer() from the pollster. Then there is run(), which runs the event loop until there's nothing left to do (no readers, no writers, no soon or later callbacks), and run_once(), which goes through the entire list of event sources once. (I think the order in which I do this isn't quite right but it works for now.) Finally, there's a helper class (ThreadRunner) here which lets you run something in a separate thread using the features of concurrent.futures. It uses the "self-pipe trick" (Google it :-) to ensure that the poll() call wakes up -- this is needed by call_in_thread() at the next layer (scheduling.py). (There may be a race condition here, but I think it can be fixed.) Note that there are no yields (or yield froms) here; that's for the next layer: scheduling.py: http://code.google.com/p/tulip/source/browse/scheduling.py This is the scheduler for PEP-380 style coroutines. I started with a Scheduler class and operations along the lines of Greg Ewing's design, with a Scheduler instance as a global variable, but ended up ripping it out in favor of a Task object that represents a single stack of generators chained via yield-from. There is a Context object holding the event loop and the current task in thread-local storage, so that multiple threads can (and must) have independent event loops. Most user (and much library) code in this system should be written as generators invoking other generators directly using yield from. However to run something as an independent task, you wrap the generator call in a Task() constructor, possibly giving it a timeout, and then calling its start() method. A Task also acts a little like a future -- you can wait() for it, add done-callbacks, and it preserves the return value of the generator call. This can be used to introduce concurrency or to give something a separate timeout. (There are also primitives to wait for the first N completed of a bunch of Tasks.) To invoke a primitive I/O operation, you call the current task's block() method and then immediately yield (similar to Greg Ewing's approach). There are helpers block_r() and block_w() that arrange for a task to block until a file descriptor is ready for reading/writing. Examples of their use are in sockets.py. There is also call_in_thread() which integrates with polling.ThreadRunner to run a function in a separate thread and wait for it. Also used in sockets.py. In the docstrings I use the prefix "COROUTINE:" to indicate public APIs that should be invoked using yield from. sockets.py: http://code.google.com/p/tulip/source/browse/sockets.py This implements some internet primitives using the APIs in scheduling.py (including block_r() and block_w()). I call them transports but they are different from transports Twisted; they are closer to idealized sockets. SocketTransport wraps a plain socket, offering recv() and send() methods that must be invoked using yield from. SslTransport wraps an ssl socket (luckily in Python 2.6 and up, stdlib ssl sockets have good async support!). Then there is a BufferedReader class that implements more traditional read() and readline() coroutines (i.e., to be invoked using yield from), the latter handy for line-oriented transports. Finally there are some functions for connecting sockets, the highest-level one create_transport(). These use call_in_thread() to run socket.getaddrinfo() in a thread (this provides IPv6 support). I don't particularly care about the exact abstractions in this module; they are convenient and I was surprised how easy it was to add SSL, but still these mostly serve as somewhat realistic examples of how to use scheduling.py. (Afterthought: I think the SocketTransport's recv() and send() methods could be made more similar to SslTransport.) More examples in the final file: main.py: http://code.google.com/p/tulip/source/browse/main.py There is a simplistic HTTP client here built on top of the sockets.*Transport abstractions. And the main code exercises this by spawning four tasks fetching a variety of URLs (more when you uncomment a block of code) and waiting for their results. The code is a bit of a mess because I used it as a place to try out various APIs. I'm most interested in feedback on the design of polling.py and scheduling.py, and to a lesser extent on the design of sockets.py; main.py is just an example of how this style works out in practice. Sorry for the brain-dump style; I would like to write it all up better, but at the same time waiting longer doesn't necessarily make it better, so here it is, for all to see. (I also have a list of problems I had to debug during the development and what I learned from that; but that's too raw to post right now.) -- --Guido van Rossum (python.org/~guido) [View Less]

20 96

os.path.commonpath()
by Serhiy Storchaka Nov. 7, 2012

Nov. 7, 2012

See http://bugs.python.org/issue10395. os.path.commonpath() should be a function which returns right longest common sub-path for specified paths (os.path.commonprefix() is completely useless for this). There are some open questions about details of *right* behavior. What should be a common prefix of '/var/log/apache2' and '/var//log/mysql'? What should be a common prefix of '/usr' and '//usr'? What should be a common prefix of '/usr/local/' and '/usr/local/'? What should be a common … [View More]

6 10

The "in"-statement
by Markus Unterwaditzer Nov. 7, 2012

Nov. 7, 2012

This is my first post on this mailinglist and i haven't lurked a long time yet, so please be gentle. While mocking objects, i got annoyed by the following code pattern i had to use when modifying multiple attributes on a single object:: obj.first_attr = "value" obj.second_attr = "value2" some_other = "lel" I thought it would be neat if i could do:: in obj: first_attr = "value" second_attr = "value2" some_other = "lel" # indenting this would cause it to … [View More]

13 14

Code snippet signature (Was: Publishing ideas on ActiveState recipe site)
by anatoly techtonik Nov. 7, 2012

Nov. 7, 2012

Speaking about snippets in the thread about ActiveState recipe site reminded me of this idea. Snippet signature is a checksum over normalized source code with stripped comments. The signature looks like `# snippet:site:hash` added before function as a comment. Normalized means that code is unindented and PEPified. Comments are also stripped as they may be custom. Signatures are useful to: 1. identify piece of code repeatably used in Python apps for potential inclusion in Python library 2. … [View More]measure popularity of snippets using automated code analysis 3. automatically check for updates using lookup on the given hash 4. automatically check if local code has improvements over the original snippet by recalculating and comparing hash 5. lookup if snippets were integrated into standard (or some other library) 6. compile a Python library with mostly reused snippets on the server side -- anatoly t. On Wed, Nov 7, 2012 at 9:46 AM, anatoly techtonik <techtonik(a)gmail.com> wrote: > On Tue, Nov 6, 2012 at 1:22 AM, Steven D'Aprano <steve(a)pearwood.info> wrote: >> On 05/11/12 23:52, Ned Batchelder wrote: >>> >>> Get everything the way you want it, and then propose it. >> >> >> +1 >> >> Also consider publishing it as a recipe on ActiveState, where many >> people will view it, use it, and offer feedback. This has many >> benefits: >> >> * You will gauge community interest; >> >> * Many eyeballs make bugs shallow; >> >> * You are providing a useful recipe that others can use, even >> if it doesn't get included in the std lib. >> >> Some of the most useful parts of the std lib, like namedtuple, >> started life on ActiveState. >> >> http://code.activestate.com/recipes/langs/python/ > > Why I don't use ActiveState: > > 1. StackOverflow is much easier to access - just one click to login > with Google Account versus several clicks, data entry and copy/paste > operations to remind the password on ActiveState - I want to login > there with Python account > 2. StackOverflow is problem search oriented - not recipe catalog > oriented, which makes it better for solving problems, which I do more > often than reading the recipe book (although I must admin when I was > starting Python - the Cookbook from O'Reilly in CHM format was mega > awesome) > 3. I post the code as gists as it includes the notion of history, > unlike ActiveState, which interface looks a little outdated - it was > not obvious for me that recipes have history until today > 4. Recipes are licensed, which is a too much of a burden for a snippet > 5. ActiveState site makes it clear that it is ActiveState site - the > 20% of my screen is taken by ActiveState header, so it looks like > company site - not a site for community > > Otherwise the idea of community recipe site is very nice. [View Less]

1 0

os.path.split(path, maxsplit=1)
by anatoly techtonik Nov. 7, 2012

Nov. 7, 2012

Why? Because it is critical when comparing paths and non-trivial. http://stackoverflow.com/questions/4579908/cross-platform-splitting-of-path… http://stackoverflow.com/questions/3167154/how-to-split-a-dos-path-into-its… http://www.gossamer-threads.com/lists/python/dev/654410 -- anatoly t.

3 5

with-statement syntactic quirk
by Barry Warsaw Nov. 5, 2012

Nov. 5, 2012

with-statements have a syntactic quirk, which I think would be useful to fix. This is true in Python 2.7 through 3.3, but it's likely not fixable until 3.4, unless of course it's a bug <wink>. Legal: >>> with open('/etc/passwd') as p1, open('/etc/passwd') as p2: pass Not legal: >>> with (open('/etc/passwd') as p1, open('/etc/passwd') as p2): pass Why is this useful? If you need to wrap this onto multiple lines, say to fit it within line length limits. IWBNI you … [View More]

17 31

Function in shutil to clear contents of a directory
by Vinay Sajip Nov. 1, 2012

Nov. 1, 2012

A couple of times recently on different projects, I've had the need to clear out the contents of an existing directory. There doesn't appear to be any function in shutil to do this. I've used def clear_directory(path): for fn in os.listdir(path): fn = os.path.join(path, fn) if os.path.islink(fn) or os.path.isfile(fn): os.remove(fn) else: shutil.rmtree(fn) One could just do shutil.rmtree(path) followed by os.mkdir(path), but that fails if … [View More]

2 1