Mailman 3 December 2012 - Python-ideas

Graph class
by Mark Adam 10 Dec '12

10 Dec '12

I have a decent semi-recursive Graph class that I think could be a good addition to the Collections module. It probably needs some refactoring, but I'm posting here to see if there's any interest. For those who aren't too abreast of CS theory, a graph is one of the most abstract data structures in computer science, encompassing trees, and lists. I'm a bit surprised that no one's offered one up yet, so I'll present mine. The code is at http://github.com/theProphet/Social-Garden under the pangaia directly called graph.py. It has a default dictionary (defdict.py) dependency that I made before Python came up with it on it's own (another place for refactoring). Cheers, MarkJ

6 6

Conventions for function annotations
by Thomas Kluyver 10 Dec '12

10 Dec '12

Function annotations (PEP 3107) are a very interesting new feature, but so far have gone largely unused. The only project I've seen using them is plac, a command-line option parser. One reason for this is that because function annotations can be used to mean anything, we're wary of doing anything in case we interfere with some other use case. A recent thread on ipython-dev touched on this [1], and we'd like to suggest some conventions to make annotations useful for everyone. 1. Code inspecting annotations should be prepared to ignore annotations it can't understand. 2. Code creating annotations should use wrapper classes to indicate what the annotation means. For instance, we are contemplating a way to specify options for a parameter, to be used in tab completion, so we would do something like this: from IPython.core.completer import options def my_io(filename, mode: options('read','write') ='read'): ... 3. There are a couple of important exceptions to 2: - Annotations that are simply a string can be used like a docstring, to be displayed to the user. Inspecting code should not expect to be able to parse any machine-readable information out of these strings. - Annotations that are a built-in type (int, str, etc.) indicate that the value should always be an instance of that type. Inspecting code may use these for type checking, introspection, optimisation, or other such purposes. Note that for now, I have limited this to built-in types, so other types can be used for other purposes, but this could be extended. For instance, the ABCs from collections (collections.Mapping et al.) could well be added to this category. 4. There should be a convention for attaching multiple annotations to one value. I propose that all code using annotations expects to handle tuples/lists of annotations. (We also considered dictionaries, but the result is long and ugly). So in this definition: def my_io(filename, mode: (options('read','write'), str, 'The mode in which to open the file') ='read'): ... the mode parameter has a set of options (ignored by frameworks that don't recognise it), should always be a string, and has a description. Any thoughts and suggestions are welcome. As an aside, we may also create a couple of decorators to fill in __annotations__ on Python 2, something like: @return_annotation('A file obect') @annotations(mode=(options('read','write'), str, 'The mode in which to open the file')) def my_io(filename, mode='read'): ... [1] http://mail.scipy.org/pipermail/ipython-dev/2012-November/010697.html Thanks, Thomas

4 4

Re: [Python-ideas] Conventions for function annotations
by Nick Coghlan 07 Dec '12

07 Dec '12

On Sun, Dec 2, 2012 at 4:26 PM, Robert McGibbon <rmcgibbo(a)gmail.com> wrote: > By being *tolerant and well behaved when confronted with annotations that > our library doesn't understand I*, I > think we can use function annotations without a short-range decorator that > translates their information in some other > structure. If other annotation-using libraries are also willing to ignore > our tabbing annotations if/when they encounter them, > then can't we all get along smoothly? > * > * > (For reference, the feature will look/work something like this) > > * > In[1]: def foo(filename : tab_glob('*.txt')): # tab completion that > recommends files/directories that match a glob pattern > ... pass > ... > In[2]: foo(<TAB> > 'a.txt' 'b.txt' > 'c.txt' 'dir/' > * > You're missing the other key reason for requiring decorators that interpret function annotations: they're there for the benefit of *readers*, not just other software. Given your definition above, I don't know what the annotations are for, except by recognising the "tab_glob" call. However, that then breaks as soon as the expression is put into a named variable earlier in the file: def foo(filename : text_files): # What does this mean? pass But the reader can be told *explicitly* what the annotations are related to via a decorator: @tab_expansion def foo(filename : text_files): # Oh, it's just a tab expansion specifier pass Readers no longer have to guess from context, and if the tab_expansion decorator creates IPython-specific metadata, then the interpreter doesn't need to guess either. (Note that you *can* use ordinary mechanisms like class decorators, metaclasses, post-creation modification of classes and IDE snippet inclusion to avoid the need to type out the "this is what these annotations mean" decorator explicitly. However, that's just an application of Python's standard abstraction tools, rather than a further special case convention) Mixing annotations intended for different consumers is a fundamentally bad idea, as it encourages unreadable code and complex dances to avoid stepping on each other's toes. It's better to design a *separate* API that supports composition by passing the per-parameter details directly to a decorator factory (which then adds appropriate named attributes to the function), with annotations used just as syntactic sugar for simple cases where no composition is involved. The important first question to ask is "How would we solve this if annotations didn't exist?" and only *then* look at the shorthand case for function-annotations. For cases where function annotations make code more complex or less robust, *don't use them*. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

17 35

A bind protocol (was Re: New __reference__ hook)
by Eric Snow 06 Dec '12

06 Dec '12

On Wed, Dec 5, 2012 at 11:09 AM, Sturla Molden <sturla(a)molden.no> wrote: > But apart from that, I think allowing overloading of the binding operator > "=" might be a good idea. A special method __bind__ could return the object > to be bound: > > a = b > > should then bind the name "a" to the return value of > > b.__bind__() > > if b implements __bind__. Keep in mind that descriptors already give you that for classes. There are other workarounds if you *really* have to have this functionality. You're right that globals (module body namespace) and locals (function body namespace) do not have that capability[1]. The main case I've heard for a generic "bind" protocol is for DRY. For instance, you create a new object with some name as an argument and then bind that object to that name in the current running namespace. This has been brought up before[2], with the canonical example of namedtuple (along with arguments on why it's not a big deal[3]). I'd expect such an API to look something like this: object.__bind__(name, namespace) object.__unbind__(name, namespace, replacement=None) namespace is the mapping for the locals/object (a.k.a. vars()) where the name is going to be bound. When an object is already bound to a name, __unbind__() would be called first on the current object. In that case, replacement would be the object that is replacing the currently bound one. At a high level the whole binding operation would look something like this: def bind(ns, name, obj): if name in ns: ns[name].__unbind__(name, ns, obj) obj.__bind__(name, ns) ns[name] = obj # or whatever If you wanted to get fancy, both methods could return a boolean indicating that the name should *not* be bound/unbound (respectively): def bind(ns, name, obj): if name in ns: if not ns[name].__unbind__(name, ns, obj): return if obj.__bind__(name, ns): ns[name] = obj # or whatever The bind protocol could also be used in the fallback behavior of augmented assignment operations. Ultimately, considering how often things are bound/unbound, I'd worry that it would be too expensive for any bind API to see the light of day. -eric [1] You *can* use your own module class to get it for "globals", sort of. This wouldn't quite work for the globals associated with functions defined in the module. [2] http://mail.python.org/pipermail/python-ideas/2011-March/009233.html, and others. [3] http://mail.python.org/pipermail/python-ideas/2011-March/009277.html

2 2

New __reference__ hook
by haael＠interia.pl 06 Dec '12

06 Dec '12

Hi, guys. Python 3 is very close to become a holy grail of programming languages in the sense that almost everything could be redefined. However, there is still one thing missing: the immutable copy-on-assign numeric types. Consider this part of code: a = 1 b = a a += 1 assert a == b + 1 The object "1" gets assigned to the "a" variable, then another independent copy gets assigned to the "b" variable, then the value in the "a" variable gets modified without affecting the second. The problem is - this behaviour can not be recreated in user-defined classes: a = MyInteger(1) b = a a += 1 assert a == b + 1 The "a" and "b" variables both point to the same object. This is a difference on what one might expect with numeric types. My proposal is to define another hook that gets called when an object is referenced. def MyInteger: def __reference__(self, context): return copy.copy(self) Each time when a reference count of an object would normally get incremented, this method should be called and the returned object will be referenced. The default implementation would be of course to return self. The context argument will give the object some information of the reason how we are being referenced. This will allow easy implementation of such concepts as singletons, copy-on-write, immutables and even simplify things like reference loops. The most obvious use-case would be implementations of some mathematical types like vectors, polynomials and so on. I've encountered this problem when I was writing a simple vector library and I had to explicitly copy each object on assignment, which is particulary annoying. The programmer's intuition for numeric-like types is for them to be immutable, yet they should have augmented asignment operators. This is a thing that can not be implemented transparently in the current Python, so there is my proposal. Cheers, haael.

14 15

Re: [Python-ideas] Conventions for function annotations
by Paul Moore 03 Dec '12

03 Dec '12

Sorry, should have gone to the list On 3 December 2012 09:30, Paul Moore <p.f.moore(a)gmail.com> wrote: > On 3 December 2012 09:21, Antoine Pitrou <solipsis(a)pitrou.net> wrote: >> In short, we have discovered that declarative typing isn't very >> useful :-) > > .. but haven't thought of any other useful applications of > annotations, and nor has the collective community on PyPI. > > Annotations seem like a solution looking for a problem, to me. (Which > is a shame, as they look like a pretty cool solution) > Paul

3 2

WSAPoll and tulip
by Trent Nelson 03 Dec '12

03 Dec '12

A weekend or two ago, I was planning on doing some work on some ideas I had regarding IOCP and the tulip/async-IO discussion. I ended up getting distracted by WSAPoll. WSAPoll is a method that Microsoft introduced with Vista/2008 that is intended to be semantically equivalent to poll() on UNIX. I decided to play around and see what it would take to get it available via select.poll() on Windows, eventually hacking it into a working state. Issue: http://bugs.python.org/issue16507 Patch: http://bugs.python.org/file28038/wsapoll.patch So, it basically works. poll() on Windows, who would have thought. It's almost impossible to test with our current infrastructure; all our unit tests seem to pass pipes and other non-Winsock-backed-socks to poll(), which, like select()-on-Windows, isn't supported. I suspect Twisted's test suite would give it a much better work out (CC'd Glyph just so it's on his radar). I ended up having to verify it worked with some admittedly-hacky dual-python-console sessions, one poll()'ing as a server, the other connecting as a client. It definitely works, so, it's worth keeping it in mind for the future. It's still equivalent to poll()'s O(N) on UNIX, but that's better than the 64/512 limit select is currently stuck with on Windows. Didn't have much luck trying to get the patched Python working with tulip's PollReactor, unfortunately, so I just wanted to provide some feedback on that experience. First bit of feedback: man, debugging `yield from` stuff is *hard*. Tulip just flat out didn't work with the PollReactor from the start but it was dying in a non-obvious way. So, I attached both a Pdb debugger and Visual Studio debugger and tried to step through everything to figure out why the first call to poll() was blowing up (can't remember the exact error message but it was along the lines of "you can't poll() whatever it is you just asked me to poll(), it's defo' not a socket"). I eventually, almost by pure luck, traced the problem to the fact that PollReactor's __init__ eventually results in code being called that calls poll() on two os.pipe() objects (in EventLoop I think). However, when I was looking at the code, it appeared as though the first poll() came from the getaddrinfo(). So all my breakpoints and whatnot were geared towards that, yet none of them were being hit, yet poll() was still being called somehow, somewhere. I ended up having to spend ages traipsing through every line in Visual Studio's debugger to try figure out what the heck was going on. I believe the `yield from` aspect made that so much more of an arduous affair -- one moment I'm in selectmodule.c's getaddrinfo(), then I'm suddenly deep in the bowels of some cryptic eval frame black magic, then one 'step' later, I'm over in some completely different part of selectmodule.c, and so on. I think the reason I found it so tough was because when you're single stepping through each line of a C program, you can sort of always rely on the fact you know what's going to happen when you "step" the next line. In this case though, a step of an eval frame would wildly jump to seemingly unrelated parts of C code. As far as I could tell, there was no easy/obvious way to figure the details out before stepping that instruction either (i.e. probing the various locals and whatnot). So, that's the main feedback from that weekend, I guess. Granted, it's more of a commentary on `yield from` than tulip per se, but I figured it would be worth offering up my experience nevertheless. I ended up with the following patch to avoid the initial poll() against os.pipe() objects: --- a/polling.py Sat Nov 03 13:54:14 2012 -0700 +++ b/polling.py Tue Nov 27 07:05:10 2012 -0500 @@ -41,6 +41,7 @@ import os import select import time +import sys class PollsterBase: @@ -459,6 +460,10 @@ """ def __init__(self, eventloop, executor=None): + if sys.platform == 'win32': + # Work around the fact that we can't poll pipes on Windows. + if isinstance(eventloop.pollster, PollPollster): + eventloop = EventLoop(SelectPollster()) self.eventloop = eventloop self.executor = executor # Will be constructed lazily. self.pipe_read_fd, self.pipe_write_fd = os.pipe() By that stage it was pretty late in the day and I accepted defeat. My patch didn't really work, it just allowed the test to run to completion without the poll OSError exception being raised. Trent.

7 14

An async facade? (was Re: [Python-Dev] Socket timeout and completion based sockets)
by Trent Nelson 01 Dec '12

01 Dec '12

[ It's tough coming up with unique subjects for these async discussions. I've dropped python-dev and cc'd python-ideas instead as the stuff below follows on from the recent msgs. ] TL;DR version: Provide an async interface that is implicitly asynchronous; all calls return immediately, callbacks are used to handle success/error/timeout. class async: def accept(): def read(): def write(): def getaddrinfo(): def submit_work(): How the asynchronicity (not a word, I know) is achieved is an implementation detail, and will differ for each platform. (Windows will be able to leverage all its async APIs to full extent, Linux et al can keep mimicking asynchronicity via the usual non-blocking + multiplexing (poll/kqueue etc), thread pools, etc.) On Wed, Nov 28, 2012 at 11:15:07AM -0800, Glyph wrote: > On Nov 28, 2012, at 12:04 PM, Guido van Rossum <guido(a)python.org> wrote: > I would also like to bring up <https://github.com/lvh/async-pep> again. So, I spent yesterday working on the IOCP/async stuff. The saw this PEP and the sample async/abstract.py. That got me thinking: why don't we have a low-level async facade/API? Something where all calls are implicitly asynchronous. On systems with extensive support for asynchronous 'stuff', primarily Windows and AIX/Solaris to a lesser extent, we'd be able to leverage the platform-provided async facilities to full effect. On other platforms, we'd fake it, just like we do now, with select, poll/epoll, kqueue and non-blocking sockets. Consider the following: class Callback: __slots__ = [ 'success', 'failure', 'timeout', 'cancel', ] class AsyncEngine: def getaddrinfo(host, port, ..., cb): ... def getaddrinfo_then_connect(.., callbacks=(cb1, cb2)) ... def accept(sock, cb): ... def accept_then_write(sock, buf, (cb1, cb2)): ... def accept_then_expect_line(sock, line, (cb1, cb2)): ... def accept_then_expect_multiline_regex(sock, regex, cb): ... def read_until(fd_or_sock, bytes, cb): ... def read_all(fd_or_sock, cb): return self.read_until(fd_or_sock, EOF, cb) def read_until_lineglob(fd_or_sock, cb): ... def read_until_regex(fd_or_sock, cb): ... def read_chunk(fd_or_sock, chunk_size, cb): ... def write(fd_or_sock, buf, cb): ... def write_then_expect_line(fd_or_sock, buf, (cb1, cb2)): ... def connect_then_expect_line(..): ... def connect_then_write_line(..): ... def submit_work(callable, cb): ... def run_once(..): """Run the event loop once.""" def run(..): """Keep running the event loop until exit.""" All methods always take at least one callback. Chained methods can take multiple callbacks (i.e. accept_then_expect_line()). You fill in the success, failure (both callables) and timeout (an int) slots. The engine will populate cb.cancel with a callable that you can call at any time to (try and) cancel the IO operation. (How quickly that works depends on the underlying implementation.) I like this approach for two reasons: a) it allows platforms with great async support to work at their full potential, and b) it doesn't leak implementation details like non-blocking sockets, fds, multiplexing (poll/kqueue/select, IOCP, etc). Those are all details that are taken care of by the underlying implementation. getaddrinfo is a good example here. Guido, in tulip, you have this implemented as: def getaddrinfo(host, port, af=0, socktype=0, proto=0): infos = yield from scheduling.call_in_thread( socket.getaddrinfo, host, port, af, socktype, proto ) That's very implementation specific. It assumes the only way to perform an async getaddrinfo is by calling it from a separate thread. On Windows, there's native support for async getaddrinfo(), which we wouldn't be able to leverage here. The biggest benefit is that no assumption is made as to how the asynchronicity is achieved. Note that I didn't mention IOCP or kqueue or epoll once. Those are all implementation details that the writer of an asynchronous Python app doesn't need to care about. Thoughts? Trent.

7 12