Mailman 3 December 2012 - Python-ideas

async: feedback on EventLoop API
by Shane Green 22 Dec '12

22 Dec '12

Ignoring the API overlap with Futures/Promises for a moment, let me throw out this straw man approach to the event loop that seems to my naive eye like it pull together a lot of these ideas… Rather than passing in your callbacks, factories, etc., asynchronous APIs return a lightweight object you register your callback with. Unlike promises, deferrers, etc., this is a one-time thing: only one callback can register with it. However, it can be chained. The registered callback is invoked with the output of the operation when it completes. Timer.wait(20).then(callme, *args, **kw) # I could do Timer.wait(20).then(callme, *args, **kw).then(piped_from_callme) #I could not do handler = Timer.wait(20) handler.then(callme) handler.then(callme2) # this would throw an exception. # I/O example… sock.accept().then(handle_connection) # invokes handle_connection(conn, addr) # Read some data conn.read(1024).then(handle_incoming) # handle_incoming invoked with up to 1024 bytes, read asynchronously. # Write some data conn.write("data").then(handle_written) # handle_written invoked with up number 5, giving number of bytes written async. # Connect HTTP channel and add it to HTTP dispatcher. channel.connect((hostname,80)).then(dispatcher.add_channel) # Listen to FD's for I/O events descriptors.select(r, w, e).then(handle) # handle(readable, writables, oobs) It seems like only supporting a single callback per returned handle lets us circumvent a lot of the weight associated with normal promise/future/deferred pattern type implementations, but the chaining could come in handy as it may cover some of the use-cases being considered when multiple events per fd came up, plus chaining is pretty powerful, especially when it comes at little cost. The API would be much more extensive than "then()", of course, with things like "every", etc. we'd have to pull examples from everything already discussed. Just wanted to throw out there to get beat up about ;-) Shane Green www.umbrellacode.com 805-452-9666 | shane(a)umbrellacode.com

3 15

PEP 3156 feedback
by Antoine Pitrou 20 Dec '12

20 Dec '12

Hello, Here is my own feedback on the in-progress PEP 3156. Please discard it if it's too early to give feedback :-)) Event loop API -------------- I would like to say that I prefer Tornado's model: for each primitive provided by Tornado, you can pass an explicit Loop instance which you instantiated manually. There is no module function or policy object hiding this mechanism: it's simple, explicit and flexible (in other words: if you want a per-thread event loop, just do it yourself using TLS :-)). There are some requirements I've found useful: - being able to instantiate multiple loops, either at the same time or serially (this is especially nice for unit tests; Twisted has to use a dedicated test runner just because their reactor doesn't support multiple instances or restarts) - being able to stop a loop explicitly: having to unregister all handlers or delayed calls is a PITA in non-trivial situations (for example you might have multiple protocol instances, each with a bunch of timers, some perhaps even in third-party libraries; keeping track of all this is the event loop's job) * The optional sock_*() methods: how about having different ABCs, e.g. the EventLoop ABC for basic behaviour, and the NetworkedEventLoop ABC adding the socket helpers? Protocols and transports ------------------------ We probably want to provide a Protocol base class and encourage people to inherit it. It can provide useful functionality (perhaps write() and writelines() shims? it can make mocking easier). My own opinion about Twisted's API is that the Factory class is often useless, and adds a cognitive burden. If you need a place to track all protocols of a given kind (e.g. all connections), you can do it yourself. Also, the Factory implies that you don't control how exactly your protocol gets instantiated (unless you override some method on the Factory I'm missing the name of: it is cumbersome). So, when creating a client, I would pass it a protocol instance. When creating a server, I would pass it a protocol class. Here the base Protocol class comes into play, its __init__() could take the transport as argument and set the "transport" attribute with it. Further args could be optionally passed to the constructor: class MyProtocol(Protocol): def __init__(self, transport, my_personal_attribute): Protocol.__init__(self, transport) self.my_personal_attribute = my_personal_attribute ... def listen(ioloop): # Each new connection will instantiate a MyProtocol with "foobar" # for my_personal_attribute. ioloop.listen_tcp(("0.0.0.0", 8080), MyProtocol, "foobar") (The hypothetical listen_tcp() is just a name: perhaps it's actually start_serving(). It should accept any callable, not just a class: therefore, you can define complex behaviour if you like) I think the transport / protocol registration must be done early, not in connection_made(). Sometimes you will want to do things on a protocol before you know a connection is established, for example queue things to write on the transport. An use case is a reconnecting TCP client: the protocol will continue existing at times when the connection is down. Unconnected protocols need their own base class and API: data_received()'s signature should be (data, remote_addr) or (remote_addr, data). Same for write(). * writelines() sounds ambiguous for datagram protocols: does it send those "lines" as a single datagram, or one separate datagram per "line"? The equivalent code suggests the latter, but which one makes more sense? * connection_lost(): you definitely want to know whether it's you or the other end who closed the connection. Typically, if the other end closed the connection, you will have to run some cleanup steps, and perhaps even log an error somewhere (if the connection was closed unexpectedly). Actually, I'm not sure it's useful to call connection_lost() when you closed the connection yourself: are there any use cases? Regards Antoine.

5 13

Re: [Python-ideas] context aware execution
by Bart Thate 20 Dec '12

20 Dec '12

Thanks for your response Chris ! Ha ! the job of the mad man is todo the things the are not "advisable" and see what gives. Like why it is not advisable and, if possible, their are ways to achieve things that are previously overseen. i already do a lot of travelling of the callstack to see from where a function is called. mostely for logging purposes, like what plugin registered this callback etc. lately i also had the need to log the variable name of a object, and the thought of be able to "break out of the namespace" concept got me thinking what i am thinking of is code that can examine the context it is run in. The object, when called can figure out in what kind of space it is living in and discover what kind of API other objects in the space offer. This is a pre function that gets called before the actual function/method and can thus determine if a certain request can be fullfilled. I bet a decorator could be made, in which i can assign certain caller context references into variables in the function/method ? I use 1 generic parameter object, in which i can stuff lots of things, but i rather have the function be able to see what is around ;] Think of sending JSON over the wire, reconstruct an object with it and then let the object figure out what it can and cannot do in this external environment. Code i use now is this: # life/plugs/context.py # # """ show context. """ ## basic import import sys ## context command def context(event): result = [] frame = sys._getframe() code = frame.f_back.f_code for i in dir(code): print("%s => %s" % (i, getattr(code, i))) del frame context.cmnd = "context" So much to explore ;]

1 0

Re: [Python-ideas] Tree as a data structure (Was: Graph class)
by Jeff Jenkins 20 Dec '12

20 Dec '12

trying again, this email address was apparently not on the list: My experience dealing with trees is always that the "tree" part is always so simple that it isn't a big deal to re-implement it. The problem is dealing with all of the extra stuff that you need and the details of what happens when you do different operations. I think it makes more sense to have an interface for a kind of thing you want to do with a tree (e.g. sorted sets, or ordered maps) rather than the tree itself. On Wed, Dec 19, 2012 at 10:55 AM, Jeff Jenkins <jeff(a)jeffreyjenkins.ca>wrote: > My experience dealing with trees is always that the "tree" part is always > so simple that it isn't a big deal to re-implement it. The problem is > dealing with all of the extra stuff that you need and the details of what > happens when you do different operations. I think it makes more sense to > have an interface for a kind of thing you want to do with a tree (e.g. > sorted sets, or ordered maps) rather than the tree itself. > > > On Wed, Dec 19, 2012 at 10:11 AM, anatoly techtonik <techtonik(a)gmail.com>wrote: > >> On Sun, Dec 16, 2012 at 6:41 PM, Guido van Rossum <guido(a)python.org>wrote: >> >>> I think of graphs and trees as patterns, not data structures. >> >> >> In my world strings, ints and lists are 1D data types, and tree can be a >> very important 2D data structure. Even if it is a pattern, this pattern is >> vital for the transformation of structured data, because it allows to >> represent any data structure in canonical format. >> >> Speaking of tree as a data structure, I assume that it has a very basic >> definition: >> >> 1. tree consists of nodes >> 2. some nodes are containers for other nodes >> 3. every node has properties >> 4. every node has 0 or 1 parent >> 5. every container has 1+ children >> 6. tree has a single starting root node >> 7. no child of a parent can be its ancestor >> (no cyclic dependencies between elements) >> >> List of trees is a forest. Every subtree is a complete tree. >> >> >> To see which tree data type would be really useful in Python distribution >> (e.g. provides a simple, extendable and intuitive interface), I see only >> one way - is to scratching some itches relevant to Python and then try to >> scale it to other problems. The outcome should be the answer - >> what for native tree type is not suitable? >> >> More ideas: >> [ ] Much experience for working with trees can be brought from XML and >> DOM manipulation practices (jQuery and friends) >> [ ] every element in a tree can be accessed by its address specificator >> as 'root/node[3]/last' >> [ ] but it is also convenient to access tree data using node names as >> 'mylib.books[:1]' >> [ ] and of course, you can run queries over trees >> >> [ ] Tree is the base for any "Data Transformation Framework" as it allows >> to jump from "data type conversion" to "data structure conversion and >> mapping" >> [ ] Trees can be converted to other trees and to more complicated >> structures >> [ ] Conversion can be symmetrical and non-symmetrical >> [ ] Conversion can be lossy and lossless >> [ ] Conversion can be lossless and non-symmetrical at the same time >> >> Trees can be used, for example, for realtime migration of issues from one >> tracker to another. For managing changesets with additional meta >> information. For presenting package dependencies and working with them. For >> atomic (transactional) file management. For managing operating system >> capability information. For logging setup. For debugging structures in >> Python. For working and converting binary file formats. For the common AST >> transformation and query interface. For the understanding how 2to3 fixers >> work. For the common ground of visual representation, comparison and >> transformation of data structures. That's probably enough of my itches. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> >> >

1 0

async: feedback on EventLoop API
by Geert Jansen 19 Dec '12

19 Dec '12

Hi, below is some feedback on the EventLoop API as implemented in tulip. I am interested in this for an (alternate) dbus interface that I've written for Python that supports evented IO. I'm hoping tulip's EventLoop could be an abstraction as well as a default implementation that allows me to support just one event interface. I looked at it from two angles: 1. Does EventLoop provide everything that is needed from a library writer point of view? 2. Can EventLoop efficiently expose a subset of the functionality of some of the main event loop implementations out there today (i looked at libuv, libev and Qt). First some code pointers... * https://github.com/geertj/looping - Here i've implemented the EventLoop interface for libuv, libev and Qt. It includes a slightly modified version of tulip's "polling.py" where I've implemented some of the suggestions below. It also adds support for Python 2.6/2.7 as the Python Qt interface (PySide) doesn't support Python 3 yet. * https://github.com/geertj/python-dbusx - A Python interface for libdbus that supports evented IO using an EventLoop interface. This module is also tests all the different loops from "looping" by doing D-BUS tests with them (looping itself doesn't have tests yet). My main points of feedback are below: * It would be nice to have repeatable timers. Repeatable timers are expected for example by libdbus when integrating it with an event loop. Without repeatable timers, I could emulate a repeatable timer by using call_later() and adding a new timer every time the timer fires. This would be an inefficient interface though for event loops that natively support repeatable timers. This could possibly be done by adding a "repeat" argument to call_later(). * It would be nice to be a way to call a callback once per loop iteration. An example here is dispatching in libdbus. The easiest way to do this is to call dbus_connection_dispatch() every iteration of the loop (a more complicated way exists to get notifications when the dispatch status changes, but it is edge triggered and difficult to get right). This could possibly be implemented by adding a "repeat" argument to call_soon(). * A useful semantic for run_once() would be to run the callbacks for readers and writers in the same iteration as when the FD got ready. This allows for the idiom below when expecting a single event to happen on a file descriptor from outside the event loop: # handle_read() sets the "ready" flag loop.add_reader(fd, handle_read) while not ready: loop.run_once() I use this idiom for example in a blocking method_call() method that calls into a D-BUS method. Currently, the handle_read() callback would be called in the iteration *after* the FD became readable. So this would not work, unless some more IO becomes available. As far as I can see libev, libuv and Qt all work like this. * If remove_reader() / remove_writer() would accept the DelayedCall instance returned by their add_xxx() cousins, then that would allow for multiple callbacks per FD. Not all event loops support this (libuv doesn't, libev and Qt do), but for the ones that do could have their functionality could be exposed like this. For event loops that don't support this, an exception could be raised when adding multiple callbacks per FD. Support for multiple callbacks per FD could be advertised as a capability. * After a DelayedCall is cancelled, it would also be very useful to have a second method to enable it again. Having that functionality is more efficient than creating a new event. For example, the D-BUS event loop integration API has specific methods for toggling events on and off that you need to provide. * (Nitpick) Multiplexing absolute and relative timeouts for the "when" argument in call_later() is a little too smart in my view and can lead to bugs. With some input, I'd be happy to produce patches. Regards, Geert Jansen

9 26

Fwd: Graph class
by Mark Adam 19 Dec '12

19 Dec '12

Meant this to go to the whole list. Sorry. On Sun, Dec 9, 2012 at 5:40 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > On 9 December 2012 01:29, Mark Adam <dreamingforward(a)gmail.com> wrote: >> All very interesting. I'm going to suggest a sort of "meta-discussion" >> about why -- despite the power of graphs as a data structure -- such a >> feature has not stabilized into a workable solution for inclusion in a >> high-level language like Python. >> >> I identity the following points of "wavery": >> >> 1) the naming of methods (add_edge, vs add(1,2)): aesthetic grounds, >> 2) what methods to include (degree + neighbors or the standard dict's >> __len__ + __getitem__): API grounds >> 3) how much flexibility to be offered (directed, multi-graphs, edge weights >> with arbitrary labeling, etc.): functionality grounds >> 4) what underlying data structure to use (sparse adjacency dicts, matrices, >> etc): representation conflicts. > > 4) Whether the library requires some sort of "Vertex" type, or works > with arbitrary values, similarly whether there is a defined "Edge" > class or edges can be labelled, weighted, etc with arbitrary Python > values. This I put under #3 (functionality grounds) "edge weights with arbitrary labeling", Vertex's with abitrary values i think would be included. > 5) Granularity - if all I want is a depth-first search algorithm, why > pull in a dependency on 100 graph algorithms I'm not interested in? Hmm, I would call this "5) comprehensiveness: whether to include every graph algorithm known to mankind." > My feeling is that graphs are right on the borderline of a data > structure that is simple enough that people invent their own rather > than bother conforming to a "standard" model but complex enough that > it's worth using library functions rather than getting the details > wrong. But this is also why (on both counts) it would be good to include it in the standard library. The *simplicity* of a graph makes everyone re-implement it, or (worse) work with some cruder work-around (like a dict of dicts but not at all clear you're dealing with an actual graph). But imagine if, for example, Subversion used a python graph class to track all branches and nodes in it's distributed revision control system. Then how easy it would be for third parties to make tools to view repos or other developers to come in and work with the dev team: they're already familiar with the standard graph class structure. And for the *complex enough* case, obviously it helps to have a standard library help you out and just provide the sophistication of a graph class for you. There are a lot of obvious uses for a graph, but if you don't know of it, a beginning programmer won't *think* of it and make some crude work-around. Mark

9 12

Re: [Python-ideas] Python-ideas Digest, Vol 73, Issue 38
by Shane Green 18 Dec '12

18 Dec '12

Sending the demultiplexed data through 15 pipes so the application actually is dealing with 15 streams of data using single callback notifications from the event loop seems like the more KISS approach, in this case… Shane Green www.umbrellacode.com 805-452-9666 | shane(a)umbrellacode.com On Dec 17, 2012, at 11:21 PM, python-ideas-request(a)python.org wrote: > Send Python-ideas mailing list submissions to > python-ideas(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request(a)python.org > > You can reach the person managing the list at > python-ideas-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > Today's Topics: > > 1. Re: Graph class (Nick Coghlan) > 2. Re: async: feedback on EventLoop API (Guido van Rossum) > 3. Re: async: feedback on EventLoop API (Nick Coghlan) > > From: Nick Coghlan <ncoghlan(a)gmail.com> > Subject: Re: [Python-ideas] Graph class > Date: December 17, 2012 7:26:38 PM PST > To: Hannu Krosing <hannu(a)krosing.net> > Cc: Vinay Sajip <vinay_sajip(a)yahoo.co.uk>, "python-ideas(a)python.org" <python-ideas(a)python.org> > > > On Mon, Dec 17, 2012 at 9:28 AM, Hannu Krosing <hannu(a)krosing.net> wrote: > On 12/16/2012 04:41 PM, Guido van Rossum wrote: >> I think of graphs and trees as patterns, not data structures. > > How do you draw line between what is data structure and what is pattern ? > > A rough rule of thumb is that if it's harder to remember the configuration options in the API than it is to just write a purpose-specific function, it's probably better as a pattern that can be tweaked for a given use case than it is as an actual data structure. > > More generally, ABCs and magic methods are used to express patterns (like iteration), which may be implemented by various data structures. > > A graph library that focused on defining a good abstraction (and adapters) that allowed graph algorithms to be written that worked with multiple existing Python graph data structures could be quite interesting. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia > > > > From: Guido van Rossum <guido(a)python.org> > Subject: Re: [Python-ideas] async: feedback on EventLoop API > Date: December 17, 2012 8:01:18 PM PST > To: Nick Coghlan <ncoghlan(a)gmail.com>, Antoine Pitrou <solipsis(a)pitrou.net> > Cc: python-ideas(a)python.org > > > On Mon, Dec 17, 2012 at 7:20 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> On Tue, Dec 18, 2012 at 10:40 AM, Guido van Rossum <guido(a)python.org> wrote: > > [A better name for DelayedCall] >>> Anyway, Handler sounds like a pretty good name. Let me think it over. > >> Is DelayedCall a subclass of Future, like Task? If so, FutureCall might >> work. > > No, they're completely related. (I'm even thinking of renaming its > cancel() to avoid the confusion? > > I still like Handler best. In fact, if I'd thought of Handler before, > I wouldn't have asked for a better name. :-) > > Going once, going twice... > > [Wall-clock timers] >> If someone really does want a wall-clock timer with a given granularity, it >> can be handled by adding a repeating timer with that granularity (with the >> obvious consequences for low power modes). > > +1. > > [Multiple calls per FD] >>> That makes sense. If we wanted to be fancy we could have several >>> different APIs: add (must not be set), set (may be set), replace (must >>> be set). But I think just offering the add and remove APIs is nicely >>> minimalistic and lets you do everything else with ease. (I'll make the >>> remove API return True if it did remove something, False otherwise.) > >> Perhaps the best bet would be to have the standard API allow multiple >> callbacks, and emulate that on systems which don't natively support multiple >> callbacks for a single event? > > Hm. AFAIK Twisted doesn't support this either. Antoine, do you know? I > didn't see it in the Tornado event loop either. > >> Otherwise, I don't see how an event loop could efficiently expose access to >> the multiple callback APIs without requiring awkward fallbacks in the code >> interacting with the event loop. Given that the natural fallback >> implementation is reasonably clear (i.e. a single callback that calls all of >> the other callbacks), why force reimplementing that on users rather than >> event loop authors? > > But what's the use case? > > I don't think our goal should be to offer APIs for any feature that > any event loop might offer. It's not quite a least-common denominator > either though -- it's about offering commonly needed functionality, > and interoperability. > > Also, event loop implementations are allowed to offer additional APIs > on their implementation. If the need for multiple handlers per FD only > exists on those platforms where the platform's event loop supports it, > no harm is done if the functionality is only available through a > platform-specific API. > > But still, I don't understand the use case. Possibly it is using file > descriptors as a more general signaling mechanism? That sounds pretty > platform specific anyway (on Windows, FDs must represent sockets). > > If someone shows me a real-world use case I may change my mind. > >> Related, the protocol/transport API design may end up needing to consider >> the gather/scatter problem (i.e. fanning out data from a single transport to >> multiple consumers, as well as feeding data from multiple producers into a >> single underlying transport). Actual *implementations* of such tools >> shouldn't be needed in the standard suite, but at least understanding how >> you would go about writing multiplexers and demultiplexers can be a good >> test of a stacked I/O design. > > Twisted supports this for writing through its writeSequence(), which > appears in Tulip and PEP 3156 as writelines(). (Though IIRC Glyph told > me that Twisted rarely uses the platform's scatter/gather primitives, > because they are so damn hard to use, and the kernel implementation > often just joins the buffers together before passing it to the regular > send()...) > > But regardless, I don't think scatter/gather would use multiple > callbacks per FD. > > I think it would be really hard to benefit from reading into multiple > buffers in Python. > >>>> Just enabling/disabling these events is a bit more friendly to the >>>> programmer IMHO than having to cancel and recreate them when needed. >>> >>> The methods on the Transport class take care of this at a higher >>> level: pause() and resume() to suspend reading, and the write() method >>> takes care of buffering and so on. > >> And the main advantage of handling that at a higher level is that suitable >> buffering designs are going to be transport specific. > > +1 > > -- > --Guido van Rossum (python.org/~guido) > > > > > From: Nick Coghlan <ncoghlan(a)gmail.com> > Subject: Re: [Python-ideas] async: feedback on EventLoop API > Date: December 17, 2012 11:21:37 PM PST > To: Guido van Rossum <guido(a)python.org> > Cc: Antoine Pitrou <solipsis(a)pitrou.net>, python-ideas(a)python.org > > > On Tue, Dec 18, 2012 at 2:01 PM, Guido van Rossum <guido(a)python.org> wrote: > On Mon, Dec 17, 2012 at 7:20 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote:Also, event loop implementations are allowed to offer additional APIs > on their implementation. If the need for multiple handlers per FD only > exists on those platforms where the platform's event loop supports it, > no harm is done if the functionality is only available through a > platform-specific API. > > Sure, but since we know this capability is offered by multiple event loops, it would be good if there was a defined way to go about exposing it. > > But still, I don't understand the use case. Possibly it is using file > descriptors as a more general signaling mechanism? That sounds pretty > platform specific anyway (on Windows, FDs must represent sockets). > > If someone shows me a real-world use case I may change my mind. > > The most likely use case that comes to mind is monitoring and debugging (i.e. the event loop equivalent of a sys.settrace). Being able to tap into a datastream (e.g. to dump it to a console or pipe it to a monitoring process) can be really powerful, and being able to do it at the Python level means you have this kind of capability even without root access to the machine to run Wireshark. > > There are other more obscure signal analysis use cases that occur to me, but those could readily be handled with a custom transport implementation that duplicated that data stream, so I don't think there's any reason to worry about those. > > > Related, the protocol/transport API design may end up needing to consider > > the gather/scatter problem (i.e. fanning out data from a single transport to > > multiple consumers, as well as feeding data from multiple producers into a > > single underlying transport). Actual *implementations* of such tools > > shouldn't be needed in the standard suite, but at least understanding how > > you would go about writing multiplexers and demultiplexers can be a good > > test of a stacked I/O design. > > Twisted supports this for writing through its writeSequence(), which > appears in Tulip and PEP 3156 as writelines(). (Though IIRC Glyph told > me that Twisted rarely uses the platform's scatter/gather primitives, > because they are so damn hard to use, and the kernel implementation > often just joins the buffers together before passing it to the regular > send()...) > > But regardless, I don't think scatter/gather would use multiple > callbacks per FD. > > I think it would be really hard to benefit from reading into multiple > buffers in Python. > > Sorry, I wasn't quite clear on what I meant by gather/scatter and it's more a protocol thing than an event loop thing. > > Specifically, gather/scatter interfaces are most useful for multiplexed transports. The ones I'm particularly familiar with are traditional telephony transports like E1 links, with 15 time-division-multiplexed channels on the wire (and a signalling timeslot), as well a few different HF comms protocols. When reading from one of those, you have a demultiplexing component which is reading the serial data coming in on the wire and making it look like 15 distinct data channels from the application's point of view. Similarly, the output multiplexer takes 15 streams of data from the application and interleaves them into the single stream on the wire. > > The rise of packet switching means that sharing connections like that is increasingly less common, though, so gather/scatter devices are correspondingly less useful in a networking context. The only modern use cases I can think of that someone might want to handle with Python are things like sharing a single USB or classic serial connection amongst multiple data streams. However, I suspect the standard transport and protocol API definitions already proposed should also suffice for the gather/scatter use case, as such a component would largely work like any other protocol-as-transport adapter, with the difference being that there would be a many-to-one relationship between the number of interfaces on the application side and those on the communications side. > > (Technically, gather/scatter components can also be used the other way around to distribute a single data stream across multi transports, but that use case is even less likely to come up when programming in Python. Multi-channel HF data comms is the only possibility that really comes to mind) > > -- > Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia > > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas

1 1

Late to the async party (PEP 3156)
by Jason Tackaberry 18 Dec '12

18 Dec '12

Hi python-ideas, I've been somewhat living under a rock for the past few months and consequently I missed the ideal window of opportunity to weigh in on the async discussions this fall that culminated into PEP 3156. I've been reading through those discussions in the archives. I've not finished digesting it all, and I'm somewhat torn in that I feel I should shut up until I read everything to date so as not to decrease the SNR, but on the other hand, knowing myself, I strongly suspect this would result in my never speaking up. And so, at risk of lowering the SNR ... First let me say that PEP 3156 makes me very, very happy. Over the past few years I've been exploring these very ideas with a little-used library called Kaa. I'm not offering it up as a paragon of proper async library design, but I wanted to share some of my experiences in case they could be useful to the PEP. https://github.com/freevo/kaa-base/ http://api.freevo.org/kaa-base/ It does seem like many similar design choices were made. In particular, I'm happy that an explicit yield will be used rather than the greenlet style of implicit suspension/reentry. Even after I've been using them for years, coroutines often feel like a form of magic, and an explicit yield is more aligned with the principle of least surprise. With Kaa, our future-style object is called an InProgress (so forgive the differing terminology in the remainder of this post): http://api.freevo.org/kaa-base/async/inprogress.html A couple properties of InProgress objects that I've found have practical value: * they can be aborted, which raises a special InProgressAborted inside the coroutine function so it can perform cleanup actions o what makes this tricky is the question of what to do to any currently yielded tasks? If A yields B and A is aborted, should B be aborted? What if the same B task is being yielded by C? Should C also be aborted, even if it's considered a sibling of A? (For example, suppose B is a task that is refreshing some common cache that both A and C want to make sure is up-to-date before they move on.) o if the decision is B should be aborted, then within A, 'yield B' will raise an exception because A is aborted, but 'yield B' within C will raise because B was aborted. So there needs to be some mechanism to distinguish between these cases. (My approach was to have an origin attribute on the exception.) o if A yields B, it may want to prevent B from being aborted if A is aborted. (My approach was to have a noabort() method in InProgress objects to return a new, unabortable InProgress object that A can then yield.) o alternatively, the saner implementation may be to do nothing to B when A is aborted and require A catch InProgressAborted and explicitly abort B if that's the desired behaviour o discussion in the PEP on cancellation has some TBDs so perhaps the above will be food for thought * they have a timeout() method, which returns a new InProgress object representing the task that will abort when the timeout elapses if the task doesn't finish o it's noteworthy that timeout() returns a /new/ InProgress and the original task continues on even if the timeout occurs -- by default that is, unless you do timeout(abort=True) o I didn't see much discussion in the PEP on timeouts, but I think this is an important feature that should be standardized Coroutines in Kaa use "yield" rather than "yield from" but the general approach looks very similar to what's been proposed: http://api.freevo.org/kaa-base/async/coroutines.html The @coroutine decorator causes the decorated function to return an InProgress. Coroutines can of course yield other coroutines, but, more fundamentally, anything else that returns an InProgress object, which could be a @threaded function, or even an ordinary function that explicitly creates and returns an InProgress object. There are some features of Kaa's implementation that could be worth considering: * it is possible to yield a special object (called NotFinished) that allows a coroutine to "time slice" as a form of cooperative multitasking * coroutines can have certain policies that control invocation behaviour. The most obvious ones to describe are POLICY_SYNCHRONIZED which ensures that multiple invocations of the same coroutine are serialized, and POLICY_SINGLETON which effectively ignores subsequent invocations if it's already running * it is possible to have a special progress object passed into the coroutine function so that the coroutine's progress can be communicated to an outside observer Once you've standardized on a way to manage the lifecycle of an in-progress asynchronous task, threads are a natural extension: http://api.freevo.org/kaa-base/async/threads.html The important element here is that @threaded decorated functions can be yielded by coroutines. This means that truly blocking tasks can be wrapped in a thread but invocation from a coroutine is identical to any other coroutine. Consequently, a threaded task could later be implemented as a coroutine (or more generally via event loop hooks) without any API changes. I think I'll stop here. There's plenty more definition, discussion, and examples in the links above. Hopefully some ideas can be salvaged for PEP 3156, but even if that's not the case, I'll be happy to know they were considered and rejected rather than not considered at all. Cheers, Jason.

5 8

Docstrings for namedtuple
by Serhiy Storchaka 17 Dec '12

17 Dec '12

What interface is better for specifying namedtuple field docstrings? Point = namedtuple('Point', 'x y', doc='Point: 2-dimensional coordinate', field_docs=['abscissa', 'ordinate']) or Point = namedtuple('Point', [('x', 'absciss'), ('y', 'ordinate')], doc='Point: 2-dimensional coordinate') ? http://bugs.python.org/issue16669

9 16

Python is not perfect - let's add 'Wart' status to track
by anatoly techtonik 12 Dec '12

12 Dec '12

I want to query all warts for specific Python 2.x versions to see how are they fixed in 3.x. Right now these warts are hidden beneath the "invalid" labels, which IMHO does as much damage to the language development as BC breaks. How about adding 'Wart' resolution to the closed status on tracker? -- anatoly t.

2 1