Mailman 3 February 2013 - Python-ideas

libuv based eventloop for tulip experiment
by Saúl Ibarra Corretgé 08 Feb '13

08 Feb '13

Hi all! I haven't been able to keep up with all the tulip development on the mailing list (hopefully I will now!) so please excuse me if something I mention has already been discussed. For those who may not know it, libuv is the platform layer library for nodejs, which implements a uniform interface on top of epoll, kqueue, event ports and iocp. I wrote Python bindings [1] for it a while ago, and I was very excited to see Tulip, so I thought I'd give this a try. Here [2] is the source code, along with some notes I took during the implementation. I know that the idea is not to re-implement the PEP itself but for people to create different EventLoop implementations. On rose I bundled tulip just to make a single package I could play with easily, once tulip makes it to the stdlib only the EventLop will remain. Here are some thoughts (in no particular order): - add_connector / remove_connector seem to be related to Windows, but being exposed like that feels a bit like leaking an implementation detail. I guess there was no way around it. - libuv implements a type of handle (Poll) which provides level-triggered file descriptor polling which also works on Windows, while being highly performant. It uses something called AFD Polling apparently, which is only available on Windows >= Vista, and a select thread on XP. I'm no Windows expert, but thanks to this the API is consistent across all platforms, which is nice. mAybe it's worth investigating? [3] - The transport abstraction seems quite tight to socket objects. pyuv provides a TCP and UDP handles, which provide a completion-style API and use a better approach than Poll handles. They should give better performance since EINTR in handled internally and there are less roundtrips between Python-land and C-land. Was it ever considered to provide some sort of abstraction so that transports can be used on top of something other than regular sockets? For example I see no way to get the remote party from the transport, without checking the underlying socket. Thanks for reading this far and keep up the good work. Regards, [1]: https://github.com/saghul/pyuv [2]: https://github.com/saghul/rose [3]: https://github.com/joyent/libuv/blob/master/src/win/poll.c -- Saúl Ibarra Corretgé http://saghul.net/blog | http://about.me/saghul

4 15

PEP 3156/Tulip: Extensible EventLoop interface
by Ben Darnell 07 Feb '13

07 Feb '13

The event loop interface in PEP 3156 has an extensibility problem. It seems likely that it will have a method like listen_udp() by the time it's done, but suppose that doesn't make it into the first official release. Third-party event loop implementations may want to provide UDP support as an extension, but the most consistent way to provide that extension is by adding new methods on the event loop object, where various extensions risk conflicting with each other or with new methods that become standardized later. The PEP specifies the add_reader family of methods in part so that core protocol implementations can be shared across all EventLoop implementations that support these methods. However, if those transports are implemented in a common base class (like Twisted's PosixReactorBase), there is no way for third-party transports to take advantage of a similar structure. I'd like to make it possible for transports to be developed independent of a particular EventLoop implementation in a way that is consistent with the way the core transports work. (This is a bigger concern for tulip than it is for twisted because twisted can update PosixReactorBase more frequently than the stdlib can change) I propose turning the interface around so that transport creation uses static functions that take the event loop as an argument, with a double-dispatch mechanism to allow the event loop to provide the actual implementation when it can: def create_connection(protocol_factory, host, port, event_loop=None): if event_loop is None: event_loop = get_event_loop() # Note the use of a fully-qualified name in the registry impl = event_loop.get_implementation('tulip.create_connection') return impl(protocol_factory, host, port) New third-party transports could provide fallbacks for event loops that don't have their own implementations: if impl is None: # These supports_*() functions are placeholders for a to-be-determined introspection interface. if supports_fd_interface(event_loop): return posix_udp_implementation(*args) elif supports_iocp_interface(event_loop): return iocp_udp_implementation(*args) else: raise Exception("This transport is not supported on this event loop") Or they could plug into the event loop's implementation registry: LibUVEventLoop.register_implementation('mymodule.listen_udp', libuv_udp_implementation) This does introduce a little magic (is there any precedent for this kind of multiple-dispatch in the standard library?), but I like the way it keeps the event loop interface from getting too big and monolithic. Third-party transports can avoid naming conflicts without looking fundamentally different from standard ones, and there's a clean path from doing something that's platform-specific (e.g. with add_reader and friends) to supporting multiple event loops to full standardization. -Ben

10 30

PEP 3156 / Tulip: Remove 'when' from Handler
by Saúl Ibarra Corretgé 05 Feb '13

05 Feb '13

Hi, While working on some changes for rose (an eventloop for tulip based on libuv) I realized the Handler class is a bit too coupled to the event loop implementation currently present in Tulip. pyuv (and pyev, and probably others) provide timers already, so there is no need to implement them "manually" and keep a heap with them. This basically means that Handler objects will always have the 'when' value set to None, and the comparison functions don't really apply, moreover __eq__ would even be incorrect in its current form. IMHO it would be good to have a base Handler class which doesn't take time into account and then a Timer class which does. Thoughts? Also, I see that the PEP does mention the possibility of providing a __call__ method on the Handler class itself, which would basically call the callback and suppress exceptions (I guess by logging them). Was this ever discussed? Sounds like a good idea to me :-) Regards, -- Saúl Ibarra Corretgé http://saghul.net/blog | http://about.me/saghul

2 2

Extend module objects to support properties
by Larry Hastings 05 Feb '13

05 Feb '13

Properties are a wonderful facility. But they only work on conventional objects. Specifically, they *don't* work on module objects. It would be nice to extend module objects so properties worked there too. For example, Victor Stinner's currently proposed PEP 433 adds two new methods to the sys module: sys.getdefaultcloexc() and sys.setdefaultcloexc(). What are we, Java? Surely this would be much nicer as a property, sys.defaultcloexc. //arry/

10 20

Specification of procedures to store cryptographic secrets
by desnacked＠riseup.net 04 Feb '13

04 Feb '13

Some hours ago I sent an email to python-crypto asking how to securely wipe cryptographic secrets from memory: http://mail.python.org/pipermail/python-crypto/2013-February/001170.html Antoine said that cryptographic secret wiping could be achieved if one uses bytearrays carefully and then overwrites their contents after use. I agree that this sounds reasonable, but I think it would be even better if that was a documented property of bytearrays. If that property of bytearrays was specified in the Python standards, it would be easier for people who write cryptographic applications and libraries to use bytearrays correctly, and it would also guarantee that this property won't change in future versions of Python. Furthermore, it would help authors of cryptographic libraries to design their APIs and internal functions in a way that would allow the secure erasure of sensitive data. Would this make sense or am I asking too much from Python?

7 7

for line in input with open(path) as input...
by Shane Green 04 Feb '13

04 Feb '13

with open(path) as input: for line in input: do(line) Using with to create reference to opened file returned by open() so it could temporarily be assigned to input for the sole purpose of iterating its contents never sat very well with me. * The context manager returned by open() exists only to create the context and return reference "input"; * the context and code block created by the "with" only exists for inner "for" loop's code block to execute in. Now, given a generator function: def iterwith(cm): with cm as context: if context is None: context = cm for item in context: yield item The previous code can be turned into: for line in iterwith(open(path)): do(line) So, questions: - Is there anything inherently wrong with the idea, or does it exist? - Is it a generally useful tool, or are the examples limited to files? - Is it possible a more general mechanism could have value, such as: for line in file with open(path) as file: do(line) The preceding could be leveraged to different effect: for line in file with locked(path): write(path + ".out", line) Or, for line in input with nested(open(path),lock,open(opath)) as input,locked,output: output.write(line) To revisit the original purpose of "with", this seems to cleanly address a very common scenario wherein: resource = create_resource() try: for item in resource: do_something(resource, item) except: raise finally: cleanup() # Standard with approach... with create_resource() as resource: for item in resource: do_something(resource, item) # With for loop as context... for item in resource with create_resource() as resource: do_something(resource, item) And, given the translation into statements, maybe even crazy stuff... [line for line in file with open(path) as file] J/K, I think.

8 14

Re: [Python-ideas] for line in input with open(path) as input...
by Shane Green 02 Feb '13

02 Feb '13

*From:* Serhiy Storchaka <storchaka(a)gmail.com> *Date:* February 2, 2013 5:50 AM *To:* python-ideas(a)python.org *Subject:* [Python-ideas] for line in input with open(path) as input... When should iterwith() call __exit__()? *From:* Shane Green <shane(a)umbrellacode.com> *Date:* February 2, 2013 5:55 AM *To:* Serhiy Storchaka <storchaka(a)gmail.com> *CC:* python-ideas(a)python.org *Subject:* [Python-ideas] for line in input with open(path) as input... iterwith() doesn't need to: __exit__() is invoked automatically when iteration completes and the loop exits. > I hadn't thought about the fact the context was created inside the > generator... Nonetheless, cleanup would be automatic, always; and the > generator should exit eventually on close() if not exhausted, I > believe (need to review the details on gen cleanup). > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas >

1 0

Re: [Python-ideas] PEP 3156: getting the socket or peer name from the transport
by Antoine Pitrou 02 Feb '13

02 Feb '13

Le dimanche 27 janvier 2013 à 14:12 +0200, Yuval Greenfield a écrit : > On Sun, Jan 27, 2013 at 1:21 PM, Antoine Pitrou <solipsis(a)pitrou.net> > wrote: > > Most protocols should be written independent of transport. > But it seems to > > me that a user might write an entire app as a "protocol". > > > Well, such an assumption can fall flat. For example, > certificate > checking in HTTPS expects that the transport is some version > of TLS or > SSL: http://tools.ietf.org/html/rfc2818.html#section-3.1 > > > I'm not sure I understood your reply. You'd be for an api that exposes > the underlying transport? I meant to say that "an entire app" entails > control over the subtleties of the underlying transport. What I meant is that the HTTP protocol needs to know that it is running over a secure transport, and it needs to fetch the server certificate from that transport (or, alternatively, it needs to have one of its callbacks called by the transport when the certificate is known). That's not entirely transport-agnostic. Regards Antoine.

4 9

Definition Symmetry
by Jason Keene 01 Feb '13

01 Feb '13

Why do function definitions require parens? >>> class MyClass: ... pass ... >>> def my_func: File "<stdin>", line 1 def my_func: ^ SyntaxError: invalid syntax This seems to me to break a symmetry with class definitions. I assume this is just a hold off from C, perhaps there is a non-historical reason tho. I believe in the past we've forced parens in list comprehensions to create a symmetry between comprehensions/generator expressions. Why not for this?

12 16