[Twisted-Python] So... Python 3.4 is getting its own async I/O system
In Guido's keynote at PyCon 2013, apparently he talked about adding an async I/O module to Python 3.4. It looks like his slides can be viewed here: https://www.dropbox.com/s/xknbe58zcvjhzhv/PyCon2013.pptx ...while this is the PEP he's talking about: http://www.python.org/dev/peps/pep-3156/ At first glance, the proposed reactor API looks very much like Twisted's (or, to be fair, GTK's, or possibly any number of other async event loop I'm less familiar with) but rather than Deferreds and callbacks, the API will be based around Futures (similar, but not identical, to Python 3.2's concurrent.futures.Future class), and an inlineCallbacks-style decorator for generators. I know Deferreds are awesome, and I don't know much about Futures (and I know Twisted core developers have given negative reviews of other Deferred/Promise/Future implementations in, say, JavaScript libraries before), and inlineCallbacks seems to have a negative reputation among experienced Twisted users. Is there anybody on this list who knows more about this new PEP (maybe somebody who's at PyCon and saw the talk in person) who can give us an informed comparison with the current state of Twisted?
On Tue, Mar 19, 2013 at 11:00 AM, Tim Allen <screwtape@froup.com> wrote:
In Guido's keynote at PyCon 2013, apparently he talked about adding an async I/O module to Python 3.4. It looks like his slides can be viewed here:
https://www.dropbox.com/s/xknbe58zcvjhzhv/PyCon2013.pptx
...while this is the PEP he's talking about:
http://www.python.org/dev/peps/pep-3156/
At first glance, the proposed reactor API looks very much like Twisted's (or, to be fair, GTK's, or possibly any number of other async event loop I'm less familiar with) but rather than Deferreds and callbacks, the API will be based around Futures (similar, but not identical, to Python 3.2's concurrent.futures.Future class), and an inlineCallbacks-style decorator for generators.
I know Deferreds are awesome, and I don't know much about Futures (and I know Twisted core developers have given negative reviews of other Deferred/Promise/Future implementations in, say, JavaScript libraries before), and inlineCallbacks seems to have a negative reputation among experienced Twisted users. Is there anybody on this list who knows more about this new PEP (maybe somebody who's at PyCon and saw the talk in person) who can give us an informed comparison with the current state of Twisted?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi inlineCallbacks is ok to me. I use it daily. I can keep code snippets succinct. no more callbacks, deep nested callbacks (they let me recall those days with javascript & node.js). The only thing I complain is it doesn't work with cython. Regards gelin yan
Not relavent to your questions, but there's a mailing list for discussing the specifics of PEP 3156, which is worth subscribing to if you want to provide specific feedback on the API. https://groups.google.com/forum/?fromgroups#!forum/python-tulip On Tue, Mar 19, 2013 at 3:00 AM, Tim Allen <screwtape@froup.com> wrote:
In Guido's keynote at PyCon 2013, apparently he talked about adding an async I/O module to Python 3.4. It looks like his slides can be viewed here:
https://www.dropbox.com/s/xknbe58zcvjhzhv/PyCon2013.pptx
...while this is the PEP he's talking about:
http://www.python.org/dev/peps/pep-3156/
At first glance, the proposed reactor API looks very much like Twisted's (or, to be fair, GTK's, or possibly any number of other async event loop I'm less familiar with) but rather than Deferreds and callbacks, the API will be based around Futures (similar, but not identical, to Python 3.2's concurrent.futures.Future class), and an inlineCallbacks-style decorator for generators.
I know Deferreds are awesome, and I don't know much about Futures (and I know Twisted core developers have given negative reviews of other Deferred/Promise/Future implementations in, say, JavaScript libraries before), and inlineCallbacks seems to have a negative reputation among experienced Twisted users. Is there anybody on this list who knows more about this new PEP (maybe somebody who's at PyCon and saw the talk in person) who can give us an informed comparison with the current state of Twisted?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, Mar 18, 2013 at 11:00 PM, Tim Allen <screwtape@froup.com> wrote:
In Guido's keynote at PyCon 2013, apparently he talked about adding an async I/O module to Python 3.4. It looks like his slides can be viewed here:
https://www.dropbox.com/s/xknbe58zcvjhzhv/PyCon2013.pptx
...while this is the PEP he's talking about:
http://www.python.org/dev/peps/pep-3156/
At first glance, the proposed reactor API looks very much like Twisted's (or, to be fair, GTK's, or possibly any number of other async event loop I'm less familiar with) but rather than Deferreds and callbacks, the API will be based around Futures (similar, but not identical, to Python 3.2's concurrent.futures.Future class), and an inlineCallbacks-style decorator for generators.
There's two parts: one is a standardized, pluggable API for event loops. It's explicitly based on Twisted's, and will be easy to add a Twisted implementation. The generator thing is layered on top, and somewhat optional, in theory. -Itamar
On Mar 18, 2013, at 8:00 PM, Tim Allen <screwtape@froup.com> wrote:
In Guido's keynote at PyCon 2013, apparently he talked about adding an async I/O module to Python 3.4.
At first glance, the proposed reactor API looks very much like Twisted's (or, to be fair, GTK's, or possibly any number of other async event loop I'm less familiar with)
Importantly, the proposed loop API is like Twisted (in that it has callbacks for the receipt of data and the emptying of send buffers) and _not_ like GTK (in that it doesn't call handle_read() and hope you can handle a socket object).
but rather than Deferreds and callbacks, the API will be based around Futures (similar, but not identical, to Python 3.2's concurrent.futures.Future class), and an inlineCallbacks-style decorator for generators.
Mostly accurate as far as I know.
I know Deferreds are awesome, and I don't know much about Futures (and I know Twisted core developers have given negative reviews of other Deferred/Promise/Future implementations in, say, JavaScript libraries before), and inlineCallbacks seems to have a negative reputation among experienced Twisted users. Is there anybody on this list who knows more about this new PEP (maybe somebody who's at PyCon and saw the talk in person) who can give us an informed comparison with the current state of Twisted?
The interesting thing about the proposal from the perspective of the Twisted project is that it is an exciting opportunity for interoperability. The availability of a standard-library, "blessed" event-loop API will give us something to write against, rather than Twisted and Tornado and Pulsar all having alternative main-loop APIs, and partial adapters between them. In the future, it might make things like the GTK reactor obsolete, as it's possible that the PyGTK project would include their own Python-standard event-loop adapter. Twisted is an amazingly full-featured and mature event-driven framework; it doesn't make sense (and isn't fair) to compare a minimalistic main-loop to it. Tulip does not contain an IMAP client or an SSH server; it does not contain any XMPP logic. As far as 'yield from' coroutines are concerned, I'm personally ambivalent, but I'm happy to ignore them. The best part about this proposal is that anyone who wants to get this new thing does not need to abandon the extensive capabilities of Twisted over some tweak of syntax. With only a little bit of specification-implementation glue, we can interoperate not just with things that do the equivalent of connectTCP/listenTCP, but we can bridge between Deferreds and Futures (as Guido's new not-quite-concurrent.-Future explicitly drops the concurrent.Future that Deferred can't implement, because it's serving the purpose of Deferred). -glyph
On Mar 20, 2013, at 1:51 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
In Guido's keynote at PyCon 2013, apparently he talked about adding an async I/O module to Python 3.4. It looks like his slides can be viewed
Will this tackle async file I/O also or only network?
Since async file I/O is not a thing you can implement[1], presumably it will only be networking. You probably want to join this group: <https://groups.google.com/forum/?fromgroups=#!forum/python-tulip> -glyph [1]: <http://blog.libtorrent.org/2012/10/asynchronous-disk-io/>
On Wed, Mar 20, 2013, at 9:36, Glyph wrote: ...
Since async file I/O is not a thing you can implement[1], ...
[1]: <http://blog.libtorrent.org/2012/10/asynchronous-disk-io/>
Understood that it isn't possible to use native asynchronous I/O. But an implementation that used threads behind the scenes and returned a Deferred (for small files) or a Protocol (for big ones) would be a nice thing to have. Peter.
On Mon, Mar 25, 2013 at 11:59 AM, Peter Westlake <peter.westlake@pobox.com>wrote:
Understood that it isn't possible to use native asynchronous I/O. But an implementation that used threads behind the scenes and returned a Deferred (for small files) or a Protocol (for big ones) would be a nice thing to have.
For streaming files, you want something pausable; there are existing producers which don't use threads, and there will presumably be a tubes equivalent. Maybe having the tubes one use threads by default is a good idea. For getting a file as a string, you can just do: d = deferToThread(lambda path: FilePath(path).getContent()) For more complex usage patterns, you'll probably want to write custom code using deferToThread/deferToThreadPool anyway. -- Itamar Turner-Trauring, Future Foundries LLC http://futurefoundries.com/ — Twisted consulting, training and support.
On 07:47 pm, itamar@futurefoundries.com wrote:
On Mon, Mar 25, 2013 at 11:59 AM, Peter Westlake <peter.westlake@pobox.com>wrote:
Understood that it isn't possible to use native asynchronous I/O. But an implementation that used threads behind the scenes and returned a Deferred (for small files) or a Protocol (for big ones) would be a nice thing to have.
For streaming files, you want something pausable; there are existing producers which don't use threads, and there will presumably be a tubes equivalent. Maybe having the tubes one use threads by default is a good idea.
For getting a file as a string, you can just do:
d = deferToThread(lambda path: FilePath(path).getContent())
Not that `FilePath` is guaranteed to be thread-safe... Jean-Paul
For more complex usage patterns, you'll probably want to write custom code using deferToThread/deferToThreadPool anyway.
-- Itamar Turner-Trauring, Future Foundries LLC http://futurefoundries.com/ — Twisted consulting, training and support.
Since async file I/O is not a thing you can implement[1], ...
[1]: <http://blog.libtorrent.org/2012/10/asynchronous-disk-io/>
Understood that it isn't possible to use native asynchronous I/O. But an implementation that used threads behind the scenes and returned a Deferred (for small files) or a Protocol (for big ones) would be a nice thing to have.
Yeah, reading a single block from disk can take somewhere between roughly 10us (FusionIO) to 10ms (magnetic platter), and if I need only single blocks (not streaming .. say I access an on-disk key-value store), it would be nice if that would be encapsulated in a deferred, while the reactor drives other stuff. Ideally without a background worker thread pool on platforms that have sufficient/sane support. And worked around via thread pool on others.
On Mar 25, 2013, at 8:59 AM, Peter Westlake <peter.westlake@pobox.com> wrote:
On Wed, Mar 20, 2013, at 9:36, Glyph wrote: ...
Since async file I/O is not a thing you can implement[1], ...
[1]: <http://blog.libtorrent.org/2012/10/asynchronous-disk-io/>
Understood that it isn't possible to use native asynchronous I/O. But an implementation that used threads behind the scenes and returned a Deferred (for small files) or a Protocol (for big ones) would be a nice thing to have.
For what it's worth, I completely agree. If someone has, or someone ever does, implement a nice async file I/O system, a nice abstract API for Twisted to do it would allow us to swap in such a thing and get a performance boost without anyone changing their code. As Itamar says though, a Fount would be an ideal way to present this once I merge the Tubes branch. (It'll be up for review any day now, thanks to all the help I got at PyCon...) -glyph
Will this tackle async file I/O also or only network?
Since async file I/O is not a thing you can implement[1], presumably it will only be networking.
Ok, so AIO is broken on Linux both implementation- and API-wise, but Windows/IOCP and/or *BSD/kqueue might be a different story. I haven't tested myself though. Probably would give kqueue on FreeBSD/ZFS a run ..
participants (8)
-
exarkun＠twistedmatrix.com -
Gelin Yan -
Glyph -
Itamar Turner-Trauring -
Peter Westlake -
Stephen Thorne -
Tim Allen -
Tobias Oberstein