On Wed, Sep 26, 2012 at 10:02:24AM -0700, Josiah Carlson wrote:
Go ahead and read PEP 3153, we will wait.
A careful reading of PEP 3153 will tell you that the intent is to make a "light" version of Twisted built into Python. There isn't any discussion as to *why* this is a good idea, it just lays out the plan of action. Its ideas were gathered from the experience of the Twisted folks.
Their experience is substantial, but in the intervening 1.5+ years since Pycon 2011, only the barest of abstract interfaces has been defined (https://github.com/lvh/async-pep/blob/master/async/abstract.py), and no discussion has taken place as to forward migration of the (fairly large) body of existing asyncore code.
it doesn't look like twisted-light to me, more like a interface suggestion for a small subset of twisted. in particular, it doesn't talk about main loops / reactors / registration-in-the-first-place. you mention interaction with the twisted people. is there willingness, from the twisted side, to use a standard python middle layer, once it exists and has sufficiently high quality?
To the point, Giampaolo already has a reactor that implements the interface (more or less "idea #3" from his earlier message), and it's been used in production (under staggering ftp(s) load). Even better, it offers effectively transparent replacement of the existing asyncore loop, and supports existing asyncore-derived classes. It is available: https://code.google.com/p/pyftpdlib/source/browse/trunk/pyftpdlib/lib/ioloop...
i've had a look at it, but honestly can't say more than that it's good to have a well-tested asyncore compatible main loop with scheduling support, and i'll try it out for my own projects.
Again, at this point in time what you're proposing looks too vague, ambitious and premature to me.
please don't get me wrong -- i'm not proposing anything for immediate action, i just want to start a thinking process towards a better integrated stdlib.
I am curious as to what you mean by "a better integrated stdlib". A new interface that doesn't allow people to easily migrate from an existing (and long-lived, though flawed) standard library is not better integration. Better integration requires allowing previous users to migrate, while encouraging new users to join in with any later development. That's what Giampaolo's suggested interface offers on the lowest level; something to handle file-handle reactors, combined with a scheduler.
a new interface won't make integration automatically happen, but it's something the standard library components can evolve on. whether, for example urllib2 will then automatically work asynchronously in that framework or whether we'll wait for urllib3, we'll see when we have it. @migrate from an existing standard library: is there a big user base for the current asyncore framework? my impression from is that it is not very well known among python users, and most that could use it use twisted.
we've talked about many things we'd need in a python asynchronous interface (not implementation), so what are the things we *don't* need? (so we won't start building a framework like twisted). i'll start:
* high-level protocol handling (can be extra modules atop of it) * ssl * something like the twisted delayed framework (not sure about that, i guess the twisted people will have good reason to use it, but i don't see compelling reasons for such a thing in a minimal interface from my limited pov) * explicit connection handling (retries, timeouts -- would be up to the user as well, eg urllib might want to set up a timeout and retries for asynchronous url requests)
I disagree with the last 3. If you have an IO loop, more often than not you want an opportunity to do something later in the same context. This is commonly the case for bandwidth limiting, connection timeouts, etc., which are otherwise *very* difficult to do at a higher level (which are the reasons why schedulers are built into IO loops). Further, SSL in async can be tricky to get right. Having the 20-line SSL layer as an available class is a good idea, and will save people time by not having them re-invent it (poorly or incorrectly) every time.
i see; those should be provided, then. i'm afraid i don't completely get the point you're making, sorry for that, maybe i've missed important statements or lack sufficiently deep knowledge of topics affected and got lost in details. what is your opinion on the state of asynchronous operations in python, and what would you like it to be? thanks for staying with this topic chrysn -- To use raw power is to make yourself infinitely vulnerable to greater powers. -- Bene Gesserit axiom