[Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)

[glyph at divmod.com]

On 02:20 am, greg.ewing at canterbury.ac.nz wrote:

>If Twisted is designed so that it absolutely *has* to
>use its own special event mechanism, and everything else
>needs to be modified to suit its requirements, then it's
>part of the problem, not part of the solution.

I've often heard this complaint, usually of the form "that's twisted-specific".  The thing is, Twisted isn't specific.  Its event mechanism isn't "special".  In fact it's hard to imagine how it might be made less "special" than it currently is.

Whatever your event dispatch mechanism, *some* code has to be responsible for calling the OS API of select(), WaitForMultipleEvents(), g_main_loop_run(), or whatever.  Twisted actually imposes very few requirements for code to participate in this, and was designed from day one specifically to be a generalized mainloop mechanism which would not limit you to one underlying multiplexing implementation, event-dispatch mechanism, or operating system if you used its API.

There have even been a few hacks to integrate Twisted with the asyncore mainloop, but almost everyone who knows both asyncore and Twisted has rapidly decided to just port all of their code to Twisted rather than maintain that bridge.

In fact, Twisted includes fairly robust support for threads, so if you really need it you can mix and match event-driven and blocking styles.  Again, most people who try this find that it is just nicer to write straight to the Twisted API, but for those that need really it, such as Zope and other WSGI-contained applications, it is available.

Aside from the perennial issue of restartable reactors (which could be resolved as part of a stdlib push), Twisted's event loop imposes very few constraints on your code.  It provides a lot of tools, sure, but few of them are required.  You don't even *need* to use Deferreds.  Now, "Twisted", overall, can be daunting.  It has a lot of conventions, a lot of different interfaces to memorize and deal with, but if you are using the main loop you don't have to necessarily care about our SSH, ECMA 48, NNTP, OSCAR or WebDAV implementation.  Those are all built at a higher level.

It may seem like I'm belaboring the point here, but every time a thread comes up on this list mentioning the possibility of a "standard" event mechanism, people who do not know it very well or haven't used it start in with implied FUD that Twisted is a big, complicated, inseparable hairy mess that has no place in the everyday Python programmer's life.  It's tangled-up Deep Magic, only for Wizards Of The Highest Order.  Alternatively, more specific to this method, it's highly specific and intricate and very tightly coupled.

Nothing could be further from the truth.  It is *strictly* layered to prevent pollution of the lower levels by higher level code, and all the dependencies are one-way.  Maybe our introductory documentation isn't very good.  Maybe event-driven programming is confusing for those expecting something else.  Maybe the cute names of some of the modules are offputting.  Still, all that aside, if you are looking for an event-driven networking engine, Twisted is about as straightforward as you can get without slavishly coding to one specific platform API.

When you boil it down, Twisted's event loop is just a notification for "a connection was made", "some data was received on a connection", "a connection was closed", and a few APIs to listen or initiate different kinds of connections, start timed calls, and communicate with threads.  All of the platform details of how data is delivered to the connections are abstracted away.  How do you propose we would make a less "specific" event mechanism?

I strongly believe that there is room for a portion of the Twisted reactor API to be standardized in the stdlib so that people can write simple event-driven code "out of the box" with Python, but still have the different plug-in implementations live in Twisted itself.  The main thing blocking this on my end (why I am not writing PEPs, advocating for it more actively, etc) is that it is an extremely low priority, and other, higher level pieces of Twisted have more pressing issues (such as the current confusion in the "web" universe).  Put simply, although it might be nice, nobody really *needs* it in the stdlib, so they're not going to spend the effort to get it there.

If someone out there really had a need for an event mechanism in the standard library, though, I encourage them to look long and hard at how the existing interfaces in Twisted could be promoted to the standard library and continue to be maintained compatibly in both places.

At the very least, standardizing on something very much like IProtocol would go a long way towards making it possible to write async clients and servers that could run out of the box in the stdlib as well as with Twisted, even if the specific hookup mechanism (listenTCP, listenSSL, et. al.) were incompatible - although a signature compatible callLater would probably be a must.

As I said, I don't have time to write the PEPs myself, but I might fix some specific bugs if there were a clear set of issues preventing this from moving forward.  Better integration with the standard library would definitely be a big win for both Twisted and Python.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/e3f6eb5b/attachment.html