On Fri, 2005-07-01 at 10:56 +0100, Matt Goodall wrote:

Obviously, having the event loop part in the std lib has a number of benefits but it should also put pressure on all event driven toolkits to conform to the common API. Everything would then work together and the world would be a happier, shinier place ;-).

Which event driven toolkits are you talking about, besides Twisted?

GvR suggested that someone should start a PEP to talk around, which seems like a sensible thing to do, although I don't know if anyone with enough knowledge of the reactor and working with other event loops has time right now.

On the whole I don't think now, if ever, is the time to add Twisted or a Twisted-based event loop to the Python standard library. There are a number of issues: 1. Release cycles: Python releases on its own schedule, which may not match up to the needs of bug fixes or feature additions to Twisted. One solution is a separate Twisted package you can install, overriding the stdlib's. Since Twisted, even split up, is quite large, this would probably be necessary for most non-trivial functionality anyway, which leads to: 2. Installing separate package issues. As PyXML has shown, having a separate package which overrides or extends a built-in stdlib API works terribly. http://www.amk.ca/conceit/xmlplus.html covers some of the issues, but there's also - 3. Limited usefulness of minimal APIs: lets say we add Twisted to the stdlib. Because it's large, we don't want to add *everything*. We can't add SSL probably cause that depends on pyOpenSSL which is LGPLed, probably don't want the pygtk reactor because that occasionally needs updates for new pygtk releases, etc.. So Jane User decides to build a network application, plays a bit with the TCP client code, and now says "OK great, now to add a GUI and TLS support" - and suddenly she needs to install the full Twisted framework to do that. I would argue that Python needs less in the stdlib, not more; specific external software packages will probably grow into de-facto standard packages, but there's no reason for them to be in stdlib. PyXML is usually installed separately because people need the extended features the full package has, and the same would happen to Twisted. The only real benefit to inclusion in the stdlib is ease of distribution, and a better way to deal with that is to put more work into things like Eggs (http://peak.telecommunity.com/DevCenter/PythonEggs). -- Itamar Shtull-Trauring http://itamarst.org

Itamar Shtull-Trauring wrote:

On Fri, 2005-07-01 at 10:56 +0100, Matt Goodall wrote:

...

GvR suggested that someone should start a PEP to talk around, which seems like a sensible thing to do, although I don't know if anyone with enough knowledge of the reactor and working with other event loops has time right now.

Well, it's a subject that comes up periodically, and I have some new thoughts, so let me play devil's advocate here:

On the whole I don't think now, if ever, is the time to add Twisted or a Twisted-based event loop to the Python standard library. There are a number of issues:

1. Release cycles: Python releases on its own schedule, which may not match up to the needs of bug fixes or feature additions to Twisted.

This will always be a concern, but Python manages to get releases out every so often, and the stdlib *does* change sometimes. We've been pretty good about preserving compatibility for a release or two ourselves. I think it is a reasonable concern but not a reason to stop this entirely.

One solution is a separate Twisted package you can install, overriding the stdlib's. Since Twisted, even split up, is quite large, this would probably be necessary for most non-trivial functionality anyway, which leads to:

IMHO the point here is the trivial functionality. As far as I can tell from the few examples of event-driven applications that I use which I didn't write :-), there is a progression that many event-driven Python programs go through these days: - Implement something with asyncore - realize asyncore's deficiencies, start enhancing it - discover Twisted, re-implement networking code (from scratch) If the event loop in Python provided similar interfaces to the one in Twisted, the amount of work involved in stage 3 could be greatly reduced.

2. Installing separate package issues. As PyXML has shown, having a separate package which overrides or extends a built-in stdlib API works terribly. http://www.amk.ca/conceit/xmlplus.html covers some of the issues, but there's also -

So let's not pretend that they're the same thing. 'from eventloop import listenTCP' -> 'from twisted.internet import reactor'. They can just share a set of common interfaces. The big problem with xmlplus was that it changed the behavior of a built-in package, so let's not ever do that again. amk's article specifically recommends this solution.

3. Limited usefulness of minimal APIs: lets say we add Twisted to the stdlib. Because it's large, we don't want to add *everything*. We can't add SSL probably cause that depends on pyOpenSSL which is LGPLed, probably don't want the pygtk reactor because that occasionally needs updates for new pygtk releases, etc.. So Jane User decides to build a network application, plays a bit with the TCP client code, and now says "OK great, now to add a GUI and TLS support" - and suddenly she needs to install the full Twisted framework to do that.

Right. But, her code uses Twisted-like interfaces so she doesn't have to re-write it all. Given that a lot of network applications are trivial one-offs, this step will not always happen.

I would argue that Python needs less in the stdlib, not more; specific external software packages will probably grow into de-facto standard packages, but there's no reason for them to be in stdlib. PyXML is usually installed separately because people need the extended features the full package has, and the same would happen to Twisted. The only real benefit to inclusion in the stdlib is ease of distribution, and a better way to deal with that is to put more work into things like Eggs (http://peak.telecommunity.com/DevCenter/PythonEggs).

I don't think the issue is "less" or "more" but "better". The Python standard library is of a highly non-uniform quality. Whether or not Twisted gets added, I think asyncore should be removed, or at least strongly deprecated.

On Jul 1, 2005, at 5:56 AM, Matt Goodall wrote:

Hi,

I know some of you guys have already thinking about this but ...

A couple of weeks ago we had a bit of a chat (flame? ;-)) in #twisted about the idea of moving parts of Twisted, specifically the event loop, into the std lib. So, at Europython I had a quick chat with GvR about it too.

GvR would really, really like to get a simple, flexible event loop into the std lib and thinks that Twisted is likely the best starting point since Twisted probably integrates with more event driven toolkits than anything else.

Obviously, having the event loop part in the std lib has a number of benefits but it should also put pressure on all event driven toolkits to conform to the common API. Everything would then work together and the world would be a happier, shinier place ;-).

GvR suggested that someone should start a PEP to talk around, which seems like a sensible thing to do, although I don't know if anyone with enough knowledge of the reactor and working with other event loops has time right now.

I think adding a subset of Twisted to the python stdlib would be a disservice to both the stdlib users and Twisted users, especially now that Python is on the cusp of having an easy way to distribute, find, and install 3rd party packages automatically. Saying that Twisted should be a "starting point" bothers me especially, as that seems to indicate that the two will diverge over time. That is worrisome. Who is going to maintain the stdlib fork? I would suggest instead that python should distribute a selected set of "recommended extras" as part of the installer, which would basically be complete 3rd party packages, included just for convenience. Twisted core would make a good package to include there, as would many others. James

Hi,

IMHO the point here is the trivial functionality. As far as I can tell from the few examples of event-driven applications that I use which I didn't write :-), there is a progression that many event-driven Python programs go through these days:

- Implement something with asyncore - realize asyncore's deficiencies, start enhancing it - discover Twisted, re-implement networking code (from scratch)

If the event loop in Python provided similar interfaces to the one in Twisted, the amount of work involved in stage 3 could be greatly reduced.

As a humble lurker on this list, I just wanna say that this argument rings a bell. I had to rewrite a P2P program which had chosen to implement its own event loop instead of using Twisted... ;) It would be nice if more people knew about Twisted. Regards Antoine.

On Jul 1, 2005, at 9:39 AM, Antoine Pitrou wrote:

...

IMHO the point here is the trivial functionality. As far as I can tell from the few examples of event-driven applications that I use which I didn't write :-), there is a progression that many event-driven Python programs go through these days:

- Implement something with asyncore - realize asyncore's deficiencies, start enhancing it - discover Twisted, re-implement networking code (from scratch)

If the event loop in Python provided similar interfaces to the one in Twisted, the amount of work involved in stage 3 could be greatly reduced.

As a humble lurker on this list, I just wanna say that this argument rings a bell. I had to rewrite a P2P program which had chosen to implement its own event loop instead of using Twisted... ;)

I did this once too, s/P2P program/XML Socket Server/ It turned out that doing one in Twisted was so easy, that I never bothered to package another one again. Everything else I had in there was more or less YAGNI because Twisted already did it, or made implementing the features so trivial as it wasn't worth including it. These days I just whip up app-specific LineReceivers with a '\0' delimiter that parse/generate with ElementTree.. takes like 5 minutes. -bob

Bob Ippolito wrote:

I just wanted to point out that I agree with James' position too, but was speaking towards the "what if ideas from Twisted were to be included", rather than "what should actually happen".

Python's standard library should dissolve into just the bits required to support simple scripts and EasyInstall/eggs.

Rat on! And the library docs will hopefully evolve into a nice distributed documentation system, too ... :) Steve

On Jul 1, 2005, at 3:54 PM, David wrote:

I am not yet a twisted programmer, so my contribution to this thread has more to do with English interpretation than with twisted.

A principle problem with event mainloops is that each toolkit wants to provide one, and the program can only have one running at a time (barring multithreading fanciness). When I read the original post referring to the discussion with GvR, I thought, cool, a resolution to the different-mainloops-for-different-tool kits problem: a mainloop that can integrate various tool kits.

It seems that all other respondents have interpreted it as a call to include twisted, or parts of it _as is_, in the stdlib. I did not see in the original post a call for twisted _networking_ inclusion at all.

When I read about twisted on other lists (the pygame list comes to mind), the examples almost always use the poll() reactor method, because it allows you to use twisted from your own mainloop; you can then integrate other tool kits into that home-spun mainloop by whatever provisions they allow for doing so.

We shouldn't have to be that creative. There should be one std dist mainloop that can accommodate, via callbacks perhaps, whatever (multiple) services need a mainloop. So a tkinter-twisted mainloop and a gtk-twisted mainloop would not be needed as separate components.

I am too naive to write such a library, nor even to write a PEP, so I won't be running with this, but is there something impossible about my interpretation?

That's a nice idea, but it can't possibly work. If you want a general way to interoperate with different toolkits, you basically have to use multithreaded fanciness, because not many of them have timers or socket notifiers built in worth their salt. Starting a single thread to sit around and wait for select() or equivalent is pretty sane, and makes the interface a lot more flexible and generic since you only need one facility in the toolkit -- namely a thread- safe way to call back into the main thread. Fortunately, there is one in Twisted 2.1, threadedselectreactor, which DOES integrate with various runloops (pygame and wxPython included) without polling. However, it's not the end of the road. threadedselectreactor should be refactored into a general interface (interleave) such that other backends (epoll, kqueue, windows junk, etc.) can be used rather than select. -bob