Steve Holden wrote:

If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in.

Since there should only be one reactor at a time in any given application, it shouldn't have to be passed in -- it could be held in a global variable deep inside the library. Only the code which creates the reactor initially needs to know about that variable, or even that there is such a thing as a reactor.

"Incorporating some piece of event-driven code written by someone else" implies specific assumptions about event types and delivery, surely.

It requires agreement on how to specify the event types and what to do in response, but that's all it should require. The way I envisage it, setting up an event callback should be like opening a file -- there's only one way to do it, and you don't have to worry about what the rest of the application is doing. You don't have to get passed an object that knows how to open files -- it's a fundamental service provided by the system. You just use it.

That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change.

Just in case it's not clear, the events I'm talking about are things like file and socket I/O, not GUI events. Trying to use two different GUIs at once is not something I'm addressing. Rather, you should be able to write code that does e.g. some async socket I/O, and embed it in a GUI app using e.g. gtk, without having to modify it to take account of the fact that it's working in a gtk environment, or having to parameterise it to allow for such things.

You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms.

I'm arguing that as much of the platform dependency as possible should be in the asyncore library (or whatever replaces it). The main application code *might* have to give it a hint such as "this app uses gtk", but no more than that. And ideally, I'd prefer it not to even have to do that -- pygtk should do whatever is necessary to hook itself into asyncore if at all possible, not the other way around.

Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion.

Okay, but one of the suggestions made seemed to be "why not just use the Twisted API". I'm putting forward a possible reason. -- Greg

On 2/14/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

glyph@divmod.com wrote:

...

On 08:52 pm, theller@ctypes.org wrote:

...

When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more.

Only the very top-most level decides which reactor the application will use.

This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level.

That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application.

This is not my idea of what another poster called a "one-stop shop" -- a common API that different pieces of code can call independently without having to know about each other.

To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used.

Eh, your own example seems to argue the opposite. If the choice for reactor was made somewhere deep inside the library, how does it know to use the GTK reactor? Only the code using GTK knows that it's going to need the GTK reactor. The generic code just uses whatever reactor was selected, automatically. The reason the decision should be made at the topmost level is that this is the most visible and adaptive location. A library or framework has to handle a boundless number of variations of environment, use and co-operation with other libraries and frameworks, whereas the topmost script knows exactly what it wants to do. A library that uses PyGTK can say 'I need the GTK reactor', but it would be wrong. All it needs is a reactor that interacts nicely with GTK. If there are multiple reactors that integrate nicely with GTK, there is no reason for the library to insist on one particular implementation. And it could be in the way, if a second library needs a different kind of integration and it knows of a reactor that supports both that *and* GTK integration. And if the whole system breaks, for some reason, the first place the programmer will look is in his own code, not deep inside another library. For what it's worth, I think a single event-handling mainloop API in the standard library is a *great* idea. Using part of Twisted for it (suitably adapted) is an even greater idea: a whole lot of work and brainpower went into Twisted's core, and it wasn't wasted. Duplicating the effort would be a big mistake and a terrible waste. A from-scratch implementation of an event-handling mechanism will not solve any of the "issues" most people have with Twisted. It will have many more bugs than Twisted (all the ones Twisted had and fixed, sometimes very obscure and hard-to-find bugs), it will not be significantly less complex (Twisted is as complex as it is because it had to be, to do what it is supposed to do) and it will not require any less of a brain-warp to understand (because that's just what event-driven programming takes.) If I wasn't short on free time (spending most of it on Py3k and, soon, py3k-forward-compatibility) I would do all the necessary Twisted-integration work myself :-) -- Thomas Wouters <thomas@python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On 2/14/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

I know that will be hard to do, but it's the only way out of this mess that I can see.

That depends on what you consider messy about it. *I* don't like the idea of something in the Python installation deciding which reactor to use. It's my application, and I'm damn well going to tell it what to do. If that means it doesn't work as I expected, it's my own fault :-) In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model. While those issues are being settled, we'll have to cope with selecting the right reactor manually. It's not all that different from what you want, in any case. The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code. -- Thomas Wouters <thomas@python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Greg Ewing schrieb:

On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it.

That is insufficient. The gtk main loop has more input sources than just the connection to X: - timers can be registered, which are called when the time comes - idle handlers can be registered which are called when there are no other events - child handlers are invoked when a child process terminates - additional file descriptors can be registered (probably used for sockets primarily) - a generalzed 'event source' can be hooked into it, with C functions for prepare, check, dispatch, and finalize See http://www.gtk.org/api/2.6/glib/glib-The-Main-Event-Loop.html Regards, Martin

On 2/14/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Thomas Wouters wrote:

...

*I* don't like the idea of something in the Python installation deciding which reactor to use.

I wouldn't mind if some way were provided of changing the reactor if you want. I'd just like to see a long term goal of making it unnecessary as far as possible.

...

In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model.

On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it.

A similar strategy ought to work for any X11-based toolkit that exposes a function to perform one iteration of its main loop.

Mileage on other platforms may vary.

...

The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code.

Sure.

The other side to all this is the client side, i.e. the code that installs event callbacks. At the moment there's no clear direction to take, so everyone makes their own choice -- some use asyncore, some use Twisted, some use the gtk event loop, some roll their own, etc.

There is no single PerfectReactor. There are several use cases where you need to wait on >1 different event systems, which guarantees at least two OS threads (and two event loops). In general it's nice to have a single Python event loop ("the reactor") to act on said threads (e.g. something just sitting on a mutex waiting for messages) but waiting for IO to occur should *probably* happen on one or more ancillary threads -- one per event system (e.g. select, GTK, WaitForMultipleEvents, etc.) -bob

Larry Hastings schrieb:

Bob Ippolito wrote:

...

There is no single PerfectReactor. There are several use cases where you need to wait on >1 different event systems, which guarantees at least two OS threads (and two event loops). In general it's nice to have a single Python event loop ("the reactor") to act on said threads (e.g. something just sitting on a mutex waiting for messages) but waiting for IO to occur should *probably* happen on one or more ancillary threads -- one per event system (e.g. select, GTK, WaitForMultipleEvents, etc.) Why couldn't PerfectReactor be a reactor for other reactors? A sort of concentrator for these multiple event systems and multiple threads.

Because you can't write a single function that blocks (idle) until an event in one of the systems occurs. None of the system is truly asynchronous. All APIs, eventually, rely on synchronous, blocking, operating system calls. In Unix, the call to make is typically select or poll, and you know it is safe to make because you have all even sources as file descriptors and you know nothing else needs to be done before one of the events occurs. Now, for these generalized event loops, it may not be possible anymore to combine all event sources into a single blocking call. If you have timeouts, you have to find out what the minimum timeout is (to make it the timeout for the blocking call, assuming that supports a timeout). If you have idle tasks, you have to invoke all idle tasks of all reactors (which you may not be able to get because there is no API to fetch them). Even more difficult: if these use different OS event mechanisms (on systems that have multiple such systems). E.g. try combinging a Win32 GetMessage/PostMessage loop with a WaitForMultipleObjects loop with a WinSock select call; then add the Win32 RPC libraries to that (i.e. try to determine the named pipe handles you have to pass to WaitForMultipleObjects).

You ask to listen to sockets, so it instantiates a singleton PerfectReactor which instantiates a select() reactor and listens to it directly in a single-threaded manner. If you then ask to listen to Win32 messages, the PerfectReactor instantiates a GetMessage() reactor. Then, realizing it has two "event systems", it spawns a thread for each child reactor with a listener that serializes the incoming events into the PerfectReactor's queue.

Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages. (This is indeed one of the biggest limitations of USER32, which was only overcome in Vista when the "native" GUI programming model is based on DirectX - the problem is then of course solved only for applications that use the DirectX UI API).

Bingo, your application doesn't need to be written thread-safe, PerfectReactor is platform-agnostic, and you don't have to know in advance all the event types you might ever listen to.

Sorry if this is a dumb question (or if I'm mangling the terminology),

Integrating with threads might be a solution in some cases, and a problem in others. You can't assume it is a universal solution. Regards, Martin

...

Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages.

I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable.

I've tried something similar on Linux, with gtk an wx. You can run the gtk main loop in its own thread, but because gtk is not thread safe, you have to grab a mutex everytime you run gtk code outside the thread the mainloop is running in. So you have to surround your calls to the gtk api with calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of course, because they are executed in the main thread... Doable, but not fun. The same goes for wx. Then all hell breaks loose when you try to use both gtk and wx at the same time. That's because on Linux, the wx main loop calls the gtk mainloop behind the scenes. As far as I know, that problem can not be solved from python. So yes that strategy can work, but it's no silver bullet. Cheers, Baptiste

Larry Hastings schrieb:

Oof! I'm embarrassed to have forgotten that. But that's not a fatal problem. It means that on Windows the PerfectReactor must service the blocking GetMessage loop, and these other threads notify the PerfectReactor of new events by sending a message. ... I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable.

As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe. Speaking of which: Tcl has a "multiple interpreter" model where each interpreter is single-threaded. You can build Tcl either single- or multi-threaded; for the multi-threaded Tcl, you still can't share data across threads directly, but instead, you create an interpreter per thread. Then, all operations on that interpreter must occur in this thread, including the reactor calls. So if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl even loop in the Tcl thread, which is normally the main thread (in Tkinter, it's the thread where you originally created the tkapp object). OTOH, PerfectReactor may also need to run some Win32 event loop in the same thread. Regards, Martin

As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe.

I'm not sure what your point is here. Let me see if I can clarify things, at least from my side. I suggest it is possible to implement a PerfectReactor. I define this PerfectReactor as a singleton top-level reactor which runs in the main thread of a Python application. All incoming events would be serialized to this single reactor, which would dispatch them to their appropriate listeners. When asked to monitor disparate event systems, the PerfectReactor would act as a concentrator of multiple other reactors, with various ugly things going on hidden in the implementation. The goal would be that modular libraries of event listeners could be written in an agnostic manner, simply requesting to listen to whatever events they need to, and the main application wouldn't have to know in advance what reactors to configure or how to configure them. You seem to assert it is not possible to implement such a PerfectReactor, and list counter-examples. So far so good. If you're saying "it is impossible for any implementation to simultaneously support more than one of the Win32, gtk, and wx reactors": fine, this hypothetical PerfectReactor can't do the impossible, but that's not an argument against a PerfectReactor. If it *is* possible to support more than one of these simultaneously through precise configuration, I assert that a PerfectReactor could support that configuration too, and the goal would be to have it automatically configure itself. You point out the gtk reactor and the wx reactor must be run in the main thread, because they run their own message pumps. What I had in mind was that, when you asked to start listening to Win32 events, the Win32 reactor would tell the PerfectReactor "I must be run in the main thread", and it would (re-) configure that to be the main thread's listener; the notification mechanism for other threads saying "there's a new event in your queue" would switch to using Win32 messages. If the gtk and wx reactors had the same restriction, then whichever was first introduced to the PerfectReactor first would win, and introducing subsequent reactors would throw an exception. (Again, can't do the impossible.) However, if gtk and wx expose enough of the plumbing of their message pumps, this central Win32 message pump could identify messages intended for the gtk and wx reactor message pumps and inject them. The gtk and wx reactors wouldn't call GetMessage, but they could call the rest of their message loops. Therefore they could all be run in the main thread. (And yes, "behind the scenes" is correct.)

if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl even loop in the Tcl thread, which is normally the main thread (in Tkinter, it's the thread where you originally created the tkapp object). OTOH, PerfectReactor may also need to run some Win32 event loop in the same thread.

Again, I'm not sure of your point. If you're saying "it is impossible to run multithreaded Tcl and any message pump (Win32, gtk, wx) in the same application", then fine, PerfectReactor cannot do the impossible. If it is only possible to do with clever configuration, then I maintain a PerfectReactor could self-configure to support this configuration at runtime. Cheers, /larry/

At 01:31 PM 2/15/2007 +1300, Greg Ewing wrote:

To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used.

*shudder*. I, on the other hand, prefer to assume that there is no one "top level" and certainly no requirement for a single event loop or reactor. peak.events, for example, lets you have multiple event loops running in the same or different threads. One of these can be Twisted's reactor, if you like, but the framework doesn't impose this singleton-ness on you. (IIUC, Twisted uses a singleton for at least two fairly good reasons: ease of programming, and the fact that certain platforms demand it for performance reasons. I just don't happen to agree that this limitation should be applied across *all* platforms. And there are good solutions for context-specific pseudo-singletons to address the API issue, although they might not be as high-performance as Twisted applications might prefer. Everything's a trade-off.)

On 2/14/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Phillip J. Eby wrote:

...

peak.events, for example, lets you have multiple event loops running in the same or different threads.

Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads.

And... how do you run multiple event loops simultaneously in the *same* thread? That sounds self-contradictory to me.

If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop. Or you can design all mainloops as coroutines and have them call each other. (I haven't looked at Phillip's approach at all, but something tells me coroutines are involved :-) -- Thomas Wouters <thomas@python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Thomas Wouters schrieb:

If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop.

That doesn't really work (and neither do variations involving coroutines. Either the 'run one iteration' method blocks until one even arrives, in which case it may block for a long time while other loops have events pending (which then won't get processed). Or each 'run one iteration' method has a polling mode (e.g. with a time-out), in which case you get busy wait (you can fiddle with the time-out value to trade business vs. responsiveness). The 'run one iteration' approach doesn't support writing applications that are single-threaded, responsive, and idle when there is nothing to do. I can't believe that coroutines help in any way here. Regards, Martin

Phillip J. Eby wrote:

At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote:

...

...

peak.events, for example, lets you have multiple event loops running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use

Phillip J. Eby wrote: threads.

And... how do you run multiple event loops simultaneously in the *same* thread?

When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI "accept" event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished.

I actually have some code that works a little bit like this. I have a SelectLoop class that the application knows about and explicitly attaches transport objects to, and queues events on the loop. I have two ways to perform some actions in a "blocking" way. 1) I can create a new SelectLoop and move some transports temporarily from the main loop to the temporary loop to ignore all other events for a while. 2) I can re-enter the SelectLoop run() method and keep reacting to all events. Once the right event occurs - I exit from the nested run() and continue on. This does require a lot more mental effort though. Note that my code is nowhere near as ambitious as Twisted, but I watned to have the above flexibility. The code really only has to work on AIX and Linux, and in a limited way, Windows. - Dave