Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 08:52 pm, theller@ctypes.org wrote:
When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more.
Yes. That's intentional. Each of those systems offers its own event loop, and each reactor implements the basic operations in terms of those loops. They all have the same API. Application code does 'from twisted.internet import reactor; reactor.listenTCP', 'reactor.callLater', etc. Only the very top-most level decides which reactor the application will use.
And they didn't even work too well. The problems I remember were that the win32reactor was limited to a only handful of handles, the wxreactor didn't process events when a wx modal dialog boy was displayed, and so on. Has this changed?
win32eventreactor is limited to 64 handles because WaitForMultipleObjects is limited to 64 handles. wxreactor's event loop is ghastly and actually did pause when a modal dialog box is displayed (that has since been fixed). Process support on win32 now works in the default select reactor as well as the gtk reactor, so win32reactor is mostly a curiosity at this point (useful mainly if you want to implement your own GDI-based GUI, as PyUI did at one point), and its limitations are not as serious for Twisted as a whole. In other words, Twisted exposes the platform limitations in its platform-specific event loop implementations, and only works around them where it's possible to do so without masking platform functionality. For servers, the epoll, poll, and select reactors work just fine. The select reactor does have a maximum of FD_SETSIZE simultaneous sockets as well, but it is very easy to switch reactors if you need something more scalable. For clients, the best GUI toolkit for Twisted applications at this point is GTK, but WX, QT and Cocoa can all be made to work with a minimum of hassle.
glyph@divmod.com wrote:
On 08:52 pm, theller@ctypes.org wrote:
When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more.
Only the very top-most level decides which reactor the application will use.
This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level. That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application. This is not my idea of what another poster called a "one-stop shop" -- a common API that different pieces of code can call independently without having to know about each other. To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used. So at this point I'm skeptical that the Twisted API for these things should be adopted as-is. -- Greg
Greg Ewing wrote:
glyph@divmod.com wrote:
On 08:52 pm, theller@ctypes.org wrote:
When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more.
Only the very top-most level decides which reactor the application will use.
This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level.
That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application.
If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in. If the borrowed code uses a fixed reactor then it's difficult to see how changes to the Twisted API could help. "Incorporating some piece of event-driven code written by someone else" implies specific assumptions about event types and delivery, surely. That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change.
This is not my idea of what another poster called a "one-stop shop" -- a common API that different pieces of code can call independently without having to know about each other.
To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used.
You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms. Glyph seems to prefer the ability for the library caller to pass in handlers for platform-dependent features.
So at this point I'm skeptical that the Twisted API for these things should be adopted as-is.
Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007
Steve Holden wrote:
If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in.
Since there should only be one reactor at a time in any given application, it shouldn't have to be passed in -- it could be held in a global variable deep inside the library. Only the code which creates the reactor initially needs to know about that variable, or even that there is such a thing as a reactor.
"Incorporating some piece of event-driven code written by someone else" implies specific assumptions about event types and delivery, surely.
It requires agreement on how to specify the event types and what to do in response, but that's all it should require. The way I envisage it, setting up an event callback should be like opening a file -- there's only one way to do it, and you don't have to worry about what the rest of the application is doing. You don't have to get passed an object that knows how to open files -- it's a fundamental service provided by the system. You just use it.
That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change.
Just in case it's not clear, the events I'm talking about are things like file and socket I/O, not GUI events. Trying to use two different GUIs at once is not something I'm addressing. Rather, you should be able to write code that does e.g. some async socket I/O, and embed it in a GUI app using e.g. gtk, without having to modify it to take account of the fact that it's working in a gtk environment, or having to parameterise it to allow for such things.
You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms.
I'm arguing that as much of the platform dependency as possible should be in the asyncore library (or whatever replaces it). The main application code *might* have to give it a hint such as "this app uses gtk", but no more than that. And ideally, I'd prefer it not to even have to do that -- pygtk should do whatever is necessary to hook itself into asyncore if at all possible, not the other way around.
Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion.
Okay, but one of the suggestions made seemed to be "why not just use the Twisted API". I'm putting forward a possible reason. -- Greg
On Thu, 15 Feb 2007 15:47:39 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Steve Holden wrote:
If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in.
Since there should only be one reactor at a time in any given application, it shouldn't have to be passed in -- it could be held in a global variable deep inside the library. Only the code which creates the reactor initially needs to know about that variable, or even that there is such a thing as a reactor.
Whether or not the premise here is accurate may be out of scope for this thread. Or it may not be. I dunno. However, I do want to point out that it is not necessarily correct that there should be only one reactor at a time in a given application. PJE has already explained that peak.events can have multiple reactors. Twisted is tied to one, but this may not always be the case. Whether there is a default reactor for applications that don't care about the ability to have more than one at a time is yet another question which may be worth examining. These are the kinds of things which should be spelled out in a PEP, including the rationale for any particular policy decisions (which should be kept to an absolute minimum) are made.
"Incorporating some piece of event-driven code written by someone else" implies specific assumptions about event types and delivery, surely.
It requires agreement on how to specify the event types and what to do in response, but that's all it should require.
The way I envisage it, setting up an event callback should be like opening a file -- there's only one way to do it, and you don't have to worry about what the rest of the application is doing. You don't have to get passed an object that knows how to open files -- it's a fundamental service provided by the system. You just use it.
If we suppose that files and sockets are supported in roughly the same way, and we suppose that sockets are supported in the way that Twisted supports them, then there is no difficulty supporting files in this way. :)
That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change.
Just in case it's not clear, the events I'm talking about are things like file and socket I/O, not GUI events. Trying to use two different GUIs at once is not something I'm addressing.
Alright, good. Getting two different GUI libraries to play together is a pretty hairy task indeed, and well worth keeping separate from this one. :)
Rather, you should be able to write code that does e.g. some async socket I/O, and embed it in a GUI app using e.g. gtk, without having to modify it to take account of the fact that it's working in a gtk environment, or having to parameterise it to allow for such things.
Excellent. To be clear, this is how the Twisted model works, with respect to integration with GUI toolkits. I would not enjoy working with a system in which this was not the case.
You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms.
I'm arguing that as much of the platform dependency as possible should be in the asyncore library (or whatever replaces it).
Certainly. Library code doesn't care if the event loop is driven by select or poll or epoll or /dev/poll or kqueue or aio or iocp or win32 events or realtime signals or kaio or whatever gnarly thing is hidden in gtk or whatever gnarly thing is hidden inside qt or whatever gnarly thing is hidden inside COM or whatever gnarly thing is hidden inside wxWidgets. It cares about what features are available. It requests them somehow, and uses them. If they are unavailable, then it can decide whether the lack is catastrophic and give up or if it can be worked around somehow. The way a Twisted application does this is based on interfaces. Assuming interfaces continue to not be present in the stdlib, a stdlib event loop would have to find some other API for presenting this information, but it is not a very hard problem to solve.
The main application code *might* have to give it a hint such as "this app uses gtk", but no more than that. And ideally, I'd prefer it not to even have to do that -- pygtk should do whatever is necessary to hook itself into asyncore if at all possible, not the other way around.
There is some advantage to declaring things up front, lest you get into the situation where you are partway through using code which will suddenly begin to demand Gtk at the same time as you are partway through using code which will suddenly begin to demand Qt, at which point you are in trouble. But this is another minor point.
Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion.
Okay, but one of the suggestions made seemed to be "why not just use the Twisted API". I'm putting forward a possible reason.
So far, it sounds like the objections to using the Twisted API are primarily based on misunderstandings of it, and the actual desired API looks pretty much the same as Twisted's. Jean-Paul
On 2/14/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
glyph@divmod.com wrote:
On 08:52 pm, theller@ctypes.org wrote:
When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more.
Only the very top-most level decides which reactor the application will use.
This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level.
That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application.
This is not my idea of what another poster called a "one-stop shop" -- a common API that different pieces of code can call independently without having to know about each other.
To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used.
Eh, your own example seems to argue the opposite. If the choice for reactor was made somewhere deep inside the library, how does it know to use the GTK reactor? Only the code using GTK knows that it's going to need the GTK reactor. The generic code just uses whatever reactor was selected, automatically. The reason the decision should be made at the topmost level is that this is the most visible and adaptive location. A library or framework has to handle a boundless number of variations of environment, use and co-operation with other libraries and frameworks, whereas the topmost script knows exactly what it wants to do. A library that uses PyGTK can say 'I need the GTK reactor', but it would be wrong. All it needs is a reactor that interacts nicely with GTK. If there are multiple reactors that integrate nicely with GTK, there is no reason for the library to insist on one particular implementation. And it could be in the way, if a second library needs a different kind of integration and it knows of a reactor that supports both that *and* GTK integration. And if the whole system breaks, for some reason, the first place the programmer will look is in his own code, not deep inside another library. For what it's worth, I think a single event-handling mainloop API in the standard library is a *great* idea. Using part of Twisted for it (suitably adapted) is an even greater idea: a whole lot of work and brainpower went into Twisted's core, and it wasn't wasted. Duplicating the effort would be a big mistake and a terrible waste. A from-scratch implementation of an event-handling mechanism will not solve any of the "issues" most people have with Twisted. It will have many more bugs than Twisted (all the ones Twisted had and fixed, sometimes very obscure and hard-to-find bugs), it will not be significantly less complex (Twisted is as complex as it is because it had to be, to do what it is supposed to do) and it will not require any less of a brain-warp to understand (because that's just what event-driven programming takes.) If I wasn't short on free time (spending most of it on Py3k and, soon, py3k-forward-compatibility) I would do all the necessary Twisted-integration work myself :-) -- Thomas Wouters <thomas@python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
Thomas Wouters wrote:
If the choice for reactor was made somewhere deep inside the library, how does it know to use the GTK reactor?
In my ideal world, there wouldn't actually be a gtk reactor -- there would only be a Linux reactor, a MacOSX reactor, a Windows reactor, etc. Things like pygtk would be adapted to hook into the platform's standard reactor. Less ideal would be for pygtk to intall a gtk reactor when it gets imported. The danger with this approach is that two libraries could fight over which kind of reactor to use. You suggest that the top level could choose some other reactor that is compatible with both libraries. That seems like a rather hit-and-miss approach -- such a reactor might exist, or it might not. If not, you're out of luck. And the chance of finding a suitable reactor gets smaller as the number of libraries increases.
The reason the decision should be made at the topmost level is that this is the most visible and adaptive location.
This is where my vision is fundamentally different: you shouldn't have to *make* a decision in the first place. All event-driven libraries should be made to use the same substrate on any given platform. Then they can coexist without the need for any top-level choices. I know that will be hard to do, but it's the only way out of this mess that I can see. -- Greg
On 2/14/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I know that will be hard to do, but it's the only way out of this mess that I can see.
That depends on what you consider messy about it. *I* don't like the idea of something in the Python installation deciding which reactor to use. It's my application, and I'm damn well going to tell it what to do. If that means it doesn't work as I expected, it's my own fault :-) In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model. While those issues are being settled, we'll have to cope with selecting the right reactor manually. It's not all that different from what you want, in any case. The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code. -- Thomas Wouters <thomas@python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
Thomas Wouters wrote:
*I* don't like the idea of something in the Python installation deciding which reactor to use.
I wouldn't mind if some way were provided of changing the reactor if you want. I'd just like to see a long term goal of making it unnecessary as far as possible.
In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model.
On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it. A similar strategy ought to work for any X11-based toolkit that exposes a function to perform one iteration of its main loop. Mileage on other platforms may vary.
The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code.
Sure. The other side to all this is the client side, i.e. the code that installs event callbacks. At the moment there's no clear direction to take, so everyone makes their own choice -- some use asyncore, some use Twisted, some use the gtk event loop, some roll their own, etc. If it were made known that asyncore or some other thing in the stdlib was intended to become the standard, then it would give people some guidance as to how to write future event-driven code. -- Greg
Greg Ewing schrieb:
On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it.
That is insufficient. The gtk main loop has more input sources than just the connection to X: - timers can be registered, which are called when the time comes - idle handlers can be registered which are called when there are no other events - child handlers are invoked when a child process terminates - additional file descriptors can be registered (probably used for sockets primarily) - a generalzed 'event source' can be hooked into it, with C functions for prepare, check, dispatch, and finalize See http://www.gtk.org/api/2.6/glib/glib-The-Main-Event-Loop.html Regards, Martin
Martin v. Löwis wrote:
That is insufficient. The gtk main loop has more input sources than just the connection to X: - timers - idle handlers - child handlers - additional file descriptors - a generalzed 'event source'
When gtk is not the central event mechanism, there's no need to use the gtk event loop for these things -- you can just use the central event mechanism directly. The pygtk APIs for setting these up can redirect them to the appropriate place, to accommodate existing code that uses the gtk event loop for them. -- Greg
On 2/14/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Thomas Wouters wrote:
*I* don't like the idea of something in the Python installation deciding which reactor to use.
I wouldn't mind if some way were provided of changing the reactor if you want. I'd just like to see a long term goal of making it unnecessary as far as possible.
In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model.
On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it.
A similar strategy ought to work for any X11-based toolkit that exposes a function to perform one iteration of its main loop.
Mileage on other platforms may vary.
The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code.
Sure.
The other side to all this is the client side, i.e. the code that installs event callbacks. At the moment there's no clear direction to take, so everyone makes their own choice -- some use asyncore, some use Twisted, some use the gtk event loop, some roll their own, etc.
There is no single PerfectReactor. There are several use cases where you need to wait on >1 different event systems, which guarantees at least two OS threads (and two event loops). In general it's nice to have a single Python event loop ("the reactor") to act on said threads (e.g. something just sitting on a mutex waiting for messages) but waiting for IO to occur should *probably* happen on one or more ancillary threads -- one per event system (e.g. select, GTK, WaitForMultipleEvents, etc.) -bob
Bob Ippolito wrote:
There is no single PerfectReactor. There are several use cases where you need to wait on >1 different event systems, which guarantees at least two OS threads (and two event loops). In general it's nice to have a single Python event loop ("the reactor") to act on said threads (e.g. something just sitting on a mutex waiting for messages) but waiting for IO to occur should *probably* happen on one or more ancillary threads -- one per event system (e.g. select, GTK, WaitForMultipleEvents, etc.) Why couldn't PerfectReactor be a reactor for other reactors? A sort of concentrator for these multiple event systems and multiple threads.
You ask to listen to sockets, so it instantiates a singleton PerfectReactor which instantiates a select() reactor and listens to it directly in a single-threaded manner. If you then ask to listen to Win32 messages, the PerfectReactor instantiates a GetMessage() reactor. Then, realizing it has two "event systems", it spawns a thread for each child reactor with a listener that serializes the incoming events into the PerfectReactor's queue. Bingo, your application doesn't need to be written thread-safe, PerfectReactor is platform-agnostic, and you don't have to know in advance all the event types you might ever listen to. Sorry if this is a dumb question (or if I'm mangling the terminology), /larry/
Larry Hastings schrieb:
Bob Ippolito wrote:
There is no single PerfectReactor. There are several use cases where you need to wait on >1 different event systems, which guarantees at least two OS threads (and two event loops). In general it's nice to have a single Python event loop ("the reactor") to act on said threads (e.g. something just sitting on a mutex waiting for messages) but waiting for IO to occur should *probably* happen on one or more ancillary threads -- one per event system (e.g. select, GTK, WaitForMultipleEvents, etc.) Why couldn't PerfectReactor be a reactor for other reactors? A sort of concentrator for these multiple event systems and multiple threads.
Because you can't write a single function that blocks (idle) until an event in one of the systems occurs. None of the system is truly asynchronous. All APIs, eventually, rely on synchronous, blocking, operating system calls. In Unix, the call to make is typically select or poll, and you know it is safe to make because you have all even sources as file descriptors and you know nothing else needs to be done before one of the events occurs. Now, for these generalized event loops, it may not be possible anymore to combine all event sources into a single blocking call. If you have timeouts, you have to find out what the minimum timeout is (to make it the timeout for the blocking call, assuming that supports a timeout). If you have idle tasks, you have to invoke all idle tasks of all reactors (which you may not be able to get because there is no API to fetch them). Even more difficult: if these use different OS event mechanisms (on systems that have multiple such systems). E.g. try combinging a Win32 GetMessage/PostMessage loop with a WaitForMultipleObjects loop with a WinSock select call; then add the Win32 RPC libraries to that (i.e. try to determine the named pipe handles you have to pass to WaitForMultipleObjects).
You ask to listen to sockets, so it instantiates a singleton PerfectReactor which instantiates a select() reactor and listens to it directly in a single-threaded manner. If you then ask to listen to Win32 messages, the PerfectReactor instantiates a GetMessage() reactor. Then, realizing it has two "event systems", it spawns a thread for each child reactor with a listener that serializes the incoming events into the PerfectReactor's queue.
Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages. (This is indeed one of the biggest limitations of USER32, which was only overcome in Vista when the "native" GUI programming model is based on DirectX - the problem is then of course solved only for applications that use the DirectX UI API).
Bingo, your application doesn't need to be written thread-safe, PerfectReactor is platform-agnostic, and you don't have to know in advance all the event types you might ever listen to.
Sorry if this is a dumb question (or if I'm mangling the terminology),
Integrating with threads might be a solution in some cases, and a problem in others. You can't assume it is a universal solution. Regards, Martin
Martin v. Löwis wrote:
Now, for these generalized event loops, it may not be possible anymore to combine all event sources into a single blocking call.
Right, that's why my proposal assumed that each disparate event source would need its own thread.
Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages.
Oof! I'm embarrassed to have forgotten that. But that's not a fatal problem. It means that on Windows the PerfectReactor must service the blocking GetMessage loop, and these other threads notify the PerfectReactor of new events by sending a message. (Either that, or, it could poll GetMessage and its incoming event queue without ever blocking. But that is obviously suboptimal.) I think I've done this sort of thing before, in fact. Of course, in the absence of any windows, the Windows PerfectReactor could fall back to a mutex. Abstract this inside PerfectReactor and its event sources wouldn't notice the difference.
Integrating with threads might be a solution in some cases, and a problem in others. You can't assume it is a universal solution.
Universal? Yeah, I doubt it too. But perhaps it would be "good enough for nearly all cases". In the cases where it wasn't, it could throw an "I can't listen to that type of event right now" exception, forcing you to fall back to preconfiguring your central reactor by hand. Anyway, like many folks I'm hoping this whole conversation results in establishing basic standard duck-typed conventions--what other languages might call "interfaces"--for event senders and receivers. In which case this non-universal threaded approach would simply be one of several to choose from. I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable. Thanks for the thoughtful reply, /larry/
Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages.
I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable.
I've tried something similar on Linux, with gtk an wx. You can run the gtk main loop in its own thread, but because gtk is not thread safe, you have to grab a mutex everytime you run gtk code outside the thread the mainloop is running in. So you have to surround your calls to the gtk api with calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of course, because they are executed in the main thread... Doable, but not fun. The same goes for wx. Then all hell breaks loose when you try to use both gtk and wx at the same time. That's because on Linux, the wx main loop calls the gtk mainloop behind the scenes. As far as I know, that problem can not be solved from python. So yes that strategy can work, but it's no silver bullet. Cheers, Baptiste
On 2/15/07, Baptiste Carvello <baptiste13@altern.org> wrote:
Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages.
I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable.
I've tried something similar on Linux, with gtk an wx.
You can run the gtk main loop in its own thread, but because gtk is not thread safe, you have to grab a mutex everytime you run gtk code outside the thread the mainloop is running in. So you have to surround your calls to the gtk api with calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of course, because they are executed in the main thread... Doable, but not fun.
The same goes for wx. Then all hell breaks loose when you try to use both gtk and wx at the same time. That's because on Linux, the wx main loop calls the gtk mainloop behind the scenes. As far as I know, that problem can not be solved from python.
So yes that strategy can work, but it's no silver bullet.
And it's worse on Windows and Mac OS X where some GUI API calls *must* happen on a particular thread or they simply don't work. -bob
Larry Hastings schrieb:
Oof! I'm embarrassed to have forgotten that. But that's not a fatal problem. It means that on Windows the PerfectReactor must service the blocking GetMessage loop, and these other threads notify the PerfectReactor of new events by sending a message. ... I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable.
As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe. Speaking of which: Tcl has a "multiple interpreter" model where each interpreter is single-threaded. You can build Tcl either single- or multi-threaded; for the multi-threaded Tcl, you still can't share data across threads directly, but instead, you create an interpreter per thread. Then, all operations on that interpreter must occur in this thread, including the reactor calls. So if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl even loop in the Tcl thread, which is normally the main thread (in Tkinter, it's the thread where you originally created the tkapp object). OTOH, PerfectReactor may also need to run some Win32 event loop in the same thread. Regards, Martin
"Martin v. Löwis" <martin@v.loewis.de> wrote:
Larry Hastings schrieb:
Oof! I'm embarrassed to have forgotten that. But that's not a fatal problem. It means that on Windows the PerfectReactor must service the blocking GetMessage loop, and these other threads notify the PerfectReactor of new events by sending a message. ... I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable.
As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe.
At least on the wx side of things, there exist ways of scheduling calls to be executed inside its event dispatching loop (wx.CallAfter, wx.FutureCall, wx.PostEvent), which allows for worker threads to hand data back to the main thread (the creator of the wx.App instance, like Tk). I would be surprised if the other toolkits didn't have something similar, couldn't be manipulated into using Queues to communicate in some way, or couldn't be manipulated into directly sending a message on win32. - Josiah
As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe.
I'm not sure what your point is here. Let me see if I can clarify things, at least from my side. I suggest it is possible to implement a PerfectReactor. I define this PerfectReactor as a singleton top-level reactor which runs in the main thread of a Python application. All incoming events would be serialized to this single reactor, which would dispatch them to their appropriate listeners. When asked to monitor disparate event systems, the PerfectReactor would act as a concentrator of multiple other reactors, with various ugly things going on hidden in the implementation. The goal would be that modular libraries of event listeners could be written in an agnostic manner, simply requesting to listen to whatever events they need to, and the main application wouldn't have to know in advance what reactors to configure or how to configure them. You seem to assert it is not possible to implement such a PerfectReactor, and list counter-examples. So far so good. If you're saying "it is impossible for any implementation to simultaneously support more than one of the Win32, gtk, and wx reactors": fine, this hypothetical PerfectReactor can't do the impossible, but that's not an argument against a PerfectReactor. If it *is* possible to support more than one of these simultaneously through precise configuration, I assert that a PerfectReactor could support that configuration too, and the goal would be to have it automatically configure itself. You point out the gtk reactor and the wx reactor must be run in the main thread, because they run their own message pumps. What I had in mind was that, when you asked to start listening to Win32 events, the Win32 reactor would tell the PerfectReactor "I must be run in the main thread", and it would (re-) configure that to be the main thread's listener; the notification mechanism for other threads saying "there's a new event in your queue" would switch to using Win32 messages. If the gtk and wx reactors had the same restriction, then whichever was first introduced to the PerfectReactor first would win, and introducing subsequent reactors would throw an exception. (Again, can't do the impossible.) However, if gtk and wx expose enough of the plumbing of their message pumps, this central Win32 message pump could identify messages intended for the gtk and wx reactor message pumps and inject them. The gtk and wx reactors wouldn't call GetMessage, but they could call the rest of their message loops. Therefore they could all be run in the main thread. (And yes, "behind the scenes" is correct.)
if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl even loop in the Tcl thread, which is normally the main thread (in Tkinter, it's the thread where you originally created the tkapp object). OTOH, PerfectReactor may also need to run some Win32 event loop in the same thread.
Again, I'm not sure of your point. If you're saying "it is impossible to run multithreaded Tcl and any message pump (Win32, gtk, wx) in the same application", then fine, PerfectReactor cannot do the impossible. If it is only possible to do with clever configuration, then I maintain a PerfectReactor could self-configure to support this configuration at runtime. Cheers, /larry/
On Thu, 15 Feb 2007 16:18:40 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
[snip]
This is where my vision is fundamentally different: you shouldn't have to *make* a decision in the first place. All event-driven libraries should be made to use the same substrate on any given platform. Then they can coexist without the need for any top-level choices.
I know that will be hard to do, but it's the only way out of this mess that I can see.
Thomas already pointed this out, but I'm repeating it anyway. This vision represents an impossible reality at present. You will not get Gtk or Qt or wxWidgets to use Python's event notification API. If you are really very interested in solving this problem, go to the developers of each platform those toolkits run on and sell them on a unified event notification API. Once they have adopted, implemented, and deployed it, you can go to the Gtk, Qt, and wxWidgets teams and tell them to port all of their code to that new API. Then, you can have a unified model in Python. Until then, the practical compromise with almost zero negative consequences (sometimes, one extra piece of configuration will be required - compare this to how the logging module works ;) is to optionally allow explicit reactor selection. Jean-Paul
At 01:31 PM 2/15/2007 +1300, Greg Ewing wrote:
To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used.
*shudder*. I, on the other hand, prefer to assume that there is no one "top level" and certainly no requirement for a single event loop or reactor. peak.events, for example, lets you have multiple event loops running in the same or different threads. One of these can be Twisted's reactor, if you like, but the framework doesn't impose this singleton-ness on you. (IIUC, Twisted uses a singleton for at least two fairly good reasons: ease of programming, and the fact that certain platforms demand it for performance reasons. I just don't happen to agree that this limitation should be applied across *all* platforms. And there are good solutions for context-specific pseudo-singletons to address the API issue, although they might not be as high-performance as Twisted applications might prefer. Everything's a trade-off.)
Phillip J. Eby wrote:
peak.events, for example, lets you have multiple event loops running in the same or different threads.
Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads. And... how do you run multiple event loops simultaneously in the *same* thread? That sounds self-contradictory to me. -- Greg
On 2/14/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Phillip J. Eby wrote:
peak.events, for example, lets you have multiple event loops running in the same or different threads.
Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads.
And... how do you run multiple event loops simultaneously in the *same* thread? That sounds self-contradictory to me.
If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop. Or you can design all mainloops as coroutines and have them call each other. (I haven't looked at Phillip's approach at all, but something tells me coroutines are involved :-) -- Thomas Wouters <thomas@python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
Thomas Wouters wrote:
If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop.
But without some way of blocking until an event arrives for *either* loop, you have to resort to some kind of busy polling, which is not elegant. -- Greg
Thomas Wouters schrieb:
If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop.
That doesn't really work (and neither do variations involving coroutines. Either the 'run one iteration' method blocks until one even arrives, in which case it may block for a long time while other loops have events pending (which then won't get processed). Or each 'run one iteration' method has a polling mode (e.g. with a time-out), in which case you get busy wait (you can fiddle with the time-out value to trade business vs. responsiveness). The 'run one iteration' approach doesn't support writing applications that are single-threaded, responsive, and idle when there is nothing to do. I can't believe that coroutines help in any way here. Regards, Martin
At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote:
Phillip J. Eby wrote:
peak.events, for example, lets you have multiple event loops running in the same or different threads.
Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads.
And... how do you run multiple event loops simultaneously in the *same* thread?
When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI "accept" event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished.
Phillip J. Eby wrote:
At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote:
peak.events, for example, lets you have multiple event loops running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use
Phillip J. Eby wrote: threads.
And... how do you run multiple event loops simultaneously in the *same* thread?
When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI "accept" event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished.
I actually have some code that works a little bit like this. I have a SelectLoop class that the application knows about and explicitly attaches transport objects to, and queues events on the loop. I have two ways to perform some actions in a "blocking" way. 1) I can create a new SelectLoop and move some transports temporarily from the main loop to the temporary loop to ignore all other events for a while. 2) I can re-enter the SelectLoop run() method and keep reacting to all events. Once the right event occurs - I exit from the nested run() and continue on. This does require a lot more mental effort though. Note that my code is nowhere near as ambitious as Twisted, but I watned to have the above flexibility. The code really only has to work on AIX and Linux, and in a limited way, Windows. - Dave
On 2/14/07, Phillip J. Eby <pje@telecommunity.com> wrote:
When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI "accept" event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished.
This would also let you wrap sys.stdout.write() in a nested event loop so as to allow print statements to still work while you use have it set to non-blocking mode, but I could see it argued that using print statements at all is wrong at that point. -- Adam Olsen, aka Rhamphoryncus
participants (13)
-
"Martin v. Löwis" -
Adam Olsen -
Baptiste Carvello -
Bob Ippolito -
Dave Cole -
glyph@divmod.com -
Greg Ewing -
Jean-Paul Calderone -
Josiah Carlson -
Larry Hastings -
Phillip J. Eby -
Steve Holden -
Thomas Wouters