[Twisted-Python] removing unsupported reactors in twisted 2.6: qt, corefoundation, threadedselect, wx
There are a few reactors in the twisted codebase which currently have no maintainer and do not pass the test suite. I've listed them in the subject of this message. I'm comfortable to leave half-supported or potentially broken reactors in the Twisted distribution, as long as they were clearly marked as such and there was some at least potential path to making them work. IOCP, for example, is good enough for many uses despite not passing the full test suite (and there are rumours of a new buildbot for it appearing soon...) However, leaving a pile of known-to-be-broken and unmaintained reactors in Twisted is just going to give the impression that Twisted is buggy, and increase the amount of dead code that people reading the codebase can be confused by. The reactors presented in the subject all seem to have been abandoned. Even our friends at Apple are mainly concerned with the use of Twisted for server work, and while they've been attempting to help us find someone to poke around GUI <-> reactor integration on OS X, it's not their area of expertise. At this point, I believe that setting up a buildslave for any of these reactors should be possible with our existing build infrastructure. Of course, contributions of additional buildslaves are always welcome :), but we can set up slaves for wx and corefoundation ourselves if someone is going to actually start trying to fix them. I know that there are people in the Twisted community integrating it with wxpython and qt. I'd really like it if someone could commit to at least contributing some patches in the future. If I don't hear anything on these fronts before the Twisted 2.6 release, those reactors will be removed from SVN trunk@HEAD and not be included in the 2.6 release. (The next planned release is Twisted 2.5, so you still have some breathing room.) Currently qtreactor is being tested on the "reactors" buildslave and is keeping it red, despite the fact that all the other reactors on that slave work fine (modulo a few elusive race conditions). wxreactor is not being tested at all. I believe we should split qt out so that it can have its own red column and not interfere with the gtk and poll reactors, and if we run a wx slave it should also be separate. Although I know of Qt and WX users, I do not know of anyone currently using the corefoundation reactor in an application. Are you out there? Is it being used? The last word I heard on the corefoundation reactor was from its author, Bob Ippolito, who suggested that it was not the correct way to integrate Twisted with GUI applications. However, his suggested replacement (threadedselectreactor) is buggy, also unmaintained, and IMHO misdesigned, so in the absence of other knowledge I believe that continuing to provide some corefoundation support would be the best way to support the mac. After receiving a shiny new computer from them I feel inclined to do the best job possible to support Apple's platform, but I simply don't have the time, knowledge, or energy to do the development myself. With the aforementioned hardware, I believe it would not be much trouble for us to set up a corefoundation reactor buildslave of our own, but we still need someone who will actually work on fixing the bugs, otherwise, the code should be removed. threadedselectreactor is an interesting concept but hopelessly broken and untested in its current implementation. I believe it should definitely be removed as a reactor of its own, although as a supporting basis for other reactors it might be good to leave around. While not entirely correct, it is certainly _less_ broken than the previous ghastly attempt at a wxreactor. I think it would be a good idea to remove threadedselect from the reactors list in Twisted 2.6, although if someone wants to convince me otherwise with a use-case (and a similar promise of future patches), I'd be happy to hear it. Once all these reactors have dedicated buildslaves and pass the tests, it is far less important that we have dedicated maintainers. We can simply insist that all future commits keep all the buildslaves green. I say this because I want to be clear that I am not asking for a lifetime commitment or we are going to instantly desupport your platform: I'm asking for help getting to the point where we can support multiple platforms without having to be experts in all of them.
Hi Glyph, I completely agree with your position on moving those reactors out of the production Twisted codebase. The only one remotely interesting to me is the wx reactor, but having thought about this issue for a long time, I've decided to try the Pyro connection (proposed by someone). If I can get that to work (which I haven't had time to play with yet), it would seem to offer several advantages: (1) running wx and twisted event loops in completely separate processes, (2) making relatively minimal use of threads in the wx process (well-supported by the wx libraries) to drive Pyro, and using the usual twisted paradigm in the twisted process. I would appreciate any advice, but I don't see any show-stoppers. I would do the Pyro communications on 127.0.0.1, since Pyro has essentially zero security, keeping it completely within the local machine. Twisted would handle all external protocols. Thoughts? Cheers, Steve
On Sat, 23 Sep 2006 23:33:11 -0400, Stephen Waterbury
If I can get that to work (which I haven't had time to play with yet), it would seem to offer several advantages: (1) running wx and twisted event loops in completely separate processes, (2) making relatively minimal use of threads in the wx process (well-supported by the wx libraries) to drive Pyro, and using the usual twisted paradigm in the twisted process. I would appreciate any advice, but I don't see any show-stoppers. I would do the Pyro communications on 127.0.0.1, since Pyro has essentially zero security, keeping it completely within the local machine. Twisted would handle all external protocols.
Thanks for the suggestion, Stephen. I don't think I like this approach though. The major reason is that it introduces a process boundary between Twisted code and WX code. In every other toolkit Twisted supports, you can handle GUI events and network events in the same process, in the same thread. That makes it possible to have individual state machines which can emit output in the form of either GUI updates and writes to streams, and you can similarly receive GUI events or data from the network as input. Keep in mind that each of the toolkits that WX backends to already *has* either a supported reactor or a proof of concept for Twisted integration: GTK+, the Win32 event loop, and Cococa. It's not that we can't get network I/O notifications from the platform. The real problem here is that WX obscures the underlying APIs for integrating sockets. (I say "obscures" rather than "does not wrap" because the last time I tried to use gtk reactor with a gtk build of wx, the process segfaulted after emitting a pile of dynamic linker errors. I have no idea what would happen on Windows or the Mac.) Whatever we're going to do at this point without getting some new code from the WX developers is going to be a workaround. That said, the mechanism that you propose may be the best way to practically integrate Twisted and WX today. If it is, I think that it would work better as a separate project, because it is a workaround with a lot of pointy edges and not a real solution. One major reason I'd like a WX reactor to continue to exist in some form is OSAF's usage of Twisted and WX. They have a similarly baroque model for handing data back and forth between Twisted code and GUI code, and I'd love to have something to propose to them that was (A) supported by Twisted (B) well tested, and (C) didn't involve lots of potential race conditions, as all shared-state-threading code was likely to do. I haven't looked at it in a while, but if I remember correctly their separation between Twisted code and GUI code was fairly clean and well-thought-out, but the separation would have been unnecessary if wx had native Twisted integration. There are other problems with a Pyro-driven Twisted subprocess, which are relevant as the goal here is a well-supported and well-tested path for integration. Multiprocess communication and process spawning, especially on Windows, is fraught with peril. We've been working on reliable event-driven process spawning in Twisted for years and years, and only recently have we gotten to the point where the published API works reliably cross-platform. Given that Twisted would be left out of this loop, I imagine the process control would be rather ad-hoc, so getting the Twisted and WX processes synchronized would be problematic, and extremely hard to unit test. Although it would be slightly easier if the Twisted proces is 'on top', you still have to inject test code into both the parent and child processes, and have a way to verify the state of both. The last release of pyro is almost a year old. That suggests it is not actively maintained. Pyro doesn't have a buildbot, or (as far as I can tell from the release) any automated tests of its own. That would make it a wash in terms of tested, robust integration support. You'd have to re-implement the Pyro protocol on top of Twisted to communicate with it without spawning lots of threads. That's a bunch of new code which is going to need a home, again, possibly a whole new project. Even after that's done, Pyro is going to be spawning threads and blocking inside the WX process because the Twisted implementation isn't running there.
glyph@divmod.com wrote:
On Sat, 23 Sep 2006 23:33:11 -0400, Stephen Waterbury
wrote: If I can get that to work (which I haven't had time to play with yet), it would seem to offer several advantages: (1) running wx and twisted event loops in completely separate processes, (2) making relatively minimal use of threads in the wx process (well-supported by the wx libraries) to drive Pyro, and using the usual twisted paradigm in the twisted process. I would appreciate any advice, but I don't see any show-stoppers. I would do the Pyro communications on 127.0.0.1, since Pyro has essentially zero security, keeping it completely within the local machine. Twisted would handle all external protocols.
Thanks for the suggestion, Stephen. I don't think I like this approach though.
The major reason is that it introduces a process boundary between Twisted code and WX code.
I see that as a *good* thing. ;)
In every other toolkit Twisted supports,
... of which there's really only *one* (GTK) that's at all interesting to anyone doing cross-platform apps ... and even that one, IMO, isn't very cross-platform, although that could certainly be argued either way ...
you can handle GUI events and network events in the same process, in the same thread. That makes it possible to have individual state machines which can emit output in the form of either GUI updates and writes to streams, and you can similarly receive GUI events or data from the network as input.
Okay. I'm willing to do without that.
Keep in mind that each of the toolkits that WX backends to already *has* either a supported reactor or a proof of concept for Twisted integration: GTK+, the Win32 event loop, and Cococa. It's not that we can't get network I/O notifications from the platform. The real problem here is that WX obscures the underlying APIs for integrating sockets. (I say "obscures" rather than "does not wrap" because the last time I tried to use gtk reactor with a gtk build of wx, the process segfaulted after emitting a pile of dynamic linker errors. I have no idea what would happen on Windows or the Mac.) Whatever we're going to do at this point without getting some new code from the WX developers is going to be a workaround.
I think *that* is the real problem ... not technical, just interest from the right people. I've waited enough years (about 5) that I've given up on that.
That said, the mechanism that you propose may be the best way to practically integrate Twisted and WX today. If it is, I think that it would work better as a separate project, because it is a workaround with a lot of pointy edges and not a real solution.
What's reality? :) OTOH ... I completely agree that it should not be part of Twisted at all ... but if I succeed with it, I'll certainly document what I do.
One major reason I'd like a WX reactor to continue to exist in some form is OSAF's usage of Twisted and WX. They have a similarly baroque model for handing data back and forth between Twisted code and GUI code, ...
... and everything else (sorry, couldn't resist ;) ...
... and I'd love to have something to propose to them that was (A) supported by Twisted (B) well tested, and (C) didn't involve lots of potential race conditions, as all shared-state-threading code was likely to do. I haven't looked at it in a while, but if I remember correctly their separation between Twisted code and GUI code was fairly clean and well-thought-out,
... hmmm ... I've been on all the OSAF Chandler/Scooby/Cosmo lists since April (and that's a lot of freakin' mail ;), and not *once* in that interval has "twisted" occurred in the subject line of a message, and no substantive discussion of it in the bodies either. I know about their zanshin twisted-based framework for WebDAV/CalDAV, and that it's used in their Sharing architecture, which is used by the Chandler wx GUI, but I just updated my chandler svn checkout, and I counted 17 occurrences of the zanshin "blockUntil" function in the Sharing.py module ... if it's so clean, how come they need those? (In zanshin's docs, "blockUntil" is billed as "for test purposes" ... ;)
but the separation would have been unnecessary if wx had native Twisted integration.
Well, yeah, but I don't think OSAF has the resources to put into it either, and Robin *works* for OSAF.
There are other problems with a Pyro-driven Twisted subprocess, which are relevant as the goal here is a well-supported and well-tested path for integration.
Multiprocess communication and process spawning, especially on Windows, is fraught with peril. We've been working on reliable event-driven process spawning in Twisted for years and years, and only recently have we gotten to the point where the published API works reliably cross-platform. Given that Twisted would be left out of this loop, I imagine the process control would be rather ad-hoc, so getting the Twisted and WX processes synchronized would be problematic, and extremely hard to unit test. Although it would be slightly easier if the Twisted proces is 'on top', you still have to inject test code into both the parent and child processes, and have a way to verify the state of both.
The last release of pyro is almost a year old. That suggests it is not actively maintained.
Or else it just works. ;)
Pyro doesn't have a buildbot, or (as far as I can tell from the release) any automated tests of its own. That would make it a wash in terms of tested, robust integration support.
As if there's going to be some other "tested, robust integration support" ... nothing on the horizon that I can see (and as I say, I've been watching for a long time).
You'd have to re-implement the Pyro protocol on top of Twisted to communicate with it without spawning lots of threads. That's a bunch of new code which is going to need a home, again, possibly a whole new project. Even after that's done, Pyro is going to be spawning threads and blocking inside the WX process because the Twisted implementation isn't running there.
Process spawning! Rusty code! Spawning threads, *and* blocking! My, you do paint a rosy picture. All your handwaving and FUD makes me want to try it, because it's probably quicker and easier than discussing it. But thanks for the warnings, anyway! ;) Cheers, Steve
On Sun, 24 Sep 2006 04:53:54 -0400, Stephen Waterbury
glyph@divmod.com wrote:
On Sat, 23 Sep 2006 23:33:11 -0400, Stephen Waterbury
wrote:
In every other toolkit Twisted supports,
... of which there's really only *one* (GTK)
Fair enough. I meant to say that the way Twisted apps are designed is to treat all events as events. This would require writing extra glue code for certain tyes of events.
Okay. I'm willing to do without that.
In that case, more power to you :)
I think *that* is the real problem ... not technical, just interest from the right people. I've waited enough years (about 5) that I've given up on that.
I've done a lot of waiting too. Now I'm starting with the yelling and hollering, see if that strategy works better :).
I completely agree that it should not be part of Twisted at all ... but if I succeed with it, I'll certainly document what I do.
Sounds great to me.
One major reason I'd like a WX reactor to continue to exist in some form is OSAF's usage of Twisted and WX. They have a similarly baroque model for handing data back and forth between Twisted code and GUI code, ...
... and everything else (sorry, couldn't resist ;) ...
Heh. Let's just keep it straight on the record - *I* didn't say it! :-)
... hmmm ... I've been on all the OSAF Chandler/Scooby/Cosmo lists since April (and that's a lot of freakin' mail ;), and not *once* in that interval has "twisted" occurred in the subject line of a message, and no substantive discussion of it in the bodies either.
Yeah, I don't think that this is really a high priority issue for them. Their application works fine in its current state, and they have a LOT of programmers working on it. A change to the architecture at this point would be so expensive that any other benefits which might come along with it wouldn't be worth it.
I know about their zanshin twisted-based framework for WebDAV/CalDAV, and that it's used in their Sharing architecture, which is used by the Chandler wx GUI, but I just updated my chandler svn checkout, and I counted 17 occurrences of the zanshin "blockUntil" function in the Sharing.py module ... if it's so clean, how come they need those?
I was just trying to be nice :-P.
(In zanshin's docs, "blockUntil" is billed as "for test purposes" ... ;)
Oh yeah, I remember that "test" function... heh heh. I wish them much luck in the 5-year 1000-man project it takes to remove it.
Well, yeah, but I don't think OSAF has the resources to put into it either, and Robin *works* for OSAF.
I just don't think it's a relevant issue for them. They're not even using the code I'm talking about removing.
As if there's going to be some other "tested, robust integration support" ... nothing on the horizon that I can see (and as I say, I've been watching for a long time).
Understandable...
Process spawning! Rusty code! Spawning threads, *and* blocking! My, you do paint a rosy picture. All your handwaving and FUD makes me want to try it, because it's probably quicker and easier than discussing it. But thanks for the warnings, anyway! ;)
What can I say? Everything is completely terrible. Still, good luck trying to make that work; let us know how it goes.
I haven't received the original email, for some reason, so apologies for respoding in wrong place in thread. Removing is a bad idea, although adding quality warning seems like a good idea: qt is failing *7* tests out of 3034, all but one i; that doesn't seem to merit removing it. If there were serious problems I would expect we would have many bug reports from qt users, but we don't. cf... dunno. tsreactor is private in twisted - starts with _, so it doesn't really guarantee anything or make any promises about working, and it's very useful for integrating with broken event loops. wxreactor is a tsreactor wrapper which makes it suck less for the wx case. AFAICT wx is one of the most popular ui toolkits for python, so people will want to use it; wxreactor for all it faults is in most cases going to be way better than whatever people try to do on their own, especially if the branch I have lying around is ever merged. The basic argument here is the same: 1. People want to use these GUI toolkits. 2. They will thus either drop Twisted and write their own networking framework, or they will write their own GUI integration code. Both of these cases result in code that is likely will be worse than our current providing. So as long we are up front and visible about existing limitations in these reactors, our users benefit by us including them.
On Sun, 24 Sep 2006 10:54:15 -0400, Itamar Shtull-Trauring
Removing is a bad idea, although adding quality warning seems like a good idea:
Where would you suggest adding that warning?
qt is failing *7* tests out of 3034, all but one i; that doesn't seem to merit removing it. If there were serious problems I would expect we would have many bug reports from qt users, but we don't.
I don't expect that we'll ever hear from users about anything, unless we threaten them directly :). Qt is definitely in a better situation than the other reactors we're talking about. However, it's very likely that code could be added or changed which would make it "redder" and nobody would notice until well after the fact. Can we set those tests to 'skip' or 'todo' in the context of the Qt reactor and call that "good enough"? tsreactor failed something like 30 tests the first time I ran trial on it and something like 100 the second time. The third run did not complete. I assume it's mainly race conditions which are killing it, but I don't know where they are.
tsreactor is private in twisted - starts with _, so it doesn't really guarantee anything or make any promises about working, and it's very useful for integrating with broken event loops.
It's exposed to users as a reactor on the command line. Maybe *that* should have an "_" added to it as well. However, if we're going to keep it around, it should have a buildbot, end of story.
wxreactor is a tsreactor wrapper which makes it suck less for the wx case. AFAICT wx is one of the most popular ui toolkits for python, so people will want to use it;
If it's so popular, why does nobody care enough to contribute a few patches to make it work properly?
wxreactor for all it faults is in most cases going to be way better than whatever people try to do on their own,
I agree, but mainly because the average case of "whatever people try to do on their own" is going to be ad-hoc and untested. Right now the alternative we are proposing is also ad-hoc and untested, just in a smaller way.
especially if the branch I have lying around is ever merged.
What branch is that? Should it be marked for review? At any rate, it sounds like YOU care enough about wxreactor to put some effor into it, which is enough to satisfy me. Should we add a --reactor option for wx, and set up a buildbot for wxreactor?
The basic argument here is the same:
1. People want to use these GUI toolkits. 2. They will thus either drop Twisted and write their own networking framework, or they will write their own GUI integration code. Both of these cases result in code that is likely will be worse than our current providing.
This is true for the moment, but eventually some of the code in quesiton is going to decay to the point where it is so seriously out of sync with the rest of Twisted that it won't even import.
So as long we are up front and visible about existing limitations in these reactors, our users benefit by us including them.
I don't think we have nearly enough visibility on the quality or stability of those reactors. "Stability: Unstable" means "we might change this", not "it is known to be broken and to fail numerous tests".
glyph@divmod.com wrote:
On Sun, 24 Sep 2006 10:54:15 -0400, Itamar Shtull-Trauring [...]
So as long we are up front and visible about existing limitations in these reactors, our users benefit by us including them.
I don't think we have nearly enough visibility on the quality or stability of those reactors. "Stability: Unstable" means "we might change this", not "it is known to be broken and to fail numerous tests".
As a tangent, people seem to get confused by this. Perhaps it would be less ambiguous to replace "Stability: Unstable" with something like: API status: in flux or: API status: unsupported and "Stability: stable" with: API status: supported Just to remove the ambiguity? -Andrew.
On 9/24/06, Andrew Bennetts
As a tangent, people seem to get confused by this. Perhaps it would be less ambiguous to replace "Stability: Unstable" with something like:
API status: in flux
or:
API status: unsupported
and "Stability: stable" with:
API status: supported
Just to remove the ambiguity?
I heartily second "in flux". I've gotten lots of e-mail from t.w.p.oscar users asking what "unstable" means. -p -- Paul Swartz paulswartz at gmail dot com http://z3p.livejournal.com/ AIM: z3penguin
On Sun, 24 Sep 2006 22:46:20 -0400, Paul Swartz
On 9/24/06, Andrew Bennetts
wrote: As a tangent, people seem to get confused by this. Perhaps it would be less ambiguous to replace "Stability: Unstable" with something like:
API status: in flux
I heartily second "in flux". I've gotten lots of e-mail from t.w.p.oscar users asking what "unstable" means.
I suspect that the phrase "in flux" may be equally jargonesque, although it does make more sense to me personally. There are 3 problems here though: 1. Modules' stability markers are not kept up-to-date as community perception changes. Not much that we can do about that except have periodic docstring reviews. 2. The terminology doesn't immediately make much sense. Changing the terminology in the way that Andrew suggests could probably improve this situation, but I think that it would not be a complete solution to the problem, because: 3. There's no good description of what we mean by the specialized terms we have chosen to describe a module's status. I think that means we need to update this document: http://twistedmatrix.com/projects/core/documentation/howto/faq.html#auto5 regardless of any change (or no change) to the terminology.
On 9/25/06, Andrew Bennetts
As a tangent, people seem to get confused by this. Perhaps it would be less ambiguous to replace "Stability: Unstable" with something like:
API status: in flux
I reckon this is a good idea. However, it would be good to know exactly what 'supported' and 'unsupported' mean -- particularly in terms of DeprecationWarnings, releases etc. cheers, jml
2006/9/23, glyph@divmod.com
Although I know of Qt and WX users, I do not know of anyone currently using the corefoundation reactor in an application. Are you out there? Is it being used? [snip]
So the only GUI toolkits that will be officialy supported are going to be Gtk, Tk and PyUI? -- Felipe.
On Sun, 24 Sep 2006 00:44:03 -0300, Felipe Almeida Lessa
So the only GUI toolkits that will be officialy supported are going to be Gtk, Tk and PyUI?
Thanks for reminding me. Neither Tk nor PyUI have buildbots. However, they're both implemented using polling timed events running on top of another reactor... I don't know how much "support" those really count for. It might be good to remove them as well, but they're very little code. GTK is supported, has a buildbot, and is currently green. Frankly, the people who currently do at least 90% of the maintenance on Twisted (myself included) only care about this toolkit, so it's the one that is likely to remain the best supported. It is my sincere hope that someone will port GTK to the Cocoa APIs for OS X at some point in the near future. Then I can cease caring even about corefoundation ;). The Win32 event loop is also supported, although its buildbot still isn't green yet. That is a "toolkit" of sorts. I think that it might even work with MFC but I'm a bit murky on the details. So, yes, the situation is even worse than you suggest; unless someone else volunteers to help, we will have 2 supported toolkits: GTK, and GDI. Twisted's architecture was developed with GUI toolkits in mind, and I would like it to support as many as possible. I hope that one day, all toolkit developers will actually care whether Twisted supports their code or not. Until now, though, it seems that we have been unable to attract the attention of even one member of each community with the appropriate experience and motivation to help out.
2006/9/24, glyph@divmod.com
Frankly, the people who currently do at least 90% of the maintenance on Twisted (myself included) only care about this toolkit, so it's the one that is likely to remain the best supported.
I'm developing a small Twisted application and was planning to write it using wx to learn how to use this toolkit, that's why I asked about it. Looking at the docs and the source, it sounds like wxreactor works nicely -- the author says that tests are expected to fail --, so it's somewhat disappointing to know that it's going to be removed from the new Twisted version. Nevertheless, I don't have the knowledge to fix whatever bugs it have. So maybe I'll stick with Gtk =). Thanks, -- Felipe.
Well, for me the threadedselectreactor works quite nicely with wx. It has it's flaws and I've spent some time working around them. On the other hand if that reactor is dropped, I'd rather drop Twisted from my application than rewrite the GUI part with a toolkit that kind of sucks and looks like crap on most platforms (i.e. GTK or TK). I like Twisted PB and I like wx - that's pretty much all I have to say about this. Let's say it this way: the network part of an application is what makes it work, the GUI part is what makes the money - guess what's dropped first :-( UC On Saturday 23 September 2006 21:31, glyph@divmod.com wrote:
On Sun, 24 Sep 2006 00:44:03 -0300, Felipe Almeida Lessa
wrote: So the only GUI toolkits that will be officialy supported are going to be Gtk, Tk and PyUI?
Thanks for reminding me.
Neither Tk nor PyUI have buildbots. However, they're both implemented using polling timed events running on top of another reactor... I don't know how much "support" those really count for. It might be good to remove them as well, but they're very little code.
GTK is supported, has a buildbot, and is currently green. Frankly, the people who currently do at least 90% of the maintenance on Twisted (myself included) only care about this toolkit, so it's the one that is likely to remain the best supported. It is my sincere hope that someone will port GTK to the Cocoa APIs for OS X at some point in the near future. Then I can cease caring even about corefoundation ;).
The Win32 event loop is also supported, although its buildbot still isn't green yet. That is a "toolkit" of sorts. I think that it might even work with MFC but I'm a bit murky on the details.
So, yes, the situation is even worse than you suggest; unless someone else volunteers to help, we will have 2 supported toolkits: GTK, and GDI.
Twisted's architecture was developed with GUI toolkits in mind, and I would like it to support as many as possible. I hope that one day, all toolkit developers will actually care whether Twisted supports their code or not. Until now, though, it seems that we have been unable to attract the attention of even one member of each community with the appropriate experience and motivation to help out.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- UC -- Open Source Solutions 4U, LLC 1618 Kelly St Phone: +1 707 568 3056 Santa Rosa, CA 95401 Cell: +1 650 302 2405 United States Fax: +1 707 568 6416
On Sat, 23 Sep 2006 22:41:23 -0700, "Uwe C. Schroeder"
Well, for me the threadedselectreactor works quite nicely with wx. It has it's flaws and I've spent some time working around them. On the other hand if that reactor is dropped, I'd rather drop Twisted from my application than rewrite the GUI part with a toolkit that kind of sucks and looks like crap on most platforms (i.e. GTK or TK).
You have lots of other options. 1. Continue using the wxreactor code by copying it into your application. We could even provide a separate page with "unmaintained reactors" for download. That way you could fix the reactor I'm just talking about removing the code from Twisted, not ordering soldiers to destroy it everywhere that it exists. In fact, this would probably be better for you anyway, if you intend to keep up with future versions of Twisted, because it would allow you to apply hacks to keep your particular application working even if the reactor in question breaks with a new release of Twisted. 2. Help fix the problem. This is really what I'd prefer. You don't have to have all the appropriate knowledge right now, just sufficient interest and energy to commit to *caring* that the buildbot is red in the future, and trying to fix it. 3. Find someone else to fix the problem. Do you know any other wx/twisted users who don't subscribe to this list, but know something about networking?
I like Twisted PB and I like wx - that's pretty much all I have to say about this. Let's say it this way: the network part of an application is what makes it work, the GUI part is what makes the money - guess what's dropped first :-(
4. Finance someone else to fix the problem, since you say you're making money from this. Maybe you can find a university student with enough time on their hands who would do it for a few hundred bucks. If you want to make money off of other people's work without contributing anything back, you may find that occasionally they do not do things in the most convenient way possible for you :).
glyph@divmod.com wrote:
You have lots of other options.
1. Continue using the wxreactor code by copying it into your application. We could even provide a separate page with "unmaintained reactors" for download.
For me, if you don't find someone that help you to maintain the wx reactor, this are a good solution, since there are a lot of person, like me, that use the (fantastic) twisted library and wx like GUI.
2. Help fix the problem. This is really what I'd prefer. You don't have to have all the appropriate knowledge right now, just sufficient interest and energy to commit to *caring* that the buildbot is red in the future, and trying to fix it.
Just for see if I can help, I tried the "trial --reactor threadedselect twisted", but after some (10 min) work, trial eat all my ram! and I must close it with e "kill -9". Are there another method for execute trial? Bye, Michele P.s.: michele:~$ python Python 2.4.4c0 (#2, Jul 30 2006, 15:43:58) [GCC 4.1.2 20060715 (prerelease) (Debian 4.1.1-9)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import twisted twisted.__version__ '2.4.0'
On Mon, 25 Sep 2006 18:06:50 +0200, Michele Petrazzo
glyph@divmod.com wrote:
You have lots of other options.
1. Continue using the wxreactor code by copying it into your application. We could even provide a separate page with "unmaintained reactors" for download.
For me, if you don't find someone that help you to maintain the wx reactor, this are a good solution, since there are a lot of person, like me, that use the (fantastic) twisted library and wx like GUI.
2. Help fix the problem. This is really what I'd prefer. You don't have to have all the appropriate knowledge right now, just sufficient interest and energy to commit to *caring* that the buildbot is red in the future, and trying to fix it.
Just for see if I can help, I tried the "trial --reactor threadedselect twisted", but after some (10 min) work, trial eat all my ram! and I must close it with e "kill -9". Are there another method for execute trial?
As a comparison, have you run trial with the default reactor? If this fails the same way, your machine have an odd configuration which is fouling up the tests, or it may simply not have enough ram to run the full test suite. On the other hand, if this runs successfully, then what you are seeing is basically one aspect of the current problem with threadedselect reactor :) Jean-Paul
Jean-Paul Calderone wrote:
Just for see if I can help, I tried the "trial --reactor threadedselect twisted", but after some (10 min) work, trial eat all my ram! and I must close it with e "kill -9". Are there another method for execute trial?
As a comparison, have you run trial with the default reactor? If this fails the same way, your machine have an odd configuration which is fouling up the tests, or it may simply not have enough ram to run the full test suite.
My top say: Mem: 776656k total, 592024k used, 184632k free, 31956k buffers and after two minutes of "trial twisted", 5273 michele 25 0 250m 203m 5012 R 95.9 26.8 1:13.57 trial What can I try, or what can I modify for make the tests? All my packages are default packages from debian testing repository! Michele
On Mon, 25 Sep 2006 18:49:12 +0200, Michele Petrazzo
Jean-Paul Calderone wrote:
Just for see if I can help, I tried the "trial --reactor threadedselect twisted", but after some (10 min) work, trial eat all my ram! and I must close it with e "kill -9". Are there another method for execute trial?
As a comparison, have you run trial with the default reactor? If this fails the same way, your machine have an odd configuration which is fouling up the tests, or it may simply not have enough ram to run the full test suite.
My top say: Mem: 776656k total, 592024k used, 184632k free, 31956k buffers
and after two minutes of "trial twisted", 5273 michele 25 0 250m 203m 5012 R 95.9 26.8 1:13.57 trial
What can I try, or what can I modify for make the tests? All my packages are default packages from debian testing repository!
Do you get any output at all? It is not uncommon for the full test suite to take as long as ten minutes. It might take even longer, depending on how fast the machine running the tests is. You may also want to try running a subset of the tests. To run only the core tests, "trial twisted.test". To run only the web tests, "trial twisted.web", etc. Jean-Paul
Do you get any output at all? Yes, I see a lot of input
It is not uncommon for the full test suite to take as long as ten minutes. It might take even longer, depending on how fast the machine running the tests is.
AMD 2.6, 750 MB ram
You may also want to try running a subset of the tests. To run only the core tests, "trial twisted.test". To run only the web tests, "trial twisted.web", etc.
michele:~$ trial twisted.web <cut code> Ran 186 tests in 4.249s PASSED (skips=22, expectedFailures=6, successes=158) What subset of test need I to execute for see the wxreactor problems, since I can't try the whole one? Michele
Hi all. I am also using the threadedselectreactor in wx without problems. I have only just read these posts and cannot understand any reasoning to drop it. What am I missing? Regards, David Uwe C. Schroeder wrote:
Well, for me the threadedselectreactor works quite nicely with wx. It has it's flaws and I've spent some time working around them. On the other hand if that reactor is dropped, I'd rather drop Twisted from my application than rewrite the GUI part with a toolkit that kind of sucks and looks like crap on most platforms (i.e. GTK or TK). I like Twisted PB and I like wx - that's pretty much all I have to say about this. Let's say it this way: the network part of an application is what makes it work, the GUI part is what makes the money - guess what's dropped first :-(
UC
On Saturday 23 September 2006 21:31, glyph@divmod.com wrote:
On Sun, 24 Sep 2006 00:44:03 -0300, Felipe Almeida Lessa
wrote: So the only GUI toolkits that will be officialy supported are going to be Gtk, Tk and PyUI? Thanks for reminding me.
Neither Tk nor PyUI have buildbots. However, they're both implemented using polling timed events running on top of another reactor... I don't know how much "support" those really count for. It might be good to remove them as well, but they're very little code.
GTK is supported, has a buildbot, and is currently green. Frankly, the people who currently do at least 90% of the maintenance on Twisted (myself included) only care about this toolkit, so it's the one that is likely to remain the best supported. It is my sincere hope that someone will port GTK to the Cocoa APIs for OS X at some point in the near future. Then I can cease caring even about corefoundation ;).
The Win32 event loop is also supported, although its buildbot still isn't green yet. That is a "toolkit" of sorts. I think that it might even work with MFC but I'm a bit murky on the details.
So, yes, the situation is even worse than you suggest; unless someone else volunteers to help, we will have 2 supported toolkits: GTK, and GDI.
Twisted's architecture was developed with GUI toolkits in mind, and I would like it to support as many as possible. I hope that one day, all toolkit developers will actually care whether Twisted supports their code or not. Until now, though, it seems that we have been unable to attract the attention of even one member of each community with the appropriate experience and motivation to help out.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, 25 Sep 2006 01:17:14 -0300, David Pratt
Hi all. I am also using the threadedselectreactor in wx without problems. I have only just read these posts and cannot understand any reasoning to drop it. What am I missing?
"If it's not tested, it's broken." The point is not that I want to drop this code from Twisted. In fact, far from it, I would like Twisted to support as many toolkits as possible. The point is that these reactors are currently excluded from automated test coverage, and have no maintainers. That means that they do not currently work as well as the other reactors, but more seriously, if they were to be *completely* broken (say, start raising exceptions upon import), nobody on the Twisted team would know until users started complaining, and most likely users would not complain until well after we had already produced a broken release. We could add automated test coverage relatively easily, *IF* someone would volunteer to at least look at the results of that coverage and contribute some fixes. Perhaps you were not even aware that there are reliability problems with threadedselectreactor. To see an example of the problem, run "trial --reactor threadedselect twisted". Would you volunteer to have a look at some of these problems and try to fix them? I have no idea how many people are using Twisted in conjunction with wxwindows today, but you would be a hero to all of them, I'm sure :).
Hi Glyph. I am not sure if I am the right person, but at the same time I don't wish to see the code removed. I see deprecation warnings and failed tests. Not great. I am willing to help but I am not sure that I possess the expertise to solve all the issues. I am willing to commit to trying though, hopefully with the help of others. Perhaps I'll post on wxpython list to see if there are others willing to help. wxpython and twisted are both important toolkits. Twisted integration is important to me for my own projects. I am hoping someone from the Chandler project thinks so too and feels that sharing in this effort is the right thing to do. I know there are a number of folks using wxreactor or threadedselectreactor in their wxpython code. It is reasonable that we shoulder some responsibility to help ourselves. Regards, David glyph@divmod.com wrote:
On Mon, 25 Sep 2006 01:17:14 -0300, David Pratt
wrote: Hi all. I am also using the threadedselectreactor in wx without problems. I have only just read these posts and cannot understand any reasoning to drop it. What am I missing?
"If it's not tested, it's broken."
The point is not that I want to drop this code from Twisted. In fact, far from it, I would like Twisted to support as many toolkits as possible.
The point is that these reactors are currently excluded from automated test coverage, and have no maintainers. That means that they do not currently work as well as the other reactors, but more seriously, if they were to be *completely* broken (say, start raising exceptions upon import), nobody on the Twisted team would know until users started complaining, and most likely users would not complain until well after we had already produced a broken release.
We could add automated test coverage relatively easily, *IF* someone would volunteer to at least look at the results of that coverage and contribute some fixes. Perhaps you were not even aware that there are reliability problems with threadedselectreactor. To see an example of the problem, run "trial --reactor threadedselect twisted".
Would you volunteer to have a look at some of these problems and try to fix them? I have no idea how many people are using Twisted in conjunction with wxwindows today, but you would be a hero to all of them, I'm sure :).
On Mon, 25 Sep 2006 02:32:03 -0300, David Pratt
Hi Glyph. I am not sure if I am the right person, but at the same time I don't wish to see the code removed. I see deprecation warnings and failed tests. Not great.
Now you can see how I feel about this code :).
I am willing to help but I am not sure that I possess the expertise to solve all the issues. I am willing to commit to trying though, hopefully with the help of others. Perhaps I'll post on wxpython list to see if there are others willing to help.
Thanks a lot. This is as much as anyone could reasonably ask. I'm sure that if you start trying to track down the issues within Twisted and asking questions, you will find that people on this mailing list and #twisted on chat.freenode.net would be more than happy to help you learn more about the reactor's internals.
wxpython and twisted are both important toolkits. Twisted integration is important to me for my own projects. I am hoping someone from the Chandler project thinks so too and feels that sharing in this effort is the right thing to do. I know there are a number of folks using wxreactor or threadedselectreactor in their wxpython code. It is reasonable that we shoulder some responsibility to help ourselves.
I really appreciate this sentiment. I'm glad you're interested in helping out. However, I think that Chandler uses Twisted in a rather idiosyncratic way; it doesn't necessarily make sense for them to help, since this isn't code that they're using. If you can interest them in doing so, of course, that's great :) If you are aware of other uses of wxreactor though, the more the merrier. Now that you've volunteered to at least start looking at the problems here, I'll see if I can set up a buildslave to test these reactors. However, I will probably only be able to set it up on one platform, and given wx's nature we'll need a few other buildbots to make sure it works consistently on mac/win/linux, so if any wxreactor users want to contribute an additional buildbot that would be great. Please keep us all appraised of your progress and let us know if you need help!
glyph@divmod.com wrote:
Now that you've volunteered to at least start looking at the problems here, I'll see if I can set up a buildslave to test these reactors. However, I will probably only be able to set it up on one platform, and given wx's nature we'll need a few other buildbots to make sure it works consistently on mac/win/linux, so if any wxreactor users want to contribute an additional buildbot that would be great.
I'll volunteer to set up a wxreactor buildbot on the machine I use to do my windows builds. It's not my platform of choice, but the vast majority of the user community I'm supporting use it (shudder), so it's the most important platform for my wx app. I haven't run buildbot before, but I want to run it for my own app, too, and a windows bot would make the most sense for that, since I develop on Linux. I'll be away on business the rest of this week but will get started as soon as I get back. Maybe we can get some twisted/wx momentum going. :) Cheers, Steve
glyph@divmod.com wrote:
Now that you've volunteered to at least start looking at the problems here, I'll see if I can set up a buildslave to test these reactors. However, I will probably only be able to set it up on one platform, and given wx's nature we'll need a few other buildbots to make sure it works consistently on mac/win/linux, so if any wxreactor users want to contribute an additional buildbot that would be great.
I'll volunteer to set up a wxreactor buildbot on the machine I use to do my windows builds. It's not my platform of choice, but the vast majority of the user community I'm supporting use it (shudder), so it's the most important platform for my wx app. Because of the way tsreactor and thus Wxreactor works you can't use
Stephen Waterbury wrote: trial with it... which is a good reason why we should figure out a better way to support wx.
Hi Glyph. I have recently added a post to the wxpython list to see if other folks can make themselves available to assist with this effort. http://lists.wxwidgets.org/cgi-bin/ezmlm-cgi?11:mss:55602:200609:hafkbkhbehm... Perhaps the first thing will be the deprecation warnings since these should be relatively simple. Then try and see what is happening with the failures. If the failures are as the result of reactor iterations for tests not relevant to the reactor, what direction should be taken? I raise this since this seems to run through Bob's comments and others concerning the qt reactor. Please let us know about the buildbot once it is set up. I see Stephen has offered support for a buildbot for windows which is excellent. Thank you Stephen! Stephen - my sentiments about windows too but I am also in the same boat as far as potential consumers of my app. I am also trying to keep my code crossplatform for mac (second most important) and linux. Regards, David glyph@divmod.com wrote:
On Mon, 25 Sep 2006 02:32:03 -0300, David Pratt
wrote: Hi Glyph. I am not sure if I am the right person, but at the same time I don't wish to see the code removed. I see deprecation warnings and failed tests. Not great.
Now you can see how I feel about this code :).
I am willing to help but I am not sure that I possess the expertise to solve all the issues. I am willing to commit to trying though, hopefully with the help of others. Perhaps I'll post on wxpython list to see if there are others willing to help.
Thanks a lot.
This is as much as anyone could reasonably ask. I'm sure that if you start trying to track down the issues within Twisted and asking questions, you will find that people on this mailing list and #twisted on chat.freenode.net would be more than happy to help you learn more about the reactor's internals.
wxpython and twisted are both important toolkits. Twisted integration is important to me for my own projects. I am hoping someone from the Chandler project thinks so too and feels that sharing in this effort is the right thing to do. I know there are a number of folks using wxreactor or threadedselectreactor in their wxpython code. It is reasonable that we shoulder some responsibility to help ourselves.
I really appreciate this sentiment. I'm glad you're interested in helping out.
However, I think that Chandler uses Twisted in a rather idiosyncratic way; it doesn't necessarily make sense for them to help, since this isn't code that they're using. If you can interest them in doing so, of course, that's great :)
If you are aware of other uses of wxreactor though, the more the merrier.
Now that you've volunteered to at least start looking at the problems here, I'll see if I can set up a buildslave to test these reactors. However, I will probably only be able to set it up on one platform, and given wx's nature we'll need a few other buildbots to make sure it works consistently on mac/win/linux, so if any wxreactor users want to contribute an additional buildbot that would be great.
Please keep us all appraised of your progress and let us know if you need help!
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi Glyph. I have recently added a post to the wxpython list to see if other folks can make themselves available to assist with this effort.
http://lists.wxwidgets.org/cgi-bin/ezmlm-cgi?11:mss: 55602:200609:hafkbkhbehmbhikjmmaf
I'll be extremely embarrassed if I was wrong about the errors with 2.7 :) All I did was run my (currently working on 2.6.3.3) code with 2.7, noticed an error on startup, and reverted to 2.6.3.3. Update: I reinstalled the version for my platform (Mac OS X, python 2.4 unicode PPC - actually I now have both versions installed) and wrote a simple script and got this traceback. It would seem the problem might just be that wxreactor is using a depreciated way to use wxPython (i.e. was wxSOMETHING now is wx.SOMETHING), thanks to the helpful depreciation warning. *** My script *** import wxversion #I have both 2.6.3.3 and 2.7 installed. Change this to 2.6.3.3 and the example works wxversion.select("2.7") from twisted.internet import wxreactor wxreactor.install() import wx class MyApp(wx.App): def OnInit(self): frame = wx.Frame(None, -1, "Hello wxPython world") frame.Show(True) self.SetTopWindow(frame) return True if __name__ == "__main__": app = MyApp(False) app.MainLoop() *** The traceback: *** moko:~/sp rgravina$ pythonw wxtest.py /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/twisted/internet/wxreactor.py:37: DeprecationWarning: The wxPython compatibility package is no longer automatically generated or activly maintained. Please switch to the wx package as soon as possible. from wxPython.wx import wxApp, wxCallAfter, wxEventLoop, wxFrame, NULL Traceback (most recent call last): File "wxtest.py", line 4, in ? from twisted.internet import wxreactor File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/twisted/internet/wxreactor.py", line 37, in ? from wxPython.wx import wxApp, wxCallAfter, wxEventLoop, wxFrame, NULL File "//Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/wx-2.7.0-mac-unicode/wxPython/__init__.py", line 15, in ? import _wx File "//Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/wx-2.7.0-mac-unicode/wxPython/_wx.py", line 7, in ? from _controls import * File "//Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/wx-2.7.0-mac-unicode/wxPython/_controls.py", line 441, in ? wxFRAME_EX_CONTEXTHELP = wx._controls.FRAME_EX_CONTEXTHELP AttributeError: 'module' object has no attribute 'FRAME_EX_CONTEXTHELP' Robert
That means that they do not currently work as well as the other reactors, but more seriously, if they were to be *completely* broken (say, start raising exceptions upon import)
Hate to break it to you, but I tried a preview build of soon-to-be- released (I think) wxPython 2.7, and the twisted code raised exceptions on import :) I reinstalled 2.6.3.3 so can't show you the traceback. Dailies are available here if anyone wants to see what I mean. http://starship.python.net/crew/robind/wxPython/daily/ Anyway, I am currently using wxPython and Twisted in a project and I would sorely miss either of them!!! I just wanted to cast my vote as another developer using this (killer) combination! wx and Twisted (especially PB) are making short work of writing a multi-user client/ server app. From what I've been able to find, wx is the only GUI toolkit that is cross platform and rich. Qt isn't an option for me because of it's expensive commercial licensing, and Tk just looks plain horrible. There must be many other users wx/Twisted, surely. I haven't experienced any problems using wxPython and Twisted together, except perhaps for this. Normally, a wx app will shutdown when the last window in your applciation closes. Since the reactor stops this from happening I workaround this by maintaining a list of open windows and on close of a window I remove it from the list and then check if it's empty. If it is, I shutdown the reactor. Robert
Robert wrote:
That means that they do not currently work as well as the other reactors, but more seriously, if they were to be *completely* broken (say, start raising exceptions upon import)
Hate to break it to you, but I tried a preview build of soon-to-be-released (I think) wxPython 2.7, and the twisted code raised exceptions on import :) I reinstalled 2.6.3.3 so can't show you the traceback. Dailies are available here if anyone wants to see what I mean. http://starship.python.net/crew/robind/wxPython/daily/
I don't know what platform are you using, but here wx 2.7 (2.7.0.0pre.20060712) and linux (gtk) work well. Work also well into my win box... Simple tried replacing 2.6 with 2.7 and executing my _not so simple_ applications that use wx, twisted for tcp comunication and adbapi for db connections. Michele
Michele Petrazzo wrote:
I don't know what platform are you using, but here wx 2.7 (2.7.0.0pre.20060712) and linux (gtk) work well. Work also well into my win box...
Sorry, but I was using threadselectreactor direcly... This is my own patch for wxreactor that work with wx 2.7. Hope this help, Michele michele:/usr/lib/python2.4/site-packages/twisted/internet# diff wxreactor.py wxreactor_ok.py 37c37 < import wx ---
from wxPython.wx import wxApp, wxCallAfter, wxEventLoop, wxFrame, NULL 40c40 < class DummyApp(wx.App):
class DummyApp(wxApp): 84c84 < self.interleave(wx.CallAfter)
self.interleave(wxCallAfter)
92,93c92,93 < ev = wx.EventLoop() < wx.EventLoop.SetActive(ev) ---
ev = wxEventLoop() wxEventLoop.SetActive(ev)
On Sep 25, 2006, at 12:17 AM, David Pratt wrote:
Hi all. I am also using the threadedselectreactor in wx without problems. I have only just read these posts and cannot understand any reasoning to drop it. What am I missing?
That it's broken according to the test suite, and that nobody has volunteered to fix it. Perhaps the bugs are in the tests (a cursory glance at the qt failures suggests this might be the case for that reactor), or perhaps the tested functionality is something that should be expected to not work for that reactor, but in any case, either the tests or the reactor should be fixed. Tests that have been failing for a long time indicates that nobody cares enough about that code to fix them. Clearly the best resolution of this would be to have all the mentioned reactors fixed. Then, nobody would propose that they be removed. James
On 9/24/06, James Y Knight
On Sep 25, 2006, at 12:17 AM, David Pratt wrote:
Hi all. I am also using the threadedselectreactor in wx without problems. I have only just read these posts and cannot understand any reasoning to drop it. What am I missing?
That it's broken according to the test suite, and that nobody has volunteered to fix it.
Perhaps the bugs are in the tests (a cursory glance at the qt failures suggests this might be the case for that reactor), or perhaps the tested functionality is something that should be expected to not work for that reactor, but in any case, either the tests or the reactor should be fixed. Tests that have been failing for a long time indicates that nobody cares enough about that code to fix them.
Clearly the best resolution of this would be to have all the mentioned reactors fixed. Then, nobody would propose that they be removed.
The biggest issue is that the tests work by reactor iteration, which is not how some of the reactors actually work in practice. Tests passing or failing don't necessarily reflect how the reactor is going to behave under normal circumstances. -bob
On Sun, 24 Sep 2006 22:19:32 -0700, Bob Ippolito
On 9/24/06, James Y Knight
wrote: On Sep 25, 2006, at 12:17 AM, David Pratt wrote:
Hi all. I am also using the threadedselectreactor in wx without problems. I have only just read these posts and cannot understand any reasoning to drop it. What am I missing?
Clearly the best resolution of this would be to have all the mentioned reactors fixed. Then, nobody would propose that they be removed.
The biggest issue is that the tests work by reactor iteration, which is not how some of the reactors actually work in practice. Tests passing or failing don't necessarily reflect how the reactor is going to behave under normal circumstances.
Over the last year, jml has been making good progress towards grinding down the unfortunate protuberances in Trial which make it less like a real application. See the following tickets: http://twistedmatrix.com/trac/ticket/1781 http://twistedmatrix.com/trac/ticket/2090 It's been a gradual process but we've been moving in this direction for a while. If we can attribute specific test failures to this issue, we should try to work around it (which is possible in many cases), otherwise .todo or .skip them as appropriate until trial's metamorphosis is complete.
On Sunday 24 September 2006 14:31, glyph@divmod.com wrote:
On Sun, 24 Sep 2006 00:44:03 -0300, Felipe Almeida Lessa
wrote: So the only GUI toolkits that will be officialy supported are going to be Gtk, Tk and PyUI?
Thanks for reminding me.
Neither Tk nor PyUI have buildbots. However, they're both implemented using polling timed events running on top of another reactor... I don't know how much "support" those really count for. It might be good to remove them as well, but they're very little code.
I'd be opposed to removing Tk - it allows you to build (ok, ugly looking)
simple UIs on top of twisted+python without needing to download anything
other than the core Python and twisted code. As you noted, the Tk code isn't
a real reactor of it's own, so what does "untested" mean, in this case?
Anthony
--
Anthony Baxter
I am on the IPython development team and one of the features of IPython that is highly praised by users is its ability to use various GUI toolkits interactively (wc, qt3, qt4, tk, gtk). Currently, this is not done using Twisted...but we are completely redesigning IPython from the ground up to use Twisted to allow us to have IPython run remotely, and lots of other interesting things. One of the reason that we went with Twisted is that it looked like it would be easy to integrate Twisted with the various GUI run loops - a feature we need to continue to support. We did some basic tests with wx and threadedselect reactor and it worked well for what we tried....but that was when tsr first came out and I guess it has never matured enough to be stable. I completely understand the desire to have code that passes tests (especially reactors). But I think the importance of these GUI reactors is being greatly underestimated if they are under consideration for removal. Minimally, they should remain somewhere in the repository and be well documented as to why they are there and their history. I don't have time to dig into this code right, but eventually, I will need to come back to this and get our code working with lots of different toolkits. I guess I would vote to leave them in, but clearly and verbosely document their status. This will encourage folks to work on them whereas removing them will lead people to simply create more half baked GUI reactors.
On Fri, 29 Sep 2006 00:27:36 -0600, Brian Granger
I am on the IPython development team and one of the features of IPython that is highly praised by users is its ability to use various GUI toolkits interactively (wc, qt3, qt4, tk, gtk). Currently, this is not done using Twisted...but we are completely redesigning IPython from the ground up to use Twisted to allow us to have IPython run remotely, and lots of other interesting things.
Thanks for thinking of Twisted when you did this :).
I completely understand the desire to have code that passes tests (especially reactors). But I think the importance of these GUI reactors is being greatly underestimated if they are under consideration for removal. Minimally, they should remain somewhere in the repository and be well documented as to why they are there and their history. I don't have time to dig into this code right, but eventually, I will need to come back to this and get our code working with lots of different toolkits.
A major complaint that many users have about Twisted is that it is sprawling. I've repeatedly heard complaints that it's very difficult to look at the source tree and have any idea what's going on. In many cases I think this complaint is miguided, because you don't have to understand everything to use some things, but there's no point in bloating the repository with code that is decaying with each release, which is never looked at by any developers, and which could silently break due to an unrelated internal API change with no test coverage or automated builds to bring anyone's attention to it.
I guess I would vote to leave them in, but clearly and verbosely document their status. This will encourage folks to work on them whereas removing them will lead people to simply create more half baked GUI reactors.
I am inviting anyone who would like to improve them to speak up now, and a few people have. Right now it seems like we have at least a few volunteers who will keep these reactors alive, but of course any additional contribution would be welcome.
glyph@divmod.com writes:
Although I know of Qt and WX users, I do not know of anyone currently using the corefoundation reactor in an application. Are you out there? Is it being used?
The last word I heard on the corefoundation reactor was from its author, Bob Ippolito, who suggested that it was not the correct way to integrate Twisted with GUI applications. However, his suggested replacement (threadedselectreactor) is buggy, also unmaintained, and IMHO misdesigned, so in the absence of other knowledge I believe that continuing to provide some corefoundation support would be the best way to support the mac. After receiving a shiny new computer from them I feel inclined to do the best job possible to support Apple's platform, but I simply don't have the time, knowledge, or energy to do the development myself.
It would be annoying to me if I could not use Twisted and PyObjC together. Currently I have a couple of noddy applications (the GUI equivalent of 5 line shell scripts) doing this (using tsr) and they work fine. I understand why you want to have less untrustworthy code in Twisted, but I think throwing out both tsr and cfreactor in 2.6 would be a step back, not forwards. I could claim that I'll try to look at cfreactor, but it's not especially realistic... Cheers, mwh -- "Also, does the simple algorithm you used in Cyclops have a name?" "Not officially, but it answers to "hey, dumb-ass!" -- Neil Schemenauer and Tim Peters, 23 Feb 2001
On Sun, 24 Sep 2006 10:28:08 +0100, Michael Hudson
glyph@divmod.com writes:
I understand why you want to have less untrustworthy code in Twisted, but I think throwing out both tsr and cfreactor in 2.6 would be a step back, not forwards.
The alternative is to eventually ship a version of Twisted where that reactor doesn't work at all. (Heck, maybe it doesn't work now, I don't know. I haven't heard from anyone using it yet.) As I said to Uwe, I am saying that these reactors should not be distributed with Twisted because we don't know whether they work; I am not saying that we should aggressively attempt to destroy all copies of this code present anywhere.
I could claim that I'll try to look at cfreactor, but it's not especially realistic...
If you think that tsreactor is the right way to go for mac apps, by all means, have a look at that instead. It doesn't sound like you'd care if cfreactor went away. Ideally, you could write a "macreactor" which would set up the appropriate interleave and shutdown with tsreactor so that we could run a buildbot in the context of a mac gui, similar to what itamar's wxreactor does, and start fixing tests from there.
Quoting glyph@divmod.com:
Currently qtreactor is being tested on the "reactors" buildslave and is keeping it red, despite the fact that all the other reactors on that slave work fine (modulo a few elusive race conditions). wxreactor is not being tested at all. I believe we should split qt out so that it can have its own red column and not interfere with the gtk and poll reactors, and if we run a wx slave it should also be separate.
The next QT version (4.2, currently in RC1) contains support for the Glib eventloop. This has to be tested, but it promises use of the gtk reactor with Qt application. Qt 4.2 (and PyQt) is not released but should be by the time of Twisted 2.6; we could then write a documentation about integration. The drawback is that you'll need glib (ie, python-gtk) to write a Qt app. We also have a qt4 reactor (http://twistedmatrix.com/trac/ticket/1770) that needs a maintener. Last time I checked few things were needed to make the test suite pass. -- Thomas ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
On Sun, 24 Sep 2006 15:20:19 +0200, Thomas HERVE
Quoting glyph@divmod.com:
Currently qtreactor is being tested on the "reactors" buildslave and is keeping it red, despite the fact that all the other reactors on that slave work fine (modulo a few elusive race conditions). wxreactor is not being tested at all. I believe we should split qt out so that it can have its own red column and not interfere with the gtk and poll reactors, and if we run a wx slave it should also be separate.
The next QT version (4.2, currently in RC1) contains support for the Glib eventloop. This has to be tested, but it promises use of the gtk reactor with Qt application. Qt 4.2 (and PyQt) is not released but should be by the time of Twisted 2.6; we could then write a documentation about integration. The drawback is that you'll need glib (ie, python-gtk) to write a Qt app.
We also have a qt4 reactor (http://twistedmatrix.com/trac/ticket/1770) that needs a maintener. Last time I checked few things were needed to make the test suite pass.
AFAICT, Qt4 also fixes the problem which is causing the current qtreactor test failures. Jean-Paul
On Sun, 24 Sep 2006 15:20:19 +0200, Thomas HERVE
Quoting glyph@divmod.com:
Currently qtreactor is being tested on the "reactors" buildslave and is keeping it red, despite the fact that all the other reactors on that slave work fine (modulo a few elusive race conditions). wxreactor is not being tested at all. I believe we should split qt out so that it can have its own red column and not interfere with the gtk and poll reactors, and if we run a wx slave it should also be separate.
The next QT version (4.2, currently in RC1) contains support for the Glib eventloop. This has to be tested, but it promises use of the gtk reactor with Qt application. Qt 4.2 (and PyQt) is not released but should be by the time of Twisted 2.6; we could then write a documentation about integration.
That would be great. Is that going to be the default mainloop or an optional add-on?
The drawback is that you'll need glib (ie, python-gtk) to write a Qt app.
Some might not call that a drawback ;).
We also have a qt4 reactor (http://twistedmatrix.com/trac/ticket/1770) that needs a maintener. Last time I checked few things were needed to make the test suite pass.
Are you willing to have a crack at fixing that code?
Quoting glyph@divmod.com:
The next QT version (4.2, currently in RC1) contains support for the Glib eventloop. This has to be tested, but it promises use of the gtk reactor with Qt application. Qt 4.2 (and PyQt) is not released but should be by the time of Twisted 2.6; we could then write a documentation about integration.
That would be great. Is that going to be the default mainloop or an optional add-on?
It's optional. It seems C++ apps will have a command-line switch. I don't know about PyQt, though.
The drawback is that you'll need glib (ie, python-gtk) to write a Qt app.
Some might not call that a drawback ;).
Well, it might be a drawback if you want to use Qt under Mac :).
We also have a qt4 reactor (http://twistedmatrix.com/trac/ticket/1770) that needs a maintener. Last time I checked few things were needed to make the test suite pass.
Are you willing to have a crack at fixing that code?
I'm no Qt expert, but I work with some so it could help. Consider it assigned. -- Thomas ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
Thomas HERVE wrote:
The next QT version (4.2, currently in RC1) contains support for the Glib eventloop. This has to be tested, but it promises use of the gtk reactor with Qt application. Qt 4.2 (and PyQt) is not released but should be by the time of Twisted 2.6; we could then write a documentation about integration. The drawback is that you'll need glib (ie, python-gtk) to write a Qt app.
We also have a qt4 reactor (http://twistedmatrix.com/trac/ticket/1770) that needs a maintener. Last time I checked few things were needed to make the test suite pass.
As far as I am aware (and please correct me if I am wrong) all these recent qt-specific reactors have the same systematic problem on win32 because they rely on hooks into the Qt main loop. This means that the reactor only processes events when the Qt main loop is running, and it stalls if the main thread blocks in some nested non-Qt message loop. This is a reasonable thing to happen in a GUI application; nested message loops are used in MessageBox, drap/drop, window resizing, and non-Qt dialog boxes (like the standard file chooser). The nested message loop will be doing the win32 GetMessage/DispatchMessage dance. This is sufficient to keep the GUI updating smoothly, and it should be sufficient for the reactor too. IMHO threadedselectreactor seems like the best option for getting *this* right. glyph wrote:
threadedselectreactor is an interesting concept but hopelessly broken and untested in its current implementation.
Im currently using threadedselectreactor (plus private patches) and a derivative threadedwin32eventreactor in production, so this statement leaves me with a fairly big itch to scratch. I will check out the failing tests.
On Sep 23, 2006, at 7:20 PM, glyph@divmod.com wrote:
The reactors presented in the subject all seem to have been abandoned. Even our friends at Apple are mainly concerned with the use of Twisted for server work, and while they've been attempting to help us find someone to poke around GUI <-> reactor integration on OS X, it's not their area of expertise.
To be clear, the need for tsr or cfr is not limited to GUI applications; it's needed by anything using CoreFoundation and therefore requiring CFRunLoop. Many non-GUI applications require a CFRunLoop. Server applications that want to use any of the above-BSD OS X toolkits (eg. via PyObjC) will fall into this camp. I expect to be in this category eventually, though I have been avoiding it to date because PyObjC isn't presently included in OS X.
tsreactor is private in twisted - starts with _, so it doesn't really guarantee anything or make any promises about working, and it's very useful for integrating with broken event loops.
It's exposed to users as a reactor on the command line. Maybe *that* should have an "_" added to it as well. However, if we're going to keep it around, it should have a buildbot, end of story.
Well, you just got this spiffy XServe. Is it not capable of running a tsr buildbot? Would it be hard to maintain? (I seriously have no clue.) -wsv
On Mon, 25 Sep 2006 19:41:06 -0700, Wilfredo Sánchez Vega
It's exposed to users as a reactor on the command line. Maybe *that* should have an "_" added to it as well. However, if we're going to keep it around, it should have a buildbot, end of story.
Well, you just got this spiffy XServe. Is it not capable of running a tsr buildbot? Would it be hard to maintain? (I seriously have no clue.)
One difficulty is that trial cannot currently run the test suite with tsr. tsr is not really a reactor in the same way that poll, select, and gtk are reactors: it has a run method, but calling it isn't how you invoke the magic secrets of tsr. Instead, you set it up to cooperate with another event loop. So we need to have a way to run the tests at all first. Then we can set up a buildslave (which will presumably fail most of the test suite). Then someone needs to start fixing it :) Jean-Paul
participants (20)
-
Andrew Bennetts
-
Anthony Baxter
-
Bob Ippolito
-
Brian Granger
-
David Pratt
-
Felipe Almeida Lessa
-
glyph@divmod.com
-
Itamar Shtull-Trauring
-
James Y Knight
-
Jean-Paul Calderone
-
Jonathan Lange
-
Michael Hudson
-
Michele Petrazzo
-
Paul Swartz
-
Robert Gravina
-
Stephen Waterbury
-
Thomas HERVE
-
Toby Dickenson
-
Uwe C. Schroeder
-
Wilfredo Sánchez Vega