glyph@divmod.com wrote:
On Sat, 23 Sep 2006 23:33:11 -0400, Stephen Waterbury <waterbug@pangalactic.us> wrote:
If I can get that to work (which I haven't had time to play with yet), it would seem to offer several advantages: (1) running wx and twisted event loops in completely separate processes, (2) making relatively minimal use of threads in the wx process (well-supported by the wx libraries) to drive Pyro, and using the usual twisted paradigm in the twisted process. I would appreciate any advice, but I don't see any show-stoppers. I would do the Pyro communications on 127.0.0.1, since Pyro has essentially zero security, keeping it completely within the local machine. Twisted would handle all external protocols.
Thanks for the suggestion, Stephen. I don't think I like this approach though.
The major reason is that it introduces a process boundary between Twisted code and WX code.
I see that as a *good* thing. ;)
In every other toolkit Twisted supports,
... of which there's really only *one* (GTK) that's at all interesting to anyone doing cross-platform apps ... and even that one, IMO, isn't very cross-platform, although that could certainly be argued either way ...
you can handle GUI events and network events in the same process, in the same thread. That makes it possible to have individual state machines which can emit output in the form of either GUI updates and writes to streams, and you can similarly receive GUI events or data from the network as input.
Okay. I'm willing to do without that.
Keep in mind that each of the toolkits that WX backends to already *has* either a supported reactor or a proof of concept for Twisted integration: GTK+, the Win32 event loop, and Cococa. It's not that we can't get network I/O notifications from the platform. The real problem here is that WX obscures the underlying APIs for integrating sockets. (I say "obscures" rather than "does not wrap" because the last time I tried to use gtk reactor with a gtk build of wx, the process segfaulted after emitting a pile of dynamic linker errors. I have no idea what would happen on Windows or the Mac.) Whatever we're going to do at this point without getting some new code from the WX developers is going to be a workaround.
I think *that* is the real problem ... not technical, just interest from the right people. I've waited enough years (about 5) that I've given up on that.
That said, the mechanism that you propose may be the best way to practically integrate Twisted and WX today. If it is, I think that it would work better as a separate project, because it is a workaround with a lot of pointy edges and not a real solution.
What's reality? :) OTOH ... I completely agree that it should not be part of Twisted at all ... but if I succeed with it, I'll certainly document what I do.
One major reason I'd like a WX reactor to continue to exist in some form is OSAF's usage of Twisted and WX. They have a similarly baroque model for handing data back and forth between Twisted code and GUI code, ...
... and everything else (sorry, couldn't resist ;) ...
... and I'd love to have something to propose to them that was (A) supported by Twisted (B) well tested, and (C) didn't involve lots of potential race conditions, as all shared-state-threading code was likely to do. I haven't looked at it in a while, but if I remember correctly their separation between Twisted code and GUI code was fairly clean and well-thought-out,
... hmmm ... I've been on all the OSAF Chandler/Scooby/Cosmo lists since April (and that's a lot of freakin' mail ;), and not *once* in that interval has "twisted" occurred in the subject line of a message, and no substantive discussion of it in the bodies either. I know about their zanshin twisted-based framework for WebDAV/CalDAV, and that it's used in their Sharing architecture, which is used by the Chandler wx GUI, but I just updated my chandler svn checkout, and I counted 17 occurrences of the zanshin "blockUntil" function in the Sharing.py module ... if it's so clean, how come they need those? (In zanshin's docs, "blockUntil" is billed as "for test purposes" ... ;)
but the separation would have been unnecessary if wx had native Twisted integration.
Well, yeah, but I don't think OSAF has the resources to put into it either, and Robin *works* for OSAF.
There are other problems with a Pyro-driven Twisted subprocess, which are relevant as the goal here is a well-supported and well-tested path for integration.
Multiprocess communication and process spawning, especially on Windows, is fraught with peril. We've been working on reliable event-driven process spawning in Twisted for years and years, and only recently have we gotten to the point where the published API works reliably cross-platform. Given that Twisted would be left out of this loop, I imagine the process control would be rather ad-hoc, so getting the Twisted and WX processes synchronized would be problematic, and extremely hard to unit test. Although it would be slightly easier if the Twisted proces is 'on top', you still have to inject test code into both the parent and child processes, and have a way to verify the state of both.
The last release of pyro is almost a year old. That suggests it is not actively maintained.
Or else it just works. ;)
Pyro doesn't have a buildbot, or (as far as I can tell from the release) any automated tests of its own. That would make it a wash in terms of tested, robust integration support.
As if there's going to be some other "tested, robust integration support" ... nothing on the horizon that I can see (and as I say, I've been watching for a long time).
You'd have to re-implement the Pyro protocol on top of Twisted to communicate with it without spawning lots of threads. That's a bunch of new code which is going to need a home, again, possibly a whole new project. Even after that's done, Pyro is going to be spawning threads and blocking inside the WX process because the Twisted implementation isn't running there.
Process spawning! Rusty code! Spawning threads, *and* blocking! My, you do paint a rosy picture. All your handwaving and FUD makes me want to try it, because it's probably quicker and easier than discussing it. But thanks for the warnings, anyway! ;) Cheers, Steve