One thing of which I'm certain: I do NOT like documentation entries that say things like "select() doesn't really work except under Unix" (still true? Maybe that's been fixed?).

Hm, that's bogus. It works well under Windows -- with the restriction that it only works for sockets, but for sockets it works as well as on Unix. it also works well on the Mac. I wonder where that note came from (it's probably 6 years old :-). Fred...?

As a user, I just find that intolerable. Sufficiently intolerable that I'll help change the situation? Well, I'm working on that part now ...

--Guido van Rossum (home page: http://www.python.org/~guido/)

...

One thing of which I'm certain: I do NOT like documentation entries that say things like "select() doesn't really work except under Unix" (still true? Maybe that's been fixed?).

Hm, that's bogus. It works well under Windows -- with the restriction that it only works for sockets, but for sockets it works as well as on Unix. it also works well on the Mac. I wonder where that note came from (it's probably 6 years old :-).

I'm pretty sure I know where it came from -- it came from Sam Rushing's tutorial on how to use Medusa, which was more or less cut & pasted into the doc, probably at the time that asyncore and asynchat were added to the Python core. IMO, it's not the best part of the Python doc -- it is much too low-to-the ground, and assumes the reader already understands much about I/O, sync/async issues, and cares mostly about high performance. All of which are true of wonderful Sam, most of which are not true of the average Python user. While we're complaining about doc, asynchat is not documented, I believe. Alas, I'm unable to find the time to write up said documentation. --david PS: I'm not sure that multiplexing can be made _easy_. Issues like block/nonblocking communications channels, multithreading etc. are hard to ignore, as much as one might want to.

On Fri, 19 May 2000, Guido van Rossum wrote:

Hm, that's bogus. It works well under Windows -- with the restriction that it only works for sockets, but for sockets it works as well as on Unix. it also works well on the Mac. I wonder where that note came from (it's probably 6 years old :-).

Is that still in there? If I could get a pointer from someone I'll be able to track it down. I didn't see it in the select or socket module documents, and a quick grep did't find 'really work'. It's definately fixable if we can find it. ;) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

From alexandre.ferrieux@cnet.francetelecom.fr Mon May 22 03:40:13 2000 . . . > Alex, it's disappointing to me too! There just isn't anything > currently in the library to do this, and I haven't written apps that > needs this often enough to have a good feel for what kind of > abstraction is needed. Thanks for the empathy. Apologies for my slight overreaction. > However perhaps we can come up with a design for something better? Do > you have a suggestion here? Yup. One easy answer is 'just copy from Tcl'... Seriously, I'm really too new to Python to suggest the details or even the *style* of this 'level 2 API to multiplexing'. However, I can sketch the implementation since select() (from C or Tcl) is the one primitive I most depend on ! Basically, as shortly mentioned before, the key problem is the heterogeneity of seemingly-selectable things in Windoze. On unix, not only does select() work with all descriptor types on which it makes sense, but also the fd used by Xlib is accessible; hence clean multiplexing even with a GUI package is trivial. Now to the real (rotten) meat, that is M$'s. Facts: 1. 'Handle' types are not equal. Unnames pipes are (surprise!) not selectable. Why ? Ask a relative in Redmond... 2. 'Handle' types are not equal (bis). Socket 'handles' are *not* true handles. They are selectable, but for example you can't use'em for redirections. Okay in our case we don't care. I only mention it cause its scary and could pop back into your face some time later. 3. The GUI API doesn't expose a descriptor (handle), but fortunately (though disgustingly) there is a special syscall to wait on both "the message queue" and selectable handles: MsgWaitForMultipleObjects. So its doable, if not beautiful. The Tcl solution to (1.), which is the only real issue, is to have a separate thread blockingly read 1 byte from the pipe, and then post a message back to the main thread to awaken it (yes, ugly code to handle that extra byte and integrate it with the buffering scheme). In summary, why not peruse Tcl's hard-won experience on selecting-on-windoze-pipes ? Then, for the API exposed to the Python programmer, the Tclly exposed one is a starter: fileevent $channel readable|writable callback ... vwait breaker_variable Explanation for non-Tclers: fileevent hooks the callback, vwait does a loop of select(). The callback(s) is(are) called without breaking the loop, unless $breaker_variable is set, at which time vwait returns. One note about 'breaker_variable': I'm not sure I like it. I'd prefer something based on exceptions. I don't quite understand why it's not already this way in Tcl (which has (kindof) first-class exceptions), but let's not repeat the mistake: let's suggest that (the equivalent of) vwait loops forever, only to be broken out by an exception from within one of the callbacks. . . . I've copied everything Alex wrote, because he writes for me, also. As much as I welcome it, I can't answer Guido's question, "What should the API look like?" I've been mulling this over, and concluded I don't have sufficiently deep know- ledge to be trustworthy on this. Instead, I'll just give a bit of personal testimony. I made the rather coy c.l.p posting, in which I sincerely asked, "How do you expert Pythoneers do it?" (my para- phrase), without disclosing either that Alex and I have been discussing this, or that the Tcl interface we both know is simply a delight to me. Here's the delight. Guido asked, approximately, "What's the point? Do you need this for more than the keeping- the-GUI-responsive-for-which-there's-already-a-notifier- around case?" The answer is, yes. It's a good question, though. I'll repeat what Alex has said, with my own em- phasis: Tcl gives a uniform command API for * files (including I/O ports, ...) * subprocesses * TCP socket connections and allows the same fcntl()-like configuration of them all as to encodings, blocking, buffering, and character translation. As a programmer, I use this stuff CONSTANTLY, and very happily. It's not just for GUIs; several of my mission-critical delivered products have Tcl-coded daemons to monitor hardware, manage customer transactions, ... It's simply wonderful to be able to evolve a protocol from a socket connection to an fopen() read to ... Tcl is GREAT at "gluing". Python can do it, but Tcl has a couple of years of refinement in regard to portability issues of managing subprocesses. I really, *really* miss this stuff when I work with a language other than Tcl. I don't often whine, "Language A isn't language B." I'm happy to let individual character come out. This is, for me, an exceptional case. It's not that Python doesn't do it the Tcl way; it's that the Tcl way is wonderful, and moreover that Python doesn't feel to me to have much of an alternative answer. I conclude that there might be some- thing for Python to learn here. A colleague has also write an even higher-level wrapper in Tcl for asynchronous sockets. I'll likely explain more about it URL:http://www-users.cs.umn.edu/~dejong/tcl/EasySocket.tar.gz in a follow-up. Conclusion for now: Alex and I like Python so much that we want you guys to know that better piping-gluing-networking truly is possible, and even worthwhile. This is sort of like the emigrants who've reported, "Yeah, here's the the stuff about CPAN that's cool, and how we can have it, too." Through it all, we absolutely want Python to continue to be Python.

Gordon McMillan wrote:

Alexandre Ferrieux wrote:

...

The Tcl solution to (1.), which is the only real issue, is to have a separate thread blockingly read 1 byte from the pipe, and then post a message back to the main thread to awaken it (yes, ugly code to handle that extra byte and integrate it with the buffering scheme).

What's the actual mechanism here? A (dummy) socket so "select" works? The WSAEvent... stuff (to associate sockets with waitable events) and WaitForMultiple...? The WSAAsync... stuff (creates Windows msgs when socket stuff happens) with MsgWait...? Some other combination?

Other. Forget about sockets here, we're talking about true anonymous pipes, under 95 and NT. Since they are not waitable nor peekable, the only remaining option is to read in blocking mode from a dedicated thread. Then of course, this thread reports back to the main MsgWaitForMultiple with PostThreadMessage.

Is the mechanism different if it's a console app (vs GUI)?

No. Why should it ?

I'd assume in a GUI, the fileevent-checker gets integrated with the message pump.

The converse: MsgWaitForMultiple integrates the thread's message queue which is a superset of the GUI's event stream. -Alex

Neil Hodgson wrote:

...

Other. Forget about sockets here, we're talking about true anonymous pipes, under 95 and NT. Since they are not waitable nor peekable, the only remaining option is to read in blocking mode from a dedicated thread. ...

Anonymous pipes are peekable on both 95 and NT with PeekNamedPipe.

Hmmm... You're right, it's documented as such. But I seem to recall we encountered a problem when actually using it. I'll check with Gordon Chaffee (Cc of this msg). -Alex

Alexandre Ferrieux wrote:

Other. Forget about sockets here, we're talking about true anonymous pipes, under 95 and NT. Since they are not waitable nor peekable,

I thought PeekNamedPipe worked just fine on anonymous pipes. or are "true anonymous pipes" not the same thing as anonymous pipes created by CreatePipe? </F>

Mark Hammond wrote:

...

Other. Forget about sockets here, we're talking about true anonymous pipes, under 95 and NT. Since they are not waitable nor peekable, the only remaining option is to read in blocking mode from a dedicated thread. Then of course, this thread reports back to the main MsgWaitForMultiple with PostThreadMessage.

Or maybe just with SetEvent(), as the main thread may just be using WaitForMultipleObjects() - it really depends on whether the app has a message loop or not.

Yes but why emphasize the differences when you can instead wipe them out by using MsgWaitForMultiple which integrates all sources ? Even if there's no message stream, it's fine !

...

...

Is the mechanism different if it's a console app (vs GUI)?

No. Why should it ?

Because it generally wont have a message loop. This is also commonly true for NT services - they only wait on settable objects and if they dont create a window generally dont need a message loop. However, it is precisely these apps that the proposal offers the most benefits to.

Yes, but see above: how would it hurt them to call MsgWait* instead of Wait* ?

...

...

I'd assume in a GUI, the fileevent-checker gets integrated with the message pump.

The converse: MsgWaitForMultiple integrates the thread's message queue which is a superset of the GUI's event stream.

But what happens when we dont own the message loop? Eg, IDLE is based on Tk, Pythonwin on MFC, wxPython on wxWindows, and so on. Generally, the primary message loops are coded in C/C++, and wont provide this level of customization.

Can you be more precise ? Which one(s) do(es)/n't fulfill the two conditions mentioned earlier ? I do agree with the fact that the primary msg loop of a random GUI package is a black box, however it must use one of the IPC mechanisms provided by the OS. Unifying them is not uniformly trivial (that's the point of this discussion), but since even on Windows it is doable (MsgWait*), I fail to see by what magic a GUI package could bypass its supervision.

Ironically, Tk seems to be one of the worst for this.

Possibly. Personally I don't like Tk very much, at least from an implementation standpoint. But precisely, the fact that the model described so far can accomodate *even* Tk is a proof of generality !

and I recently(ish) both added threading support to our respective IDEs. MFC was quite simple to do, as it used a "standard" windows message loop. From all accounts, Guido had quite a difficult time due to some of the assumptions made in the message loop. The other anecdote I have relates to debugging. The Pythonwin debugger is able to live happily under most other GUI applications - eg, those written in VB, Delphi, etc. Pythonwin creates a new "standard" message loop under these apps, and generally things work well. However, Tkinter based apps remain un-debuggable using Pythonwin due to the assumptions made by the message loop. This is probably my most oft-requested feature addition!!

As you said, all this is due to the assumptions made in Tk. Clearly a mistake not to repeat, and also orthogonal to the issue of unifying IPC mechanisms and the API to their multiplexing. -Alex

Mark Hammond wrote:

...

Can you be more precise ? Which one(s) do(es)/n't fulfill the two conditions mentioned earlier ? I do agree with the fact that the primary msg loop of a random GUI package is a black box, however it must use one of the IPC mechanisms provided by the OS. Unifying them is not uniformly trivial (that's the point of this discussion), but since even on Windows it is doable (MsgWait*), I fail to see by what magic a GUI package could bypass its supervision.

The only way I could see this working would be to use real, actual Windows messages on Windows. Python would need to nominate a special message that it knows will not conflict with any GUI environments Python may need to run in.

Why use a special message ? MsgWait* does multiplex true Windows Message *and* other IPC mechanisms. So if a package uses messages, it will awaken MsgWait* by its 'message queue' side, while if the package uses a socket or a pipe, it will awaken it by its 'waitable handle' side (provided, of course, that you can get your hands on that handle and pass it in th elist of objects to wait for...).

Each GUI package maintainer would then need to add some special logic in their message hooking code. When their black-box message loop delivers this special message, the framework would need to enter the Python "event-loop", where it does its stuff - until a new message arrives.

The key is that there wouldn't be two separate Python/GUI evloops. That's the reason for the (a) condition: be able to awaken a multiplexing syscall.

Unless Im missing something?

I believe the next thing to do is to enumerate which GUI packages fullfill the following conditions ((a) updated to (a') to reflect the first paragraph of this msg): (a') Its internal event source is either the vanilla Windows Message queue, or an IPC channel which can be exposed to the outer framework (for enlisting in a select()-like call), like the socket of an X connection. (b) Its queue can be Peek'ed (to check for buffered msgs before blockigng again) HTH, -Alex

Mark Hammond wrote:

...

I'd really like to challenge that 'almost'...

Sure - but the problem is simply that MFC has a non-standard message - so it _must_ be like I described in my message - and you will agree that sounds messy.

If such a set of multiplexing primitives took off, and people found them useful, and started complaining they dont work in Pythonwin, then I will surely look at it again. Its too much work and too intrusive for a proof-of-concept effort.

Okay, fine.

...

I understand. Maybe I underestimated some of the difficulties. However, I'd still like to separate what can be separated. The unfriendliness to Python debuggers is sad news to me, but is not strictly related to the problem of heterogeneous multiplexing: if I were to design a debugger from scratch for a random language, I believe I'd arrange for the IPC channel used to be more transparent. IOW, the very fact of using the message queue for the debugging IPC *is* the culprit ! In unix, the ptrace() or /proc interfaces have never walked on the toes of any package, GUI or not...

The unfriendliness is purely related to Pythonwin, and not the general Python debugger. I agree 100% that an RPC type mechanism is far better for a debugger. It was just an anecdote to show how fickle these message loops can be (and therefore the complex requirements they have).

Okay, so maybe it's time to summarize what we agreed on: (1) 'tearing open' the main loop of a GUI package is tricky in the general case. (2) perusing undefined WM_* messages requires care... (3) on the other hand, all other IPC channels are multiplexable. Even for the worst case (pipes on Windows) at least 1 (1.5?) method has been identified. The temporary conclusion as far as I understand, is that nobody in the Python community has the spare time and energy to tackle (1), that (2) is tricky due to an unfortunate choice in the implementation of some debuggers, and that the seemingly appealing unification outlined by (3) is not enough of a motivation... Under these conditions, clearly the only option is to put the blackbox GUI loop inside a separate thread and arrange for it to use a well-chosen IPC channel to awaken (something like) the Watcher.go() proposed by Ka-Ping Yee. Now there's still the issue of actually making select.select() crossplatform. Any takers ? -Alex

Guido van Rossum wrote:

...

Yup. One easy answer is 'just copy from Tcl'...

Tcl seems to be your only frame of reference.

Nope, but I'll welcome any proof of existence of similar abstractions (for multiplexing) elsewhere.

I think it's too early to say that borrowing Tcl's design is right for Python. Don't forget that part of Tcl's design was guided by the desire for backwards compatibility with Tcl's strong (stronger than Python I find!) Unix background.

I don't quite get how the 'unix background' comes into play here, since [fileevent] is now implemented and works correctly on all platforms. If you are talinkg about the API as seen from above, I don't understand why 'hooking a callback' and 'multiplexing event sources' are a unix specificity, and/or why it should be avoided outside unix.

...

Seriously, I'm really too new to Python to suggest the details or even the *style* of this 'level 2 API to multiplexing'. However, I can sketch the implementation since select() (from C or Tcl) is the one primitive I most depend on !

Basically, as shortly mentioned before, the key problem is the heterogeneity of seemingly-selectable things in Windoze. On unix, not only does select() work with all descriptor types on which it makes sense, but also the fd used by Xlib is accessible; hence clean multiplexing even with a GUI package is trivial. Now to the real (rotten) meat, that is M$'s. Facts:

Note that on Windows, select() is part of SOCKLIB, which explains why it only understands sockets. Native Windows code uses the wait-for-event primitives that you are describing, and these are powerful enough to wait on named pipes, sockets, and GUI events. Complaining about the select interface on Windows isn't quite fair.

Sorry, you missed the point. Here I used the term 'select()' as a generic one (I didn't want to pollute a general discussion with OS-specific names...). On windows it means MsgWaitForMultipleObjects. Now as you said "these are powerful enough to wait on named pipes, sockets, and GUI events"; I won't deny the obvious truth. However, again, they don't work on *unnamed pipes* (which are the only ones in '95). That's my sole reason for complaining, and I'm afraid it is fair ;-)

...

1. 'Handle' types are not equal. Unnames pipes are (surprise!) not selectable. Why ? Ask a relative in Redmond...

Can we cut the name-calling?

Yes we can :^P

...

2. 'Handle' types are not equal (bis). Socket 'handles' are *not* true handles. They are selectable, but for example you can't use'em for redirections. Okay in our case we don't care. I only mention it cause its scary and could pop back into your face some time later.

Handles are a much more low-level concept than file descriptors. get used to it.

Take it easy, I meant to help. Low level as they be, can you explain why *some* can be passed to CreateProcess as redirections, and *some* can't ? Obviously there *is* some attempt to unify things in Windows (if only the single name of 'handle'); and just as clearly it is not completely successful.

...

3. The GUI API doesn't expose a descriptor (handle), but fortunately (though disgustingly) there is a special syscall to wait on both "the message queue" and selectable handles: MsgWaitForMultipleObjects. So its doable, if not beautiful.

The Tcl solution to (1.), which is the only real issue,

Why is (1) the only issue?

Because for (2) we don't care (no need for redirections in our case) and for (3) the judgement is only aesthetic.

Maybe in Tcl-land...

Come on, I'm emigrating from Tcl to Python with open palms, as Cameron puts it. I've already mentioned the outstanding beauty of Python's internal design, and in comparison Tcl is absolutely awful. Even at the (script) API level, Some of the early choices in Tcl are disgusting (and some recent ones too...). I'm really turning to Python with the greatest pleasure - please don't interpret my arguments as yet another Lang1 vs. Lang2 flamewar.

...

is to have a separate thread blockingly read 1 byte from the pipe, and then post a message back to the main thread to awaken it (yes, ugly code to handle that extra byte and integrate it with the buffering scheme).

Or the exposed API could deal with this in a different way.

Please elaborate ?

...

In summary, why not peruse Tcl's hard-won experience on selecting-on-windoze-pipes ?

Because it's designed for Tcl.

I said 'why not' as a positive suggestion. I didn't expect you to actually say why not... Moreover, I don't understand 'designed for Tcl'. What's specific to Tcl in unifying descriptor types ?

...

Then, for the API exposed to the Python programmer, the Tclly exposed one is a starter:

fileevent $channel readable|writable callback ... vwait breaker_variable

Explanation for non-Tclers: fileevent hooks the callback, vwait does a loop of select(). The callback(s) is(are) called without breaking the loop, unless $breaker_variable is set, at which time vwait returns.

Sorry, you've lost me here. Fortunately there's more info at http://dev.scriptics.com/man/tcl8.3/TclCmd/fileevent.htm. It looks very complicated,

Ahem, self-destroying argument: "Fortunately ... very complicated". While I agree the fileevent manpage is longer than it should be, I fail to see what's complicated in the model of 'hooking a callback for a given kind of events'.

and I'm not sure why you rejected my earlier suggestion to use threads outright as "too complicated".

Not on the same level. You're complaining about the script-level API (or its documentation, more precisely !). I dismissed the thread-based *implementation* as an overkill in terms of resource consumption (thread context + switching + ITC) on platforms which can use select() (for anon pipes on Windows, as already explained, the thread is unavoidable).

After reading that man page, threads seem easy compared to the caution one has to exert when using non-blocking I/O.

Oh, I get it. The problem is, *that* manpage unfortunately tries to explain event-based and non-blocking I/O at the same time (presumably because the average user will never follow the 'See Also' links). That's a blatant pedagogic mistake. Let me try: fileevent <channel> readable|writable <script> Hooks <script> to be called back whenever the given <channel> becomes readable|writable. 'Whenever' here means from within event processing primitives (vwait, update). Example: # whenever a new line comes down the socket, display it. set s [socket $host $port] fileevent $s readable gotdata proc gotdata {} {global s;puts "New data: [gets $s]"} vwait forever To answer a potential question about blockingness, yes, in the example above the [gets] will block until a complete line is received. But mentioning this fact in the manpage is uselessly misleading because the fileevent mechanism obviously allows to implement any kind of protocol, line-based or not, terminator- or size-header- based or not. Uses with blocking and nonblocking [read] and mixes thereof are immediate consequences of this classification. Hope this helps.

Vwait seems to be part of the Tcl event model.

Hardly. It's just the Tcl name for the primitive that (blockingly) calls select() (generic term - see above)

Maybe we would need to think about an event model for Python?

With pleasure - please define 'model'. Do you mean callbacks vs. explicit decoding of an event strucutre ? Do you mean blocking select() vs. something more asynchronous like threads or signals ?

On the other hand, Python is at the mercy of the event model of whatever GUI package it is using -- which could be Tk, or wxWindows, or Gtk, or native Windows, or native MacOS, or any of a number of other event models.

Why should Python be alone to be exposed to this diversity ? Don't assume that Tk is the only option for Tcl. The Tcl/C API even exposes the proper hooks to integrate any new event source, like a GUI package. Again, I'm not interested in Tcl vs. Python here (and anyway Python wins !!!). I just want to extract what's truly orthogonal to specific design choices. As it turns out, what you call 'the Tcl event model' can happily be transported to any (imperative) lang. I can even be more precise: a random GUI package can be used this way iff the two following conditions hold: (a) Its queue can awaken a select()-like primitive. (b) Its queue can be Peek'ed (to check for buffered msgs before blockigng again)

Perhaps this is an issue that each GUI package available to Python will have to deal with separately...

The characterization is given just above. To me it looks generic enough to build an abstraction upon it. It's been done for Tcl, and is utterly independent from its design peculiarities. Now everything depends on whether abstraction is sought or not... -Alex

On Mon, 22 May 2000, Cameron Laird wrote:

Tcl's event model has been more successful than any of you probably realize. You deserve to know that.

Events are a very powerful concurrency model (arguably more reliable because they are easier to understand than threads). My friend Mark Miller has designed a language called E (http://www.erights.org/) that uses an event model for all object messaging, and i would be interested in exploring how we can apply those ideas to improve Python.

Should Python have an event model? I'm not con- vinced.

Indeed. This would be a huge core change, way too large to be feasible. But i do think it would be excellent to simply provide more facilities for helping people use whatever model they want, and given the toolkit we let people build great things. What you described sounded like it could be implemented fairly easily with some functions like register(handle, mode, callback) or file.register(mode, callback) Put 'callback' in a dictionary of files to be watched for mode 'mode'. mainloop(timeout) Repeat (forever or until 'timeout') a 'select' on all the files that have been registered, and do calls to the callbacks that have been registered. Presumably there would be some exception that a callback could raise to quietly exit the 'select' loop. 1. How does Tcl handle exiting the loop? Is there a way for a callback to break out of the vwait? 2. How do you unregister these callbacks in Tcl? -- ?!ng

On Tue, 23 May 2000, Alexandre Ferrieux wrote:

Great !!! That's exactly the kind of Pythonic translation I was expecting. Thanks !

Here's a straw man. Try the attached module. To test it, run: python ./watcher.py 10203 then telnet to port 10203 on the local machine. You can open several telnet connections to port 10203 at once. In one session: skuld[1041]% telnet localhost 10203 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. >>> 1 + 2 3 >>> spam = 3 In another session: skuld[1008]% telnet localhost 10203 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. >>> spam 3

We just register an empty string as the callback name (script). But this is just a random API choice. Anything more Pythonic is welcome (an explicit unregister function is okay for me).

So is there no way to register more than one callback on a particular file? Do you ever find yourself wanting to do that? -- ?!ng

On Mon, 22 May 2000, Guido van Rossum wrote:

Can we cut the name-calling?

Hey, what's life without a MS bashing now and then <wink>?

Vwait seems to be part of the Tcl event model. Maybe we would need to think about an event model for Python? On the other hand, Python is at the mercy of the event model of whatever GUI package it is using -- which could be Tk, or wxWindows, or Gtk, or native Windows, or native MacOS, or any of a number of other event models.

But that's sort of the point: Python needs a non-GUI event model, to use with daemons which need to handle many files. Every GUI package would have its own event model, and Python will have one event model that's not tied to a GUI package. that-only-proves-we-have-a-problem-ly y'rs, Z. -- Moshe Zadka http://www.oreilly.com/news/prescod_0300.html http://www.linux.org.il -- we put the penguin in .com

On Mon, 22 May 2000, Cameron Laird wrote:

I've got a whole list of "higher-level abstractions around OS stuff" that I've been collecting. Maybe I'll make it fit for others to see once we're through this affair

Absolutely! I've thought about this too. A nice "child process management" module would be very convenient to have -- i've done such stuff before -- though i don't know enough about Windows semantics to make one that works on multiple platforms. Some sort of (hypothetical) delegate.spawn(function) - return a child object or id delegate.kill(id) - kill child etc. could possibly free us from some of the system dependencies of fork, signal, etc. I currently have a module called "delegate" which can run a function in a child process for you. It uses pickle() to send the return value of the function back to the parent (via an unnamed pipe). Again, Unix-specific -- but it would be very cool if we could provide this functionality in a module. My module provides just two things, but it's already very useful: delegate.timeout(function, timeout) - run the 'function' in a child process; if the function doesn't finish in 'timeout' seconds, kill it and raise an exception; otherwise, return the return value of the function delegate.parallelize(function, [work, work, work...]) - fork off many children (you can specify how many if you want) and set each one to work calling the 'function' with one of the 'work' items, queueing up work for each of the children until all the work gets done. Return the results in a dictionary mapping each 'work' item to its result. -- ?!ng