[Python-Dev] Trial balloon: microthreads library in stdlib

["Martin v. Löwis"]

Richard Tew schrieb:
> It depends what you mean by mechanism.  I can't go into low level details
> because I do not have the necessary experience or knowledge to know
> which approach would be best.

Indeed, low-level details is what I'm interested in.

> I currently use Windows.  By using asyncore and monkeypatching in a
> replacement socket module based on it [1] I can give users of Stackless
> socket operations which invisibly block at the microthread level allowing
> the scheduler to continue to run.  However there is no support for
> asynchronous file IO.

That's actually a limitation of Windows: in a UNIX system, you can pass
sockets and file descriptors alike to select(2) or poll(2) (in fact,
sockets *are* file descriptors).

> Now I can monkeypatch in a replacement file
> object which uses IO completion ports [2]

Ok, so you want to use IO completion ports on Windows. This is the
mechanism I was asking for (overlapped IO would have been another
mechanism available on Win32).

Adding support for IO completion ports is certainly something
that would be worthwhile.

> but at that stage I need to
> poll two different resources for events.  In order to avoid this it makes
> sense to stop using asyncore for sockets and to write a new
> replacement socket object based on IO completion ports.

I don't know whether/how this is possible. The underlying 
WaitForMultipleObjects routine does not accept WinSock objects,
to my knowledge (perhaps it does in Vista, with the rewriting
of WinSock?).

However, if it would be possible, I would suggest to add support
for this into the select module, and allow then to pass non-socket
objects to select on Windows (just as it is possible on Unix).
If this can't work (as you need to associate the file handle
with a io completion port, after which you can't use ReadFile
anymore), a new poll-like routine may be added, and then support
for that may be added to asyncore.

Notice that WaitForMultipleObjects is limited to MAXIMUM_WAIT_OBJECTS,
which I believe is 64. So this is somewhat limiting.

> Perhaps there is a better way.  And I of course have no concept of
> how this might be done on other platforms.

For file descriptors, other platforms are ahead of Windows.
For child process IDs, it's somewhat more tricky: you get a
POSIX signal when some of them terminates, and any system
call should terminate with EINTR. So the universal poll
routine would abort when a child dies, and could then check
a global variable set by the signal handler.

In any case, one needs to specify such a "universal event
loop" precisely, or else it will just fail. Look at
Tcl's even mechanism for an example that has failed (IMO).

Regards,
Martin