Stackless, thread paradigm and C extensions

[Frederic Giacometti]

"Michael Hudson" <mwh at python.net> wrote in message
news:u8zdaodu4.fsf at python.net...
> "Frederic Giacometti" <frederic.giacometti at arakne.com> writes:
>
> > "Michael Hudson" <mwh at python.net> wrote in message
> > news:uu1w0e0vo.fsf at python.net...
> [...]
> > > I must be missing something.  Can you sketch how, say, socket.send()
> > > would be implemented?  Would you spawn a new OS thread for every C API
> > > call?  I'm afraid I don't understand you to this point...
> >
> > As I mentioned, two thread pools would be maintained.  To the extent
> > that send() is reentrant on the underlying OS, it would beexecuted
> > in one of the threads of the reentrant thread pool.  This is the
> > 'thread pool' pattern; where threads are kept up (i.e. active) from
> > one call to the next. It's a standard pattern and concurrent
> > programming algorithm.
>
> So when Python code executes
>
>    sock.recv(data)
>
> the interpreter would take a thread form the pool, and in effect say
> "here, run this function".  Then the interpreter thread would go off
> and execute pending Python threads, and when the sock.recv call
> returned, it would add the thread that called it back to the set of
> pending interpreter threads?  (I think this would be easier to discuss
> with paper and pencil...)
>
> What does this buy us again?  It still makes writing C code that calls
> Python code a bit of a pain, doesn't it?  Oh, maybe not.  More
> thinking required...

A possible implementation would consist in using the method flag to mark C
python methods/functions to run outside the interpreter thread. This way,
the entire function runs in a separate thread.

The programmer does not have to explicitely manipulate OS threads at any
point; the continuation/OS thread correspondance would be taken care of
automatically by the threaded stackless implementation.
One would have parallel threads of execution through just continuations.
C extensions would run entirely either in a parallel thread, or in the
interpreter thread. There would be no more interpreter lock; and when a C
thread needs some service by the interpreter, it would just post a
continuation to the interpreter scheduler.

This will simplify programming when using Python callbacks, too; since the
currently active Python continuation would be the continuation active in the
thread.