[Python-ideas] micro-threading PEP proposal (long) -- take 2!

[Bruce Frederiksen]

Antoine Pitrou wrote:
> Having separate PEPs also makes it easier to discuss the issues piecewise rather
> than a whole big chunk of additions.
>   
I am preparing 3 new PEPs now.  One for the C level, one for 
micro-threads which provides some basic capabilities, and one for 
micro-pipes which I expect will be the one that will change the most and 
take the longest to put to bed.  I'm also concentrating on APIs rather 
than implementations and will include examples of using these APIs.
>> They seem necessary to handle exception situations.  For example, when a 
>> reader thread on a pipe dies with an exception, how is the write thread 
>> notified?  What mechanism knows that this pipe was being read by the 
>> errant thread so that it will never be read from again?
>>     
>
> A pipe could be divided into two half-objects: the receiving end and the sending
> end, each independently managed by the standard reference counting mechanism,
> and referencing each other through weakrefs. That way, when e.g. the last
> reference to the receiving end dies, the sending end will notice that and can
> raise an exception when a thread tries to write to it.
>   
I don't think that this would be very portable to other flavors of 
python (jython/ironpython/pypy) that don't use reference counting.  It 
makes the thread termination dependent on the implementation of the 
garbage collector.
> In any case, I don't think creating standard "stdin" and "stdout" pipes for each
> thread makes things any easier. You still have to handle the case of whatever
> non-stdin/stdout pipes get created.
>   
I imagine that the python programmer would not be allowed to create 
micro-pipes directly.  I'm thinking that there would be a num_stdout (= 
0) parameter on the micro_thread constructor that creates that many 
micro_pipes, and then the python programmer can connect these to the 
stdin of other micro-threads.  The reason for this is so that the 
underlying implementation always knows how the dots are connected so 
that it can provide sensible exception/abort semantics.

I'm working on another project that uses generators a _lot_ and there 
are problems there because 'for' loops don't call 'close' on generators 
to clean things up.  I've also hit problems where the code works fine on 
CPython, but fails on jython and ironpython because I'm relying on the 
reference counting to immediately collect abandoned generators and run 
their 'finally' clauses.
>   
>> Good suggestion!  I was hoping that some might show up here, but ...  I 
>> guess I need to go looking for them!
>>     
>
> A Twisted developer told me that he had tried to read your PEP (the first
> version) but found it difficult to understand.
>   
I hope that the new versions will be easier to follow!

Thanks!

-bruce