[Python-ideas] micro-threading PEP proposal (long) -- take 2!
dangyogi at gmail.com
Fri Aug 29 14:35:45 CEST 2008
Antoine Pitrou wrote:
> Having separate PEPs also makes it easier to discuss the issues piecewise rather
> than a whole big chunk of additions.
I am preparing 3 new PEPs now. One for the C level, one for
micro-threads which provides some basic capabilities, and one for
micro-pipes which I expect will be the one that will change the most and
take the longest to put to bed. I'm also concentrating on APIs rather
than implementations and will include examples of using these APIs.
>> They seem necessary to handle exception situations. For example, when a
>> reader thread on a pipe dies with an exception, how is the write thread
>> notified? What mechanism knows that this pipe was being read by the
>> errant thread so that it will never be read from again?
> A pipe could be divided into two half-objects: the receiving end and the sending
> end, each independently managed by the standard reference counting mechanism,
> and referencing each other through weakrefs. That way, when e.g. the last
> reference to the receiving end dies, the sending end will notice that and can
> raise an exception when a thread tries to write to it.
I don't think that this would be very portable to other flavors of
python (jython/ironpython/pypy) that don't use reference counting. It
makes the thread termination dependent on the implementation of the
> In any case, I don't think creating standard "stdin" and "stdout" pipes for each
> thread makes things any easier. You still have to handle the case of whatever
> non-stdin/stdout pipes get created.
I imagine that the python programmer would not be allowed to create
micro-pipes directly. I'm thinking that there would be a num_stdout (=
0) parameter on the micro_thread constructor that creates that many
micro_pipes, and then the python programmer can connect these to the
stdin of other micro-threads. The reason for this is so that the
underlying implementation always knows how the dots are connected so
that it can provide sensible exception/abort semantics.
I'm working on another project that uses generators a _lot_ and there
are problems there because 'for' loops don't call 'close' on generators
to clean things up. I've also hit problems where the code works fine on
CPython, but fails on jython and ironpython because I'm relying on the
reference counting to immediately collect abandoned generators and run
their 'finally' clauses.
>> Good suggestion! I was hoping that some might show up here, but ... I
>> guess I need to go looking for them!
> A Twisted developer told me that he had tried to read your PEP (the first
> version) but found it difficult to understand.
I hope that the new versions will be easier to follow!
More information about the Python-ideas