[Python-ideas] micro-threading PEP proposal (long) -- take 2!

[Antoine Pitrou]

Hi,

> I had considered that.  But the core by itself accomplishes nothing, 
> except to serve as a foundation for some kind of higher-level 
> constructs, so I put them together.  I guess having separate PEPs allows 
> them to evolve more independently.  (I'm new to this PEP process).

Having separate PEPs also makes it easier to discuss the issues piecewise rather
than a whole big chunk of additions.

> If I split them, so I keep posting updated versions on python-ideas?  Or 
> do I just accumulate the changes offline and post the completed PEP much 
> later?

I think it's better to post updated versions, at least as long as there seems to
be some interest.

> So micro_pipes don't store a sequence of values (like lists or deques), 
> but pass individual values on from one thread to another.  The 
> implementation proposed in the Python level pseudo code only stores one 
> value and will block the writer when it tries to write a second value 
> before the reader has read the first value.  This buffer size of one 
> could be expanded, but I've been working on the premise that this should 
> be kept as simple as possible for a first out, and then allowed to grow 
> after more experience is gained with it.

Yes, it's the kind of things that can be discussed as part of the implementation
rather than as part of the spec itself.

> They seem necessary to handle exception situations.  For example, when a 
> reader thread on a pipe dies with an exception, how is the write thread 
> notified?  What mechanism knows that this pipe was being read by the 
> errant thread so that it will never be read from again?

A pipe could be divided into two half-objects: the receiving end and the sending
end, each independently managed by the standard reference counting mechanism,
and referencing each other through weakrefs. That way, when e.g. the last
reference to the receiving end dies, the sending end will notice that and can
raise an exception when a thread tries to write to it.

In any case, I don't think creating standard "stdin" and "stdout" pipes for each
thread makes things any easier. You still have to handle the case of whatever
non-stdin/stdout pipes get created.

> > In this example, can you give the C pseudo-code and the equivalent Twisted
> > Python (pseudo-)code?
> >   
> Do you mean the pseudo code of the deferred implementation, or the 
> pseudo code for using the deferreds?

I mean the pseudo code for using the deferreds in the particular example which
is outlined. Also, if possible, the corresponding Twisted code, to see where and
how the two idioms diverge.

You can of course give pseudo-code for the implementation as well, but I think
discussing the implementation is prematurate if the API hasn't been discussed
first :-)

> Good suggestion!  I was hoping that some might show up here, but ...  I 
> guess I need to go looking for them!

A Twisted developer told me that he had tried to read your PEP (the first
version) but found it difficult to understand.

Regards

Antoine.