[Python-Dev] Trial balloon: microthreads library in stdlib

[glyph at divmod.com]

On 04:49 pm, richard.m.tew at gmail.com wrote:
>On 2/13/07, glyph at divmod.com <glyph at divmod.com> wrote:
>>Tristan is correct: this should be a patch against Twisted, or perhaps as a
>>separate library that could implement a reactor.
>
>I think there is some confusion here.

Quite.

>I am not writing a competing event driven mechanism.  What I was doing
>was feeling out whether there was any interest in better support for
>asynchronous calls.

Perhaps you could describe what you were proposing a bit better, because both to Tristan and myself, it sounded (and frankly, still sounds) like you were describing that would directly replace Twisted's mainloop core API.  "asynchronous calls" is a very vague term with at least three completely separate definitions.

>Yes I have looked at Twisted.  It was the first place I looked, several
>months ago, and what drew me to it was the IOCP reactor.  However
>as I also explained in my reply to Tristan it is not suitable for my
>needs.

As far as I can tell, you still haven't even clearly expressed what your needs are, let alone whether or not Twisted is suitable.  In the reply you're citing, you said that "this" sounded like something "low level" that "twisted would be written on top of" - but the "this" you were talking about, based on your previous messages, sounded like monkeypatching the socket and asyncore modules to provide asynchronous file I/O based on the platform-specific IOCP API for Windows.

Twisted can already provide that functionality in a _much_ cleaner fashion already, although you might need to edit some code or monkeypatch some objects specifically to get asynchronous file I/O, at least Twisted is *designed* for such applications.

To quote you from the original message that Tristan replied to:

>> I can't go into low level details because I do not have the necessary 
>> experience or knowledge to know which approach would be best. ... 
>> I of course have no concept of how this might be done on other platforms.

As someone who _does_ have the necessary experience and knowledge, I can tell you that Twisted *IS* the "one stop shop" for events that you're describing.  I do know how one would do such a thing on other platforms and it is not a simple task - so much so that async file I/O is still not available in Twisted today.

>It is a large dependency and it is a secondary framework.

Has it occurred to you that it is a "large" dependency not because we like making bloated and redundant code, but because it is doing something that is actually complex and involved?

>And I did not feel the need to verify the implication that it wasn't
>ready because this matched my own recollections.

It meaning... Twisted?  Twisted's IOCP support?  Ready for... what?  IOCPReactor definitely not up to the standard of much of the rest of the code in Twisted, but it's clearly ready for _something_ since BitTorrent uses it.

>But I hope you realise that asserting that things should be within
>Twisted without giving any reason,

I have suggested that work proceed in Twisted rather than in Python because adding async file I/O to Twisted would be much easier than adding an entirely new event-loop core to Python, and then adding async file I/O to *that*.

I thought that I provided several reasons before as well, but let me state them as clearly as I can here.  Twisted is a large and mature framework with several years of development and an active user community.  The pluggable event loop it exposes is solving a hard problem, and its implementation encodes a lot of knowledge about how such a thing might be done.  It's also tested on a lot of different platforms.

Writing all this from scratch - even a small subset of it - is a lot of work, and is unlikely to result in something robust enough for use by all Python programs without quite a lot of effort.  It would be a lot easier to get the Twisted team involved in standardizing event-driven communications in Python.  Every other effort I'm aware of is either much smaller, entirely proprietary, or both.  Again, I would love to be corrected here, and shown some other large event-driven framework in Python that I was heretofore unaware of.  Standardization is much easier to achieve when you have multiple interested parties with different interests to accommodate.  As Yitzhak Rabin used to say, "you don't engage in API standardization with your friends, you engage in API standardization with your enemies" - or... something like that.

>especially when the first person
>saying it just stated that the matching work in Twisted wasn't even
>ready, feels like Twisted is trying to push itself forward by bullying
>people to work within it whenever something can be asserted as
>laying within whatever domain it is which Twisted covers.

You say that you weren't proposing an alternate implementation of an event loop core, so I may be reacting to something you didn't say at all.  However, again, at least Tristan thought the same thing, so I'm not the only one.

I think that *that* could be described as bullying.  I -- and many others -- have a considerable investment in Twisted, and attempts to "standardize" on another, new main-loop within the standard library is an attempt to subvert that investment and pre-empt honest competition between different event-driven frameworks, by de-facto forcing all Python users (and Twisted itself) to learn and deal with what the standard library is doing.  Ignoring the Twisted API while doing this adds insult to injury.  There are some clear and very deep misunderstandings in this discussion about what the Twisted reactor is, how it works, and what it does, so I do really feel like it is being ignored.

Not all of this is a direct reaction to things you've said, but to the continuing pattern of posts to this list which encourage development of new event-driven mechanisms or changes to asyncore to add features, while still ignoring Twisted and all the lessons learned from it.

If we're talking about inclusion in the standard library, maybe Twisted's event loop *isn't* ready.  Certainly all the _rest_ of Twisted isn't.  I have admitted that already in this thread.  That doesn't mean another from-scratch toy event loop, even one that Twisted could be implemented "on top of" _is_ ready.

Stackless - I'd say that's certainly ready, although I understand the other concerns voiced in this thread.  It's got a proven track-record and it, also, solves a difficult problem.  Re-implementing microthreads for python from scratch, while ignoring stackless, specifically for inclusion in the standard library would also, presumably, be a bad idea.

>Even if it doesn't suit their needs.

If someone doesn't "have the necessary experience or knowledge to know which approach would be best", then how, exactly, do they know whether things suit their needs or not?  You specifically claimed ignorance, and then said you were being "bullied" when someone tried to give you the information that you are lacking.

>I don't think Tristan intended
>that, but when followed up with your reply and JP's interesting
>response to Greg, it feels like it.

I thought JP's response to Greg was useful, since much of this discussion has been hopelessly vague.  For example, your earlier claim that "it wasn't ready".

I've tried to restrict my comments here to the possibility of a new event-driven core in the Python standard library.  There are a lot of other features also being discussed here - a microthread scheduler API, for example - which I am _not_ advocating the use of Twisted for.  Twisted could definitely make use of such a scheduler API if it existed in the stdlib.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/6756bc17/attachment.html