[Python-Dev] Trial balloon: microthreads library in stdlib

[Richard Tew]

On 2/13/07, "Martin v. Löwis" <martin at v.loewis.de> wrote:
> Steve Holden schrieb:
> > The only things that concern me are a) whether it could make sense to
> > add Stackless in bits and pieces and b) whether the BDFL (or even the
> > developer community en masse) would object in principle, thereby
> > rendering such efforts useless.
>
> I think I need to try again. The 'developer community en masse' does
> not have a single voice, so it won't object. Not sure about the BDFL,
> but I personally don't object 'to a change like that' 'in principle',
> primarily because I don't know what the change is.
>
> I know I'm almost certainly object to the change 'incorporate
> Stackless Python into Python as-is', because history shows that
> any change of that size would need multiple iterations until it
> was acceptable (despite the code being in-use for multiple years).
> Very few people can get away with getting their code unedited
> into Python (ctypes and elementtree being the most prominent
> examples); in these cases, I could accept what I consider flaws
> in the code because:
> a) the authors promised to maintain the code, and react to
>     actual bug reports, and
> b) these come as extension modules, so it's easy to ignore
>     them if you don't like them.

This makes a lot of sense.  You have given a lot of food for
thought and a path forward.

If I gave an impression in my mails that I expected a consensus
of objection, it was because I believed objection to Stackless
to be an established thing and that it didn't matter whether
there was consensus or not because someone would step
forward and reiterate that.  I didn't want to push the issue or
invest wasted effort if that proved to be the case as mistakenly
expected.

At no time have I expected that Stackless would be added as-is.

> > My (limited) understanding is that with Stackless installed *all*
> > existing programs that don't import stackless should continue to run
> > unchanged.  If that's true then it seems to me it would be a desirable
> > addition as long as maintainability wasn't compromised. I suppose only
> > the patches will allow a sensible judgment on that issue.
>
> My understanding is that incorporating Stackless will reduce the
> portability: it cannot work on certain platforms, and even for
> platforms it works on, it cannot work with certain compilers
> or compiler switches (the Windows SEH being the primary example;
> the SPARC register stack another one - although this might now
> be supported through assembler code). On platforms where it isn't
> supported, it still may compile, but might crash the interpreter
> when used.

If there is no Stackless 'hard switching' available (the stack
switching done with assistance of assembler) for the platform it is
being compiled on, then compilation proceeds without Stackless
compiled into Python.  The module is not available.  The process
of adding support for additional platforms is rather straightforward
and with access to them I can do it if required.

I don't know about SEH but I believe support for SPARC was added
in 2002.
http://svn.python.org/view/stackless/trunk/Stackless/platf/switch_sparc_sun_gcc.h

> Please understand that this is from a shallow inspection a few
> years ago (actually, a description of the working mechanics that
> Christian gave me). It would make me feel uncomfortable if Python
> had modules that may randomly crash the interpreter, and there
> is no procedure to tell beforehand whether a certain applications
> is supported or not.

Right.  I imagine that if there are still problems like these, they
will be brought to light when and if patches are submitted. But for
now I will address what you mention just so that people don't
assume they are still the case, especially since Stackless has
been rewritten since then.

> Also, I would not like to see modules that monkey-patch other
> modules in the standard library. If socket.connect is to
> behave differently, the code that makes that so should be
> *in* socket.connect, and the documentation of socket.connect
> should state that it has that modified behavior. Of course,
> people may object to massive library changes of the nature
> "if running in stackless mode, this library routine behaves
>   differently".

Perhaps my pursuit of better support for asynchronous calls
has led to some confusion.  Stackless is an implementation
with minimal changes to the core.  It does not include any modules
which monkey patch.

However I have a replacement socket module based on asyncore
which is not part of the Stackless distribution.  If monkey patched
it allows people to use sockets suitably in their microthreads.
I am experimenting with this in order to see if I can make
using Stackless as close to being as straightforward as writing
normal Python code as possible, while still gaining the benefits
which use of microthreads should allow.

It was considered and quickly dismissed to modify the socket and
file support in the core at C level to do what was needed. Primarily
because it would complicate the limited changes we make to the
core.

Thanks,
Richard.