The Future of Python Threading

[Justin T.]

On Aug 10, 10:34 am, Jean-Paul Calderone <exar... at divmod.com> wrote:

> >I'm not an expert, but I understand that much. What greenlets do is
> >force the programmer to think about concurrent programming. It doesn't
> >force them to think about real threads, which is good, because a
> >computer should take care of that for you.Greenlets are nice because
> >they can run concurrently, but they don't have to. This means you can
> >safely divide them up among many threads. You could not safely do this
> >with just any old python program.
>
> There may be something to this.  On the other hand, there's no _guarantee_
> that code written with greenlets will work with pre-emptive threading instead
> of cooperative threading.  There might be a tendency on the part of developers
> to try to write code which will work with pre-emptive threading, but it's just
> that - a mild pressure towards a particular behavior.  That's not sufficient
> to successfully write correct software (where "correct" in this context means
> "works when used with pre-emptive threads", of course).
Agreed. Stackless does include a preemptive mode, but if you don't use
it, then you don't need to worry about locking at all. It would be
quite tricky to get around this, but I don't think it's impossible.
For instance, you could just automatically lock anything that was not
a local variable. Or, if you required all tasklets in one object to
run in one thread, then you would only have to auto-lock globals.

>
> One also needs to consider the tasks necessary to really get this integration
> done.  It won't change very much if you just add greenlets to the standard
> library.  For there to be real consequences for real programmers, you'd
> probably want to replace all of the modules which do I/O (and maybe some
> that do computationally intensive things) with versions implemented using
> greenlets.  Otherwise you end up with a pretty hard barrier between greenlets
> and all existing software that will probably prevent most people from changing
> how they program.

If the framework exists to efficiently multi-thread python, I assume
that the module maintainers will slowly migrate over if there is a
performance benefit there.
>
> Then you have to worry about the other issues greenlets introduce, like
> invisible context switches, which can make your code which _doesn't_ use
> pre-emptive threading broken.

Not breaking standard python code would definitely be priority #1 in
an experiment like this. I think that by making the changes at the
core we could achieve it. A standard program, after all, is just 1
giant tasklet.
>
> All in all, it seems like a wash to me.  There probably isn't sufficient
> evidence to answer the question definitively either way, though.  And trying
> to make it work is certainly one way to come up with such evidence. :)

::Sigh:: I honestly don't see myself having time to really do anything
more than experiment with this. Perhaps I will try to do that though.
Sometimes I do grow bored of my other projects. :)

Justin