The Future of Python Threading
jmtulloss at gmail.com
Sat Aug 11 01:17:52 CEST 2007
On Aug 10, 10:34 am, Jean-Paul Calderone <exar... at divmod.com> wrote:
> >I'm not an expert, but I understand that much. What greenlets do is
> >force the programmer to think about concurrent programming. It doesn't
> >force them to think about real threads, which is good, because a
> >computer should take care of that for you.Greenlets are nice because
> >they can run concurrently, but they don't have to. This means you can
> >safely divide them up among many threads. You could not safely do this
> >with just any old python program.
> There may be something to this. On the other hand, there's no _guarantee_
> that code written with greenlets will work with pre-emptive threading instead
> of cooperative threading. There might be a tendency on the part of developers
> to try to write code which will work with pre-emptive threading, but it's just
> that - a mild pressure towards a particular behavior. That's not sufficient
> to successfully write correct software (where "correct" in this context means
> "works when used with pre-emptive threads", of course).
Agreed. Stackless does include a preemptive mode, but if you don't use
it, then you don't need to worry about locking at all. It would be
quite tricky to get around this, but I don't think it's impossible.
For instance, you could just automatically lock anything that was not
a local variable. Or, if you required all tasklets in one object to
run in one thread, then you would only have to auto-lock globals.
> One also needs to consider the tasks necessary to really get this integration
> done. It won't change very much if you just add greenlets to the standard
> library. For there to be real consequences for real programmers, you'd
> probably want to replace all of the modules which do I/O (and maybe some
> that do computationally intensive things) with versions implemented using
> greenlets. Otherwise you end up with a pretty hard barrier between greenlets
> and all existing software that will probably prevent most people from changing
> how they program.
If the framework exists to efficiently multi-thread python, I assume
that the module maintainers will slowly migrate over if there is a
performance benefit there.
> Then you have to worry about the other issues greenlets introduce, like
> invisible context switches, which can make your code which _doesn't_ use
> pre-emptive threading broken.
Not breaking standard python code would definitely be priority #1 in
an experiment like this. I think that by making the changes at the
core we could achieve it. A standard program, after all, is just 1
> All in all, it seems like a wash to me. There probably isn't sufficient
> evidence to answer the question definitively either way, though. And trying
> to make it work is certainly one way to come up with such evidence. :)
::Sigh:: I honestly don't see myself having time to really do anything
more than experiment with this. Perhaps I will try to do that though.
Sometimes I do grow bored of my other projects. :)
More information about the Python-list