[Python-Dev] Pythonic concurrency

[Guido van Rossum]

On 10/10/05, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> I'm wondering whether Python threads should be
> non-preemptive by default. Preemptive threading is
> massive overkill for many applications. You don't
> need it, for example, if you just want to use threads
> to structure your program, overlap processing with I/O,
> etc.

I recall using a non-preemptive system in the past; in Amoeba, to be precise.

Initially it worked great.

But as we added more powerful APIs to the library, we started to run
into bugs that were just as if you had preemptive scheduling: it
wouldn't always be predictable whether a call into the library would
need to do I/O or not (it might use some sort of cache) so it would
sometimes allow other threads to run and sometimes not. Or a change to
the library would change this behavior (making a call that didn't use
to block into sometimes-blocking).

Given the tendency of Python developers to build layers of
abstractions I don't think it will help much.

> Preemptive threading would still be there as an option
> to turn on when you really need it.
>
> Or perhaps there could be a priority system, with a
> thread only able to be preempted by a thread of higher
> priority. If you ignore priorities, all your threads
> default to the same priority, so there's no preemption.
> If you want a thread that can preempt others, you give
> it a higher priority.

If you ask me, priorities are worse than the problem they are trying to solve.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)