[Python-Dev] Pythonic concurrency

[Shane Hathaway]

Bruce Eckel wrote:
> But. I do happen to have contact with a lot of people who are at the
> forefront of the threading world, and *none* of them (many of whom
> have written the concurrency libraries for Java 5, for example) ever
> imply that threading is easy. In fact, they generally go out of their
> way to say that it's insanely difficult.

What's insanely difficult is really locking, and locking is driven by 
concurrency in general, not just threads.  It's hard to reason about 
locks.  There are only general rules about how to apply locking 
correctly, efficiently, and without deadlocks.  Personally, to be 
absolutely certain I've applied locks correctly, I have to think for 
hours.  Even then, it's hard to express my conclusions, so it's hard to 
be sure future maintainers will keep the locking correct.

Java uses locks very liberally, which is to be expected of a language 
that provides locking using a keyword.  This forces Java programmers to 
deal with the burden of locking everywhere.  It also forces the 
developers of the language and its core libraries to make locking 
extremely fast yet safe.  Java threads would be easy if there wasn't so 
much locking going on.

Zope, OTOH, is far more conservative with locks.  There is some code 
that dispatches HTTP requests to a worker thread, and other code that 
reads and writes an object database, but most Zope code isn't aware of 
concurrency.  Thus locking is hardly an issue in Zope, and as a result, 
threading is quite easy in Zope.

Recently, I've been simulating high concurrency on a PostgreSQL 
database, and I've discovered that the way you reason about row and 
table locks is very similar to the way you reason about locking among 
threads.  The big difference is the consequence of incorrect locking: in 
PostgreSQL, using the serializable mode, incorrect locking generally 
only leads to aborted transactions; while in Python and most programming 
languages, incorrect locking instantly causes corruption and chaos. 
That's what hurts developers.  I want a concurrency model in Python that 
acknowledges the need for locking while punishing incorrect locking with 
an exception rather than corruption.  *That* would be cool, IMHO.

Shane