[Python-Dev] Making python C-API thread safe (try 2)

[Paul Winkler]

Harri Pesonen <fuerte at sci.fi> wrote in message news:<EIL8b.6564$ZB4.1083 at reader1.news.jippii.net>...
> A.M. Kuchling wrote:
> > On Sat, 13 Sep 2003 13:55:59 GMT, 
> > 	Mitch Chapman <mitchchapman at earthlink.net> wrote:
> It's not useless, but it is not optimal. That's a fact. If you have a 
> multiprocessor machine, you just waste it because Python can't run two 
> threads at the same time. So if you want to create a high performance 
> web site, then Python is out of the question for the scripting language.

You're making an awful lot of assumptions there. I'm going to
concentrate on one of them.

First, you assume that this hypothetical high-performance website
really is CPU-bound. I've never had one of those to deal with, but I
won't argue.

Second, I'll assume that SMP really is the most cost-effective way
available to scale up your CPU power. Probably depends on the rack
space you have available and maybe somebody already provided an SMP
box for the job.

Third, I'll assume that cacheing is already being used to the extent
that it's applicable. Every website I've worked on is much more
read-intensive than write-intensive. For these sort of sites,
appropriate cacheing buys you a LOT.
Disk space and RAM are cheap. But maybe you have an app where this
isn't the case and you really can't avoid heavy CPU useage.

Fourth, you seem to think that threading is the only / best-practice
way to handle a lot of requests. This is the point I'm going to argue
against.

I'll give one counter-example because it's the one I'm most familiar
with.
In the Zope world, it's common to run a load balancer in front of one
or more ZEO clients (processes) per processor, all connected to a
single ZEO server. This is a matter of very simple configuration. It's
*completely* transparent to the application, which has no idea whether
you're running ZEO or have a direct connection to a dedicated ZODB
database. ZEO has the added benefit of being equally applicable
whether the CPUs are on one box or not. You can't say that about
threading. This approach could certainly be used outside the zope
world. ZEO is not rocket science, it's about 6500 lines of code plus
3100 lines of unit tests. Note also that ZEO's primary purpose is to
remove one of the single points of failure; handling more requests is
almost a side effect. Free threading won't help you with the
point-of-failure issue.

I will also mention that it's quite possible to handle a lot of
requests without using threads *at all*, and the resulting code can
look very nice.  See twistedmatrix.com.

Sure, it would be nice if a single python process could take advantage
of SMP, but it's demonstrably false that python is currently "out of
the question" for "high performance" web use. In fact python performs
very well on the web. To say otherwise is nothing but FUD.

I think the primary problem with the GIL is the bad publicity that
results from faulty assumptions like this.