[Python-3000] Kill GIL?

[Calvin Spealman]

On 9/18/06, Ivan Krstić <krstic at solarsail.hcs.harvard.edu> wrote:
> > What use case *requires* sharing?
>
> Strictly speaking, it's always avoidable. But in setup-heavy systems,
> avoiding SHM is a massive and costly pain. Consider web applications;
> ideally, you can preload one copy of all of your translations, database
> information, and other static information, into RAM -- and have worker
> threads do reads from this table as they're processing individual
> requests. Without SHM, you'd have to either duplicate the static set in
> memory for each CPU, or make individual requests for each desired piece
> of information to the master process that keeps the static set in RAM.
>
> I've seen a number of computationally-bound systems that require an
> authoritative copy of a (large) dataset in RAM, and are OK with paying
> the cost of a read waiting on a lock during a write (and since writes
> only happen at the completion of complex calculations, they generally
> want to use locking like that provided by brlocks in the Linux kernel).
> All of this is workable without SHM, but some of it gets really unwieldy.

So reload the information you want available to worker tasks and pass
that information along to them, or provide a mechanism for them to
request it from its preloaded housing. Shared memory isn't the only or
best way to share resources between the tasks involved. Very rarely
would any worker task need more than a few rows of any large preloaded
table.

Alternatively, one could say you dont usually want any preloaded data
because there is simply too much information to preload and reusable
worker tasks can provide their own, more effectively targetted caches.
One might even consider some setup where-by worker threads report
their cache contents to a controller, which distributes tasks to
workers it knows to have the information required already cached.

But in the end, you have to realize this is all at a higher level than
we would really need to even consider for the discussion at hand.