2.6, 3.0, and truly independent intepreters

["Martin v. Löwis"]

> Because then we're back into the GIL not permitting threads efficient
> core use on CPU bound scripts running on other threads (when they
> otherwise could).

Why do you think so? For C code that is carefully written, the GIL
allows *very well* to write CPU bound scripts running on other threads.
(please do get back to Jesse's original remark in case you have lost
the thread :-)

> An example would be python scripts that generate video programatically
> using an initial set of params and use an in-house C module to
> construct frame (which in turn make and modify python C objects that
> wrap to intricate codec related data structures).  Suppose you wanted
> to render 3 of these at the same time, one on each thread (3
> threads).  With the GIL in place, these threads can't anywhere close
> to their potential.  Your response thus far is that the C module
> should release the GIL before it commences its heavy lifting.  Well,
> the problem is that if during its heavy lifting it needs to call back
> into its interpreter.

So it should reacquire the GIL then. Assuming the other threads
all do their heavy lifting, it should immediately get the GIL,
fetch some data, release the GIL, and continue to do heavy lifting.
If it's truly CPU-bound, I hope it doesn't spend most of its time
in Python API, but in true computation.

> It's turns out that this isn't an exotic case
> at all: there's a *ton* of utility gained by making calls back into
> the interpreter. The best example is that since code more easily
> maintained in python than in C, a lot of the module "utility" code is
> likely to be in python.

You should really reconsider writing performance-critical code in
Python. Regardless of the issue under discussion, a lot of performance
can be gained by using "flattened" data structures, less pointer,
less reference counting, less objects, and so on - in the inner loops
of the computation. You didn't reveal what *specific* computation you
perform, so it's difficult to give specific advise.

> Unsurprisingly, this is the situation myself
> and many others are in: where we want to subsequently use the
> interpreter within the C module (so, as I understand it, the proposal
> to have the C module release the GIL unfortunately doesn't work as a
> general solution).

Not if you do the actual computation in Python, no. However, this
subthread started with Jesse's remark that you *can* release the GIL
in C code.

Again, if you do heavy-lifting in Python, you should consider to rewrite
the performance-critical parts in C. You may find that the need for
multiple CPUs goes even away.

> I appreciate your arguments these a PyC concept is a lot of work with
> some careful design work, but let's not kill the discussion just
> because of that.

Any discussion in this newsgroup is futile, except when it either
a) leads to a solution that is already possible, and the OP didn't
envision, or
b) is followed up by code contributions from one of the participants.

If neither is likely to result, killing the discussion is the most
productive thing we can do.

Regards,
Maritn