2.6, 3.0, and truly independent intepreters

[Andy O'Meara]

On Oct 24, 9:52 pm, "Martin v. Löwis" <mar... at v.loewis.de> wrote:
> >> A c-level module, on the other hand, can sidestep/release
> >> the GIL at will, and go on it's merry way and process away.
>
> > ...Unless part of the C module execution involves the need do CPU-
> > bound work on another thread through a different python interpreter,
> > right?
>
> Wrong.


Let's take a step back and remind ourselves of the big picture.  The
goal is to have independent interpreters running in pthreads that the
app starts and controls.  Each interpreter never at any point is doing
any thread-related stuff in any way.  For example, each script job
just does meat an potatoes CPU work, using callbacks that, say,
programatically use OS APIs to edit and transform frame data.

So I think the disconnect here is that maybe you're envisioning
threads being created *in* python.  To be clear, we're talking out
making threads at the app level and making it a given for the app to
take its safety in its own hands.



>
> > As far as I can tell, it seems
> > CPython's current state can't CPU bound parallelization in the same
> > address space.
>
> That's not true.
>

Well, when you're talking about large, intricate data structures
(which include opaque OS object refs that use process-associated
allocators), even a shared memory region between the child process and
the parent can't do the job.  Otherwise, please describe in detail how
I'd get an opaque OS object (e.g. an OS ref that refers to memory-
resident video) from the child process back to the parent process.

Again, the big picture that I'm trying to plant here is that there
really is a serious need for truly independent interpreters/contexts
in a shared address space.  Consider stuff like libpng, zlib, ipgjpg,
or whatever, the use pattern is always the same: make a context
object, do your work in the context, and take it down.  For most
industry-caliber packages, the expectation and convention (unless
documented otherwise) is that the app can make as many contexts as its
wants in whatever threads it wants because the convention is that the
app is must (a) never use one context's objects in another context,
and (b) never use a context at the same time from more than one
thread.  That's all I'm really trying to look at here.


Andy