Hi Eric, Antoine, all
Antoine said that what I proposed earlier was very similar to what Eric
is trying to do, but from the direction the discussion has taken so far
that appears not to be the case.
I will therefore try to clarify my proposal.
Basically, what I am suggesting is a direct translation of Javascript's
to Python.
The Web Worker API is generally considered a "share-nothing" approach, although
as we will see some state can be shared.
The basic principle is that any object lives in a single Worker (Worker = subinterpreter).
If a message is send from Worker A to Worker B, the message is not shared,
rather the so-called "structured clone" algorithm is used to create recursively a NEW message
object in Worker B. This is roughly equivalent to pickling in A and then unpickling in B,
Of course, this may become a bottleneck if large amounts of data need to be communicated.
Therefore, there is a special object type designed to provide a view upon a piece
of shared memory: SharedArrayBuffer. Notable, this only provides a view upon
raw "C"-style data (ints or floats or whatever), not on Javascript objects.
To translate this to the Python situation: each Python object is owned by a single
subinterpreter, and may only be manipulated by a thread which holds the GIL
of that particular subinterpreter. Message sending between subinterpreters will
require the message objects to be "structured cloned".
Certain C extension types may override what structured cloning means for them.
In particular, some C extension types may have a two-layer structure where
the Py_Object contains a refcounted pointer to the actual data.
The structured cloning on such an object may create a second Py_Object which
references the same underlying object.
This secondary refcount will need to be properly atomic, since it may be manipulated
from multiple subinterpreters.
In this way, interpreter-shared data structures can be implemented.
However, all the "normal" Python objects are not shared and can continue
to use the current, non-atomic refcounting implementation.
Hope this clarifies my proposal.
Stephan