On Tue, May 5, 2020 at 3:47 PM Guido van Rossum wrote:

This sounds like a significant milestone!

Is there some kind of optimized communication possible yet between subinterpreters? (Otherwise I still worry that it's no better than subprocesses -- and it could be worse because when one subinterpreter experiences a hard crash or runs out of memory, all others have to die with it.)

As far as I understand it, the subinterpreter folks have given up on optimized passing of objects, and are only hoping to do optimized (zero-copy) passing of raw memory buffers. On my laptop, some rough measurements [1] suggest that simply piping bytes between processes goes at ~2.8 gigabytes/second, and that pickle/unpickle is ~10x slower than that. So that would suggest that once subinterpreters are fully optimized, they might provide a maximum ~10% speedup vs multiprocessing, for a program that's doing nothing except passing pickled objects back and forth. Of course, any real program that's spawning parallel workers will presumably be designed so its workers spend most of their time doing work on that data, not just passing it back and forth. That makes a 10% speedup highly unrealistic; in real-world programs it will be much smaller. So IIUC, subinterpreter communication is currently about the same speed as multiprocessing communication, and the plan is to keep it that way. -n [1] Of course there are a lot of assumptions in my quick back-of-the-envelope calculation: pickle speed depends on the details of the objects being pickled, there are other serialization formats, there are other IPC methods that might be faster but are more complicated (shared memory), the stdlib 'multiprocessing' library might not be as good as it could be (the above measurements are for an ideal multiprocessing library, I haven't tested the one we currently have in the stdlib), etc. So maybe there's some situation where subinterpreters look better. But I've been pointing out this issue to Eric et al for years and they haven't disputed it, so I guess they haven't found one yet. -- Nathaniel J. Smith -- https://vorpus.org

Hi Nathaniel, Le mer. 6 mai 2020 à 04:00, Nathaniel Smith a écrit :

As far as I understand it, the subinterpreter folks have given up on optimized passing of objects, and are only hoping to do optimized (zero-copy) passing of raw memory buffers.

I think that you misunderstood the PEP 554. It's a bare minimum API, and the idea is to *extend* it later to have an efficient implementation of "shared objects". -- IMO it should easy to share *data* (object "content") between subinterpreters, but each interpreter should have its own PyObject which exposes the data at the Python level. See the PyObject has a proxy to data. It would badly hurt performance if a PyObject is shared by two interpreters: it would require locking or atomic variables for PyObject members and PyGC_Head members. It seems like right now, the PEP 554 doesn't support sharing data, so it should still be designed and implemented later. Who owns the data? When can we release memory? Which interpreter releases the memory? I read somewhere that data is owned by the interpreter which allocates the memory, and its memory would be released in the same interpreter. How do we track data lifetime? I imagine a reference counter. When it reaches zero, the interpreter which allocates the data can release it "later" (it doesn't have to be done "immediately"). How to lock the whole data or a portion of data to prevent data races? If data doesn't contain any PyObject, it may be safe to allow concurrent writes, but readers should be prepared for inconsistencies depending on the access pattern. If two interpreters access separated parts of the data, we may allow lock-free access. I don't think that we have to reinvent the wheel. threading, multiprocessing and asyncio already designed such APIs. We should to design similar APIs and even simply reuse code. My hope is that "synchronization" (in general, locks in specific) will be more efficient in the same process, than synchronization between multiple processes. -- I would be interested to have a generic implementation of "remote object": a empty proxy object which forward all operations to a different interpreter. It will likely be inefficient, but it may be convenient for a start. If a method returns an object, a new proxy should be created. Simple scalar types like int and short strings may be serialized (copied). Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On Wed, May 6, 2020 at 5:41 AM Victor Stinner wrote:

Hi Nathaniel,

Le mer. 6 mai 2020 à 04:00, Nathaniel Smith a écrit :

...

As far as I understand it, the subinterpreter folks have given up on optimized passing of objects, and are only hoping to do optimized (zero-copy) passing of raw memory buffers.

I think that you misunderstood the PEP 554. It's a bare minimum API, and the idea is to *extend* it later to have an efficient implementation of "shared objects".

No, I get this part :-)

IMO it should easy to share *data* (object "content") between subinterpreters, but each interpreter should have its own PyObject which exposes the data at the Python level. See the PyObject has a proxy to data.

So when you say "shared object" you mean that you're sharing a raw memory buffer, and then you're writing a Python object that stores its data inside that memory buffer instead of inside its __dict__: class MySharedObject: def __init__(self, shared_memview, shared_lock): self._shared_memview = shared_memview self._shared_lock = shared_lock @property def my_attr(self): with self._shared_lock: return struct.unpack_from(MY_ATTR_FORMAT, self._shared_memview, MY_ATTR_OFFSET)[0] @my_attr.setter def my_attr(self, new_value): with self._shared_lock: struct.pack_into(MY_ATTR_FORMAT, self._shared_memview, MY_ATTR_OFFSET, new_value) This is an interesting idea, but I think when most people say "sharing objects between subinterpreters", they mean being able to pass some pre-existing object between subinterpreters cheaply, while this requires defining custom objects with custom locking. So we should probably use different terms for them to avoid confusion :-). This is an interesting idea, and it's true that it's not considered in my post you're responding to. I was focusing on copying objects, not sharing objects on an ongoing basis. You can't implement this kind of "shared object" using a pipe/socket, because those create two independent copies of the data. But... if this is what you want, you can do the exact same thing with subprocesses too. OSes provide inter-process shared memory and inter-process locks. 'MySharedObject' above would work exactly the same. So I think the conclusion still holds: there aren't any plans to make IPC between subinterpreters meaningfully faster than IPC between subprocesses.

I don't think that we have to reinvent the wheel. threading, multiprocessing and asyncio already designed such APIs. We should to design similar APIs and even simply reuse code.

Or, we could simply *use* the code instead of using subinterpreters :-). (Or write new and better code, I feel like there's a lot of room for a modern 'multiprocessing' competitor.) The question I'm trying to figure out is what advantage subinterpreters give us over these proven technologies, and I'm still not seeing it.

My hope is that "synchronization" (in general, locks in specific) will be more efficient in the same process, than synchronization between multiple processes.

Hmm, I would be surprised by that – the locks in modern OSes are highly-optimized, and designed to work across subprocesses. For example, on Linux, futexes work across processes. Have you done any benchmarks? Also btw, note that if you want to use async within your subinterpreters, then that rules out a lot of tools like regular locks, because they can't be integrated into an event loop. If your subinterpreters are using async, then you pretty much *have* to use full-fledged sockets or equivalent for synchronization.

I would be interested to have a generic implementation of "remote object": a empty proxy object which forward all operations to a different interpreter. It will likely be inefficient, but it may be convenient for a start. If a method returns an object, a new proxy should be created. Simple scalar types like int and short strings may be serialized (copied).

How would this be different than https://docs.python.org/3/library/multiprocessing.html#proxy-objects ? How would you handle input arguments -- would those get proxied as well? Also, does this mean the other subinterpreter has to be running an event loop to process these incoming requests? Or is the idea that the other subinterpreter would process these inside a traditional Python thread, so users are exposed to all the classic shared-everything locking issues? -n -- Nathaniel J. Smith -- https://vorpus.org

On Wed, May 6, 2020 at 10:03 AM Antoine Pitrou wrote:

On Tue, 5 May 2020 18:59:34 -0700 Nathaniel Smith wrote:

...

On Tue, May 5, 2020 at 3:47 PM Guido van Rossum wrote:

...

This sounds like a significant milestone!

Is there some kind of optimized communication possible yet between subinterpreters? (Otherwise I still worry that it's no better than subprocesses -- and it could be worse because when one subinterpreter experiences a hard crash or runs out of memory, all others have to die with it.)

As far as I understand it, the subinterpreter folks have given up on optimized passing of objects, and are only hoping to do optimized (zero-copy) passing of raw memory buffers.

Which would be useful already, especially with pickle out-of-band buffers.

Sure, zero cost is always better than some cost, I'm not denying that :-). What I'm trying to understand is whether the difference is meaningful enough to justify subinterpreters' increased complexity, fragility, and ecosystem breakage. If your data is in large raw memory buffers to start with (like numpy arrays or arrow dataframes), then yeah, serialization costs are smaller proportion of IPC costs. And out-of-band buffers are an elegant way of letting pickle users take advantage of that speedup while still using the familiar pickle API. Thanks for writing that PEP :-). But when you're in the regime where you're working with large raw memory buffers, then that's also the regime where inter-process shared-memory becomes really efficient. Hence projects like Ray/Plasma [1], which exist today, and even work for sharing data across languages and across multi-machine clusters. And the pickle out-of-band buffer API is general enough to work with shared memory too. And even if you can't quite manage zero-copy, and have to settle for one-copy... optimized raw data copying is just *really fast*, similar to memory access speeds. And CPU-bound, big-data-crunching apps are by definition going to access that memory and do stuff with it that's much more expensive than a single memcpy. So I still have trouble figuring out how skipping a single memcpy will make subinterpreters significantly faster that subprocesses in any real-world scenario. -n [1] https://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/ https://github.com/ray-project/ray -- Nathaniel J. Smith -- https://vorpus.org

On Thu, May 7, 2020 at 2:50 AM Emily Bowman wrote:

While large object copies are fairly fast -- I wouldn't say trivial, a gigabyte copy will introduce noticeable lag when processing enough of them -- the flip side of having large objects is that you want to avoid having so many copies that you run into memory pressure and the dreaded swapping. A multiprocessing engine that's fully parallel, every fork takes chunks of data and does everything needed to them won't gain much from zero-copy as long as memory limits aren't hit. But a pipeline of processing would involve many copies, especially if you have a central dispatch thread that passes things from stage to stage. This is a big deal where stages may take longer or slower at any time, especially in low-latency applications, like video conferencing, where dispatch needs the flexibility to skip steps or add extra workers to shove a frame out the door, and using signals to interact with separate processes to tell them to do so is more latency and overhead.

Not that I'm recommending someone go out and make a pure Python videoconferencing unit right now, but it's a use case I'm familiar with. (Since I use Python to test new ideas before converting them into C++.)

Thanks for the insight, Emily (and everyone else). It's really helpful to get many different expert perspectives on the matter. I am definitely not an expert on big-data/high-performance use cases so, personally, I rely on folks like Nathaniel, Travis Oliphant, and yourself. The more, the better. :) Again, thanks! -eric

Guido van Rossum wrote:

Is there some kind of optimized communication possible yet between subinterpreters? (Otherwise I still worry that it's no better than subprocesses -- and it could be worse because when one subinterpreter experiences a hard crash or runs out of memory, all others have to die with it.)

The use case that I have in mind with subinterpreters is Windows. With its lack of fork() and the way it spawns a fresh interpreter process it always feels a bit weird to use multiprocessing on Windows. Would it be faster and/or cleaner to start a new in-process subinterpreter instead?

On Thu, 7 May 2020 at 01:34, Cameron Simpson wrote:

Maybe I'm missing something, but the example that comes to my mind is embedding a Python interpreter in an existing nonPython programme.

My pet one-day-in-the-future example is mutt, whose macro language is... crude. And mutt is single threaded.

However, it is easy to envisage a monolithic multithreaded programme which has use for Python subinterpreters to work on the larger programme's in-memory data structures.

I haven't a real world example to hand, but that is the architectural situation where I'd consider multiprocessing to be inappropriate or infeasible because the target data are all in the one memory space.

Vim would be a very good example of this. Vim has Python interpreter support, but multiprocessing would not be viable as you say. And from my recollection, experiments with threading didn't end well when I tried them :-) Paul

Le mer. 6 mai 2020 à 22:10, Serhiy Storchaka a écrit :

I am wondering how much 3.9 will be slower than 3.8 in single-thread single-interpreter mode after getting rid of all process-wide singletons and caches (Py_None, Py_True, Py_NonImplemented. small integers, strings, tuples, _Py_IDENTIFIER, _PyArg_Parser, etc). Not mentioning breaking binary compatibility.

There is no plan to remove caches like small integers, _Py_IDENTIFIER or _PyArg_Parser. The plan is to make these caches "per-interpreter". I already modified small integers to make them per-interpreter. Victor -- Night gathers, and now my watch begins. It shall not end until my death.