Due to https://www.python.org/dev/peps/pep-0554/ multi-interpreters implementation going really slow, I had the audicity to try an alternative route towards the same objective of implementing multicore support of python: instead of sharing the memory by running multiple threads, I employed an interprocess shared memory with multiple processes. I know there are multiprocessing.sharedctypes and multiprocessing.shared_memory, but I went much deeper into it by solving two problems they failed to solve: sharing of complex dynamic objects and synchronization of data access. I have a working prototype to show: https://github.com/byko3y/python-shared-objects It's a kind of software transactional memory within shared memory. It has a basic support for fundamental python types (bool, int, str, bytes, tuple, list, dict, object), for both atomic and non-atomic transactions via fine-grained RW-locks, has a built-in protection against deadlock and starvation. Most of my code can also be used for cross-interpreter communication after PEP 554 is successfully implemented, since cross-interpreter communication is still an open question.
On 25.10.2021 21:40, byko3y@gmail.com wrote:
Due to https://www.python.org/dev/peps/pep-0554/ multi-interpreters implementation going really slow, I had the audicity to try an alternative route towards the same objective of implementing multicore support of python: instead of sharing the memory by running multiple threads, I employed an interprocess shared memory with multiple processes.
I know there are multiprocessing.sharedctypes and multiprocessing.shared_memory, but I went much deeper into it by solving two problems they failed to solve: sharing of complex dynamic objects and synchronization of data access.
I have a working prototype to show: https://github.com/byko3y/python-shared-objects It's a kind of software transactional memory within shared memory. It has a basic support for fundamental python types (bool, int, str, bytes, tuple, list, dict, object), for both atomic and non-atomic transactions via fine-grained RW-locks, has a built-in protection against deadlock and starvation.
Most of my code can also be used for cross-interpreter communication after PEP 554 is successfully implemented, since cross-interpreter communication is still an open question.
This looks interesting. The 32-bit limitation is a bit of a bummer, but I suppose that can be lifted, right ? Some additional pointers for inspiration: - Here's an old project trying to do more or less the same: http://poshmodule.sourceforge.net/ - Another newer one, which is specific to numpy arrays: https://pypi.org/project/SharedArray/ - For more general purpose types, there's Apache Arrow's Plasma store: https://arrow.apache.org/docs/python/plasma.html -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 27 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
This looks interesting. The 32-bit limitation is a bit of a bummer, but I suppose that can be lifted, right ? Right. Already lifted. I wasn't actually realizing that installing additional python version could cause people so much trouble.
Here's an old project trying to do more or less the same: http://poshmodule.sourceforge.net/ Yes, in my previous article on habr I specifically mention it. I actually borrowed lots of of techniques from it (memory handles, shared heap, object copy/proxy), I even called my project similarly. However, I went far beyond the POSH implementation by solving the key problem - how to use those objects correctly. POSH provides some very basic optional locking. Countrary, I implemented real atomic transactions to manipulate shared objects with unprecedented safety and simplicity. Despite the similarities, you can see my own code is 4 time as large as POSH, so there much more differences than similarities, that's why I called it differently.
Another newer one, which is specific to numpy arrays: https://pypi.org/project/SharedArray/ That is the strangest and probably useless thing I met so far. It provides absolutely no synchronization and writes the data to a file for some reason (to just list the shared objects he could keep the files empty). It seems like the author had no idea what he was doing.
For more general purpose types, there's Apache Arrow's Plasma store: https://arrow.apache.org/docs/python/plasma.html Yes, that's the most popular thing for a general purpose sharing of memory. However, it's not python-friendly and implies no mutation on shared memory. Although it's good for communicating between python and external programs.
participants (2)
-
byko3y@gmail.com
-
Marc-Andre Lemburg