Due to https://www.python.org/dev/peps/pep-0554/ multi-interpreters implementation going really slow, I had the audicity to try an alternative route towards the same objective of implementing multicore support of python: instead of sharing the memory by running multiple threads, I employed an interprocess shared memory with multiple processes. I know there are multiprocessing.sharedctypes and multiprocessing.shared_memory, but I went much deeper into it by solving two problems they failed to solve: sharing of complex dynamic objects and synchronization of data access. I have a working prototype to show: https://github.com/byko3y/python-shared-objects It's a kind of software transactional memory within shared memory. It has a basic support for fundamental python types (bool, int, str, bytes, tuple, list, dict, object), for both atomic and non-atomic transactions via fine-grained RW-locks, has a built-in protection against deadlock and starvation. Most of my code can also be used for cross-interpreter communication after PEP 554 is successfully implemented, since cross-interpreter communication is still an open question.
On 25.10.2021 21:40, byko3y@gmail.com wrote:
This looks interesting. The 32-bit limitation is a bit of a bummer, but I suppose that can be lifted, right ? Some additional pointers for inspiration: - Here's an old project trying to do more or less the same: http://poshmodule.sourceforge.net/ - Another newer one, which is specific to numpy arrays: https://pypi.org/project/SharedArray/ - For more general purpose types, there's Apache Arrow's Plasma store: https://arrow.apache.org/docs/python/plasma.html -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 27 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 25.10.2021 21:40, byko3y@gmail.com wrote:
This looks interesting. The 32-bit limitation is a bit of a bummer, but I suppose that can be lifted, right ? Some additional pointers for inspiration: - Here's an old project trying to do more or less the same: http://poshmodule.sourceforge.net/ - Another newer one, which is specific to numpy arrays: https://pypi.org/project/SharedArray/ - For more general purpose types, there's Apache Arrow's Plasma store: https://arrow.apache.org/docs/python/plasma.html -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 27 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
participants (2)
-
byko3y＠gmail.com -
Marc-Andre Lemburg