How is this a different problem than the cache coherency problem?

https://en.wikipedia.org/wiki/Cache_coherence

Perhaps that's an unhelpful abstraction?

This hasn't gone anywhere:

https://en.wikipedia.org/wiki/Distributed_shared_memory#Directory_memory_coherence

Here's a great comparison chart for message passing vs distributed shared memory: https://en.wikipedia.org/wiki/Distributed_shared_memory#Message_Passing_vs._DSM

Could there be a multiprocessing.MemoryPool that tracks allocations, refcounts, and also locks?

A combined approach might have an IPC channel/stream/source/sinks for messages that instruct workers to invalidate/re-fetch object_id/references, but consistency and ordering result in the same issues encountered with the cache coherence problem.

Then, what is the best way to enqueue changes to shared global state (in shared memory on one node, in this instance)?

(... "Ask HN: Learning about distributed systems?" https://news.ycombinator.com/item?id=23931730 )

A solution for this could help accelerate dask and dask.distributed (which already address many parallel issues in multiprocess and distributed systems in pure python)

"Accelerate intra-node IPC with shared memory" https://github.com/dask/dask/issues/6267

On Sun, Aug 2, 2020, 3:11 PM Eric V. Smith <eric@trueblade.com> wrote:

On 8/2/2020 12:20 PM, Eric V. Smith wrote:

On Sat, 1 Aug 2020 at 22:42, Eric V. Smith <eric@trueblade.com> wrote:

While they're immutable at the Python level, strings (and all other
objects) are mutated at the C level, due to reference count updates. You
need to consider this if you're sharing objects without locking or other
synchronization.

This is interesting. What if you want to have a language that uses only immutable objects and garbage collection? Could smart pointers address this problem?

Yes, garbage collection changes the picture entirely, with or without immutable objects. But the original topic was cross-processs shared memory, and I don't know of any cross-process aware garbage collectors that support shared memory. Although such a thing could easily exist without my knowledge.

Note that I'm talking about putting Python objects into this shared memory. If that's not what people are contemplating, then my observations don't apply.

Eric

_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/KT7URLYEMMHTSB7NIBLF3667OYOCF67I/
Code of Conduct: http://python.org/psf/codeofconduct/