Known Limitations The shared memory scheduler has some notable limitations:
- It works on a single machine - The threaded scheduler is limited by the GIL on Python code, so if your operations are pure python functions, you should not expect a multi-core speedup - The multiprocessing scheduler must serialize functions between workers, which can fail - The multiprocessing scheduler must serialize data between workers and
https://docs.dask.org/en/latest/shared.html#known-limitations : the central process, which can be expensive
- The multiprocessing scheduler cannot transfer data directly between worker processes; all data routes through the master process.
... https://distributed.dask.org/en/latest/memory.html#difference-with-dask-comp... (... https://github.com/dask/dask-labextension ) On Sat, Aug 1, 2020 at 7:34 PM Wes Turner <wes.turner@gmail.com> wrote:
PyArrow Plasma object ids, "sealing" makes an object immutable, pyristent
https://arrow.apache.org/docs/python/plasma.html#object-ids https://arrow.apache.org/docs/python/plasma.html#creating-an-object-buffer
Objects are created in Plasma in two stages. First, they are created, which allocates a buffer for the object. At this point, the client can write to the buffer and construct the object within the allocated buffer.
To create an object for Plasma, you need to create an object ID, as well as give the object’s maximum size in bytes. ```python # Create an object buffer. object_id = plasma.ObjectID(20 * b"a") object_size = 1000 buffer = memoryview(client.create(object_id, object_size))
# Write to the buffer. for i in range(1000): buffer[i] = i % 128 ```
When the client is done, the client seals the buffer, making the object immutable, and making it available to other Plasma clients.
```python # Seal the object. This makes the object immutable and available to other clients. client.seal(object_id) ```
https://pypi.org/project/pyrsistent/ also supports immutable structures
On Sat, Aug 1, 2020 at 4:44 PM Eric V. Smith <eric@trueblade.com> wrote:
On 8/1/2020 1:25 PM, Marco Sulla wrote:
You don't need locks with immutable objects. Since they're immutable, any operation that usually will mutate the object, generate another immutable instead. The most common example is str: the sum of two strings in Python (and in many other languages) produces a new string.
While they're immutable at the Python level, strings (and all other objects) are mutated at the C level, due to reference count updates. You need to consider this if you're sharing objects without locking or other synchronization.
Eric
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FEJEHF... Code of Conduct: http://python.org/psf/codeofconduct/