El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka (storchaka@gmail.com) escribió:

10.10.21 17:19, Facundo Batista пише:

...

I have a long list of nums (several millions), ended up doing the following:

struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)

Why not use array('Q', nums)?

You mean `array` from the `array` module? The only way I see using it is like the following:

...

...

shm = shared_memory.SharedMemory(create=True, size=total_size) a = array.array('Q', nums) shm.buf[l_offset:r_offset] = a.tobytes()

But I don't like it because of the `tobytes` call, which will produce a huge bytearray only to insert it in the shared memory buffer. That's why I liked `pack_into`, because it will write directly into the memory view. Or I'm missing something? Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org.ar/ Twitter: @facundobatista

You can take a memory view of the array directly: memoryview(array.array("Q", range(1000))) If your exact use-case is writing to a SharedMemory, then I don't think there is any simple way to do it without creating some intermediate memory buffer. (other than using struct.pack_into, or packing the values completely manually) On 10/10/2021 16:18, Facundo Batista wrote:

El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka (storchaka@gmail.com) escribió:

...

10.10.21 17:19, Facundo Batista пише:

...

I have a long list of nums (several millions), ended up doing the following:

struct.pack_into(f'{len(nums)}Q', buf, 0, *nums) Why not use array('Q', nums)? You mean `array` from the `array` module? The only way I see using it is like the following:

...

...

...

shm = shared_memory.SharedMemory(create=True, size=total_size) a = array.array('Q', nums) shm.buf[l_offset:r_offset] = a.tobytes() But I don't like it because of the `tobytes` call, which will produce a huge bytearray only to insert it in the shared memory buffer.

That's why I liked `pack_into`, because it will write directly into the memory view.

Or I'm missing something?

Thanks!

This still isn't a completely direct write - you're still creating an array in between which is then copied into shm, whereas struct.pack_into writes directly into the shared memory with no intermediate buffer. And unfortunately you can't do shm.buf[l_offset:r_offset].cast('Q')[:] = nums where nums is a plain Python iterable; it has to already be an array, so it still requires a copy. On 10/10/2021 16:57, Serhiy Storchaka wrote:

10.10.21 18:18, Facundo Batista пише:

...

You mean `array` from the `array` module? The only way I see using it is like the following:

...

...

...

shm = shared_memory.SharedMemory(create=True, size=total_size) a = array.array('Q', nums) shm.buf[l_offset:r_offset] = a.tobytes() But I don't like it because of the `tobytes` call, which will produce a huge bytearray only to insert it in the shared memory buffer.

That's why I liked `pack_into`, because it will write directly into the memory view.

Or I'm missing something? shm.buf[l_offset:r_offset].cast('Q')[:] = a

or

shm.buf[l_offset:r_offset] = memoryview(a).cast('B')

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/A2XKNKG4... Code of Conduct: http://python.org/psf/codeofconduct/

10.10.21 19:05, Patrick Reader пише:

This still isn't a completely direct write - you're still creating an array in between which is then copied into shm, whereas struct.pack_into writes directly into the shared memory with no intermediate buffer.

And unfortunately you can't do

shm.buf[l_offset:r_offset].cast('Q')[:] = nums

where nums is a plain Python iterable; it has to already be an array, so it still requires a copy.

Yes, I tried this too. Maybe we will add a support of arbitrary iterables. There may be problems with this because we don't know the length of the iterable and can't guarantee that all items are convertible without converting them and saving results in a temporary array. If you do not want to spend additional memory, iterate nums and set items one by one. Or iterate and set them by chunks target = shm.buf[l_offset:r_offset].cast('Q') for i in range(0, len(nums), chunksize): j = min(i + chunksize, len(nums)) target[i:j] = array.array('Q', nums[i:j])

On Sun, 10 Oct 2021 11:19:44 -0300 Facundo Batista wrote:

Hello everyone!

I need to pack a long list of numbers into shared memory, so I thought about using `struct.pack_into`.

Its signature is

struct.pack_into(format, buffer, offset, v1, v2, ...)

I have a long list of nums (several millions), ended up doing the following:

struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)

However, passing all nums as `*args` is very inefficient [0]. So I started wondering why we don't have something like:

struct.pack_into(format, buffer, offset, values=values)

which would receive the list of values directly.

Just use `numpy.frombuffer` with your shared memory buffer and write into the Numpy array? https://numpy.org/doc/stable/reference/generated/numpy.frombuffer.html When you're looking to efficiently handle large volumes of primitive values such as integers, chances are Numpy already has the solution for you. Regards Antoine.

On Sun, Oct 10, 2021 at 4:23 PM Facundo Batista wrote:

struct.pack_into(format, buffer, offset, v1, v2, ...)

I've encountered this wart with pack and pack_into too. The current interface makes sense when if v1, v2 are a small number of items from a data record, but it becomes a bit silly when v1, v2, ... are the elements of an array of 10 000 integers, for example. The option to take a list-like argument like you suggest is a good idea, but maybe this should have its own function or is just outside the scope of the built-in struct module. Multiprocessing sort of added support for this via multiprocessing.Array -- see https://stackoverflow.com/questions/9754034/can-i-create-a-shared-multiarray.... I haven't looked at what multiprocessing.Array does under the hood. Summary of the StackOverflow answer for those who don't feel like clicking: mp_arr = mp.Array(c.c_double, size) # then in each new process create a new numpy array using: arr = np.frombuffer(mp_arr.get_obj()) - Simon

El lun, 11 de oct. de 2021 a la(s) 06:50, Simon Cross (hodgestar+pythondev@gmail.com) escribió:

Multiprocessing sort of added support for this via multiprocessing.Array -- see https://stackoverflow.com/questions/9754034/can-i-create-a-shared-multiarray.... I haven't looked at what multiprocessing.Array does under the hood.

Summary of the StackOverflow answer for those who don't feel like clicking:

mp_arr = mp.Array(c.c_double, size) # then in each new process create a new numpy array using: arr = np.frombuffer(mp_arr.get_obj())

Right, this is very close to what I had in mind for the "example with numpy.array" that I want to code next (as I just said in the response to Gregory, this is a series of implementations for pedagogical purposes trying to teach parallel processing). Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org.ar/ Twitter: @facundobatista

Sorry for the spam! A bunch of these were backed up in the moderation queue. I used the UI to set the list to auto-discard future messages from this address, but then clicked "Accept" in the mistaken sense of "yes, accept my request to auto-nuke this clown". But it took "Accept" to mean "sure thing, boss! I'll send this to everyone ASAP!!". Computer software. Everyone, please, get a useful job instead ;-) On Wed, Oct 20, 2021 at 10:43 PM wrote:

that is great to see this post . https://bit.ly/3C551OO _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5BMKWRYJ... Code of Conduct: http://python.org/psf/codeofconduct/