On 2021-10-10 19:20, Gregory P. Smith wrote:
On Sun, Oct 10, 2021 at 7:25 AM Facundo Batista <facundobatista@gmail.com <mailto:facundobatista@gmail.com>> wrote:
Hello everyone!
I need to pack a long list of numbers into shared memory, so I thought about using `struct.pack_into`.
Its signature is
struct.pack_into(format, buffer, offset, v1, v2, ...)
I have a long list of nums (several millions), ended up doing the following:
struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
However, passing all nums as `*args` is very inefficient [0]. So I started wondering why we don't have something like:
struct.pack_into(format, buffer, offset, values=values)
which would receive the list of values directly.
Is that because my particular case is very uncommon? Or maybe we *do* want this but we don't have it yet? Or do we already have a better way of doing this?
Thanks!
[0] https://linkode.org/#95ZZtVCIVtBbx72dURK7a4 <https://linkode.org/#95ZZtVCIVtBbx72dURK7a4>
My first reaction on seeing things like this is "Why not use a numpy.array?"
Does what you have really need to be a long list? If so, that's already a huge amount of Python object storage as it is. Is it possible for your application to have kept that in a numpy array for the entirety of the data lifetime? https://numpy.org/doc/stable/reference/routines.array-creation.html <https://numpy.org/doc/stable/reference/routines.array-creation.html>
I'm not saying the stdlib shouldn't have a better way to do this by not abusing *args as an API, just that other libraries solve the larger problem of data-memory-inefficiency in their own way already.
/(neat tricks from others regarding stdlib array, shm, & memoryview even if... not ideal)/
Maybe what's needed is to add, say, '*' to the format string to indicate that multiple values should come from an iterable, e.g.: struct.pack_into(f'{len(nums)}*Q', buf, 0, nums) in this case len(nums) from the nums argument.