Data unchanged when passing data to Python in multiprocessing shared memory
Barry
barry at barrys-emacs.org
Tue Feb 1 17:20:40 EST 2022
> On 1 Feb 2022, at 20:26, Jen Kris via Python-list <python-list at python.org> wrote:
>
> I am using multiprocesssing.shared_memory to pass data between NASM and Python. The shared memory is created in NASM before Python is called. Python connects to the shm: shm_00 = shared_memory.SharedMemory(name='shm_object_00',create=False).
>
> I have used shared memory at other points in this project to pass text data from Python back to NASM with no problems. But now this time I need to pass a 32-bit integer (specifically 32,894) from NASM to Python.
>
> First I convert the integer to bytes in a C program linked into NASM:
>
> unsigned char bytes[4]
> unsigned long int_to_convert = 32894;
>
> bytes[0] = (int_to_convert >> 24) & 0xFF;
> bytes[1] = (int_to_convert >> 16) & 0xFF;
> bytes[2] = (int_to_convert >> 8) & 0xFF;
> bytes[3] = int_to_convert & 0xFF;
> memcpy(outbuf, bytes, 4);
>
> where outbuf is a pointer to the shared memory. On return from C to NASM, I verify that the first four bytes of the shared memory contain what I want, and they are 0, 0, -128, 126 which is binary 00000000 00000000 10000000 01111110, and that's correct (32,894).
>
> Next I send a message to Python through a FIFO to read the data from shared memory. Python uses the following code to read the first four bytes of the shared memory:
>
> byte_val = shm_00.buf[:4]
> print(shm_00.buf[0])
> print(shm_00.buf[1])
> print(shm_00.buf[2])
> print(shm_00.buf[3])
>
> But the bytes show as 40 39 96 96, which is exactly what the first four bytes of this shared memory contained before I called C to overwrite them with the bytes 0, 0, -128, 126. So Python does not see the updated bytes, and naturally int.from_bytes(byte_val, "little") does not return the result I want.
>
> I know that Python refers to shm00.buf, using the buffer protocol. Is that the reason that Python can't see the data that has been updated by another language?
>
> So my question is, how can I alter the data in shared memory in a non-Python language to pass back to Python?
Maybe you need to use a memory barrier to force the data to be seen by another cpu?
Maybe use shm lock operation to sync both sides?
Googling I see people talking about using stdatomic.h for this.
But I am far from clear what you would need to do.
Barry
>
> Thanks,
>
> Jen
>
> --
> https://mail.python.org/mailman/listinfo/python-list
More information about the Python-list
mailing list