Hi Tobias,

On Wed, Jan 18, 2017 at 5:27 PM, Tobias Oberstein <tobias.oberstein@gmail.com> wrote:

Are you aware of

https://github.com/alex/zero_buffer

?

This emulates Python strings using zero-copy read-only buffer views.

yes, I saw it in the past and I considered to use it for capnpy. IIRC, I measured that at the end of the days, the overhead of using it was larger than simply doing string slicing, especially for short strings.

It might be useful for very large strings, however.

Depending of what you need to do, for zero copy you could also consider returning a memoryview slice of the original underlying buffer.

Holy grail for me (use case being IPC) would be:

Python process 1 mmap's a file shared with Python process 2.

Python process 1 puts a string into mmap'ed file, pointer to that is "somehow transferred" to process 2 (eg pushing the index into the mmap'ed file over Unix domain socket .. a single uint64), and Python code in process 2 can do stuff with this string _without_ copying - probably via zero_buffer.

Have you actually measured that copying the data between processes is the bottleneck? Using shared memory is something I tried also for a client of mine but at the end we switched back to pass messages using network because the extra complexity was not worth the gain. But again, I suppose it depends on the size of the message.

ciao,

Anto