[Python-Dev] Yet another "A better story for multi-core Python" comment

[cwillu]

On 8 September 2015 at 11:07, Gary Robinson <garyrob at me.com> wrote:
>> I guess a third possible solution, although it would probably have
>> meant developing something for yourself which would have hit the same
>> "programmer time is critical" issue that you noted originally, would
>> be to create a module that managed the data structure in shared
>> memory, and then use that to access the data from the multiple
>> processes.
>
> I think you mean, write a non-python data structure in shared memory, such as writing it in C? If so, you’re right, I want to avoid the time overhead for writing something like that. Although I have used C data in shared-memory in the past when the data structure was simple enough. It’s not a foreign concept to me — it just would have been a real nuisance in this case.
>
> An in-memory SQLLite database would have been too slow, at least if I used any kind of ORM. Without an ORM it still would have slowed things down while making for code that’s harder to read  and write. While I have used in-memory SQLite code at times, I’m not sure how much slowdown it would have engendered in this case.
>
>> Your suggestion (2), of having a non-refcounted data structure is
>> essentially this, doable as an extension module. The core data
>> structures all use refcounting, and that's unlikely to change, but
>> there's nothing to say that an extension module couldn't implement
>> fast data structures with objects allocated from a pool of
>> preallocated memory which is only freed as a complete block.
>
> Again, I think you’re talking about non-Python data structures, for instance C structures, which could be written to be “fast”? Again, I want to avoid writing that kind of code. Sure, for a production project where I had more programmer time, that would be a solution, but that wasn’t my situation. And, ideally, even if I had more time, I would greatly prefer not to have to spend it on that kind of code. I like Python because it saves me time and eliminates potential bugs that are associated with language like C but not with Python (primarily memory management related). To the extent that I have to write and debug external modules in C or C++, it doesn’t.

I've used cffi to good effect to gain some of the benefits of the
"share a lump of memory" model.