a huge shared read-only data in parallel accesses -- How? multithreading? multiprocessing?
garyrob at mac.com
Thu Dec 31 22:36:37 CET 2009
One thing I'm not clear on regarding Klauss' patch. He says it's
applicable where the data is primarily non-numeric. In trying to
understand why that would be the case, I'm thinking that the increased
per-object memory overhead for reference-counting would outweigh the
space gains from the shared memory.
Klauss's test code stores a large number of dictionaries which each
contain just 3 items. The stored items are strings, but short ones...
it looks like they take up less space than double floats(?).
So my understanding is that the point is that the overhead for the
dictionaries is big enough that the patch is very helpful even though
the stored items are small. And that the patch would be less and less
effective as the number of items stored in each dictionary became
greater and greater, until eventually the patch might do more use more
space for reference counting than it saved by shared memory.
Is this understanding correct? (I'm hoping not, because for some
applications, I'd like to be able to use it for large dictionaries
containing lots of numbers.)
More information about the Python-list