a huge shared read-only data in parallel accesses -- How? multithreading? multiprocessing?

[Klauss]

On Dec 9, 11:58 am, Valery <khame... at gmail.com> wrote:
> Hi all,
>
> Q: how to organize parallel accesses to a huge common read-only Python
> data structure?
>
> Details:
>
> I have a huge data structure that takes >50% of RAM.
> My goal is to have many computational threads (or processes) that can
> have an efficient read-access to the huge and complex data structure.
>
> <snip>
>
> 1. multi-processing
>  => a. child-processes get their own *copies* of huge data structure
> -- bad and not possible at all in my case;

How's the layout of your data, in terms # of objects vs. bytes used?
Just to have an idea of the overhead involved in refcount
externalization (you know, what I mentioned here:
http://groups.google.com/group/unladen-swallow/browse_thread/thread/9d2af1ac3628dc24
)