a huge shared read-only data in parallel accesses -- How? multithreading? multiprocessing?
Aaron Watters
aaron.watters at gmail.com
Wed Dec 9 16:06:13 EST 2009
On Dec 9, 9:58 am, Valery <khame... at gmail.com> wrote:
> Hi all,
>
> Q: how to organize parallel accesses to a huge common read-only Python
> data structure?
Use a BTree on disk in a file. A good file system will keep most of
the
pages you need in RAM whenever the data is "warm". This works
for Python or any other programming language. Generally you can
always get to any piece of data in about 4 seeks at most anyway,
so if your disk is fast your app will be fast too. The file can
be accessed concurrently without problems by any number of processes
or threads.
-- Aaron Watters
http://listtree.appspot.com
http://whiffdoc.appspot.com
===
less is more
More information about the Python-list
mailing list