problem with multiprocessing and defaultdict
Wolodja Wentland
wentland at cl.uni-heidelberg.de
Tue Jan 12 07:16:45 EST 2010
On Tue, Jan 12, 2010 at 11:48 +0100, wiso wrote:
> They sent back the object filled with data. The problem is very simple: I
> have a container, the container has a method read(file_name) that read a
> huge file and fill the container with datas. I have more then 1 file to read
> so I want to parallelize this process. The reading method is quite slow
> because it involves regex.
Take a look at multiprocessing.Manager and use one to proxy access to a
*shared* container to your container from all processes.
If your container is a dict it is as easy as:
manager = multiprocessing.Manager()
managed_dict = manager.dict()
...
--
.''`. Wolodja Wentland <wentland at cl.uni-heidelberg.de>
: :' :
`. `'` 4096R/CAF14EFC
`- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20100112/e64dd688/attachment-0001.sig>
More information about the Python-list
mailing list