multiprocessing and dictionaries

[Bjorn Meyer]

On Monday 13 July 2009 01:56:08 Piet van Oostrum wrote:

> >>>>> Bjorn Meyer <bjorn.m.meyer at gmail.com> (BM) wrote:
> >
> >BM> I am trying to convert a piece of code that I am using the thread
> > module with BM> to the multiprocessing module.
> >
> >BM> The way that I have it set up is a chunk of code reads a text file and
> > assigns BM> a dictionary key multiple values from the text file. I am
> > using locks to write BM> the values to the dictionary.
> >BM> The way that the values are written is as follows:
> >BM> 	mydict.setdefault(key, []).append(value)
> >
> >BM> The problem that I have run into is that using multiprocessing, the
> > key gets BM> set, but the values don't get appended.
> >BM> I've even tried the Manager().dict() option, but it doesn't seem to
> > work.
> >
> >BM> Is this not supported at this time or am I missing something?
>
> I think you should give more information. Try to make a *minimal* program
> that shows the problem and include it in your posting or supply a
> download link.
> --
> Piet van Oostrum <piet at cs.uu.nl>
> URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
> Private email: piet at vanoostrum.org

Here is what I have been using as a test.
This pretty much mimics what I am trying to do.
I put both threading and multiprocessing in the example which shows the output 
that I am looking for.

#!/usr/bin/env python

import threading
from multiprocessing import Manager, Process

name = ('test1','test2','test3')
data1 = ('dat1','dat2','dat3')
data2 = ('datA','datB','datC')

def thread_test(name,data1,data2, d):
  for nam in name:
    for num in range(0,3):
      d.setdefault(nam, []).append(data1[num])
      d.setdefault(nam, []).append(data2[num])
  print 'Thread test dict:',d

def multiprocess_test(name,data1,data2, mydict):
  for nam in name:
    for num in range(0,3):
      mydict.setdefault(nam, []).append(data1[num])
      mydict.setdefault(nam, []).append(data2[num])
  print 'Multiprocess test dic:',mydict

if __name__ == '__main__':
  mgr = Manager()
  md = mgr.dict()
  d = {}

  m = Process(target=multiprocess_test, args=(name,data1,data2,md))
  m.start()
  t = threading.Thread(target=thread_test, args=(name,data1,data2,d))
  t.start()
  
  m.join()
  t.join()
  
  print 'Thread test:',d
  print 'Multiprocess test:',md


Thanks
Bjorn