Persistent variable in subprocess using multiprocessing?

[Diez B. Roggisch]

mheavner schrieb:
> I'm using multiprocessing to spawn several subprocesses, each of which
> uses a very large data structure (making it impractical to pass it via
> pipes / pickling). I need to allocate this structure once when the
> process is created and have it remain in memory for the duration of
> the process. The way the multiprocessing module is set up, only the
> 'run' method runs within the subprocess - so creating a wrapper class
> with a constructor that allocates the structure in __init__ will not
> work, as far as I know, as this will still be within the parent
> process.
> 
> If I were working in C/C++, I would declare the variable "static"
> within the function body - is there any way with the multiprocessing
> module to have persistent data members within subprocesses?

Works for me, at least under OSX (and I presume *nixes in general work.) 
No idea about Windows.


The thing to keep in mind is that forking is used, and that 
interpreter-state up to the moment of the fork is the same for all 
subprocesses.



from multiprocessing import Process

class MyProcess(Process):


     def __init__(self, huge_shared_state):
         self.huge_shared_state = huge_shared_state
         super(MyProcess, self).__init__()


     def run(self):
         print self.name, len(self.huge_shared_state)




shared_state = range(1000000)

processes = []
for i in xrange(10):
     p = MyProcess(shared_state)
     p.start()
     processes.append(p)


for p in processes:
     p.join()



Diez