out of memory with processing module

[alessiogiovanni.baroni at gmail.com]

On 20 Apr, 17:03, Brian <knair... at yahoo.com> wrote:
> I'm using the third-party "processing" module in Python 2.5, which may
> have become the "multiprocessing" module in Python 2.6, to speed up
> the execution of a computation that takes over a week to run. The
> relevant code may not be relevant, but it is:
>
>             q1, q2 = processing.Queue(), processing.Queue()
>             p1 = processing.Process(target=_findMaxMatch, args=
> (reciprocal, user, clusters[1:(numClusters - 1)/2], questions,
> copy.copy(maxMatch), q1))
>             p2 = processing.Process(target=_findMaxMatch, args=
> (reciprocal, user, clusters[(numClusters - 1)/2:], questions, copy.copy
> (maxMatch), q2))
>             p1.start()
>             p2.start()
>             maxMatch1 = q1.get()[0]
>             maxMatch2 = q2.get()[0]
>             p1.join()
>             p2.join()
>             if maxMatch1[1] > maxMatch2[1]:
>                 maxMatch = maxMatch1
>             else:
>                 maxMatch = maxMatch2
>
> This code just splits up the calculation of the cluster that best
> matches 'user' into two for loops, each in its own process, rather
> than one. (It's not important what the cluster is.)
>
> The error I get is:
>
> [21661.903889] Out of memory: kill process 14888 (python) score 610654
> or a child
> [21661.903930] Killed process 14888 (python)
> Traceback (most recent call last):
> ...etc. etc. ...
>
> Running this process from tty1, rather than GNOME, on my Ubuntu Hardy
> system allowed the execution to get a little further than under GNOME.
>
> The error was surprising because with just 1 GB of memory and a single
> for loop I didn't run into the error, but with 5 GB and two processes,
> I do. I believe that in the 1 GB case there was just a lot of
> painfully slow swapping going on that allowed it to continue.
> 'processing' appears to throw its hands up immediately, instead.
>
> Why does the program fail with 'processing' but not without it? Do you
> have any ideas for resolving the problem? Thanks for your help.

If your program crashes with more of one process, maybe you handle the
Queue objects
not properly? If you can, post the code of _findMaxMatch.