Problem with copy.deepcopy and multiprocessing.Queue
James DeVincentis
admin at hexhost.net
Thu Oct 15 19:36:14 EDT 2015
I think I tracked this down and resolved it.
It appears taking an object from a multiprocess.Queue and placing it into a queue.Queue is a no-no even if the queue.Queue isn’t shared across processes.
I have a series of workers (multiprocessing) that share a multiprocess.Queue across all processes. Each worker has it’s own local queue (Queue.queue) that is shared within only it’s own threads. It appears that when taking an object from a multiprocessing queue and moving it into a Queue.queue it breaks things. I don’t know what or how, but it does. As soon as I switched the local queue to a multiprocess.Queue it fixed the issue.
Kind of odd to me. Not sure if anyone wants to look into it.
> On Oct 15, 2015, at 5:42 PM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
>
> On Thu, Oct 15, 2015 at 4:02 PM, James DeVincentis <admin at hexhost.net> wrote:
> >
> > Anyone have any ideas? I feel like this could be a bug with the garbage collector across multiprocessing.
>
> I'll second MRAB's response from yesterday: could it just be reusing space that it has recently freed?
>
> As a debugging measure, what happens if you try to keep the objects in memory, e.g. by adding them to some long-lived list? Do you start seeing different addresses?
More information about the Python-list
mailing list