Is this a bug? Python intermittently stops dead for seconds

[charlie strauss]

Steve and other good folks who replied:

I want to clarify that, on my computer, the first instance of the gap occurs way before the memory if filled. (at about 20% of physical ram).  Additionally the process monitor shows no page faults.

  Yes if you let the as-written demo program  run to completetion (all 20,000 iterations) then on many computers it would not be surprising that your computer eventually goes into forced page swapping at some point.  That would be expected and is not the issue than the one I am concerned with.

in my case starts glicthing at around iteration 1000.

1000(bars) x 100(foos)x(10 integers in array)

is nominally 
100,000 class objects and
1,000,000 array elements.

(note that the array if filled as [1]*10, so there is actually only one "integer", but 10 array elements refering to it, per foo class.)


However steve may have put his finger on the reason why the duration grows with time.  Here is my current hypothesis.  The design of the program does not have and points where significant amounts of memory are released: all objects have held references till the end.  But prehaps there are some implicitly created objects of the same size created along the way???  For example when I write

me.memory = [1]*nfoo

perhaps internally, python is allocating an array of size foo and then __copying__ it into me.memory???  Since there is no reference to the intermediate it would then marked for future garbage collection.   

If that were true then the memory would have interleaved entities of things to GC and things with references held in me.memory.

Then to remove these would require GC to scan the entire set of existing objects, which is growing.

Turning off GC would prevent this.


In any case I don't think what I'm doing is very unusual.  The original program that trigger my investigation of the bug was doing this:

foo was an election ballot holding 10 outcomes, and bar was a set of 100 ballots from 100 voting machines, and the total array was holding the ballot sets from a few thousand voting machines.  

Almost any inventory program is likely to have such a simmilar set of nested array, so it hardly seems unusual.