[Python-bugs-list] [ python-Bugs-616019 ] list(xrange(sys.maxint / 4)) -> swapping

Mon, 14 Oct 2002 14:12:15 -0700

Bugs item #616019, was opened at 2002-09-29 02:00
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=616019&group_id=5470

Category: Python Interpreter Core
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Brian Lenihan (brianl)
Assigned to: Nobody/Anonymous (nobody)
Summary: list(xrange(sys.maxint / 4)) -> swapping

Initial Comment:
Refer to #556025 list(xrange(1e9)) --> seg fault for
the history.  The test in test/test_b1.py causes OS X
10.2.1 to start swapping instantly and never stopped
until I killed the process after 5 minutes.
I wasn't brave enough to let it run for any longer.

list(xrange(1e09)) or even list(xrange(sys.maxint)) works:

Traceback (most recent call last):
  File "test_b1.py", line 553, in ?
    list(xrange(sys.maxint))
MemoryError

PhysMem:  55.7M wired,  300M active,  150M inactive, 
506M used, 6.19M free
VM: 4.19G + 76.9M   36909(35) pageins, 1505603(200)
pageouts 

  PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT
 RSHRD  RSIZE  VSIZE
 1157 python       9.1%  0:33.72   1    13  1186  
373M+ 1.55M   160M- 2.29G+

----------------------------------------------------------------------

>Comment By: Brian Lenihan (brianl)
Date: 2002-10-14 21:12

Message:
Logged In: YES 
user_id=17671

Nope.  Tim's change fixed it.  I've verified that it works
in 2.3 on OS X 10.2.1 and 2.2.2 on XP.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2002-10-14 20:49

Message:
Logged In: YES 
user_id=33168

This problem was recently addressed by changing sys.maxint /
4 to sys.maxint / 2.  Brian, do you still have the problem?

----------------------------------------------------------------------

Comment By: Brian Lenihan (brianl)
Date: 2002-10-07 22:29

Message:
Logged In: YES 
user_id=17671

Forget what I said in the last comment.  I must have been
asleep.

The problem boils down to this:  OS X does lazy memory
allocation and each process gets up to 4GB of virtual memory.

I now know a lot more about how the interpreter manages
memory.  I could have saved myself the trouble by paying
attention to what the VM: 4.19G+ in top was telling me.

The test in test_b1.py should not be run on OS X.  I am
going to avoid problems like this in the future by setting a
hard limit on per process memory of 400MB.

----------------------------------------------------------------------

Comment By: Brian Lenihan (brianl)
Date: 2002-10-06 09:31

Message:
Logged In: YES 
user_id=17671

I did a little more digging on this problem.  I don't
understand why this is happening yet, but I also think I
found a bug in obmalloc.c

I built a debug interpreter with PYMALLOC_DEBUG after I ran
test_b1.py in gdb and saw memset being called with a bogus
argument:

#1  0x0009722c in _PyObject_DebugMalloc (nbytes=2147483648)
at Objects/obmalloc.c:998

nbytes here should be sys.maxint / 4 + 16, but instead
memset is being called with sys.maxint + 16.

obmalloc.c line 981:

total = nbytes + 16
if (total < nbytes || (total >> 31) > 1) {
  ....
  return NULL

total is a uchar and nbytes is a size_t.  Neither side of
this test is true if total = sys.maxint + 16 has overflowed:
total: 2147483664 nbytes: -2147483632

It seems to me the test should be:

if (total < nbytes || (total >> 31) >= 1) {

Changing obmalloc.c as above, fixes the swapping, but brings
me no closer to finding the actual cause of the poblem.

----------------------------------------------------------------------

Comment By: Brian Lenihan (brianl)
Date: 2002-10-01 04:30

Message:
Logged In: YES 
user_id=17671

OK, I added a fprintf to the NRESIZE macro  and got the same
results on three different platforms, but both OS X builds
start swapping rather than raising a MemoryError.  I guess
the next place to look is PyMem_RESIZE

Windows-XP-5.1.2600-SP1:
Linux (glibc2.1.3):
OS X 10.2.1 (gcc3):
OS X 10.2.1 (gcc 2.95.2):

sys.maxint: 2147483647

test_b1.py:
list
_new_size: 8
_new_size: 8
_new_size: 8
_new_size: 8
_new_size: 8
_new_size: 536870912

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-09-30 16:01

Message:
Logged In: YES 
user_id=21627

Can you please find out how much memory it tries to
allocate? The easiest approach might be add a print
statement; I *think* the relevant allocation should come
from the NRESIZE macro - we'd need to find out what value
_new_size has.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=616019&group_id=5470