[Python-bugs-list] [ python-Bugs-556025 ] list(xrange(1e9)) --> seg fault

[noreply@sourceforge.net]

Bugs item #556025, was opened at 2002-05-14 11:41
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=556025&group_id=5470

Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Raymond Hettinger (rhettinger)
Assigned to: Nobody/Anonymous (nobody)
Summary: list(xrange(1e9))  -->  seg fault

Initial Comment:
>From c.lang.py:

'''
 Python 2.2.1 (#2, Apr 21 2002, 22:22:55) 
[GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)] 
on linux2
Type "help", "copyright", "credits" or "license" for 
more information.
>>> list(xrange(1e9))
Segmentation fault
'''

I've reproduced the same fault on Windows ME using 
Py2.2.0 and Py2.3a.

----------------------------------------------------------------------

>Comment By: Raymond Hettinger (rhettinger)
Date: 2002-05-17 19:12

Message:
Logged In: YES 
user_id=80475

Added a patch to add this bug to the testbank.

Shallow code review:  Patch compiles okay (applied to 
Py2.3a0). Fixes the original problem.  Passes the smell 
test.  Macro style good (only the "var" operand is used 
more than once; no side-effects except setting "var" to 
zero upon a resize error).  Passes the standard regression 
tests.  Passes regression testing versus my personal (real 
code) testbank.

Will give it a deeper look this week-end.

One other thought: should the ValueError be replaced with a 
MemoryError to make the message consistent with PyList_New?



----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2002-05-17 16:03

Message:
Logged In: YES 
user_id=33168

Ok, this time I have a patch.  The patch only fixes listobject.

I looked over the other uses of PyMem_R{ESIZE,ALLOC}() and
they don't appear to be nearly as problematic as list.  For
example, if the grammar has 1e9 nodes, there are going to be
other problems well before then (ie, memory will probably be
gone).

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2002-05-17 15:25

Message:
Logged In: YES 
user_id=33168

Oops, sorry about that last comment.  That was something I
was playing with.  The CVS version is fine for [x]range(float).

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2002-05-17 15:22

Message:
Logged In: YES 
user_id=33168

Note, this problem is even more generic in CVS version:

>>> range(1.1)
Segmentation fault (core dumped)
>>> xrange(1.1)
Segmentation fault (core dumped)

[x]xrange(float) work fine in 2.2.1.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2002-05-14 19:44

Message:
Logged In: YES 
user_id=80475

Would there be some merit to converting PyMem_RESIZE to a 
function with an overflow check?  

I would guess that the time spent on a realloc dwarfs the 
overhead of a short wrapper function.

OTOH, I'll bet this core dump only occurs in toy examples 
and never in real code.



----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-05-14 17:15

Message:
Logged In: YES 
user_id=31435

Heh.  This is an instance of a general flaw:  the 
PyMem_RESIZE macro doesn't check for int overflow in its

    (n) * sizeof(type)

subexpression. The basic deal is that 1000000000 fits in an 
int, but 4 times that (silently) overflows.  In more 
detail, for this specific test case, listobject.c'.s 
roundupsize rounds 1e9 up to 0x40000000, which silently 
underflows to 0!() when PyMem_RESIZE multiplies it by 4.

Hard to know how to fix this in general; PyMem_RESIZE 
callers don't generally worry about overflow now.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=556025&group_id=5470