[ python-Bugs-1185883 ] PyObject_Realloc bug in obmalloc.c

Mon Jul 11 00:33:07 CEST 2005

Bugs item #1185883, was opened at 2005-04-19 08:07
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1185883&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.3
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Kristján Valur (krisvale)
Assigned to: Tim Peters (tim_one)
Summary: PyObject_Realloc bug in obmalloc.c

Initial Comment:
obmalloc.c:835
If the previous block was not handled by obmalloc, and 
the realloc is for growing the block, this memcpy may 
cross a page boundary and cause a segmentation 
fault.  This scenario can happen if a previous allocation 
failed to successfully allocate from the obmalloc pools, 
due to memory starvation or other reasons, but was 
successfully allocated by the c runtime.

The solution is to query the actual size of the allocated 
block, and copy only so much memory.  Most modern 
platforms provide size query functions complementing 
the malloc()/free() calls.  on Windows, this is the _msize
() function.

----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2005-07-10 18:33

Message:
Logged In: YES 
user_id=31435

Repaired, in

Misc/NEWS 1.1312
Objects/obmalloc.c 2.54

Should be backported to 2.4 maint.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2005-05-27 05:35

Message:
Logged In: YES 
user_id=6656

Ping!

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2005-04-19 11:34

Message:
Logged In: YES 
user_id=31435

krisvale:  Thank you for the very clear explanation.  Even I 
understand this now <wink>.

We won't use _msize here -- Python has to run under dozens 
of compilers and C libraries, and it's saner to give up on 
this "optimization" completely than to introduce a rat's nest 
of #ifdefs here.  IOW, I expect the entire "if (nbytes <= 
SMALL_REQUEST_THRESHOLD)" block will go away, so 
that the platform realloc() gets called in every case obmalloc 
doesn't control the incoming block.

BTW, note that there's no plan to do another release in the 
Python 2.3 line.

----------------------------------------------------------------------

Comment By: Kristján Valur (krisvale)
Date: 2005-04-19 11:22

Message:
Logged In: YES 
user_id=1262199

The platform is windows 2000/2003 server, single threaded C 
runtime.  I have only had the chance to do postmortem 
debugging on this but it would appear to be as you describe:  
The following page is not mapped in.  Windows doesn´t use 
the setbrk() method of heap management and doesn´t 
automatically move the break.  Rather they (the multiple 
heaps) requests pages as required.   A malloc may have 
succeeded from a different page and copying to much from 
the old block close to the boundary caused an exception 
_at_ the page boundary.
Fyi, old block was 68 bytes at 0x6d85efb8.  This block ends 
at -effc.   The new size requested was 108 bytes.  Reading 
108 bytes from this address caused an exception at address 
0x6d85f000.  As you know, reading past a malloc block 
results in undefined behaviour and sometimes this can mean 
a crash.
I have patched python locally to use MIN(nbytes, _msize(p)) 
in stead and we are about to run the modified version on our 
server cluster.  Nodes were dying quite regularly because of 
this.  I'll let you know if this changes anyting in that aspect.

Btw, I work for ccp games, and we are running the MMORPG 
eve online (www.eveonline.com)

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2005-04-19 11:00

Message:
Logged In: YES 
user_id=31435

mwh:  Umm ... I don't understand what the claim is.  For 
example, what HW does Python run on where memcpy 
segfaults just because the address range crosses a page 
boundary?  If that's what's happening, sounds more like a 
bug in the platform memcpy.  I can memcpy blocks spanning 
thousands of pages on my box -- and so can you <wink>.

krisvale:  which OS and which C are you using?

It is true that this code may try to access a bit of memory 
that wasn't allocated.  If that's at the end of the address 
space, then I could see a segfault happening.  If it is, I doubt 
there's any portable way to fix it short of PyObject_Realloc 
never trying to take over small blocks it didn't control to begin 
with.  Then the platform realloc() will segfault instead <wink>.

----------------------------------------------------------------------

Comment By: Kristján Valur (krisvale)
Date: 2005-04-19 10:39

Message:
Logged In: YES 
user_id=1262199

I can only say that I´ve been seeing this happeing with our 
software.  Admittedly it's because we are eating up all 
memory due to other reasons, but we would like to deal with 
that with a MemoryError rather than a crash.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2005-04-19 10:30

Message:
Logged In: YES 
user_id=6656

Tim, what do you think?

This is a pretty unlikely scenario, it seems to me.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1185883&group_id=5470