[Python-bugs-list] [ python-Bugs-563303 ] Heap corruption in debug

[noreply@sourceforge.net]

Bugs item #563303, was opened at 2002-06-01 10:10
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=563303&group_id=5470

Category: Python Interpreter Core
Group: Python 2.3
>Status: Closed
>Resolution: Invalid
Priority: 5
Submitted By: Cliff Owen (cliffo)
Assigned to: Tim Peters (tim_one)
Summary: Heap corruption in debug

Initial Comment:
PyObject_DebugMalloc() returns a pointer that is +8 
bytes to detect for underwriting the pointer.

Sometimes, PyObject_Free() is called instead of going 
through PyObject_DebugFree(). This means the pointer 
is off by 8. PyObject_Free() takes the pointer passed in 
as being valid, and will update the pool to indicate the 
passed pointer is a pointer to the new free block.

Corruption of the heap occurs when 
PyObject_DebugMalloc() is called and there are 
multiple free blocks in a row that are properly aligned. 
Writing the FORBIDDENBYTE set to the end of the 
newly allocated block may stomp the next free pointer.

2 more allocations later on this arena, and you will AV.

Why is PyObject_Free() being called instead of 
PyObject_DebugFree() (from ste_dealloc)? I'm 
assuming it was built incorrectly or there's an ordering 
issue in objimpl.h where PyObject_Free() is defined 
before the macro, and referenced by PyObject_Del.

The fact remains that a simple alignment check in 
PyObject_Free() will handle any misaligned pointer.

If I've got this completely screwed up and it is in fact 
building incorrectly, please let me know. I've traced 
through it about 100 times. I'm fairly certain I understand 
how/why it works the way it does, but I could be wrong.

This problem would only show up in debug mode.

----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2002-06-03 12:50

Message:
Logged In: YES 
user_id=31435

Offline, it was determined that:

+ The OP was compiling Python as well as his app.

+ Some Python modules weren't getting compiled with 
_DEBUG, via an accident in his build setup.

So closing as invalid -- the problem was due to a flawed 
custom build setup.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-06-01 19:56

Message:
Logged In: YES 
user_id=31435

Could you try to describe exactly how the problem arises, 
holding off on speculation about causes and cures?  What I 
still don't grasp at all is how you get into this situation to 
begin with:  it should be impossible to enter PyObject_Free
() in a pymalloc debug build *except* when 
_PyObject_DebugFree() calls it.  Your report begins with

"""
Sometimes, PyObject_Free() is called instead of going 
through PyObject_DebugFree(). 
"""

and I'm still lost on that sentence:  it shouldn't be possible 
to do that in a debug build.  If there is a way to do it, it's a 
serious problem, and your suggested code won't fix it (it 
may hide bad symptoms by accident for a while, but the 
problem still exists and will come back to bite you later).

It's unclear to me why you mentioned ste_dealloc, but I've 
stepped thru that in the MSVC 6 debugger and it does call 
_PyObject_DebugFree in a debug build.  It's doing what it's 
supposed to do. 

1. Is it your claim that ste_dealloc actually calls 
PyObject_Free directly in *your* debug build?  "Yes" or "no" 
would be more helpful than outrage <wink>.

2. What does "a snapshot of the python 2.3 build" mean, 
exactly?  Did you compile it yourself?  If not, from where did 
you get it?

3. Are you writing your own C extensions?  If so, are you 
mixing _DEBUG with non-_DEBUG between Python and 
your own code?  If you are so mixing, it's never going to 
work (Python and extension modules on Windows must 
both be built with _DEBUG, or neither).

----------------------------------------------------------------------

Comment By: Cliff Owen (cliffo)
Date: 2002-06-01 18:22

Message:
Logged In: YES 
user_id=556609

Yes, I do have a program which demonstrates the problem. 
However, I cannot send it in because it is commercial in 
nature. :( I understand this poses a bit of a problem for 
debugging, but I'm hoping you understand. I do have other 
things that require more of my time than browsing the Python 
sources and making speculations on what I see. This did AV, 
and I did spend a while debugging it, with a debugger. Being 
the nice developer who's using a free product that's not being 
maintained by me, I thought I'd report this.

Yes, I also have seen what happens inside of objimpl.h, and 
do know that PyObject_Del is called in ste_dealloc. These 
are details I didn't think necessary to mention because you 
would obviously be familiar with the code and since I 
mentioned ste_dealloc, you might assume I would be familiar 
with these mappings as well since ste_dealloc() clearly calls 
PyObject_Del, which is nothing but a macro to 
PyObject_Free.

Regardless and bickering aside, I have added the following 
line to PyObject_Free() in my own code and it resolves the 
problem by assuring that any pointer passed in is properly 
aligned on the correct block size. This one line covers a 
multitude of sins from the caller.

p = (char *) p - ((char *)p - ((char *)pool + 
POOL_OVERHEAD)) % INDEX2SIZE(pool->szidx);

Right after:

if (ADDRESS_IN_RANGE(p, pool->arenaindex))

If you choose to use this line or not is up to you. If the user 
passes in a pointer that is aligned, this line does nothing. If 
the user passes in a pointer which resides anywhere in a 
valid block, that block is released. Is this the correct 
behavior? Probably not. It should probably raise an exception 
rather than allow the user to get away with it. This would at 
least force the user to figure out why they were passing in a 
bogus pointer in the first place.

Some additional information: I am building with MSVC 6.x 
(SP3), using a snapshot of the python 2.3 build. I am 
compiling into a static library using these defines only.

WIN32,_DEBUG,_MBCS,_LIB,USE_DL_EXPORT

If you need additional information, let me know. I'm not sure 
what else I can tell you.


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-06-01 12:35

Message:
Logged In: YES 
user_id=31435

Do you have an actual program that shows corruption, or 
are you speculating?

Note that in a debug build, it's not possible to call 
PyObject_Free:  the name PyObject_Free is remapped to 
_PyObject_DebugFree by objimpl.h then.  In a debug build 
when pymalloc is enabled, all calls to all PyMem_XYZ and 
PyObject_XYZ memory functions are redirected to 
_PyObject_DebugXYZ, except within obmalloc.c itself 
(which #undefs some name substitutions).

ste_dealloc doesn't call PyObject_Free.  It calls 
PyObject_Del.  In a debug pymalloc build, that ends up 
calling _PyObject_DebugFree.  Step through it in a 
debugger if you don't believe it <wink>.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=563303&group_id=5470