How is obmalloc safe with "Invalid read of size 4" ?
We are having random, rare, nonreproducible segfaults/hangs with python2 on ubuntu 14.04 in EC2. I've managed to attach GDB to some hung ones and there looks like clear memory corruption in the 'interned' hash table, causing lookdict_string() to spin forever because all remaining slots have a garbage 'key' pointer. This happens just loading the 'site' module dependencies, like 're' or 'codecs', before any of our code even gets run. So we then tried running it under valgrind, and we got a lot of nasty errors. Even after reading the Misc/README.valgrind, which talks about *uninitialized* reads being ok, I still don't see how reading from *freed* memory would ever be safe, and why the suppression file thinks thats ok: $ valgrind ./pymd79/bin/python -c "" ==14651== Memcheck, a memory error detector ==14651== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==14651== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==14651== Command: ./pymd79/bin/python -c ==14651== ==14651== Invalid read of size 4 ==14651== at 0x461E40: Py_ADDRESS_IN_RANGE (obmalloc.c:1911) ==14651== by 0x461EA3: PyObject_Free (obmalloc.c:994) ==14651== by 0x4789AB: tupledealloc (tupleobject.c:235) ==14651== by 0x5225BA: code_dealloc (codeobject.c:309) ==14651== by 0x4CFFC3: load_source_module (import.c:1100) ==14651== by 0x4D0E16: import_submodule (import.c:2700) ==14651== by 0x4D1E19: PyImport_ImportModuleLevel (import.c:2515) ==14651== by 0x4AE49A: builtin___import__ (bltinmodule.c:49) ==14651== by 0x422C89: PyObject_Call (abstract.c:2529) ==14651== by 0x4B12E5: PyEval_EvalFrameEx (ceval.c:3902) ==14651== by 0x4B6A47: PyEval_EvalCodeEx (ceval.c:3265) ==14651== by 0x4B6B71: PyEval_EvalCode (ceval.c:667) ==14651== Address 0x5bcd020 is 2,256 bytes inside a block of size 2,801 free'd ==14651== at 0x4C28577: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==14651== by 0x4DB2B0: PyMarshal_ReadLastObjectFromFile (marshal.c:1145) ==14651== by 0x4CFE71: load_source_module (import.c:801) - Karl
On 03/24/2015 03:28 PM, Karl Pickett wrote:
PyObject_Free() is not reading *freed* memory, it is reading memory outside (right before) the allocated range. This is, of course, undefined behavior as far as C is concerned and an invalid read in the eyes of valgrind. Still, it's the kind of thing you can get away with if you are writing a heavily optimized allocator (and if your name starts with "Tim" and ends with "Peters"). README.valgrind explains in quite some detail why this is done. In short, it allows for a very fast check whether the memory passed to PyObject_Free() was originally allocated by system malloc or by Python's pool allocator.
On 03/24/2015 03:28 PM, Karl Pickett wrote:
PyObject_Free() is not reading *freed* memory, it is reading memory outside (right before) the allocated range. This is, of course, undefined behavior as far as C is concerned and an invalid read in the eyes of valgrind. Still, it's the kind of thing you can get away with if you are writing a heavily optimized allocator (and if your name starts with "Tim" and ends with "Peters"). README.valgrind explains in quite some detail why this is done. In short, it allows for a very fast check whether the memory passed to PyObject_Free() was originally allocated by system malloc or by Python's pool allocator.
participants (2)
-
Hrvoje Niksic -
Karl Pickett