[Python-Dev] valgrind

["Martin v. Löwis"]

Herman Geza schrieb:
>> It can't. In brk, you can only manage memory in chunks of "one page"
>> (i.e. 4kiB on x86). Since we only access memory on the same page,
>> access is guaranteed to succeed.
> Yes, I'm aware of it. But logically, it is possible, isn't it?

No, it isn't.

> At malloc(), libc recognizes that brk needed, it calls sbrk(4096).
> Suppose that python releases this very same block immediately. At free(), 
> libc recognizes that sbrk(-4096) could be executed, so the freed block not 
> available anymore (even for reading)

That can't happen for a different reason. When this access occurs,
we still have a pointer to allocated memory (either allocated through
malloc, or obmalloc - we don't know at the pointer where the access
is made). The access is "invalid" only if the memory was allocated
through malloc. So when the access is made, we have a pointer p,
which is allocated through malloc, and access p-3000 (say, assuming
that p-3000 is a page boundary). Since p is still allocated, libc
*cannot* have made sbrk(p-3000), since that would have released
the still-allocated block.

> 
>>> Or the last block 
>>> in an mmap'd area - it calls unmap. So when Py_ADDRESS_IN_RANGE tries 
>>> to read from this freed memory block it gets SIGSEGV. However, I've never 
>>> got SIGSEGV from python. 
>> Likewise. This is guaranteed to work, by the processor manufacturers.
> The same: if the freed block is the last one in the mmap'd area, libc may 
> unmap it, doesn't it?

But it isn't. We still have an allocated block of memory on the
same page. The C library can't have released it.

>>> I just want to be sure that I'll never get SIGSEGV from python.
>> You least won't get SIGSEGVs from that part of the code.
> That's what I still don't understand. If valgrind is right then how can 
> python be sure that it can still reach a freed block?

valgrind knows the block is released. We know that the block is still
"mapped" to memory by the operating system. These are different
properties. To write to memory, you better have allocated it. To read
from memory, it ought to be mapped (in most applications, it is also
an error to read from released memory, even if the read operation
succeeds; valgrind reports this error as "invalid read").

>>> Note that Misc/valgrind-python.supp contains suppressions "Invalid read"'s 
>>> at Py_ADDRESS_IN_RANGE.
>> Right. This is to tell valgrind that these reads are known to work
>> as designed.
> Does this mean that python strongly depends on libc?

No. It strongly depends on a lower estimate of the page size, and that
memory is mapped on page boundaries.

> If I want to port 
> python to another platform which uses a totally different malloc, is 
> Py_ADDRESS_IN_RANGE guaranteed to work or do I have to make some changes?

It's rather unimportant how malloc is implemented. The real question
is whether you have a flat address space (Python likely won't work at
all if you don't have a flat address space), and whether the system
either doesn't have virtual memory, or, if it does, whether obmalloc's
guess of the page size is either right or an underestimation.
If some constraints fail, you can't use obmalloc (you could still
port Python, to not use obmalloc).

Notice that on a system with limited memory, you probably don't
want to use obmalloc, even if it worked. obmalloc uses arenas
of 256kiB, which might be expensive on the target system.

Out of curiosity: what is your target system?

Regards,
Martin