[issue11849] glibc allocator doesn't release all free()ed memory

[Antoine Pitrou]

Antoine Pitrou <pitrou at free.fr> added the comment:

> On my box:
> default:
> $ ./python -m timeit -s "n=300000; f=open('/tmp/10MB.bin', 'rb');
> b=bytearray(n)" "f.seek(0);f.readinto(b)"
> 1000 loops, best of 3: 640 usec per loop
> 
> default without patch ("$ hg revert -r 68258 Objects/obmalloc.c && make"):
> $ ./python -m timeit -s "n=300000; f=open('/tmp/10MB.bin', 'rb');
> b=bytearray(n)" "f.seek(0);f.readinto(b)"
> 1000 loops, best of 3: 663 usec per loop
> 
> I'm just observing a random variance (but my computer is maybe too
> slow to notice).

Hmm, quite slow indeed, are you sure you're not running in debug mode?

> However, I really don't see how the patch could play a role here.
> 
> Concerning the slight performance regression, if it's a problem, I see
> two options:
> - revert the patch
> - replace calls to malloc()/free() by mmap()/munmap() to allocate/free
> arenas (but I'm not sure anonymous mappings are supported by every OS
> out there, so this might lead to some ugly #ifdef's...)

If the performance regression is limited to read(), I don't think it's
really an issue, but using mmap/munmap explicitly would probably benicer
anyway (1° because it lets the glibc choose whatever heuristic is best,
2° because it would help release memory on more systems than just glibc
systems). I think limiting ourselves to systems which have
MMAP_ANONYMOUS is good enough.

Here is what the glibc malloc does btw:

/*
   Nearly all versions of mmap support MAP_ANONYMOUS,
   so the following is unlikely to be needed, but is
   supplied just in case.
*/

#ifndef MAP_ANONYMOUS

static int dev_zero_fd = -1; /* Cached file descriptor for /dev/zero. */

#define MMAP(addr, size, prot, flags) ((dev_zero_fd < 0) ? \
 (dev_zero_fd = open("/dev/zero", O_RDWR), \
  mmap((addr), (size), (prot), (flags), dev_zero_fd, 0)) : \
   mmap((addr), (size), (prot), (flags), dev_zero_fd, 0))

#else

#define MMAP(addr, size, prot, flags) \
 (mmap((addr), (size), (prot), (flags)|MAP_ANONYMOUS, -1, 0))

#endif

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________