[Python-Dev] RE: Program very slow to finish

[Martin von Loewis]

> Can any Python-Dev'er make time to dig into the advisability of
> making PyMalloc the default?

I was looking into making pymalloc the default a few weeks ago, when I
noticed that allocations is pretty efficient on Linux. I studied glibc
malloc a bit to see that it is indeed quite clever about allocation,
even in the presence of MT locks (with per-arena locks, and recording
a per-thread arena in a thread-local variable).

I believe the main difference in deallocation speed (which I didn't
notice then) comes from the attempt to combine subsequent memory
blocks in glibc; pymalloc doesn't attempt to do so. Also, there *is* a
lock acquire/release cycle in glibc malloc, which may account for some
overhead, too.

Looking at obmalloc.c, I see one aspect that I'd consider
questionable: detection of pool-vs-system-malloc uses a heuristics,
namely

	offset = (off_t )p & POOL_SIZE_MASK;
	pool = (poolp )((block *)p - offset);
	if (pool->pooladdr != pool || pool->magic != (uint )POOL_MAGIC) {
		_SYSTEM_FREE(p);
		return;
	}

So if the pool header has the pool magic and points to itself, it
really is a pool header. That may result in a system malloc being
recognized as a pool malloc, under rare circumstances. As a result,
the system heap will be corrupted.

Another aspect that may be troubling is that obmalloc never returns
memory to the system. It just detects free pools, and is willing to
rearrange a free pool for use with a different object size. Of course,
there is little chance that a complete arena will ever become empty,
so this is probably not that bad.

Regards,
Martin