[Python-Dev] Changing pymalloc behaviour for long running processes

["Martin v. Löwis"]

Evan Jones wrote:
>> Another is that the platform OS+libc may not shrink VM at
>> times even when memory is returned to the system free().
> 
> 
> There is absolutely nothing I can do about that, however.

You could if you wanted. Don't use malloc/free, but use
mmap/munmap, VirtualAlloc/VirtualFree, etc.

> Anyone care 
> to enlighten me?

Microsoft ships the source of its malloc(3) implementation
together with VC; you need to install the CRT source to see it.

> Let me just make sure I am clear on this: Some extensions use native 
> threads, is that why this is a problem? Because as far as I am aware, 
> the Python interpreter itself is not threaded. So how does the cyclical 
> garbage collector work? Doesn't it require that there is no execution 
> going on?

The garbage collector holds the GIL. So while there could be other
threads running, they must not manipulate any PyObject*. If they try
to, they need to obtain the GIL first, which will make them block
until the garbage collector is complete.

> It is also a waste if the core Python developers decide it is a bad 
> idea, and don't want to accept patches! :)

That will ultimately depend on the patches. The feature itself would
be fine, as Tim explains. However, patches might be rejected because:
- they are incorrect,
- their correctness cannot easily be established,
- they change unrelated aspects of the interpreter,
- they have undesirable performance properties, or
- they have other problems I can't think of right now :-)

Regards,
Martin