[Python-Dev] Memory Allocator Part 2: Did I get it right?
ejones at uwaterloo.ca
Wed Feb 16 04:02:53 CET 2005
On Feb 15, 2005, at 17:52, Tim Peters wrote:
> As I said before, I don't think we need to support this any more.
> More, I think we should not -- the support code is excruciatingly
> subtle, it wasted plenty of your time trying to keep it working, and
> if we keep it in it's going to continue to waste time over the coming
> years (for example, in the short term, it will waste my time reviewing
I do not have nearly enough experience in the Python world to evaluate
this decision. I've only been programming in Python for about two years
now, and as I am sure you are aware, this is my first patch that I have
submitted to Python. I don't really know my way around the Python
internals, beyond writing basic extensions in C. Martin's opinion is
clearly the opposite of yours.
Basically, the debate seems to boil down to maintaining backwards
compatibility at the cost of making the code in obmalloc.c harder to
understand. The particular case that is being supported could
definitely be viewed as a "bug" in the code that using obmalloc. It
also likely is quite rare. However, until now it has been supported, so
it is hard to judge exactly how much code would be affected. It would
definitely be a minor barrier to moving to Python 2.5. Is there some
sort of consensus that is possible on this issue?
>> While one thread holds the GIL, any other thread can call
>> with a pointer that was returned by the system malloc.
> What _was_ supported was more generally that
> any number of threads could call PyObject_Free with pointers that
> returned by the system malloc/realloc
> at the same time as
> a single thread, holding the GIL, was doing anything whatsoever
> executing any code inside obmalloc.c)
Okay, good, that is what I have assumed.
> Although that's a misleading way of expressing the actual intent; more
> on that below.
That's fine. It may be a misleading description of the intent, but it
is an accurate description of the required behaviour. At least I hope
> I expect it would be easier if you
> ripped out the horrid support for PyObject_Free abuse; in a sane
> world, the release-build PyMem_FREE, PyMem_Del, and PyMem_DEL would
> expand to "free" instead of to "PyObject_FREE" (via changes to
It turns out that basically the only thing that would change would be
removing the "volatile" specifiers from two of the global variables,
plus it would remove about 100 lines of comments. :) The "work" was
basically just hurting my brain trying to reason about the concurrency
issues, not changing code.
> It was never legit to do #a without holding the GIL. It was clear as
> mud whether it was legit to do #b without holding the GIL. If
> PyMem_Del (etc) change to expand to "free" in a release build, then #b
> can remain clear as mud without harming anyone. Nobody should be
> doing #a anymore. If someone still is, "tough luck -- fix it, you've
> had years of warning" is easy for me to live with at this stage.
Hmm... The issue is that case #a may not be an easy problem to
diagnose: Some implementations of free() will happily do nothing if
they are passed a pointer they know nothing about. This would just
result in a memory leak. Other implementations of free() can output a
warning or crash in this case, which would make it trivial to locate.
> I suppose the other consideration is that already-compiled extension
> modules on non-Windows(*) systems will, if they're not recompiled,
> continue to call PyObject_Free everywhere they had a
> PyMem_Del/DEL/FREE call.
Is it guaranteed that extension modules will be binary compatible with
future Python releases? I didn't think this was the case.
Thanks for the feedback,
More information about the Python-Dev