[Patches] PyMem [1/8] - intro
Tue, 7 Mar 2000 00:04:40 +0100 (CET)
Guido van Rossum wrote:
> > So your discomfort relates only with the extensions' mem buffers.
> > The lack of macros that expand directly to malloc() is in fact the
> > invisible implementation of the private Python memory concept. You don't
> > need such macros -- one can still use malloc/free directly. But if Python
> > memory is needed, the PyMem_* APIs are provided precisely for this purpose.
> I want to export both a function and a macro (for each operation) to
> use the core allocator, where the macro is normally defined as malloc
> and the function uses the macro.
Obviously, you want them both :-)
And I agree that the in-core/out-core distinction may be confusing.
It's probably not the best way to implement the concept, but still,
let me defend its design once again:
- Besides the malloc cleanup, the goal is to be able to change the
core allocator easily. If the core malloc is not changed, or if it cannot
be changed easily, the net result of the cleanup is of little added value,
- We have to think about lots of legacy extensions. Actually, these
extensions use malloc/free or PyMem_NEW/DEL. Obviously, extensions
that free() objects allocated with PyObject_NEW need to be fixed
(in source) for using them with another core malloc. But the ones using
PyMem_DEL do not require fixes in the source, just recompilation,
because this patch redirects them silently to the wrappers.
Recompilation is by far easiler than fixing the sources, especially
if one deals with 3rd party modules.
With this consideration in mind, I concluded that it would be better
to impose the wrappers for all extensions and to make the core
a "black box" for them w.r.t the core allocator. This simplifies
the understanding of the exposed interfaces for the extension writer
and presents a simple choice: use the Python APIs for Python memory
or non-Python APIs for non-Python memory.
You raise another fair point: the user is smart enough for not mixing
the APIs and is allowed to use Python APIs for memory which may not
be handled by the Python mem manager. This is what would happen if
we do not impose the "2nd class citizenship", i.e., do not impose the
wrapper redirection for *_NEW/DEL and expose PyMem_MALLOC.
(BTW, PyMem_MALLOC is non-existent for the moment)
Now I'm torn too: one of the goals is to distinguish the Python heap
from the rest, the distinction being made by the APIs. I'd like to
know which piece of memory is under Python's control and which is not;
I'd prefer to see the difference explicitely mentioned in the source.
After all, I'd like to handle more efficiently the memory used by Python,
for Python -- a mem mgt scheme which has its specificities. I know that
we'll never gain control over foreign memory, used in 3rd party code
for specific purposes (_tkinter, arrays dealing with specific hardware,
etc.). The distinction will obviously be made by the extension writer,
and we cannot discourage systematically the use of malloc/free... This
is application dependent.
The lack of a clear differentiation between what would be Python mem and
what wouldn't (with a custom malloc), is probably what disturbs me in the
APIs you propose...
Otherwise, I agree with the the safe-func/less-safe-macro arguments
> > (Think unresolved symbols in an object extension file -- if the extension
> > uses Python memory, it should be redirected to the Python memory manager,
> > i.e. have unresolved PyMem_* symbols. The extension can still use malloc()
> > and free() for private purposes, though, resolved with libc's malloc/free).
> But don't we want to discourage extensions from using malloc and free
> directly? Otherwise we'll lose some of the advantages of the custom
> allocator, right?
Right, but we'll discourage their use "if at all possible".
> (Tell me again what you need the custom allocator for?)
:-) Primarily for an overall better management of the memory used for
Python objects and internals, i.e. the one used by custom objects + the
one needed by the core objects and structures. It's about a special-purpose
malloc, not a general-purpose one.
> > What's the point of exporting PyMem_MALLOC? This would be misleading.
> Why misleading? It exists.
not yet. But once we define it, it would be hard to go back or redefine
it to something else.
> I don't like to have macros that behave differently in the core
> than outside, because the core is so often used as an example.
Understood. I don't want them to behave differently. I want them the same,
but tend to prefer exposing them as wrappers for outside. BTW, the fact that
the core is often used as an example is scary too, because the core inlines
frequently _PyObject_New... We need to emphasize the use of _PyObject_NEW/DEL
in the docs.
> > In this case, things would fail badly... The extension writer would
> > be confident in using the Python API, and would expect that by using this
> > API, the extension would work with any version of the core >= 1.6
> I'm not clear how this would "fail badly." It would simply not use
> the custom allocator in this case, but as I mentioned above, memory
> allocation and freeing would still work, using malloc.
My bad - an unweighted thought of mine.
> I hope you will give it some thought. The in-core/not-in-core
> distinction is evil in my mind.
I understand. On my side, the hic is the possibility of using the APIs
without gaining control over the memory they're associated with, plus
the issue about fixing vs. recompiling legacy C code (of the 1.x line).
Hm. Tell me it's okay once again, and before implementing it,
I'll think about it once again <wink>.
Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252