[Numpy-discussion] NEP 49: Data allocation strategies
matti.picus at gmail.com
Wed Apr 21 21:26:11 EDT 2021
See my comments interspersed in Ralf's reply. Thanks for the additional
On 21/4/21 3:10 am, Ralf Gommers wrote:
> Motivation and Scope
> Users may wish to override the internal data memory routines with
> of their
> own. Two such use-cases are to ensure data alignment and to pin
> allocations to certain NUMA cores.
> It would be great to expand a bit on these two sentences, and add some
> links. There's a lot of history here in NumPy development to refer to
> as well:
> There must also be a good amount of ideas/discussion elsewhere.
I added more context to this section, trying to focus on the large data
allocations in NumPy.
> <https://bugs.python.org/issue18835> discussed an aligned allocator
> for Python itself, with fairly detailed discussion about whether/how
> NumPy could benefit. With (I think) the conclusion it shouldn't be in
> Python, but NumPy/Arrow/others are better off doing their own thing.
> I'm wondering if improved memory profiling is a use case as well? Fil
> <https://github.com/pythonspeed/filprofiler>) for example seems to use
> such a strategy:
Thanks. I added a sentence about this as well.
> Does it interact with our tracemalloc support
I added a sentence about this. The new C-API wrapper functions preserve
the current status vis-a-vis tracemalloc support. I am not sure that
support is complete. The NEP should not change the situation for better
> User who wish to change the NumPy data memory management routines
> will use
> This is design, not motivation or scope. Try to not refer to specific
> function names in this section. I suggest moving this content to the
> "Detailed design" section (or better, a "high level design" at the
> start of that section).
More information about the NumPy-Discussion