[Numpy-discussion] numpy arrays, data allocation and SIMD alignement
david at ar.media.kyoto-u.ac.jp
Mon Aug 6 23:41:03 EDT 2007
Lisandro Dalcin wrote:
> On 8/3/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>> Here is what I can think of:
>> - adding an API to know whether a given PyArrayObject has its data
>> buffer 16 bytes aligned, and requesting a 16 bytes aligned
>> PyArrayObject. Something like NPY_ALIGNED, basically.
>> - forcing data allocation to be 16 bytes aligned in numpy (eg
>> define PyDataMem_Mem to a 16 bytes aligned allocator instead of malloc).
> All this sounds pretty similar to sdt::allocator we can found in C++
> STL (http://www.sgi.com/tech/stl/Allocators.html). Perhaps a NumPy
> array could be associated with an instance of an 'allocator' object
> (surely written in C, perhaps subclassable in Python) providing
> appropriate methos for
> alloc/dealloc(/realloc?/initialize(memset)?/copy(memcpy)?) memory.
> This would be really nice, as it is extensible (you could even write a
> custom allocator, perhaps making use of a preallocated,static pool;
> use of C++ new/delete; use of any C++ std::allocator, shared memory,
> etc. etc.). I think this is the direction to go but no idea how much
> difficult it could be to implement.
Well, when I proposed the SIMD extension, I was willing to implement the
proposal, and this was for a simple goal: enabling better integration
with many numeric libraries which need SIMD alignment.
As nice as a custom allocator might be, I will certainly not implement
it myself. For SIMD, I think the weight adding complexity / benefit
worth it (since there is not much change to the API and implementation),
and I know more or less how to do it; for custom allocator, that's an
entirely different story. That's really more complex; static pools may
be useful in some cases (but that's not obvious, since only the data are
allocated with this buffer, everything else being allocated through the
python memory allocator, and numpy arrays have pretty simple memory
More information about the NumPy-Discussion