[Numpy-discussion] Aligned / configurable memory allocation
Julian Taylor
jtaylor.debian at googlemail.com
Tue Feb 10 16:54:49 EST 2015
On 10.02.2015 22:33, Nathaniel Smith wrote:
> On 10 Feb 2015 13:10, "Antoine Pitrou" <solipsis at pitrou.net
> <mailto:solipsis at pitrou.net>> wrote:
>>
>> On Tue, 10 Feb 2015 11:26:22 -0800
>> Nathaniel Smith <njs at pobox.com <mailto:njs at pobox.com>> wrote:
>> > On 10 Feb 2015 09:11, "Antoine Pitrou" <solipsis at pitrou.net
> <mailto:solipsis at pitrou.net>> wrote:
>> > >
>> > >
>> > > Hello,
>> > >
>> > > I apologize for pinging the list, but I was wondering if there was
>> > > interest in either of https://github.com/numpy/numpy/pull/5457 (make
>> > > array data aligned by default) or
>> > > https://github.com/numpy/numpy/pull/5470 (make the array data
> allocator
>> > > configurable)?
>> >
>> > I'm not a fan of the configurable allocator. It adds new public APIs
> for us
>> > to support, and makes switching to using Python's own memory allocation
>> > APIs more complex. The feature is intrinsically dangerous, because newly
>> > installed deallocators must be able to handle memory allocated by the
>> > previous allocator. (AFAICT the included test case can crash the test
>> > process if you get unlucky and GC runs during it?).
>>
>> It's taken care of in the patch.
unfortunately it also breaks the ABI on two fronts, by adding a new
member to the public array struct which needs initializing by non api
using users and by removing the ability to use free on array pointers.
Both not particularly large breaks, but breaks nonetheless.
At least for the first issue we should (like for the proposed dtype and
ufunc changes) apply a more generic break of hiding the new internal
members in a new private structure that embeds the public structure
unchanged.
The second issue can probably be ignored, though we could retain it for
posix/c11 as those standards wisely decided to make aligned pointers
freeable with free.
That on the other hand costs us efficient calloc and realloc (standard
comities are weird sometimes ...)
>
> Ah, I see -- I missed that you added an allocator field to
> PyArrayObject. That does reduce my objections to the patch. But I'm
> still not sure what problems this is solving exactly.
>
> Also, if we do decide to add a deallocation callback to PyArrayObject
> then I think we should take advantage of the opportunity to also make
> life easier for c API users who need a custom callback on a case-by-case
> basis and currently have to jump through hoops using ->base.
>
More information about the NumPy-Discussion
mailing list