David Cournapeau wrote:

...

Hi Jarrod,

Would it be possible to merge some of the work I have done recently concerning cleaning configuration and so on (If nobody is against it, of course) ? If this is considerer too big of a change, what is the plan for a 1.1 release, if any ?

Could you review what the disadvantages are for including your changes into the trunk? Would there be any code breakage? What is the approximate size increase of the resulting NumPy? I've tried to follow the discussion, but haven't kept up with all of it. Thanks, -Travis

I've finally caught up with the discussion on aligned allocators for NumPy. In general I'm favorable to the idea, although it is not as easy to implement in 1.0.X because of the need to possibly change the C-API. The Python solution is workable and would just require a function call on the Python side (which one could call from the C-side as well with little difficulty, I believe Chuck Harris already suggested such a function). So, I think at least the Python functions are an easy addition for 1.0.4 (along with simple tests for alignment --- although a.ctypes.data % 16 is pretty simple and probably doesn't warrant a new function) I'm a bit more resistant to the more involved C-code in the patch provided with #568, because of the requested new additions to the C-API, but I understand the need. I'm currently also thinking heavily about using SIMD intrinsics in ufunc inner loops but will not likely get those in before 1.0.4. Unfortunately, all ufuncs that take advantage of SIMD instructions will have to handle the unaligned portions which may occur even if the start of the array is aligned, so the problem of thinking about alignment does not go away there with a simplified function call. A simple addition is an NPY_ALIGNED_16 and NPY_ALIGNED_32 flag for the PyArray_From_Any that could adjust the data-pointer as needed to get at least those kinds of alignment. We can't change the C-API for PyArray_FromAny to accept an alignment flag, and I'm pretty loath to do that even for 1.1. Is there a consensus? What do others think of the patch in ticket #568? Is there a need to add general-purpose aligned memory allocators to NumPy without a corresponding array_allocator? I would think the PyArray_FromAny and PyArray_NewFromDescr with aligned memory is more important which I think we could do with flag bits. -Travis O.

Travis E. Oliphant wrote:

I've finally caught up with the discussion on aligned allocators for NumPy. In general I'm favorable to the idea, although it is not as easy to implement in 1.0.X because of the need to possibly change the C-API.

The Python solution is workable and would just require a function call on the Python side (which one could call from the C-side as well with little difficulty, I believe Chuck Harris already suggested such a function). So, I think at least the Python functions are an easy addition for 1.0.4 (along with simple tests for alignment --- although a.ctypes.data % 16 is pretty simple and probably doesn't warrant a new function)

I'm a bit more resistant to the more involved C-code in the patch provided with #568, because of the requested new additions to the C-API, but I understand the need. I'm currently also thinking heavily about using SIMD intrinsics in ufunc inner loops but will not likely get those in before 1.0.4. Unfortunately, all ufuncs that take advantage of SIMD instructions will have to handle the unaligned portions which may occur even if the start of the array is aligned, so the problem of thinking about alignment does not go away there with a simplified function call.

I don't know anything about the ufunc machinery yet, but I guess you need to know the alignement of a given buffer, right ? This can be done easily. Actually, one problem I encountered (If I remember correctly) was that there is no pure C library facility in numpy: by that, I mean a simple C library, independant of python, which could be reusable by C code using numpy. For example, if we want to start thinking about using SIMD, I think it would be good to support basics in a pure C library. I don't see any downside to this approach ?

A simple addition is an NPY_ALIGNED_16 and NPY_ALIGNED_32 flag for the PyArray_From_Any that could adjust the data-pointer as needed to get at least those kinds of alignment.

We can't change the C-API for PyArray_FromAny to accept an alignment flag, and I'm pretty loath to do that even for 1.1.

Is there a consensus? What do others think of the patch in ticket #568? Is there a need to add general-purpose aligned memory allocators to NumPy without a corresponding array_allocator?

Having the NPY_ALIGNED_* flags would already be enough for most cases (for SIMD, you rarely, if ever needs more than 32 bytes alignment AFAIK). Those flags + general purposes memory allocators (in a C support library) would be enough to do everything we need to fft, for example. cheers, David

Lisandro Dalcin wrote:

On 10/19/07, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:

...

numpy.scons branch

This is a much more massive change. Scons itself adds something like 350 kb to a bzip tarball.

If numpy build system will not depend on scons (is this right?) then .. Is it strictly needed to distribute scons with numpy? Well, the choice is between requiring another dependency, with all the version problems, and including it in the sources; also, although numpy would not depend on scons in any way with the changes I suggested for 1.0.4, it would have helped detecting problems on some platforms. And I hope that in the end, scons will be used for numpy (for 1.1 ?), once I finish the conversion.

I don't see situations where adding 350 Kb in the tarball can be an issue, so could you tell me what the problem would be ? cheers, David

Lisandro Dalcin wrote:

On 10/20/07, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote: And I

...

hope that in the end, scons will be used for numpy (for 1.1 ?), once I finish the conversion.

I don't see situations where adding 350 Kb in the tarball can be an issue, so could you tell me what the problem would be ?

Then if numpy is going to use scons for build itself in the near future, then there is no problem, and including scons in numpy tarball is good from the user side.

Just as a precision: I am the only one so far working on this (using scons instead of distutils), and no decision has been made whether it will be used at all in the future.

Just a final observation. Try to make it possible to reuse a scons other than that included in numpy tarball. Packagers would be happy with this.

Yes, I though about that, it would not be too difficult to handle (and scons has a simple mechanism to ensure a given version), since scons is called externally (it could be a option given to the scons command inside distutils, for example). cheers, David