Numpy 1.9.1, zeros and alignement
Hi, I have not followed closely the changes that happen in 1.9.1, but was surprised by the following: x = np.zeros(12, "d") assert x.flags.aligned # fails This is running numpy 1.9.1 built on windows with VS 2008. Is it expected that zeros may return a non-aligned array ? David
On Tue, Nov 18, 2014 at 11:20 AM, David Cournapeau <cournape@gmail.com> wrote:
Hi,
I have not followed closely the changes that happen in 1.9.1, but was surprised by the following:
x = np.zeros(12, "d") assert x.flags.aligned # fails
This is running numpy 1.9.1 built on windows with VS 2008. Is it expected that zeros may return a non-aligned array ?
The alignment requirements may have become stricter, Julian would know. Is this on 32 or 64 Windows? Chuck
On 18.11.2014 19:20, David Cournapeau wrote:
Hi,
I have not followed closely the changes that happen in 1.9.1, but was surprised by the following:
x = np.zeros(12, "d") assert x.flags.aligned # fails
This is running numpy 1.9.1 built on windows with VS 2008. Is it expected that zeros may return a non-aligned array ?
what is the real alignment of the array? Are you on 32 bit or 64 bit? What is the alignment of doubles in windows (linux its 4 byte on 32 bit 8 byte on 64 bit (% special compiler flags)? print x.__array_interface__["data"] there are problems with complex types but doubles should be aligned even on 32 bit.
It is on windows 32 bits, but I would need to make this work for complex (pair of double) as well. Is this a bug (I assumed array creation methods would always create aligned arrays for their type) ? Seems like quite a bit of code out there would assume this (scipy itself does for example). (the context is > 100 test failures on scipy 0.14.x on top of numpy 1.9., because f2py intent(inout) fails on work arrays created by zeros, this is a windows-32 only failure). David On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:
On 18.11.2014 19:20, David Cournapeau wrote:
Hi,
I have not followed closely the changes that happen in 1.9.1, but was surprised by the following:
x = np.zeros(12, "d") assert x.flags.aligned # fails
This is running numpy 1.9.1 built on windows with VS 2008. Is it expected that zeros may return a non-aligned array ?
what is the real alignment of the array? Are you on 32 bit or 64 bit? What is the alignment of doubles in windows (linux its 4 byte on 32 bit 8 byte on 64 bit (% special compiler flags)? print x.__array_interface__["data"]
there are problems with complex types but doubles should be aligned even on 32 bit. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Additional point: it seems to always return aligned data on 1.8.1 (same platform/compiler/everything). On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau <cournape@gmail.com> wrote:
It is on windows 32 bits, but I would need to make this work for complex (pair of double) as well.
Is this a bug (I assumed array creation methods would always create aligned arrays for their type) ? Seems like quite a bit of code out there would assume this (scipy itself does for example).
(the context is > 100 test failures on scipy 0.14.x on top of numpy 1.9., because f2py intent(inout) fails on work arrays created by zeros, this is a windows-32 only failure).
David
On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:
On 18.11.2014 19:20, David Cournapeau wrote:
Hi,
I have not followed closely the changes that happen in 1.9.1, but was surprised by the following:
x = np.zeros(12, "d") assert x.flags.aligned # fails
This is running numpy 1.9.1 built on windows with VS 2008. Is it expected that zeros may return a non-aligned array ?
what is the real alignment of the array? Are you on 32 bit or 64 bit? What is the alignment of doubles in windows (linux its 4 byte on 32 bit 8 byte on 64 bit (% special compiler flags)? print x.__array_interface__["data"]
there are problems with complex types but doubles should be aligned even on 32 bit. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
< 1.9 lies about alignment it doesn't actually check for new arrays. is the array aligned? On 18.11.2014 19:37, David Cournapeau wrote:
Additional point: it seems to always return aligned data on 1.8.1 (same platform/compiler/everything).
On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote:
It is on windows 32 bits, but I would need to make this work for complex (pair of double) as well.
Is this a bug (I assumed array creation methods would always create aligned arrays for their type) ? Seems like quite a bit of code out there would assume this (scipy itself does for example).
(the context is > 100 test failures on scipy 0.14.x on top of numpy 1.9., because f2py intent(inout) fails on work arrays created by zeros, this is a windows-32 only failure).
David
On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor <jtaylor.debian@googlemail.com <mailto:jtaylor.debian@googlemail.com>> wrote:
On 18.11.2014 19:20, David Cournapeau wrote: > Hi, > > I have not followed closely the changes that happen in 1.9.1, but was > surprised by the following: > > x = np.zeros(12, "d") > assert x.flags.aligned # fails > > This is running numpy 1.9.1 built on windows with VS 2008. Is it > expected that zeros may return a non-aligned array ? >
what is the real alignment of the array? Are you on 32 bit or 64 bit? What is the alignment of doubles in windows (linux its 4 byte on 32 bit 8 byte on 64 bit (% special compiler flags)? print x.__array_interface__["data"]
there are problems with complex types but doubles should be aligned even on 32 bit. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Nov 18, 2014 at 6:40 PM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:
< 1.9 lies about alignment it doesn't actually check for new arrays.
When I do the following on 1.8.1 with win 32 bits: x = np.zeros(12, "D") print x.aligned.flags == (x.__array_interface__["data"][0] % 16 == 0) # always true print x.aligned.flags # always true but on 1.9.1: x = np.zeros(12, "D") print x.aligned.flags == (x.__array_interface__["data"][0] % 16 == 0) # always true print x.aligned.flags # not always true I wonder why numpy 1.8.1 always returned 16 bytes aligned arrays in this case (unlikely to be a coincidence, as I try quite a few times with different sizes).
is the array aligned?
On 18.11.2014 19:37, David Cournapeau wrote:
Additional point: it seems to always return aligned data on 1.8.1 (same platform/compiler/everything).
On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote:
It is on windows 32 bits, but I would need to make this work for complex (pair of double) as well.
Is this a bug (I assumed array creation methods would always create aligned arrays for their type) ? Seems like quite a bit of code out there would assume this (scipy itself does for example).
(the context is > 100 test failures on scipy 0.14.x on top of numpy 1.9., because f2py intent(inout) fails on work arrays created by zeros, this is a windows-32 only failure).
David
On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor <jtaylor.debian@googlemail.com <mailto:jtaylor.debian@googlemail.com>> wrote:
On 18.11.2014 19:20, David Cournapeau wrote: > Hi, > > I have not followed closely the changes that happen in 1.9.1, but was > surprised by the following: > > x = np.zeros(12, "d") > assert x.flags.aligned # fails > > This is running numpy 1.9.1 built on windows with VS 2008. Is it > expected that zeros may return a non-aligned array ? >
what is the real alignment of the array? Are you on 32 bit or 64 bit? What is the alignment of doubles in windows (linux its 4 byte on 32 bit 8 byte on 64 bit (% special compiler flags)? print x.__array_interface__["data"]
there are problems with complex types but doubles should be aligned even on 32 bit. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
32 bit windows should not provide 16 byte alignment, at least it doesn't for me. That is typically a property of 64 bit OS. But that does not explain why normal double is not aligned for you, that only needs 4 bytes on i386 which even 32 bit OS should provide. I though I tested scipy on 32 bit linux which has very similar properties to win32, time to rerun the test. On 18.11.2014 19:56, David Cournapeau wrote:
On Tue, Nov 18, 2014 at 6:40 PM, Julian Taylor <jtaylor.debian@googlemail.com <mailto:jtaylor.debian@googlemail.com>> wrote:
< 1.9 lies about alignment it doesn't actually check for new arrays.
When I do the following on 1.8.1 with win 32 bits:
x = np.zeros(12, "D") print x.aligned.flags == (x.__array_interface__["data"][0] % 16 == 0) # always true print x.aligned.flags # always true
but on 1.9.1:
x = np.zeros(12, "D") print x.aligned.flags == (x.__array_interface__["data"][0] % 16 == 0) # always true print x.aligned.flags # not always true
I wonder why numpy 1.8.1 always returned 16 bytes aligned arrays in this case (unlikely to be a coincidence, as I try quite a few times with different sizes).
is the array aligned?
On 18.11.2014 19:37, David Cournapeau wrote: > Additional point: it seems to always return aligned data on 1.8.1 (same > platform/compiler/everything). > > On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com> > <mailto:cournape@gmail.com <mailto:cournape@gmail.com>>> wrote: > > It is on windows 32 bits, but I would need to make this work for > complex (pair of double) as well. > > Is this a bug (I assumed array creation methods would always create > aligned arrays for their type) ? Seems like quite a bit of code out > there would assume this (scipy itself does for example). > > (the context is > 100 test failures on scipy 0.14.x on top of numpy > 1.9., because f2py intent(inout) fails on work arrays created by > zeros, this is a windows-32 only failure). > > David > > On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor > <jtaylor.debian@googlemail.com <mailto:jtaylor.debian@googlemail.com> > <mailto:jtaylor.debian@googlemail.com <mailto:jtaylor.debian@googlemail.com>>> wrote: > > On 18.11.2014 19:20, David Cournapeau wrote: > > Hi, > > > > I have not followed closely the changes that happen in 1.9.1, > but was > > surprised by the following: > > > > x = np.zeros(12, "d") > > assert x.flags.aligned # fails > > > > This is running numpy 1.9.1 built on windows with VS 2008. Is it > > expected that zeros may return a non-aligned array ? > > > > what is the real alignment of the array? Are you on 32 bit or 64 > bit? > What is the alignment of doubles in windows (linux its 4 byte on > 32 bit > 8 byte on 64 bit (% special compiler flags)? > print x.__array_interface__["data"] > > there are problems with complex types but doubles should be > aligned even > on 32 bit. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> <mailto:NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Nov 18, 2014 at 7:05 PM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:
32 bit windows should not provide 16 byte alignment, at least it doesn't for me. That is typically a property of 64 bit OS.
But that does not explain why normal double is not aligned for you, that only needs 4 bytes on i386 which even 32 bit OS should provide.
Sorry for the confusion, doubles are aligned, only complex128 are not. But I see that on linux 32 bits, this is the same as on windows (zeros output not always aligned on "D" dtype), and yet I don't see the issues with f2py not being able to I am starting to think I got on the wrong track regarding the original issue I see with scipy 0.14.x on win 32 bits... I will create a separate ticket for that. David
18.11.2014, 21:44, David Cournapeau kirjoitti:
On Tue, Nov 18, 2014 at 7:05 PM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:
32 bit windows should not provide 16 byte alignment, at least it doesn't for me. That is typically a property of 64 bit OS.
But that does not explain why normal double is not aligned for you, that only needs 4 bytes on i386 which even 32 bit OS should provide.
Sorry for the confusion, doubles are aligned, only complex128 are not. But I see that on linux 32 bits, this is the same as on windows (zeros output not always aligned on "D" dtype), and yet I don't see the issues with f2py not being able to
The scipy ticket is here, btw: https://github.com/scipy/scipy/issues/4168 The second question is whether F2py actually *needs* to check the dtype-size alignment, or is just something like sizeof(double) enough for Fortran compilers. Fortran compilers however apparently do generate code that crashes and burns if there's no alignment also on x86: https://github.com/scipy/scipy/pull/2698 All this is probably unspecified, as it's just F77 up out there. Apparently, everything has worked OK with the old Numpy behavior, or at least, nobody managed to pinpoint a crash on Win32 because of this? Can the F2py alignment checks be relaxed? Maybe it is enough to assume the Fortran compiler is happy with whatever alignment the system malloc() assures? If not, the second option is a bit nasty, since I'd believe many people have f2py code out there with complex inout arrays, and I think no-one uses special aligned allocators... -- Pauli Virtanen
On 18/11/14 23:10, Pauli Virtanen wrote:
The second question is whether F2py actually *needs* to check the dtype-size alignment, or is just something like sizeof(double) enough for Fortran compilers.
The Fortran standard does not specify an ABI so this is likely compiler dependent. The only safe way of calling Fortran from C is through a proxy Fortran routine exported to C using Fortran 2003 ISO C bindings. That is what f2py should do behind the scenes now that the Fortran 77 limitation is lifted from SciPy. With this method we can just pass an arbitrary C pointer to Fortran and construct a Fortran array pointer with c_f_function. The Fortran compiler will make an aligned copy of the array referenced by the pointer if required. No special coding will be required to ensure correct alignment. Sturla
participants (5)
-
Charles R Harris -
David Cournapeau -
Julian Taylor -
Pauli Virtanen -
Sturla Molden