[Cython] Upcoming cython/numpy breakage with stride checking

[Dag Sverre Seljebotn]

On 04/06/2013 04:19 PM, Nathaniel Smith wrote:
> Hi all,
>
> If you build current numpy master with
>    NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install
> then Cython code using ndarrays starts blowing up, e.g.:
>
> # foo.pyx
> def add_one(array):
>      cdef double[::1] a = array
>      a[0] += 1.
>      return array
>
>>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100]))
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "foo.pyx", line 2, in foo.add_one (foo.c:1210)
>      cdef double[::1] a = array
> ValueError: Buffer and memoryview are not contiguous in the same dimension.
>
> The problem (as discussed before) is that Cython has an unnecessarily
> strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1
> pretty much breaks all existing compiled Cython modules.
>
> Our plan is to make NPY_RELAXED_STRIDES_CHECKING=1 into the default
> sooner or later, and Cython is a major blocker on this plan. It may
> become the default as soon as the 1.8 pre-releases (with the
> expectation that we'll probably have to switch back again before the
> actual release, but still).
>
> References:
>
> Previous thread:
>    http://thread.gmane.org/gmane.comp.python.cython.devel/14634
> Detailed discussion of the difference between numpy/cython's current
> definition of "contiguity", and the correct definition:
>    http://thread.gmane.org/gmane.comp.python.cython.devel/14634/focus=14640
> The PR implementing NPY_RELAXED_STRIDES_CHECKING:
>    https://github.com/numpy/numpy/pull/3162
> Another test case:
>    https://github.com/numpy/numpy/issues/2956
>
> We're hoping that Cython will also switch soon to the more accurate
> check for contiguity. This shouldn't cause any backwards compatibility
> problems -- it just means Cython code would make strictly fewer
> copies, and error out due to lack of contiguity strictly less often,
> even with older numpys. And it seems like a necessary step for getting
> this untangled and minimizing user pain. What do you think?

I agree that we should follow NumPy here, but can't see myself having 
time to do the change in near future. I don't know about Mark?

I think it is a fairly simple change though if any NumPyers would like 
to do it, look at in Cython/Utility/MemoryView_C.c in the function

_pyx_memviewslice_is_contig

looks like it should just be to add a check for shape too.

I guess you have changed your implementation of PEP 3118 too slightly on 
the NumPy side? Is this undefined in the PEP or are you now not in 
strict adherence to it?

Dag Sverre