[Cython] Upcoming cython/numpy breakage with stride checking

[Sebastian Berg]

On Mon, 2013-04-08 at 08:42 +0200, Dag Sverre Seljebotn wrote:
> On 04/06/2013 04:19 PM, Nathaniel Smith wrote:
> > Hi all,
> >
> > If you build current numpy master with
> >    NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install
> > then Cython code using ndarrays starts blowing up, e.g.:
> >
> > # foo.pyx
> > def add_one(array):
> >      cdef double[::1] a = array
> >      a[0] += 1.
> >      return array
> >
> >>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100]))
> > Traceback (most recent call last):
> >    File "<stdin>", line 1, in <module>
> >    File "foo.pyx", line 2, in foo.add_one (foo.c:1210)
> >      cdef double[::1] a = array
> > ValueError: Buffer and memoryview are not contiguous in the same dimension.
> >
> > The problem (as discussed before) is that Cython has an unnecessarily
> > strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1
> > pretty much breaks all existing compiled Cython modules.
> >
> > Our plan is to make NPY_RELAXED_STRIDES_CHECKING=1 into the default
> > sooner or later, and Cython is a major blocker on this plan. It may
> > become the default as soon as the 1.8 pre-releases (with the
> > expectation that we'll probably have to switch back again before the
> > actual release, but still).
> >
> > References:
> >
> > Previous thread:
> >    http://thread.gmane.org/gmane.comp.python.cython.devel/14634
> > Detailed discussion of the difference between numpy/cython's current
> > definition of "contiguity", and the correct definition:
> >    http://thread.gmane.org/gmane.comp.python.cython.devel/14634/focus=14640
> > The PR implementing NPY_RELAXED_STRIDES_CHECKING:
> >    https://github.com/numpy/numpy/pull/3162
> > Another test case:
> >    https://github.com/numpy/numpy/issues/2956
> >
> > We're hoping that Cython will also switch soon to the more accurate
> > check for contiguity. This shouldn't cause any backwards compatibility
> > problems -- it just means Cython code would make strictly fewer
> > copies, and error out due to lack of contiguity strictly less often,
> > even with older numpys. And it seems like a necessary step for getting
> > this untangled and minimizing user pain. What do you think?
> 
> I agree that we should follow NumPy here, but can't see myself having 
> time to do the change in near future. I don't know about Mark?
> 
> I think it is a fairly simple change though if any NumPyers would like 
> to do it, look at in Cython/Utility/MemoryView_C.c in the function
> 
> _pyx_memviewslice_is_contig
> 
> looks like it should just be to add a check for shape too.
> 
> I guess you have changed your implementation of PEP 3118 too slightly on 
> the NumPy side? Is this undefined in the PEP or are you now not in 
> strict adherence to it?
> 

Hi,

maybe I will have a look at that, but not sure if I will manage.  The
PEP 3118 is a point, but it does not seem to cover this (probably we
should try to get a clarification into 3118).  I am still wondering
whether for buffers that are requested contiguous numpy should set the
strides again, since it cannot hurt.  Would that make a difference for
Cython? I expected Cython just got any buffer and then checked the
strides instead of requesting the buffer contiguous and then double
checking.

Regards,

Sebastian

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>