[Cython] Upcoming cython/numpy breakage with stride checking

[mark florisson]

On 9 April 2013 13:09, mark florisson <markflorisson88 at gmail.com> wrote:

>
>
>
> On 8 April 2013 13:04, Sebastian Berg <sebastian at sipsolutions.net> wrote:
>
>> On Mon, 2013-04-08 at 12:31 +0200, Dag Sverre Seljebotn wrote:
>> > On 04/08/2013 09:59 AM, Sebastian Berg wrote:
>> > > On Mon, 2013-04-08 at 08:42 +0200, Dag Sverre Seljebotn wrote:
>> > >> On 04/06/2013 04:19 PM, Nathaniel Smith wrote:
>> > >>> Hi all,
>> > >>>
>> > >>> If you build current numpy master with
>> > >>>     NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install
>> > >>> then Cython code using ndarrays starts blowing up, e.g.:
>> > >>>
>> > >>> # foo.pyx
>> > >>> def add_one(array):
>> > >>>       cdef double[::1] a = array
>> > >>>       a[0] += 1.
>> > >>>       return array
>> > >>>
>> > >>>>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100]))
>> > >>> Traceback (most recent call last):
>> > >>>     File "<stdin>", line 1, in <module>
>> > >>>     File "foo.pyx", line 2, in foo.add_one (foo.c:1210)
>> > >>>       cdef double[::1] a = array
>> > >>> ValueError: Buffer and memoryview are not contiguous in the same
>> dimension.
>> > >>>
>> > >>> The problem (as discussed before) is that Cython has an
>> unnecessarily
>> > >>> strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1
>> > >>> pretty much breaks all existing compiled Cython modules.
>> > >>>
>> <snip>
>> > >>
>> > >> I guess you have changed your implementation of PEP 3118 too
>> slightly on
>> > >> the NumPy side? Is this undefined in the PEP or are you now not in
>> > >> strict adherence to it?
>> > >>
>> > >
>> > > Hi,
>> > >
>> > > maybe I will have a look at that, but not sure if I will manage.  The
>> > > PEP 3118 is a point, but it does not seem to cover this (probably we
>> > > should try to get a clarification into 3118).  I am still wondering
>> > > whether for buffers that are requested contiguous numpy should set the
>> > > strides again, since it cannot hurt.  Would that make a difference for
>> > > Cython? I expected Cython just got any buffer and then checked the
>> > > strides instead of requesting the buffer contiguous and then double
>> > > checking.
>> >
>> > At least when I implemented
>> >
>> > cdef np.ndarray[double, mode='fortran'] arr
>> >
>> > that relies on PEP 3118 contiguous-flags and I did no checking myself.
>> > Lots of Cython code does this instead of memoryviews (I still write my
>> > own code that way).
>>
>> Yeah, though even if numpy "fixes" the buffer strides, `arr.strides`
>> points at the original array I believe, leaving it to the user. But that
>> just means users have to be a bit more careful with strides.
>>
>> I think I will have numpy "fix" the buffers stride anyway (when
>> requested contiguous through the buffer protocol).  I cannot think of
>> any reason not to do it and it may work around bugs in extensions which
>> may not even be aware of numpy's existence.
>>
>> >
>> > The memory views OTOH does their own checking, but I also see plenty of
>> > references to PyBUF_C_CONTIGUOUS etc. inside
>> > Cython/Utility/MemoryView.pyx, so perhaps it does both. Mark would have
>> > the definitive answer here.
>> >
>>
>> After a quick check, it seems to me that those are there for exposing
>> the memoryviews buffer and not for getting a buffer from another object.
>
>
> Yeah, I think most of the checking is in MemoryView_C.c, when obtaining
> the memoryview. I don't know how well buffer producers check these flags,
> but maybe we can simply remove some of those checks, or amend them
> otherwise. I'll try to look into it one of these days.
>

OTOH if someone else wants to take a crack at it, I'll be happy to review
patches as well.


>
>> - Sebastian
>>
>> > Dag Sverre
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > http://mail.python.org/mailman/listinfo/cython-devel
>> >
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20130409/6f7972a7/attachment-0001.html>