[Cython] NumPy test failures in Py2.4

[mark florisson]

On 24 March 2012 22:24, mark florisson <markflorisson88 at gmail.com> wrote:
> On 23 March 2012 19:40, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> after re-enabling the NumPy tests, I'm getting two NumPy (1.6.1) test
>> failure in Py2.4 (2.7 is ok):
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py24-ext/224/testReport/junit/doctest/DocTestCase/Doctest__numpy_test/
>>
>> """
>> File ".../run/c/numpy_test.so", line 207, in numpy_test
>> Failed example:
>>    print(test_partially_packed_align(np.zeros((1,), dtype=np.dtype([('a',
>> 'b'), ('b', 'i'), ('sub', np.dtype('b,i')), ('c', 'i')], align=True))))
>> Exception raised:
>>    Traceback (most recent call last):
>>      File "numpy_test.pyx", line 451, in
>> numpy_test.test_partially_packed_align (numpy_test.c:7967)
>>    ValueError: Buffer dtype mismatch; next field is at offset 13 but 16
>> expected
>> ----------------------------------------------------------------------
>> File ".../run/c/numpy_test.so", line 211, in numpy_test
>> Failed example:
>>    print(test_partially_packed_align_2(np.zeros((1,),
>> dtype=np.dtype([('a', 'b'), ('b', 'i'), ('c', 'b'), ('sub', np.dtype('b,i',
>> align=True))]))))
>> Exception raised:
>>      File "numpy_test.pyx", line 459, in
>> numpy_test.test_partially_packed_align_2 (numpy_test.c:8186)
>>    ValueError: Buffer dtype mismatch; next field is at offset 7 but 10
>> expected
>> """
>>
>> Does anyone know what to make of this?
>
> That's probably a regression from my changes to the buffer format
> parser. It should be aligning on the second integer in the substruct,
> for which the entire struct is aligned and the first character padded.
> I'll look into it, thanks.

No, there is no regression, the buffer format parser has always been
wrong in this regard. The tests have just always passed because the
buffer was obtained through numpy.pxd's __getbuffer__, which flattens
the structs. Since numpy supports the buffer interface now, it tries
to parse the actual format string but fails.

Basically the problem is that with aligned structs inside padded
structs the format parser needs to start aligning in isolation from
the actual offset. E.g. it still needs to isolate an int on a 4-byte
boundary, but in relation to the start of the aligned struct, which
may in itself not be aligned. This is not a regression so I'm leaving
it as is, as the fix is complicated and I'm rather preoccupied.
Basically I think the buffer format parser should be rewritten to use
a single stack (and so does Dag), which would make things a lot less
complicated, but that's not a trivial task.

>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel