Memory views: dereferencing pointer does break strict-aliasing rules
Hi, The simplest possible program using memory views compiles with a large number of warnings for me, even for a rather outdated version of gcc: def hello(int [:] a): print(a, "world") If I translate it with the latest released version of Cython like this: cython cpp.pyx cython --cplus cpp.pyx and compile like this: gcc -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.c -o cpp.o g++ -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.cpp -o cpp.o I get lots of warnings (see attached). It doesn't seem to be related to C++ as such, but rather it seems that the memory views code indeed somehow violates strict-aliasing rules. I'm not sure of how severe it is, but the documentation seems to suggest that this might even lead to incorrect results. Can this possibly be fixed in Cython and how important is that? Shall I create a bug report on the Trac? Is my only resort to test whether the compiler supports -fno-strict-aliasing and use that? Thanks! -- Sincerely yours, Yury V. Zaytsev
On Tue, Jul 2, 2013 at 4:54 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
Hi,
The simplest possible program using memory views compiles with a large number of warnings for me, even for a rather outdated version of gcc:
def hello(int [:] a): print(a, "world")
If I translate it with the latest released version of Cython like this:
cython cpp.pyx cython --cplus cpp.pyx
and compile like this:
gcc -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.c -o cpp.o g++ -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.cpp -o cpp.o
I get lots of warnings (see attached).
It doesn't seem to be related to C++ as such, but rather it seems that the memory views code indeed somehow violates strict-aliasing rules.
I'm not sure of how severe it is, but the documentation seems to suggest that this might even lead to incorrect results.
Can this possibly be fixed in Cython and how important is that? Shall I create a bug report on the Trac? Is my only resort to test whether the compiler supports -fno-strict-aliasing and use that?
You should compile with -fno-strict-aliasing--if you were using distutils rather than gcc directly it should add the all necessary flags for you. Aliasing different pointer types is necessary for Cython--it's how it implements inheritance (in plain C, a PyObject* could be a pointer to a list or dict or your own cdef class--pointer aliasing right there. Also with memory views (and numpy arrays), the underlying data is allocated as a char* and interpreted as a float* or int* or according to the metadata in the array. - Robert
Hi Robert, On Tue, 2013-07-02 at 09:07 -0700, Robert Bradshaw wrote:
You should compile with -fno-strict-aliasing--if you were using distutils rather than gcc directly it should add the all necessary flags for you.
Indeed, I'm using autotools to compile the module, so now I've added the AX_CHECK_COMPILE_FLAG macro from the Autoconf Archive to check whether -fno-strict-aliasing is supported by the compiler and append it to CXXFLAGS if necessary.
Aliasing different pointer types is necessary for Cython--it's how it implements inheritance (in plain C, a PyObject* could be a pointer to a list or dict or your own cdef class--pointer aliasing right there. Also with memory views (and numpy arrays), the underlying data is allocated as a char* and interpreted as a float* or int* or according to the metadata in the array.
Thank you very much for this explanation, I just wanted to make sure that this is unavoidable! -- Sincerely yours, Yury V. Zaytsev
On Tue, Jul 2, 2013 at 5:07 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
On Tue, Jul 2, 2013 at 4:54 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
Hi,
The simplest possible program using memory views compiles with a large number of warnings for me, even for a rather outdated version of gcc:
def hello(int [:] a): print(a, "world")
If I translate it with the latest released version of Cython like this:
cython cpp.pyx cython --cplus cpp.pyx
and compile like this:
gcc -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.c -o cpp.o g++ -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.cpp -o cpp.o
I get lots of warnings (see attached).
It doesn't seem to be related to C++ as such, but rather it seems that the memory views code indeed somehow violates strict-aliasing rules.
I'm not sure of how severe it is, but the documentation seems to suggest that this might even lead to incorrect results.
Can this possibly be fixed in Cython and how important is that? Shall I create a bug report on the Trac? Is my only resort to test whether the compiler supports -fno-strict-aliasing and use that?
You should compile with -fno-strict-aliasing--if you were using distutils rather than gcc directly it should add the all necessary flags for you.
Aliasing different pointer types is necessary for Cython--it's how it implements inheritance (in plain C, a PyObject* could be a pointer to a list or dict or your own cdef class--pointer aliasing right there. Also with memory views (and numpy arrays), the underlying data is allocated as a char* and interpreted as a float* or int* or according to the metadata in the array.
I think this is a relatively serious issue. Using pointer aliasing is not valid C, and we can usually get the same feature with union (which is usually supported). See http://stackoverflow.com/questions/11639947/is-type-punning-through-a-union-... I have been bitten by this recently: how difficult of a change would it be ? David
- Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
On Fri, Oct 11, 2013 at 2:15 AM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Jul 2, 2013 at 5:07 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
On Tue, Jul 2, 2013 at 4:54 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
Hi,
The simplest possible program using memory views compiles with a large number of warnings for me, even for a rather outdated version of gcc:
def hello(int [:] a): print(a, "world")
If I translate it with the latest released version of Cython like this:
cython cpp.pyx cython --cplus cpp.pyx
and compile like this:
gcc -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.c -o cpp.o g++ -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.cpp -o cpp.o
I get lots of warnings (see attached).
It doesn't seem to be related to C++ as such, but rather it seems that the memory views code indeed somehow violates strict-aliasing rules.
I'm not sure of how severe it is, but the documentation seems to suggest that this might even lead to incorrect results.
Can this possibly be fixed in Cython and how important is that? Shall I create a bug report on the Trac? Is my only resort to test whether the compiler supports -fno-strict-aliasing and use that?
You should compile with -fno-strict-aliasing--if you were using distutils rather than gcc directly it should add the all necessary flags for you.
Aliasing different pointer types is necessary for Cython--it's how it implements inheritance (in plain C, a PyObject* could be a pointer to a list or dict or your own cdef class--pointer aliasing right there. Also with memory views (and numpy arrays), the underlying data is allocated as a char* and interpreted as a float* or int* or according to the metadata in the array.
I think this is a relatively serious issue. Using pointer aliasing is not valid C, and we can usually get the same feature with union (which is usually supported). See http://stackoverflow.com/questions/11639947/is-type-punning-through-a-union-...
I have been bitten by this recently: how difficult of a change would it be ?
I don't think this change is even possible. NumPy has exactly the same issue, as will any library trying to store vectors of arbitrary data types (whose layout may even be defined at runtime) in C. The Python buffer API doesn't provide a way around it. Also, Python and Cython in general breaks pointer aliasing as objects are simultaneously generic PyObject* and PyListObject*, PyDictObject*, etc. As the set of possible types is large and open, unions won't work. This is how object oriented programming (with subclassing) is done in C. - Robert
Robert Bradshaw, 11.10.2013 19:10:
Python and Cython in general breaks pointer aliasing as objects are simultaneously generic PyObject* and PyListObject*, PyDictObject*, etc. As the set of possible types is large and open, unions won't work. This is how object oriented programming (with subclassing) is done in C.
This has been fixed in CPython 3.x. Stefan
On Fri, Oct 11, 2013 at 10:46 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Robert Bradshaw, 11.10.2013 19:10:
Python and Cython in general breaks pointer aliasing as objects are simultaneously generic PyObject* and PyListObject*, PyDictObject*, etc. As the set of possible types is large and open, unions won't work. This is how object oriented programming (with subclassing) is done in C.
This has been fixed in CPython 3.x.
Ah, I wasn't even aware of that. For the curious: http://www.python.org/dev/peps/pep-3123/ Basically, there is an exception for casting a struct pointer to a pointer of its first member. Looking into this more, there's also an exception for char* (aliasing is explicitly allowed), so I take back what I said about this not being possible (due to the looseness of "strict" aliasing in some cases) but I don't know how easy it'd be. - Robert
participants (4)
-
David Cournapeau -
Robert Bradshaw -
Stefan Behnel -
Yury V. Zaytsev