Mailman 3 HEADS UP: Compilation risk with new GCC 4.5.0 - Python-Dev

newer
configuring the buildbot to skip...

HEADS UP: Compilation risk with new GCC 4.5.0

Jesus Cea

12 May 2010 12 May '10

6:43 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 [Zlib-devel] HEADS UP: Apparent bad compilation under (just released) GCC 4.5.0 http://mail.madler.net/pipermail/zlib-devel_madler.net/2010-May/002267.html GCC Bugzilla Bug 40838 gcc shouldn't assume that the stack is aligned http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838 Short history: new GCC 4.5.0 (released a month ago), when compiling with - -O3, is adding MMX/SSE instructions that requires stack aligned to 16 byte. This is wrong, since x86 ABI only requires stack aligned to 4 bytes. If you compile EVERYTHING with GCC 4.5.0, you are safe (I guess!), but if your environment has mixed compiled code (for instance, the OS libraries), you can possibly "core dump". If you have an old compiled Python and you update libs compiled with GCC 4.5.0, you can crash in the process. Psyco is showing the issue, but it is not the culprit. It only leaves - -correctly- the stack in not 16-byte alignment. But there are plenty of examples of crashes not related to python+psyco. Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2. Warm 2.3/2.4/2.5/2.6/3.0/3.1 users. Or warm users compiling with GCC 4.5.0. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS+qpcZlgi5GaxT1NAQLAjgP/anZ5C2lED190++ffcluApF3ASF20iSnH t21ynHxrz2fgkPeOxeGRAqGEGCc3jZ0UuXchECcLLzFhI8WQS8K766EOgOKdwZLV WCt0ohWZ0FfsJZX4J5Y73x39uhjShbnl6SSek2uEvPi/tua/R4W/kVXrAZ0ZZR6/ vRqSUXpdolM= =5K+S -----END PGP SIGNATURE-----

Show replies by date

Nick Coghlan

12 May 12 May

7:02 p.m.

Jesus Cea wrote:

...

Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2.

Will this actually help? Won't there still be a problem if any extension module is compiled with GCC 4.5.0 without that option, regardless of the options used to build Python itself? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Jesus Cea

7:26 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/05/10 15:32, Nick Coghlan wrote:

...

Jesus Cea wrote:

...
Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2.

Will this actually help? Won't there still be a problem if any extension module is compiled with GCC 4.5.0 without that option, regardless of the options used to build Python itself?

When I do a "python setup.py install", the options used to compile the module are the same that used to compile python itself. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS+qzn5lgi5GaxT1NAQJ5BgQAimvcMssT57iuuLpaI9P9GXKGZf9VUAzj F4XJM/lT4Qtzu22jecIvEej0MyiMR/4qHvns8lgqLXn/30pkYrkmYcjxFigpM7Bl rOKMYeAr3ki8dN87APkoiMKOJHByXlEUZu+BowCOXcUCtGYcENLwQ+STr1NyCsUB IINfh1pJ2Nc= =GiUG -----END PGP SIGNATURE-----

James Y Knight

7:09 p.m.

On May 12, 2010, at 9:13 AM, Jesus Cea wrote:

...

Short history: new GCC 4.5.0 (released a month ago), when compiling with - -O3, is adding MMX/SSE instructions that requires stack aligned to 16 byte. This is wrong, since x86 ABI only requires stack aligned to 4 bytes.

If you compile EVERYTHING with GCC 4.5.0, you are safe (I guess!), but if your environment has mixed compiled code (for instance, the OS libraries), you can possibly "core dump". If you have an old compiled Python and you update libs compiled with GCC 4.5.0, you can crash in the process.

Psyco is showing the issue, but it is not the culprit. It only leaves - -correctly- the stack in not 16-byte alignment. But there are plenty of examples of crashes not related to python+psyco.

Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2. Warm 2.3/2.4/2.5/2.6/3.0/3.1 users. Or warm users compiling with GCC 4.5.0.

While assuming the stack is 16byte aligned is undeniably an ABI- violation in GCC, at this point, it's surely simpler to just go along: the new unofficial ABI for x86 is that the stack must always be left in 16-byte alignment... So, just change psyco to always use 16-byte-aligned stackframes. GCC has used 16byte-aligned stackframes for a heck of a long time now (so if the stack starts 16byte aligned on entry to a function it will stay that way on calls). So usually the only way people run into unaligned stacks is via hand-written assembly code or JIT compilers. I think you'll be a lot happier just modifying Psyco than making everyone else in the world change their compiler flags. James

Jesus Cea

7:31 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/05/10 15:39, James Y Knight wrote:

...

While assuming the stack is 16byte aligned is undeniably an ABI-violation in GCC, at this point, it's surely simpler to just go along: the new unofficial ABI for x86 is that the stack must always be left in 16-byte alignment...

You can not rule out other software embedding python inside, or callbacks from foreign code. For instance, Berkeley DB library can do callbacks to Python code.

...

So, just change psyco to always use 16-byte-aligned stackframes. GCC has used 16byte-aligned stackframes for a heck of a long time now (so if the stack starts 16byte aligned on entry to a function it will stay that way on calls). So usually the only way people run into unaligned stacks is via hand-written assembly code or JIT compilers.

Not all the universe is GCC based. For instance, Solaris system libraries are not compiled using GCC. The world is bigger that Linux/GCC.

...

I think you'll be a lot happier just modifying Psyco than making everyone else in the world change their compiler flags.

Would be nice if GCC 4.5.1 would solve this :). They are objectivelly breaking the x86 ABI. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS+q0rplgi5GaxT1NAQK98wP+NJoqNCpvjemP4Gv7y1G/iPkQgjuidslT uiPxDcN9Eprcluc+mGTBu6N+fCTj09xYhUCD1wWhoJq2dRyoA8b+XC1fCSyL4VXc mzsy0rGmKeQh4lyAw+7agFCqryd6n+/oyl+9aOT6YkzyLFjQd4KDEcGNZ0h+6PAf 4jtx1+p3k0c= =TzTY -----END PGP SIGNATURE-----

James Y Knight

7:57 p.m.

On May 12, 2010, at 10:01 AM, Jesus Cea wrote:

...

On 12/05/10 15:39, James Y Knight wrote:

...
While assuming the stack is 16byte aligned is undeniably an ABI-violation in GCC, at this point, it's surely simpler to just go along: the new unofficial ABI for x86 is that the stack must always be left in 16-byte alignment...

You can not rule out other software embedding python inside, or callbacks from foreign code. For instance, Berkeley DB library can do callbacks to Python code.

So? When calling callback functions, the Berkeley DB library won't un-16byte-align the stack, will it? (Assuming it's been compiled with gcc in the last 10 years)

...

Not all the universe is GCC based. For instance, Solaris system libraries are not compiled using GCC. The world is bigger that Linux/ GCC.

If the Solaris compilers don't use 16byte-aligned stackframes, and GCC on Solaris/x86 also assumes 16byte-aligned stacks, I guess GCC on Solaris/x86 is pretty broken indeed. But for Linux/x86, stacks have been de-facto 16byte aligned for so long, you can *almost* excuse the ABI violation as unimportant. But anyways, psyco should keep the stackframes 16byte aligned regardless, for performance reasons: even when accessing datatypes for which unaligned access doesn't crash, it's faster when it's aligned. James

Jeffrey Yasskin

9:28 p.m.

On Wed, May 12, 2010 at 6:39 AM, James Y Knight wrote:

...

I think you'll be a lot happier just modifying Psyco than making everyone else in the world change their compiler flags.

Aye, there's the rub. Nobody's happier modifying Psyco. :) But that just means people will gradually have to stop using psyco in favor of maintainable JITs. Gcc's not going to change its stack requirements just because some people think they know what the ABI should be better than the people defining the ABI. Btw, this has been a problem since at least gcc-4.4.

Maciej Fijalkowski

14 May 14 May

10:33 p.m.

On Wed, May 12, 2010 at 9:58 AM, Jeffrey Yasskin wrote:

...

On Wed, May 12, 2010 at 6:39 AM, James Y Knight wrote:

...
I think you'll be a lot happier just modifying Psyco than making everyone else in the world change their compiler flags.

Aye, there's the rub. Nobody's happier modifying Psyco. :) But that just means people will gradually have to stop using psyco in favor of maintainable JITs. Gcc's not going to change its stack requirements just because some people think they know what the ABI should be better than the people defining the ABI. Btw, this has been a problem since at least gcc-4.4.

Heh. psyco already does it for Mac OS X (which defines 16 byte stack alignment as ABI), so should be super trivial. Good rant Jeffrey though. Cheers, fijal

"Martin v. Löwis"

12 May 12 May

8:44 p.m.

...

Short history: new GCC 4.5.0 (released a month ago), when compiling with -O3, is adding MMX/SSE instructions that requires stack aligned to 16 byte. This is wrong, since x86 ABI only requires stack aligned to 4 bytes.

I think this is debatable. It depends on the operating system also; ultimately, it is the OS vendor who specifies the C ABI for their systems. On Linux, in absence of a vendor, the ABI is what the kernel and gcc define it to be. Regards, Martin

5089

Age (days ago)

5091

Last active (days ago)

List overview

Download

8 comments

6 participants

participants (6)

"Martin v. Löwis"
James Y Knight
Jeffrey Yasskin
Jesus Cea
Maciej Fijalkowski
Nick Coghlan

HEADS UP: Compilation risk with new GCC 4.5.0

tags

participants (6)