Hi all, I was hacking to get mingw32 builds of psyco to work and found a pretty weird thing: I used mingw32 (stable distro) to build the psyco extension on top of standard python2.6, built with Visual Studio, and got weird crashes. The reason is in objimpl.h: typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; long double dummy; /* force worst-case alignment */ } PyGC_Head; Mingw32 behaves funny here. The size of long double is 12 ! 8-) I happened to use the _PyObject_GC_UNTRACK macro in psyco, instead of the function, and that caused the errors, because psython and the extension disagreed on the location of the gc pointers... Using PyObject_GC_Untrack instead fixed the problem, but there is a bad feeling left. Question: Is that considered a mingw bug? Should we change the above union to a safer construct? Or maybe I just missed something obvious and made a fool out of me? cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Hi,
I used mingw32 (stable distro) to build the psyco extension on top of standard python2.6, built with Visual Studio, and got weird crashes.
The reason is in objimpl.h:
typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; long double dummy; /* force worst-case alignment */ } PyGC_Head;
Mingw32 behaves funny here. The size of long double is 12 !
The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees with Visual Studio on some ABI subtleties (is it expected? is mingw supposed to be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)). Regards Antoine.
On Thu, Jul 23, 2009 at 4:40 AM, Antoine Pitrou<solipsis@pitrou.net> wrote:
The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees with Visual Studio
Yes, mingw and VS do not have the same long double type. This has been the source of some problems in numpy as well, since mingw uses the MS runtime, and everything involving long double and the runtime is broken (printf, math library calls). I wish there was a way to disable this in mingw, but there isn't AFAIK.
on some ABI subtleties (is it expected? is mingw supposed to be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)).
I think mostly ABI compatible is the best description :) David
On 7/22/09 5:17 PM, David Cournapeau wrote:
On Thu, Jul 23, 2009 at 4:40 AM, Antoine Pitrou<solipsis@pitrou.net> wrote:
The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees with Visual Studio
Yes, mingw and VS do not have the same long double type. This has been the source of some problems in numpy as well, since mingw uses the MS runtime, and everything involving long double and the runtime is broken (printf, math library calls). I wish there was a way to disable this in mingw, but there isn't AFAIK.
on some ABI subtleties (is it expected? is mingw supposed to be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)).
I think mostly ABI compatible is the best description :)
Well, the source of my problems is simply that I tried to build an extension for a Visual Studio built Python, using mingw. Did that, because it seems to be common practice so far. Maybe the simple solution is to prevent building extensions with mingw, if the python executable was not also built with it? Then, all would be fine I guess. Despite the fact that Python probably has to be changed: If it is true then all the 32-bit Linux Pythons have a 12 byte GC head, IOW they are *all* badly aligned. If that matters, of course. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer <tismer <at> stackless.com> writes:
Despite the fact that Python probably has to be changed: If it is true then all the 32-bit Linux Pythons have a 12 byte GC head, IOW they are *all* badly aligned.
Why are they badly aligned? The fact that long double is 12 bytes long doesn't mean it will force a 12-byte alignment - just whatever alignment is enough for a long double on the target machine. This could be 4, 8 or 16 bytes.
On 7/23/09 2:04 AM, Antoine Pitrou wrote:
Christian Tismer<tismer<at> stackless.com> writes:
Despite the fact that Python probably has to be changed: If it is true then all the 32-bit Linux Pythons have a 12 byte GC head, IOW they are *all* badly aligned.
Why are they badly aligned? The fact that long double is 12 bytes long doesn't mean it will force a 12-byte alignment - just whatever alignment is enough for a long double on the target machine. This could be 4, 8 or 16 bytes.
Things are a bit different: Alignment is not the primary concern of the gc header structure. Note that all the objects are created by malloc (system or python's arena allocator), and therefore all objects are correctly aligned by construction. The point is: The GC header is a structure invisible to the "real" gc allocated objects. It is opaquely prepended to every gc aware object. Therefore, it *needs* to have the correct size, in order to propagate its (already correct) alignment to the real object. It appears that python, compiled with gcc for all x64 32bit Linuxen (and mingw32) produces a 12 byte GC header. Not relevant for the header itself, but all GC objects are misaligned. This may not be recognized so far, because there is no builtin GC-enabled type that embeds a double. But if you create such a type, then the double will be correctly aligned in your object's structure, but then, when the object gets allocated, it is misaligned, because the header size is not a multiple of 8. To Martin: So I disagree. The gc header is not responsible for alignment in the first place, but to propagate it, correctly. And this fails miserably (in principle) since years. Proposal: We should use a simple construct that makes the gc header size simply a multiple of 8 or 16, whatever needed. Even a byte array would be ok. But please no long double :-) cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer <tismer <at> stackless.com> writes:
The point is: The GC header is a structure invisible to the "real" gc allocated objects. It is opaquely prepended to every gc aware object. Therefore, it *needs* to have the correct size, in order to propagate its (already correct) alignment to the real object.
Indeed.
This may not be recognized so far, because there is no builtin GC-enabled type that embeds a double. But if you create such a type, then the double will be correctly aligned in your object's structure, but then, when the object gets allocated, it is misaligned, because the header size is not a multiple of 8.
I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU. Alignment is primarily important to avoid access violations on CPUs and datatypes which don't support arbitrary alignment, although it can also be useful for performance reasons. In any case, you seem to be right on this particular point: the PyGC_Head union should probably contain a "double" alternative in addition to the "long double" (and perhaps even a "long long" one). Of course, it will also make memory consumption a tad bigger for GC-enabled objects (but GC-enabled objects are generally not that small anyway). (I disagree, however, that we should remove the "long double". After all, we also want alignment of PyObjects to allow inclusion of a "long double" in them) Feel free to propose a patch on bugs.python.org, by the way. Regards Antoine.
Antoine Pitrou <solipsis <at> pitrou.net> writes:
In any case, you seem to be right on this particular point: the PyGC_Head union should probably contain a "double" alternative in addition to the "long double" (and perhaps even a "long long" one).
Sorry, I realize that this doesn't really address the point. In addition to that union, we should also have a particular mechanism to compute what the proper offset should be between the PyGC_Head and the PyObject. Probably something like: typedef struct { PyGC_Head head; union { /* ... similar union as in PyGC_head */ } body; } _PyGC_dummy; #define _PyGC_Head_OFFSET offsetof(_PyGC_dummy, body) #define _Py_AS_GC(o) ((PyGC_Head *) ((void *)(o) - _PyGC_Head_OFFSET)) Regards Antoine.
Antoine Pitrou <solipsis <at> pitrou.net> writes:
In any case, you seem to be right on this particular point: the PyGC_Head union should probably contain a "double" alternative in addition to the "long double" (and perhaps even a "long long" one).
Sorry, I realize that this doesn't really address the point.
I don't realize that. Why is your first proposal bad?
In addition to that union, we should also have a particular mechanism to compute what the proper offset should be between the PyGC_Head and the PyObject.
Why is that difficult? It's sizeof(PyGC_Head). Regards, Martin
Martin v. Löwis <martin <at> v.loewis.de> writes:
In addition to that union, we should also have a particular mechanism to
compute
what the proper offset should be between the PyGC_Head and the PyObject.
Why is that difficult? It's sizeof(PyGC_Head).
Is it enough to correctly propagate the alignment for, say, a long double in the following PyObject? I'm sorry, I'm not enough of a C lawyer. Regards Antoine.
Why is that difficult? It's sizeof(PyGC_Head).
Is it enough to correctly propagate the alignment for, say, a long double in the following PyObject? I'm sorry, I'm not enough of a C lawyer.
Yes, sizeof must always be a multiple of the alignment. Otherwise, array elements would be misaligned. Regards, Martin
On 7/23/09 2:27 PM, Antoine Pitrou wrote:
Christian Tismer<tismer<at> stackless.com> writes: ...
I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU.
I'm also not sure. Anyway, the result was neither intended nor expected, I guess.
Alignment is primarily important to avoid access violations on CPUs and datatypes which don't support arbitrary alignment, although it can also be useful for performance reasons.
Performance, performance, of course (that's my job, after all :-) ) ...
Of course, it will also make memory consumption a tad bigger for GC-enabled objects (but GC-enabled objects are generally not that small anyway).
For that reason, I don't like the addition of the opaque header too much. If there were an option to make the header explicit, we would not have to round up the object size to a multiple of (8, 16), but could arrange embedded doubles as they fit the best.
(I disagree, however, that we should remove the "long double". After all, we also want alignment of PyObjects to allow inclusion of a "long double" in them)
Well, I doubt that a 12 byte long double causes any other alignment but 4. -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Antoine Pitrou <solipsis <at> pitrou.net> writes:
Roumen Petrov <bugtrack <at> roumenpetrov.info> writes:
Antoine Pitrou wrote:
In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!).
And alignment is ?
16 bytes as well.
This is probably why, by the way, a PyGC_Head takes only 12 bytes in 32-bit mode, but 32 bytes in 64-bit mode.
Antoine Pitrou wrote:
Antoine Pitrou <solipsis <at> pitrou.net> writes:
Roumen Petrov <bugtrack <at> roumenpetrov.info> writes:
Antoine Pitrou wrote:
In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!). And alignment is ? 16 bytes as well.
This is probably why, by the way, a PyGC_Head takes only 12 bytes in 32-bit mode, but 32 bytes in 64-bit mode.
yes, 10x Roumen
For that reason, I don't like the addition of the opaque header too much. If there were an option to make the header explicit, we would not have to round up the object size to a multiple of (8, 16), but could arrange embedded doubles as they fit the best.
We could add the gc head *after* the object head. However, this will require manual changes to all types that support GC, so we can't do that before Python 4.
(I disagree, however, that we should remove the "long double". After all, we also want alignment of PyObjects to allow inclusion of a "long double" in them)
Well, I doubt that a 12 byte long double causes any other alignment but 4.
What you apparently fail to understand is this: On some platforms, sizeof(long double) == 16, and alignof(long double) == 16. It is these platforms why we need to keep the long double. Regards, Martin
This may not be recognized so far, because there is no builtin GC-enabled type that embeds a double. But if you create such a type, then the double will be correctly aligned in your object's structure, but then, when the object gets allocated, it is misaligned, because the header size is not a multiple of 8.
I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU.
That's not really the point. The question is whether the double is misaligned wrt. the compiler's ABI. The compiler (or the operating system) specifies the alignment for all primitive types, and guarantees that everything is properly aligned.
Alignment is primarily important to avoid access violations on CPUs and datatypes which don't support arbitrary alignment, although it can also be useful for performance reasons.
Alignment may also have propagated into other aspects. For example, if you have double foo; intptr_t bar = (intptr_t)&foo; then it may be guaranteed (by the ABI) that bar&7 == 0. So we really need to keep all promises wrt. alignment intact.
In any case, you seem to be right on this particular point: the PyGC_Head union should probably contain a "double" alternative in addition to the "long double" (and perhaps even a "long long" one).
I agree. Regards, Martin
To Martin: So I disagree. The gc header is not responsible for alignment in the first place, but to propagate it, correctly.
I think we are in agreement in this respect. That's the whole point of the long double: to make the gc head's alignment the maximum alignment on that platform
And this fails miserably (in principle) since years.
Here I agree. In principle, it works fine - it is just perhaps incomplete.
Proposal: We should use a simple construct that makes the gc header size simply a multiple of 8 or 16, whatever needed. Even a byte array would be ok.
How would you implement that? In particular, how do you determine what is needed?
But please no long double :-)
Well, the long double needs to stay, as for some platforms, long double is indeed the type with the maximum alignment. I propose to add another (regular) double into the union. Regards, Martin
Martin v. Löwis:
I propose to add another (regular) double into the union.
Adding a regular double as a second dummy gives the same sizes and alignments with Mingw or MSVC as the original definition with MSVC: typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; long double dummy; /* force worst-case alignment */ double dummy2; /* in case long double doesn't trigger worst-case */ } PyGC_Head; In regard to alignment penalties, a simple copy loop for doubles runs about 20% slower when misaligned on an my AMD processor. Other x86 processors can be much worse. As much as 2 to 3.25 times according to http://msdn.microsoft.com/en-us/library/aa290049%28VS.71%29.aspx Neil
Neil Hodgson <nyamatongwe <at> gmail.com> writes:
typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; long double dummy; /* force worst-case alignment */ double dummy2; /* in case long double doesn't trigger worst-case */ } PyGC_Head;
We could add a PY_LONG_LONG to the mix just in case. By the way, will this kind of thing be frozen by the PEP ABI? Regards Antoine.
We could add a PY_LONG_LONG to the mix just in case. By the way, will this kind of thing be frozen by the PEP ABI?
Yes: alignof(PyGC_HEAD) would be specified as being the maximum alignment on a platform; sizeof(PyGC_HEAD) would be frozen. The actual content would not be frozen, i.e. it could be changed in principle as long as the change would be guaranteed not to affect the structure size. Regards, Martin
Martin v. Löwis:
Yes: alignof(PyGC_HEAD) would be specified as being the maximum alignment on a platform; sizeof(PyGC_HEAD) would be frozen.
Maximum alignment currently on x86 is 16 bytes for SSE vector types. Next year AVX will add 32 byte types and while they are supposed to work OK with 16 byte alignment, performance will be better with 32 byte alignment. It is possible that some use could be found for vector instructions in core Python but it is more likely that they will only be used in specialized extensions that can take care of alignment issues for their own cases. http://en.wikipedia.org/wiki/Advanced_Vector_Extensions http://software.intel.com/en-us/forums/intel-avx-and-cpu-instructions/topic/... Neil
Maximum alignment currently on x86 is 16 bytes for SSE vector types. Next year AVX will add 32 byte types and while they are supposed to work OK with 16 byte alignment, performance will be better with 32 byte alignment.
That doesn't really matter. What matters is the platform's ABI, i.e. what the maximum alignment is that the compiler gives, and what the alignment of a malloc() result is. That is unlikely to change even if x86 evolves. Regards, Martin
On 7/25/09 7:11 AM, Neil Hodgson wrote:
Martin v. Löwis:
I propose to add another (regular) double into the union.
Adding a regular double as a second dummy gives the same sizes and alignments with Mingw or MSVC as the original definition with MSVC:
Great (checked that, too) This makes pretty much sense, also given that this waste happens under Windows, already.
In regard to alignment penalties, a simple copy loop for doubles runs about 20% slower when misaligned on an my AMD processor. Other x86 processors can be much worse. As much as 2 to 3.25 times according to
I am still unhappy with this waste of memory, just because the GC header has to be rounded up, regardlwss wether that is needed or not. We should keep Martin's hint in mind, that Python 4 could place the gc header at the end of structures, instead. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On 7/27/09 12:48 AM, Christian Heimes wrote:
Christian Tismer wrote:
We should keep Martin's hint in mind, that Python 4 could place the gc header at the end of structures, instead.
Wow, 3.1 just came out and we already have the first PEP for Python 4k? :)
Christian
Maybe it's even possible right now, with a couple of tweaks that allow to explicitly say where the header is, for a specific extension type. Might cost a few cycles, tho. Christian -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
2009/7/22 Christian Tismer <tismer@stackless.com>:
Maybe the simple solution is to prevent building extensions with mingw, if the python executable was not also built with it? Then, all would be fine I guess.
I have never had problems in practice with extensions built with mingw rather than MSVC - so while I'm not saying that the issue doesn't exist, it certainly doesn't affect all extensions, so disabling mingw support seems a bit of an extreme measure. Having a preprocessor check in the Python headers to check if this bug is triggered in the actual code, and raising a compile error in that case, may be an option (but I expect that ensuring such a check is precise enough would be hard). Now that VC Express is freely available, the importance of mingw support is less, but AFAIK many developers cross-compile Windows binaries on Linux using mingw, or develop on gcc and use mingw to build on Windows (or Wine) but wouldn't install VC. So removing mingw support in cases where it does work would cause definite issues for Windows users relying on prebuilt binaries. Paul.
Paul Moore wrote:
2009/7/22 Christian Tismer <tismer@stackless.com>:
Maybe the simple solution is to prevent building extensions with mingw, if the python executable was not also built with it? Then, all would be fine I guess.
I have never had problems in practice with extensions built with mingw rather than MSVC - so while I'm not saying that the issue doesn't exist, it certainly doesn't affect all extensions, so disabling mingw support seems a bit of an extreme measure.
For runtime compatibility purposes, mingw is no worse than using versions of MSVC other than the one that was used to build Python (i.e. primarily making sure you use the correct memory management APIs and not trying to call the FILE* APIs). If there are other things that cause mingw to get structure sizes wrong then I don't see a major problem with a config check or some #ifdef'ery to work around it. A patch added to a tracker item would be a necessary starting point for that idea though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Thu, Jul 23, 2009 at 6:49 PM, Paul Moore<p.f.moore@gmail.com> wrote:
2009/7/22 Christian Tismer <tismer@stackless.com>:
Maybe the simple solution is to prevent building extensions with mingw, if the python executable was not also built with it? Then, all would be fine I guess.
I have never had problems in practice with extensions built with mingw rather than MSVC - so while I'm not saying that the issue doesn't exist, it certainly doesn't affect all extensions, so disabling mingw support seems a bit of an extreme measure.
I am strongly against this as well. We build numpy with mingw on windows, and disabling it would make my life even more miserable on windows. One constant source of pain with MS compilers is when supporting different versions of python - 2.4, 2.5 and 2.6 require a different VS version (and free versions are available only for the last version of VS usually). I am far from a windows specialist, but I understand that quite a few problems with mingw-built extensions with python are caused by some Python decisions as well (the C API with runtime-dependent structures like FILE, etc...). So mingw is not the only to blame :) David
2009/7/23 David Cournapeau <cournape@gmail.com>:
On Thu, Jul 23, 2009 at 6:49 PM, Paul Moore<p.f.moore@gmail.com> wrote:
2009/7/22 Christian Tismer <tismer@stackless.com>:
Maybe the simple solution is to prevent building extensions with mingw, if the python executable was not also built with it? Then, all would be fine I guess.
I have never had problems in practice with extensions built with mingw rather than MSVC - so while I'm not saying that the issue doesn't exist, it certainly doesn't affect all extensions, so disabling mingw support seems a bit of an extreme measure.
I am strongly against this as well. We build numpy with mingw on windows, and disabling it would make my life even more miserable on windows. One constant source of pain with MS compilers is when supporting different versions of python - 2.4, 2.5 and 2.6 require a different VS version (and free versions are available only for the last version of VS usually).
I am far from a windows specialist, but I understand that quite a few problems with mingw-built extensions with python are caused by some Python decisions as well (the C API with runtime-dependent structures like FILE, etc...). So mingw is not the only to blame :)
In this case, the OP tries to use an API that is explicitly documented as dangerous for extension modules. The recommended function is not sensitive to compiler differences. -- Amaury Forgeot d'Arc
Is that considered a mingw bug?
Perhaps. I can't test right now: *if* VS has a long double type that is smaller, it is apparently an ABI violation for mingw to not follow VS. It appears that other people also consider it a bug: http://www.velocityreviews.com/forums/showpost.php?p=1514923&postcount=2 OTOH, they probably cannot change it without causing
Should we change the above union to a safer construct?
No. tim_one changed it to be long double in r25454 to support some system that Dave Abrahams uses, so it needs to stay that way :-) However, we can certainly acknowledge that this is a bug in MingW, and special case it. Either introduce a symbolic type gchead_align_t which gets defined to just double on MingW, or put the #ifdef right into the structure. It might also be useful to assert that sizeof(gchead_align_t) is 8 or 16, and reject 12 as a value. The point is that we need the maximum alignment, and that certainly shouldn't be 12. Regards, Martin
Martin v. Löwis wrote: [SNIP]
No. tim_one changed it to be long double in r25454 to support some system that Dave Abrahams uses, so it needs to stay that way :-)
However, we can certainly acknowledge that this is a bug in MingW, and special case it. Either introduce a symbolic type gchead_align_t which gets defined to just double on MingW, or put the #ifdef right into the structure.
No this is not GCC bug. GCC support "hardware extended precision" as implement long double and mingw w32api implement long double functions. According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is MSVC limitation.
It might also be useful to assert that sizeof(gchead_align_t) is 8 or 16, and reject 12 as a value. The point is that we need the maximum alignment, and that certainly shouldn't be 12.
So look like python bug.
Regards, Martin
Regards, Roumen
On 7/22/09 4:56 PM, Roumen Petrov wrote:
Martin v. Löwis wrote: [SNIP]
No. tim_one changed it to be long double in r25454 to support some system that Dave Abrahams uses, so it needs to stay that way :-)
However, we can certainly acknowledge that this is a bug in MingW, and special case it. Either introduce a symbolic type gchead_align_t which gets defined to just double on MingW, or put the #ifdef right into the structure.
No this is not GCC bug. GCC support "hardware extended precision" as implement long double and mingw w32api implement long double functions. According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is MSVC limitation.
It might also be useful to assert that sizeof(gchead_align_t) is 8 or 16, and reject 12 as a value. The point is that we need the maximum alignment, and that certainly shouldn't be 12.
So look like python bug.
The assumption is that the union with long double gives alignment to the largest possible integral type with a power of 2 size, which is then wrong, because of the unexpected size. What do you propose for doing proper alignment, then? I fear this needs to become yet another special case in pyconfig.h cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer wrote:
On 7/22/09 4:56 PM, Roumen Petrov wrote:
Martin v. Löwis wrote: [SNIP]
No. tim_one changed it to be long double in r25454 to support some system that Dave Abrahams uses, so it needs to stay that way :-)
However, we can certainly acknowledge that this is a bug in MingW, and special case it. Either introduce a symbolic type gchead_align_t which gets defined to just double on MingW, or put the #ifdef right into the structure.
No this is not GCC bug. GCC support "hardware extended precision" as implement long double and mingw w32api implement long double functions. According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is MSVC limitation.
It might also be useful to assert that sizeof(gchead_align_t) is 8 or 16, and reject 12 as a value. The point is that we need the maximum alignment, and that certainly shouldn't be 12.
So look like python bug.
The assumption is that the union with long double gives alignment to the largest possible integral type with a power of 2 size, which is then wrong, because of the unexpected size.
As is posted for GCC on linux(32-bit) size of structure is 12 = sizeof(struct in union) = sizeof(pointer)+sizeof(pointer)+sizeof(ssize_t). And sizeof(ssize_t) <= sizeof(long) <= sizeof(pointer) .
What do you propose for doing proper alignment, then?
May be "void* dummy[4]" is better for force alignment ? What about alignment on platforms with pointers > 32 bit ?
I fear this needs to become yet another special case in pyconfig.h
cheers - chris
Regards, Roumen
What do you propose for doing proper alignment, then?
May be "void* dummy[4]" is better for force alignment ?
That would have the same alignment as a single pointer.
What about alignment on platforms with pointers > 32 bit ?
The C compiler will choose the alignment of the union to be the alignment of the union branch that requires that largest alignment. So if the largest alignment is the one that a pointer has, then alignof(PyGC_HEAD) == alignof(gc_next). The double is in the union only in case it has larger alignment than a pointer. Regards, Martin
Roumen Petrov wrote:
Martin v. Löwis wrote: [SNIP]
No. tim_one changed it to be long double in r25454 to support some system that Dave Abrahams uses, so it needs to stay that way :-)
However, we can certainly acknowledge that this is a bug in MingW, and special case it. Either introduce a symbolic type gchead_align_t which gets defined to just double on MingW, or put the #ifdef right into the structure.
No this is not GCC bug. GCC support "hardware extended precision" as implement long double and mingw w32api implement long double functions.
And that exactly is the bug. GCC cannot simultaneously be Windows-compatible, and also support extended precision floating point under the name "long double".
According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is MSVC limitation.
Therefore, mingw should also implement that limitation, to be binary-compatible with MSVC. Since binary compatibility is an explicit goal of mingw, failing to provide binary compatibility is a bug. Regards, Martin
Christian Tismer wrote:
Hi all,
I was hacking to get mingw32 builds of psyco to work and found a pretty weird thing:
I used mingw32 (stable distro) to build the psyco extension on top of standard python2.6, built with Visual Studio, and got weird crashes.
The reason is in objimpl.h:
typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; long double dummy; /* force worst-case alignment */ } PyGC_Head;
Mingw32 behaves funny here. The size of long double is 12 ! 8-)
No it is correct. GNU C compiler for windows support 80-bit extended precision.
I happened to use the _PyObject_GC_UNTRACK macro in psyco, instead of the function, and that caused the errors, because psython and the extension disagreed on the location of the gc pointers... Using PyObject_GC_Untrack instead fixed the problem, but there is a bad feeling left.
Question: Is that considered a mingw bug?
No as sizeof(long double) = sizeof(double) is MSVC limitation.
Should we change the above union to a safer construct?
Or maybe I just missed something obvious and made a fool out of me?
cheers - chris
Regards, Roumen
Christian Tismer wrote:
Hi all,
I was hacking to get mingw32 builds of psyco to work and found a pretty weird thing:
I used mingw32 (stable distro) to build the psyco extension on top of standard python2.6, built with Visual Studio, and got weird crashes.
The reason is in objimpl.h:
typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; long double dummy; /* force worst-case alignment */ } PyGC_Head; [SNIP]
Did the crash disappear is you add "__attribute__((aligned(8)))" after variable dummy ? Regards, Roumen
On 7/24/09 5:16 AM, Roumen Petrov wrote:
Christian Tismer wrote: ... Did the crash disappear is you add "__attribute__((aligned(8)))" after variable dummy ?
Did not try. But the proposed addition of a double does it, see the dev list. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
participants (10)
-
"Martin v. Löwis" -
Amaury Forgeot d'Arc -
Antoine Pitrou -
Christian Heimes -
Christian Tismer -
David Cournapeau -
Neil Hodgson -
Nick Coghlan -
Paul Moore -
Roumen Petrov