-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
A Solaris installation contains ALWAYS 32 and 64 bits libraries. So in any Solaris you can run 32/64 bits programs, and compile in 32 and 64 bits.
For this, libraries are stores in "/usr/lib", for instance, for 32 bits, while the same 64 bits libraries are stored in "/usr/lib/64".
Currently, python do not considerate this.
We have Solaris 10 buildslaves, but they compile in 32 bits, aparently. For instance http://www.python.org/dev/buildbot/all/builders/sparc%20solaris10%20gcc%203.x/builds/2163/steps/configure/logs/stdio.
We now have 32 and 64 bits OpenIndiana buildslaves, so we can actually check this. They were deployed yesterday.
Apparently the changes would be pretty simple, adding ".../64" to library paths, to try to find the extra libraries.
What do you think?.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
On Mon, Nov 22, 2010 at 10:34, Jesus Cea jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
A Solaris installation contains ALWAYS 32 and 64 bits libraries. So in any Solaris you can run 32/64 bits programs, and compile in 32 and 64 bits.
For this, libraries are stores in "/usr/lib", for instance, for 32 bits, while the same 64 bits libraries are stored in "/usr/lib/64".
Currently, python do not considerate this.
We have Solaris 10 buildslaves, but they compile in 32 bits, aparently. For instance http://www.python.org/dev/buildbot/all/builders/sparc%20solaris10%20gcc%203.x/builds/2163/steps/configure/logs/stdio.
We now have 32 and 64 bits OpenIndiana buildslaves, so we can actually check this. They were deployed yesterday.
Apparently the changes would be pretty simple, adding ".../64" to library paths, to try to find the extra libraries.
What do you think?.
Are you asking about buildbots only or as a general policy? If you are asking about the buildbots then I definitely think we should use 64 bits. If you are asking about policy I would say it should be an option in case people are using C extensions that are not designed to work with 64 bits.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 22/11/10 20:12, Brett Cannon wrote:
Are you asking about buildbots only or as a general policy? If you are asking about the buildbots then I definitely think we should use 64 bits. If you are asking about policy I would say it should be an option in case people are using C extensions that are not designed to work with 64 bits.
The point is that building python in 64 bits under Solaris (family) is not easy, because the 64 bits libraries (zlib, openssl, berkeley db, curses, etc., etc., etc) are not is "/usr/lib", "/usr/local/lib", etc., but "/usr/lib/64", "/usr/local/lib/64", etc.
Solaris overcomes most of the issue having separate library searchpath in 32 and 64 bits (via the "crle" command). But in some cases python try to find some library in "/usr/local/lib", and my point is that it should search TOO inside "/usr/local/lib/64".
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
Solaris overcomes most of the issue having separate library searchpath in 32 and 64 bits (via the "crle" command). But in some cases python try to find some library in "/usr/local/lib", and my point is that it should search TOO inside "/usr/local/lib/64".
I don't think this will work. If the linker finds a library of the wrong ELF type, then it will choke.
Before enabling anything on a build slave, a patch needs to be contributed to make it work in the first place.
Regards, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 22/11/10 20:42, "Martin v. Löwis" wrote:
Before enabling anything on a build slave, a patch needs to be contributed to make it work in the first place.
I actually agree. I am not sure yet, but I am thinking that adding a "--build-64" parameter to "configure" could be an option under Solaris. Most OSs (let say, Linux) force you to choose 32/64 bits at install time, but Solaris can use both at the same time, and compilers allow to compile both (using -m32 or -m64).
Since choosing 32 or 64 bits when compiling python under Solaris change the requirement, paths, etc., automating it should be a goal.
PS: Martin, is there any reason to restrict the solaris 10 buildslaves to 32 bits, beside the said problems?.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
I actually agree. I am not sure yet, but I am thinking that adding a "--build-64" parameter to "configure" could be an option under Solaris. Most OSs (let say, Linux) force you to choose 32/64 bits at install time
Actually, that's not at all the case. Most systems these days support 32-bit and 64-bit applications simultaneously, and also support compiler tool chains that allow building for either mode. Solaris, Linux, and Windows are about on-par in this respect; OS X is more advanced as it allows to have a single binary that supports both 32-bit and 64-bit execution (making the need for adjusted path names irrelevant).
Since choosing 32 or 64 bits when compiling python under Solaris change the requirement, paths, etc., automating it should be a goal.
PS: Martin, is there any reason to restrict the solaris 10 buildslaves to 32 bits, beside the said problems?.
I don't see that as a restriction. I have to make a choice, and there are sooo many choices to make: - gcc vs. SunPRO - 32-bit vs. 64-bit - GNU make vs. /usr/ccs/bin/make
I picked the combination which was most easy to setup, and is therefore likely to be used by most users (except for those who think 64-bit is somehow "better" than 32-bit, when it is actually the other way 'round - IMO).
As for configuration, I personally prefer that setting CC indicates what type of build you want. Set CC to "gcc -m64" to indicate a 64-build. Ideally, you will *not* have to adjust library paths, since the other compiler will know on its own where to search things.
Regards, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 22/11/10 23:05, "Martin v. Löwis" wrote:
PS: Martin, is there any reason to restrict the solaris 10 buildslaves to 32 bits, beside the said problems?.
I don't see that as a restriction. I have to make a choice, and there are sooo many choices to make:
- gcc vs. SunPRO
- 32-bit vs. 64-bit
- GNU make vs. /usr/ccs/bin/make
I picked the combination which was most easy to setup, and is therefore likely to be used by most users (except for those who think 64-bit is somehow "better" than 32-bit, when it is actually the other way 'round - IMO).
Do not think this is a personal attack. Not at all. I am deploying 32 and 64 bits buildslaves (in the same machine) and feeling the pain. You are far more experiences than me with buildbots and python. I want to know if I am missing something.
As for configuration, I personally prefer that setting CC indicates what type of build you want. Set CC to "gcc -m64" to indicate a 64-build. Ideally, you will *not* have to adjust library paths, since the other compiler will know on its own where to search things.
The problem is not with system library paths. Compilers overcome that. The problem is with things like "/usr/local/lib" and hardcoded library paths in Python.
For example, checking http://www.python.org/dev/buildbot/all/builders/AMD64%20OpenIndiana%203.x/builds/8/steps/test/logs/stdio:
""" gcc -shared -m64 build/temp.solaris-2.11-i86pc-3.2-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/readline.o - -L/usr/lib/termcap -L/usr/local/lib -lreadline -lncursesw -o build/lib.solaris-2.11-i86pc-3.2-pydebug/readline.so ld: fatal: file /usr/local/lib/libncursesw.so: wrong ELF class: ELFCLASS32 ld: fatal: file processing errors. No output written to build/lib.solaris-2.11-i86pc-3.2-pydebug/readline.so collect2: ld returned 1 exit status """
The "-L/usr/local/lib" should be "-L/usr/local/lib/64". An example of many.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
Am 23.11.2010 00:41, schrieb Jesus Cea:
On 22/11/10 23:05, "Martin v. Löwis" wrote:
PS: Martin, is there any reason to restrict the solaris 10 buildslaves to 32 bits, beside the said problems?.
I don't see that as a restriction. I have to make a choice, and there are sooo many choices to make:
- gcc vs. SunPRO
- 32-bit vs. 64-bit
- GNU make vs. /usr/ccs/bin/make
I picked the combination which was most easy to setup, and is therefore likely to be used by most users (except for those who think 64-bit is somehow "better" than 32-bit, when it is actually the other way 'round - IMO).
Do not think this is a personal attack.
No offense taken. If you really want to know the historical background: this was the very first build slave (before I actually announced it to python-dev), and I haven't changed much from the initial setup.
I just point out that none of the binaries in /usr/bin is a 64-bit binary; this includes the Sun-provided /usr/sfw/bin/python
The "-L/usr/local/lib" should be "-L/usr/local/lib/64". An example of many.
Is that really the case? I.e. will ncurses automatically install into /usr/local/lib/64 if built with a 64-bit compiler? My installation doesn't even have a /usr/local/lib/64 folder.
In any case: this shouldn't need a configure option. Instead, Python can find out itself whether it's a 64-bit build, and make modifications it considers necessary.
Regards, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 23/11/10 01:05, "Martin v. Löwis" wrote:
No offense taken. If you really want to know the historical background: this was the very first build slave (before I actually announced it to python-dev), and I haven't changed much from the initial setup.
I do really want to know. I love trivia :-). Thanks.
I just point out that none of the binaries in /usr/bin is a 64-bit binary; this includes the Sun-provided /usr/sfw/bin/python
The "-L/usr/local/lib" should be "-L/usr/local/lib/64". An example of many.
Is that really the case? I.e. will ncurses automatically install into /usr/local/lib/64 if built with a 64-bit compiler? My installation doesn't even have a /usr/local/lib/64 folder.
A fresh Solaris 10 install doesn't even have a "/usr/local" directory :).
Sadly today most Open Source code is written like if Linux were the only Unix system out there.
I was amazed that OpenSSL 1.0 installs automatically in "/usr/local/ssl/lib" when compiled in 32 bits, and in "/usr/local/ssl/lib/64" when compiled in 64 bits. I almost cry.
In any case: this shouldn't need a configure option. Instead, Python can find out itself whether it's a 64-bit build, and make modifications it considers necessary.
I agree. Python should detect it automatically and update the paths when compiling.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 23/11/10 01:05, "Martin v. Löwis" wrote:
I just point out that none of the binaries in /usr/bin is a 64-bit binary; this includes the Sun-provided /usr/sfw/bin/python
True. This is for simplicity reasons (provide only one binary valid for 32 and 64 bits CPUs) and because 64 bits is overkill for a lot of stuff.
In my own system my only 64 bits libraries are OpenSSL, GMP, and some multimedia stuff like mencoder, vorbis, etc, where the difference is big.
And the GCC 4.5.x install, that installs libraries (fortran, stdc++, objective C, etc) automatically under "/usr/local/lib/64". GOOD.
But if we say the Python can be compiled as 64 bits under Solaris, would be nice if that was actually true. Now that we have a buildbot (under OpenIndiana) to test, it is doable.
If not, we could say that Solaris+64 bits is unsupported. I don't think we should go that way. Solaris+64 bits should be a full citizen.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
But if we say the Python can be compiled as 64 bits under Solaris, would be nice if that was actually true. Now that we have a buildbot (under OpenIndiana) to test, it is doable.
But it is true, and always has been true. The lib/64 issue did not prevent one building Python on Solaris/SPARC64 at all, including the extension modules. Just edit Modules/Setup to suit your needs - that works since 1995 (before distutils was even written).
If not, we could say that Solaris+64 bits is unsupported. I don't think we should go that way. Solaris+64 bits should be a full citizen.
There we go again: "supported". Python builds on many systems which we don't have buildbots for, including obscure systems (although Guido has ruled that we won't specifically accept code for obscure systems anymore, unlike we did before). It is never fully automatic (you always have to at least make sure manually that the dependencies are installed).
Regards, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 23/11/10 07:55, "Martin v. Löwis" wrote:
But if we say the Python can be compiled as 64 bits under Solaris,
would
be nice if that was actually true. Now that we have a buildbot (under OpenIndiana) to test, it is doable.
But it is true, and always has been true. The lib/64 issue did not prevent one building Python on Solaris/SPARC64 at all, including the extension modules. Just edit Modules/Setup to suit your needs - that works since 1995 (before distutils was even written).
Would be acceptable to change something like:
""" add_library_path("/usr/local/lib") """
to something similar to:
""" if (platform.uname()=="SunOS") and (platform.architecture()[0]=="64bits") : add_library_path("/usr/local/lib/64") else : add_library_path("/usr/local/lib") """
python-dev would consider that change OK?.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 23/11/10 13:19, Jesus Cea wrote:
Would be acceptable to change something like:
""" add_library_path("/usr/local/lib") """
to something similar to:
""" if (platform.uname()=="SunOS") and (platform.architecture()[0]=="64bits") : add_library_path("/usr/local/lib/64") else : add_library_path("/usr/local/lib") """
python-dev would consider that change OK?.
Resurrecting an old thread.
I think that, under SunOS and 64 bits, python should install the lib directory under "/usr/local/lib/64", following the convention in the platform.
In fact, under OpenIndiana 147, I see this:
""" - -bash-4.0$ type python python is /usr/bin/python - -bash-4.0$ ldd /usr/bin/python libpython2.6.so.1.0 => /usr/lib/libpython2.6.so.1.0 libc.so.1 => /usr/lib/libc.so.1 libdl.so.1 => /lib/libdl.so.1 libm.so.2 => /lib/libm.so.2 - -bash-4.0$ file /usr/bin/python /usr/bin/python: ELF 32-bit LSB executable 80386 Version 1 [FPU], dynamically linked, not stripped, no debugging information available - -bash-4.0$ ls -lA /usr/lib/|grep python lrwxrwxrwx 1 root root 19 Oct 1 10:55 libpython2.6.so -> libpython2.6.so.1.0 - -rwxr-xr-x 1 root bin 1788896 Oct 1 10:55 libpython2.6.so.1.0 lrwxrwxrwx 1 root root 22 Oct 1 10:55 libpython2.6_db.so -> libpython2.6_db.so.1.0 - -rwxr-xr-x 1 root bin 10372 Oct 1 10:55 libpython2.6_db.so.1.0 drwxr-xr-x 25 root bin 432 Oct 1 10:55 python2.6 - -bash-4.0$ ls -lA /usr/lib/64/|grep python lrwxrwxrwx 1 root root 19 Oct 1 10:55 libpython2.6.so -> libpython2.6.so.1.0 - -rwxr-xr-x 1 root bin 2217504 Oct 1 10:55 libpython2.6.so.1.0 lrwxrwxrwx 1 root root 22 Oct 1 10:55 libpython2.6_db.so -> libpython2.6_db.so.1.0 - -rwxr-xr-x 1 root bin 13336 Oct 1 10:55 libpython2.6_db.so.1.0 - -bash-4.0$ file /usr/lib/64/libpython2.6.so.1.0 /usr/lib/64/libpython2.6.so.1.0: ELF 64-bit LSB dynamic lib AMD64 Version 1 [SSE2 SSE CMOV], dynamically linked, not stripped, no debugging information available """
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I think this is probably trivial, but is there any foolproof way to detect 64 bit builds in python, beside "sys.maxint"?.
And any macro useable for conditional compilation in C?.
Checking Solaris 10 header files, I see macros like "_LP64". Portability would be nice, but in this personal case, probably unneeded...
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
On Tue, 23 Nov 2010 00:48:06 +0100 Jesus Cea jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I think this is probably trivial, but is there any foolproof way to detect 64 bit builds in python, beside "sys.maxint"?.
sys.maxsize
And any macro useable for conditional compilation in C?.
SIZEOF_VOID_P > 4
On Mon, Nov 22, 2010 at 17:48, Jesus Cea jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I think this is probably trivial, but is there any foolproof way to detect 64 bit builds in python, beside "sys.maxint"?.
import platform platform.architecture()
Am 23.11.2010 00:48, schrieb Jesus Cea:
I think this is probably trivial, but is there any foolproof way to detect 64 bit builds in python, beside "sys.maxint"?.
The canonical way is to use platform.architecture().
And any macro useable for conditional compilation in C?.
You need to be more specific than that. There are perhaps ten independent properties you may query, depending on what precise problem you try to solve. Most likely, you are looking for SIZEOF_VOID_P (but don't use that unless you literally want to know how many bytes a pointer uses, or whether it uses 4 or 8 bytes).
Regards, Martin
In article 4CEAE129.2060505@jcea.es, Jesus Cea jcea@jcea.es wrote:
On 22/11/10 20:42, "Martin v. Löwis" wrote:
Before enabling anything on a build slave, a patch needs to be contributed to make it work in the first place.
I actually agree. I am not sure yet, but I am thinking that adding a "--build-64" parameter to "configure" could be an option under Solaris. Most OSs (let say, Linux) force you to choose 32/64 bits at install time, but Solaris can use both at the same time, and compilers allow to compile both (using -m32 or -m64).
Since choosing 32 or 64 bits when compiling python under Solaris change the requirement, paths, etc., automating it should be a goal.
You might want to look at the existing --with-universal-archs=ARCH in configure for how this is done for OS X builds. It's probably both simpler and more complicated than would be needed elsewhere: on OS X, a single file can contain object codes for multiple architectures, e.g 32-bit and 64-bit, rather than having to have multiple files.
Hi,
I think this bug is related: http://bugs.python.org/issue1294959 “Problems with /usr/lib64 builds.”
Regards
Am 22.11.2010 23:51, schrieb Éric Araujo:
Hi,
I think this bug is related: http://bugs.python.org/issue1294959 “Problems with /usr/lib64 builds.”
Perhaps more closely related:
http://bugs.python.org/issue847812 http://bugs.python.org/issue1733484 http://bugs.python.org/issue1676121 http://bugs.python.org/issue1628484
Regards, Martin
-
"Martin v. Löwis" -
Antoine Pitrou -
Brett Cannon -
Brian Curtin -
Jesus Cea -
Ned Deily -
Éric Araujo