Mailman 3 Solaris family and 64 bits compiling - Python-Dev

List overview All Threads
Download

newer

Solaris family and 64 bits compiling

older

Re: [Python-Dev] cpython: Exercise...

buildbot error when pushing to 2.5...

Jesus Cea

22 Nov 2010 22 Nov '10

6:34 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

A Solaris installation contains ALWAYS 32 and 64 bits libraries. So in any Solaris you can run 32/64 bits programs, and compile in 32 and 64 bits.

For this, libraries are stores in "/usr/lib", for instance, for 32 bits, while the same 64 bits libraries are stored in "/usr/lib/64".

Currently, python do not considerate this.

We have Solaris 10 buildslaves, but they compile in 32 bits, aparently. For instance http://www.python.org/dev/buildbot/all/builders/sparc%20solaris10%20gcc%203.....

We now have 32 and 64 bits OpenIndiana buildslaves, so we can actually check this. They were deployed yesterday.

Apparently the changes would be pretty simple, adding ".../64" to library paths, to try to find the extra libraries.

What do you think?.

Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTOq3yZlgi5GaxT1NAQLQhAP9G2liX+YveYmfYDOuVjWWS8PE7r2wM/XA 5rik9mJM4Z7/wDnY4wrWjG5l3B9sSyrhhNI1YmIcXm4klfYxV9xTkG9dMNL+2bVc +s98rlTdjNlMVTf8Xc7U3tMpdkG/JK0+XWmRfWsf52ATdtxPHazI9L6KvqdYjNuZ 2w3dXNXErZE= =oYXo -----END PGP SIGNATURE-----

Show replies by date

Brett Cannon

22 Nov 22 Nov

7:12 p.m.

On Mon, Nov 22, 2010 at 10:34, Jesus Cea jcea@jcea.es wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

A Solaris installation contains ALWAYS 32 and 64 bits libraries. So in any Solaris you can run 32/64 bits programs, and compile in 32 and 64 bits.

For this, libraries are stores in "/usr/lib", for instance, for 32 bits, while the same 64 bits libraries are stored in "/usr/lib/64".

Currently, python do not considerate this.

We have Solaris 10 buildslaves, but they compile in 32 bits, aparently. For instance http://www.python.org/dev/buildbot/all/builders/sparc%20solaris10%20gcc%203.....

We now have 32 and 64 bits OpenIndiana buildslaves, so we can actually check this. They were deployed yesterday.

Apparently the changes would be pretty simple, adding ".../64" to library paths, to try to find the extra libraries.

What do you think?.

Are you asking about buildbots only or as a general policy? If you are asking about the buildbots then I definitely think we should use 64 bits. If you are asking about policy I would say it should be an option in case people are using C extensions that are not designed to work with 64 bits.

Jesus Cea

7:26 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 22/11/10 20:12, Brett Cannon wrote:

...

The point is that building python in 64 bits under Solaris (family) is not easy, because the 64 bits libraries (zlib, openssl, berkeley db, curses, etc., etc., etc) are not is "/usr/lib", "/usr/local/lib", etc., but "/usr/lib/64", "/usr/local/lib/64", etc.

Solaris overcomes most of the issue having separate library searchpath in 32 and 64 bits (via the "crle" command). But in some cases python try to find some library in "/usr/local/lib", and my point is that it should search TOO inside "/usr/local/lib/64".

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTOrD8Jlgi5GaxT1NAQJhRQP/dd4q70eXsq5AUFrleqUx3A+AagChpCcp UDHAomaX26cMl0tLFwLOd4SaKizzRMvjdTJc3GhZDIqYrF3QuqZAyLPjr5tyogP8 /4KPM73l5L2cb7IdHdSHpruwMh8f2WJ4S6+ig8DzOj6qBcttXKMymrV/skum4ENJ yb4mbpH9q/0= =Oe2G -----END PGP SIGNATURE-----

"Martin v. Löwis"

7:42 p.m.

...

I don't think this will work. If the linker finds a library of the wrong ELF type, then it will choke.

Before enabling anything on a build slave, a patch needs to be contributed to make it work in the first place.

Regards, Martin

Jesus Cea

9:31 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 22/11/10 20:42, "Martin v. Löwis" wrote:

...

Before enabling anything on a build slave, a patch needs to be contributed to make it work in the first place.

I actually agree. I am not sure yet, but I am thinking that adding a "--build-64" parameter to "configure" could be an option under Solaris. Most OSs (let say, Linux) force you to choose 32/64 bits at install time, but Solaris can use both at the same time, and compilers allow to compile both (using -m32 or -m64).

Since choosing 32 or 64 bits when compiling python under Solaris change the requirement, paths, etc., automating it should be a goal.

PS: Martin, is there any reason to restrict the solaris 10 buildslaves to 32 bits, beside the said problems?.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTOrhKZlgi5GaxT1NAQI0cAP+OUFGVDd7UV6MdHzMenBn8fO3h4M1n0dR UZrVyYJhUYvEX9p7MRBdDNFY/6LrUITb3WCVegD3PuOymQP16GgksRfIA/jGDXyl Fe+Ed5amlDgdVPeVVH/55OodrO4SuOrJZ846G6GB1wav2IjR7I9YGxZQ6PA0LR7l 4Iph6HfcMlw= =hTNy -----END PGP SIGNATURE-----

"Martin v. Löwis"

10:05 p.m.

...

Actually, that's not at all the case. Most systems these days support 32-bit and 64-bit applications simultaneously, and also support compiler tool chains that allow building for either mode. Solaris, Linux, and Windows are about on-par in this respect; OS X is more advanced as it allows to have a single binary that supports both 32-bit and 64-bit execution (making the need for adjusted path names irrelevant).

...

Since choosing 32 or 64 bits when compiling python under Solaris change the requirement, paths, etc., automating it should be a goal.

PS: Martin, is there any reason to restrict the solaris 10 buildslaves to 32 bits, beside the said problems?.

I don't see that as a restriction. I have to make a choice, and there are sooo many choices to make:

gcc vs. SunPRO
32-bit vs. 64-bit
GNU make vs. /usr/ccs/bin/make

I picked the combination which was most easy to setup, and is therefore likely to be used by most users (except for those who think 64-bit is somehow "better" than 32-bit, when it is actually the other way 'round - IMO).

As for configuration, I personally prefer that setting CC indicates what type of build you want. Set CC to "gcc -m64" to indicate a 64-build. Ideally, you will not have to adjust library paths, since the other compiler will know on its own where to search things.

Regards, Martin

Jesus Cea

11:41 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 22/11/10 23:05, "Martin v. Löwis" wrote:

...

PS: Martin, is there any reason to restrict the solaris 10 buildslaves to 32 bits, beside the said problems?.

I don't see that as a restriction. I have to make a choice, and there are sooo many choices to make:

gcc vs. SunPRO
32-bit vs. 64-bit
GNU make vs. /usr/ccs/bin/make

Do not think this is a personal attack. Not at all. I am deploying 32 and 64 bits buildslaves (in the same machine) and feeling the pain. You are far more experiences than me with buildbots and python. I want to know if I am missing something.

...

The problem is not with system library paths. Compilers overcome that. The problem is with things like "/usr/local/lib" and hardcoded library paths in Python.

For example, checking http://www.python.org/dev/buildbot/all/builders/AMD64%20OpenIndiana%203.x/bu...:

""" gcc -shared -m64 build/temp.solaris-2.11-i86pc-3.2-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/readline.o

-L/usr/lib/termcap -L/usr/local/lib -lreadline -lncursesw -o build/lib.solaris-2.11-i86pc-3.2-pydebug/readline.so ld: fatal: file /usr/local/lib/libncursesw.so: wrong ELF class: ELFCLASS32 ld: fatal: file processing errors. No output written to build/lib.solaris-2.11-i86pc-3.2-pydebug/readline.so collect2: ld returned 1 exit status """

The "-L/usr/local/lib" should be "-L/usr/local/lib/64". An example of many.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTOr/n5lgi5GaxT1NAQLzogP/Sb2VMe7UwK/YeB8/cQSxhuoKeNRre0pZ XCJDePusysqI3uXBHmH8vitEIILmUKd5kQ6vsFwErPIry7ikl2fbDHe7eQgNr2HK o5Xcul36bqtuKWGkDV+gIyBH/m9k4pkvc7Lfp3mvR7yiYTBB75V/azt64XSTC9si 7QjjetX5wnA= =NCtE -----END PGP SIGNATURE-----

"Martin v. Löwis"

23 Nov 23 Nov

12:05 a.m.

Am 23.11.2010 00:41, schrieb Jesus Cea:

...

On 22/11/10 23:05, "Martin v. Löwis" wrote:

PS: Martin, is there any reason to restrict the solaris 10 buildslaves to 32 bits, beside the said problems?.

I don't see that as a restriction. I have to make a choice, and there are sooo many choices to make:

gcc vs. SunPRO
32-bit vs. 64-bit
GNU make vs. /usr/ccs/bin/make

Do not think this is a personal attack.

No offense taken. If you really want to know the historical background: this was the very first build slave (before I actually announced it to python-dev), and I haven't changed much from the initial setup.

I just point out that none of the binaries in /usr/bin is a 64-bit binary; this includes the Sun-provided /usr/sfw/bin/python

...

The "-L/usr/local/lib" should be "-L/usr/local/lib/64". An example of many.

Is that really the case? I.e. will ncurses automatically install into /usr/local/lib/64 if built with a 64-bit compiler? My installation doesn't even have a /usr/local/lib/64 folder.

In any case: this shouldn't need a configure option. Instead, Python can find out itself whether it's a 64-bit build, and make modifications it considers necessary.

Regards, Martin

Jesus Cea

12:47 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 23/11/10 01:05, "Martin v. Löwis" wrote:

...

I do really want to know. I love trivia :-). Thanks.

...

I just point out that none of the binaries in /usr/bin is a 64-bit binary; this includes the Sun-provided /usr/sfw/bin/python

The "-L/usr/local/lib" should be "-L/usr/local/lib/64". An example of many.

Is that really the case? I.e. will ncurses automatically install into /usr/local/lib/64 if built with a 64-bit compiler? My installation doesn't even have a /usr/local/lib/64 folder.

A fresh Solaris 10 install doesn't even have a "/usr/local" directory :).

Sadly today most Open Source code is written like if Linux were the only Unix system out there.

I was amazed that OpenSSL 1.0 installs automatically in "/usr/local/ssl/lib" when compiled in 32 bits, and in "/usr/local/ssl/lib/64" when compiled in 64 bits. I almost cry.

...

In any case: this shouldn't need a configure option. Instead, Python can find out itself whether it's a 64-bit build, and make modifications it considers necessary.

I agree. Python should detect it automatically and update the paths when compiling.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTOsPBZlgi5GaxT1NAQIw+QP/ZuxpWo2WZYUUcDfARRnOtp60n4PbIGMf fqQ4ZnC9JnelzKDU9kBo0yReL2zYAw0ZwezsGwZ98M9i3XyKkFCtcJcM1vXpIsDL eBwga8kPDpab5loP/vuac5kVC0wn0Z0z8x+BRMW6mwoOMHJzd463E8GTQywdx3x1 06FUHwJ0Hv4= =PV43 -----END PGP SIGNATURE-----

Jesus Cea

12:58 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 23/11/10 01:05, "Martin v. Löwis" wrote:

...

I just point out that none of the binaries in /usr/bin is a 64-bit binary; this includes the Sun-provided /usr/sfw/bin/python

True. This is for simplicity reasons (provide only one binary valid for 32 and 64 bits CPUs) and because 64 bits is overkill for a lot of stuff.

In my own system my only 64 bits libraries are OpenSSL, GMP, and some multimedia stuff like mencoder, vorbis, etc, where the difference is big.

And the GCC 4.5.x install, that installs libraries (fortran, stdc++, objective C, etc) automatically under "/usr/local/lib/64". GOOD.

But if we say the Python can be compiled as 64 bits under Solaris, would be nice if that was actually true. Now that we have a buildbot (under OpenIndiana) to test, it is doable.

If not, we could say that Solaris+64 bits is unsupported. I don't think we should go that way. Solaris+64 bits should be a full citizen.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTOsRxplgi5GaxT1NAQKqqAP/fkiPpnPswMYOWc30Bflg3nDqRf6ih1bW ZZYHEMuJN9C8rm419LnRtoTyeAruHQYJ3o/dAoA2xDZu1xDYz8OOJKzG1L8hRVce OGm9TmziS4zuwWS4sYdmh21/ZCuD0MVq3gqD1h8zYPwrqbTTA6shYr6/He5hAo6j 5PsYWj4gIAE= =Rr80 -----END PGP SIGNATURE-----

"Martin v. Löwis"

6:55 a.m.

...

But if we say the Python can be compiled as 64 bits under Solaris, would be nice if that was actually true. Now that we have a buildbot (under OpenIndiana) to test, it is doable.

But it is true, and always has been true. The lib/64 issue did not prevent one building Python on Solaris/SPARC64 at all, including the extension modules. Just edit Modules/Setup to suit your needs - that works since 1995 (before distutils was even written).

...

If not, we could say that Solaris+64 bits is unsupported. I don't think we should go that way. Solaris+64 bits should be a full citizen.

There we go again: "supported". Python builds on many systems which we don't have buildbots for, including obscure systems (although Guido has ruled that we won't specifically accept code for obscure systems anymore, unlike we did before). It is never fully automatic (you always have to at least make sure manually that the dependencies are installed).

Regards, Martin

Jesus Cea

12:19 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 23/11/10 07:55, "Martin v. Löwis" wrote:

...

But if we say the Python can be compiled as 64 bits under Solaris, would be nice if that was actually true. Now that we have a buildbot (under OpenIndiana) to test, it is doable.

""" add_library_path("/usr/local/lib") """

to something similar to:

""" if (platform.uname()=="SunOS") and (platform.architecture()[0]=="64bits") : add_library_path("/usr/local/lib/64") else : add_library_path("/usr/local/lib") """

python-dev would consider that change OK?.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTOuxW5lgi5GaxT1NAQJuDwP/dzbhDZScanoSnPeF4Ze5XHm+WnSmowx+ x9qvM782i4bYzqYNsbpPHflshROpUwdl9dC0/dFySLFWmMYo12hYogbM6vr5RD6k vEgq1iriIfsei9yNrtt2Ou6+1LVxJ2FMsbpY0Av5hDQVfuJpvB5WRML/mbyYj4T7 9w/jmPT2+rc= =riDG -----END PGP SIGNATURE-----

Jesus Cea

17 Mar 17 Mar

1:02 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 23/11/10 13:19, Jesus Cea wrote:

...

Would be acceptable to change something like:

""" add_library_path("/usr/local/lib") """

to something similar to:

""" if (platform.uname()=="SunOS") and (platform.architecture()[0]=="64bits") : add_library_path("/usr/local/lib/64") else : add_library_path("/usr/local/lib") """

python-dev would consider that change OK?.

Resurrecting an old thread.

I think that, under SunOS and 64 bits, python should install the lib directory under "/usr/local/lib/64", following the convention in the platform.

In fact, under OpenIndiana 147, I see this:

"""

-bash-4.0$ type python python is /usr/bin/python

-bash-4.0$ ldd /usr/bin/python

  libpython2.6.so.1.0 =>  
/usr/lib/libpython2.6.so.1.0
  libc.so.1 =>     /usr/lib/libc.so.1
  libdl.so.1 =>    /lib/libdl.so.1
  libm.so.2 =>     /lib/libm.so.2

-bash-4.0$ file /usr/bin/python /usr/bin/python: ELF 32-bit LSB executable 80386 Version 1 [FPU], dynamically linked, not stripped, no debugging information available
-bash-4.0$ ls -lA /usr/lib/|grep python lrwxrwxrwx 1 root root 19 Oct 1 10:55 libpython2.6.so -> libpython2.6.so.1.0
-rwxr-xr-x 1 root bin 1788896 Oct 1 10:55 libpython2.6.so.1.0 lrwxrwxrwx 1 root root 22 Oct 1 10:55 libpython2.6_db.so -> libpython2.6_db.so.1.0
-rwxr-xr-x 1 root bin 10372 Oct 1 10:55 libpython2.6_db.so.1.0 drwxr-xr-x 25 root bin 432 Oct 1 10:55 python2.6
-bash-4.0$ ls -lA /usr/lib/64/|grep python lrwxrwxrwx 1 root root 19 Oct 1 10:55 libpython2.6.so -> libpython2.6.so.1.0
-rwxr-xr-x 1 root bin 2217504 Oct 1 10:55 libpython2.6.so.1.0 lrwxrwxrwx 1 root root 22 Oct 1 10:55 libpython2.6_db.so -> libpython2.6_db.so.1.0
-rwxr-xr-x 1 root bin 13336 Oct 1 10:55 libpython2.6_db.so.1.0
-bash-4.0$ file /usr/lib/64/libpython2.6.so.1.0 /usr/lib/64/libpython2.6.so.1.0: ELF 64-bit LSB dynamic lib AMD64 Version 1 [SSE2 SSE CMOV], dynamically linked, not stripped, no debugging information available """

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTYIGSJlgi5GaxT1NAQIrGAP/fBcLKIdFN+O8jNArCs01m30pQC5rNAhp hs1mTqlllpGkSQLDeQvcN/ctwFpR8a+mlC4yCFwYFB8CpB9TjilIKRn0RlerfOlw U0OuhezPTGc75NY95a+x8iZG4FQYytMkv7ClPmyDjG/g68CTS/zArRoKGRA7trZk +WbsLPK4lhg= =ScvG -----END PGP SIGNATURE-----

Jesus Cea

22 Nov 22 Nov

11:48 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

I think this is probably trivial, but is there any foolproof way to detect 64 bit builds in python, beside "sys.maxint"?.

And any macro useable for conditional compilation in C?.

Checking Solaris 10 header files, I see macros like "_LP64". Portability would be nice, but in this personal case, probably unneeded...

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTOsBNplgi5GaxT1NAQLkJwP+P1YyABBPGInHJXvwsU2ZLuj+u/OuZCRE m6hmbZgMajAyc5NtTie36qyHKAtVBcxFFvUdDeyfDZXV5gU+dF9Ha7/R16dclG3k b5W0CbccnGFcQJ/XypNPjH2dYPFDiqF8kCkDfeLJ7ZyL9ojA1YlRGFrgswN77/cF XM7Cwq1mh5k= =JXDq -----END PGP SIGNATURE-----

Antoine Pitrou

23 Nov 23 Nov

12:06 a.m.

On Tue, 23 Nov 2010 00:48:06 +0100 Jesus Cea jcea@jcea.es wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

I think this is probably trivial, but is there any foolproof way to detect 64 bit builds in python, beside "sys.maxint"?.

sys.maxsize

...

And any macro useable for conditional compilation in C?.

SIZEOF_VOID_P > 4

Brian Curtin

12:06 a.m.

On Mon, Nov 22, 2010 at 17:48, Jesus Cea jcea@jcea.es wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

I think this is probably trivial, but is there any foolproof way to detect 64 bit builds in python, beside "sys.maxint"?.

import platform platform.architecture()

"Martin v. Löwis"

12:12 a.m.

Am 23.11.2010 00:48, schrieb Jesus Cea:

...

I think this is probably trivial, but is there any foolproof way to detect 64 bit builds in python, beside "sys.maxint"?.

The canonical way is to use platform.architecture().

...

And any macro useable for conditional compilation in C?.

You need to be more specific than that. There are perhaps ten independent properties you may query, depending on what precise problem you try to solve. Most likely, you are looking for SIZEOF_VOID_P (but don't use that unless you literally want to know how many bytes a pointer uses, or whether it uses 4 or 8 bytes).

Regards, Martin

Ned Deily

22 Nov 22 Nov

10:12 p.m.

In article 4CEAE129.2060505@jcea.es, Jesus Cea jcea@jcea.es wrote:

...

On 22/11/10 20:42, "Martin v. Löwis" wrote:

Before enabling anything on a build slave, a patch needs to be contributed to make it work in the first place.

Since choosing 32 or 64 bits when compiling python under Solaris change the requirement, paths, etc., automating it should be a goal.

You might want to look at the existing --with-universal-archs=ARCH in configure for how this is done for OS X builds. It's probably both simpler and more complicated than would be needed elsewhere: on OS X, a single file can contain object codes for multiple architectures, e.g 32-bit and 64-bit, rather than having to have multiple files.

-- Ned Deily, nad@acm.org