Re: [Python-Dev] [issue1633863] AIX: configure ignores $CC

Hi Martin, I have started to correct quite a lot of issues I have with Python on AIX, and since I had to test quite a lot of patchs, I though it would be more convenient to setup a buildbot for that platform. So I now have a buildbot environment with 2 slaves (AIX 5.3 and 6.1) that builds and tests Python (branch py3k) with both gcc and xlc (the native AIX compiler) (I have 4 builders ("py3k-aix6-xlc", "py3k-aix5-xlc", "py3k-aix6-gcc", "py3k-aix5-gcc"). I expect to add 4 more builders for branch 2.7 in coming days. I would like to share the results of this buildbot to the Python community so that issues with AIX could be addressed more easily. R. David Murray pointed me to the page on the python wiki concerning buildbot. It is stated there that is is possible to connect some slaves to some official Python buildbot master. Unfortunately, I don't think this solution is possible for me: I don't think the security team in my company would appreciate that a server inside our network runs some arbitrary shell commands provided by some external source. Neither can I expose the buildbot master web interface. Also I had to customize the buildbot rules in order to work with some specificities of AIX (see attached master.cfg), and I can't guarantee that this buildbot will run 24 hours a day; I may have to schedule it only once at night for example if it consumes too much resources. (And the results are very unstable at the moment, mostly because of issue 9862). On the other hand, I could upload the build results with rsync or scp somewhere or setup some MailNotifier if that can help. How do you think I could share those results? regards -- Sébastien Sablé Le 15/09/2010 23:28, R. David Murray a écrit :

Hi Sebastien,
I still think this would be the best thing, and I feel that from a security point of view, it doesn't really differ from what you are doing now already - see below.
Neither can I expose the buildbot master web interface.
That shouldn't be necessary.
If you are having the build slave compile Python, I'd like to point out that you *already* run arbitrary shell commands provided by some external source: if somebody would check some commands into Python's configure.in, you would unconditionally execute them. So if it's ok that you run the Python build process at all, it should (IMO) also be acceptable to run a build slave. If there are concerns that running it under your Unix account gives it too much power, you should create a separate, locked-down account.
I'd be hesitant to support this as a special case. If the results are not in the standard locations, people won't look at them, anyway. Given that one often also needs access to the hardware in order to fix problems, it might be sufficient if only you look at the buildslave results, and create bug reports whenever you notice a problem. Regards, Martin

Hi Martin, Le 17/09/2010 14:42, "Martin v. Löwis" a écrit :
Someone messing with the configure script in python svn would probably get noticed very quickly, but I agree this is also a security risk, and the buildbot slave runs with a user with limited privileges. I will try to convince the IT Team that this is an acceptable risk and setup a chroot or something like that for the buildbot slave. That may take some time. Also could you provide me the master.cfg file (with obfuscated passwords) that is used by the Python buildbot master or tell me if it is in subversion somewhere? I would like to make my script as close as possible to yours, in order to propose a patch for the AIX specific flags that have to be used for compilation on this platform when everything will be stable enough. Regards -- Sébastien Sablé

Hi Martin, I finally got the authorization to run some buildbot slaves on our AIX servers. I made some tests with a buildbot script as close as possible to what you already use. Here is a patch that show the kind of modifications I had to do in order to get the buildbot slave to correctly run on AIX. It is also necessary to apply the patch provided in issue 9862 so that the tests won't hang forever. It also would help if the corrections provided in the following issues could be commited: 4499, 678250, 730467. Could you please take a look at those modifications in master.cfg, provide me some password for the bot slaves and apply the corrections in those issues? Once this is done, I can run 2 buildbot slaves for AIX 6.1 and AIX 5.3 with build plans for gcc and xlc. I can't guarantee that those bots will run forever, and I may have to ask to schedule them only at night if the activity it generates on the server appears to be too high. But I think it could greatly help to improve the state of Python on AIX (which is far from perfect at the moment). regards -- Sébastien Sablé Le 20/09/2010 21:21, "Martin v. Löwis" a écrit :

On Fri, 15 Oct 2010 17:38:47 +0200, =?UTF-8?B?U8OpYmFzdGllbiBTYWJsw6k=?= <sable@users.sourceforge.net> wrote:
I've committed #9862, #4499 and #678250. I'd like a confirming opinion on #730467, though it sounds reasonable to me. -- R. David Murray www.bitdance.com

Hi Martin, I got no reply concerning those modifications to the buildbot script so that I could run 2 slaves on AIX. I am able to spend some time at the moment on Python for AIX because we are migrating the product in my company from AIX 5.3 to AIX 6.1. However once this is done, by the end of November, it will be more difficult for me to justify spending too much time on this builbot installation and on Python AIX bugs. So it would be nice if I could get this slave running as soon as possible, so that I could spend a few more days solving some of the remaining issues on this platform. regards -- Sébastien Sablé Le 15/10/2010 17:38, Sébastien Sablé a écrit :

I got no reply concerning those modifications to the buildbot script so that I could run 2 slaves on AIX.
I'm really really reluctant here. The proposed/requested changes are fairly intrusive, even though AIX is just a minority platform. So I need to find some time to rewrite them, which I haven't been able to, yet. Regards, Martin

On Fri, 15 Oct 2010 17:38:47 +0200 Sébastien Sablé <sable@users.sourceforge.net> wrote:
About the master.cfg modifications: there should be no need for separate environment variables. Instead, you should be able to specify them as command-line arguments to ./configure, e.g.: ["--with-pydebug", "--without-computed-gotos", "CC=xlc", 'OPT="-O2 -qmaxmem=18000"'] Can you check this works for you? Also, there's no need to complicate the buildbot naming procedure. You should be able to run several buildslaves on a single machine, provided we give you separate credentials: one per compiler type. Regards Antoine.

Hi Antoine, I tried to provide command lines arguments to configure instead of environment variables with: configureFlags = ["--with-pydebug", "--without-computed-gotos", "CC=xlc", 'OPT="-O2 -qmaxmem=18000"'] But that would fail: on the slave, configure would run like that: ./configure --with-pydebug --without-computed-gotos CC=xlc OPT="-O2 -qmaxmem=18000" And the compilation would give some error like that: xlc -c "-O2 -qmaxmem=18000" -O2 -O2 -I. -IInclude -I./Include -I/home/cis/.buildbot/support-buildbot/include -I/home/cis/.buildbot/support-buildbot/include/ncurses -I/home/cis/.buildbot/support-buildbot/include -I/home/cis/.buildbot/support-buildbot/include/ncurses -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c xlc: 1501-216 (W) command option - -qmaxmem=18000 is not recognized - passed to ld However running 2 different slaves per host in order to distinguish xlc and gcc would be OK; though I would appreciate if they could run sequentially rather than in parallel as that would limit the host load. regards -- Sébastien Sablé Le 28/10/2010 16:45, Antoine Pitrou a écrit :

Le lundi 08 novembre 2010 à 18:46 +0100, Sébastien Sablé a écrit :
xlc: 1501-216 (W) command option - -qmaxmem=18000 is not recognized - passed to ld
Is -qmaxmem really necessary to build Python? If so, you could try passing it in CFLAGS.
If there are two separate slaves, I can't think of any simple way to run builds sequentially. Perhaps you can assign both of them to a single CPU (assuming AIX allows that). Regards Antoine.

On Mon, Nov 08, 2010 at 06:50:32PM +0100, Antoine Pitrou wrote:
You can specify a slave lock to do this, in buildbot: http://buildbot.net/buildbot/docs/0.8.1/Interlocks.html One the neat things that a master/slave system like buildbot provides... cheers, --titus -- C. Titus Brown, ctb@msu.edu

On Fri, 17 Sep 2010 11:40:12 +0200 Sébastien Sablé <sable@users.sourceforge.net> wrote:
Following on Martin's comments, you might also want to share things with the ActiveState guys who, AFAIK, maintain an AIX version of Python (but you have been the most active AIX user on the bug tracker lately; perhaps they are keeping their patches to themselves). (see http://www.activestate.com/activepython ) Regards Antoine.

Le 17/09/2010 15:05, Antoine Pitrou a écrit :
I tried to find some source package or subversion repository on their web site, but they only distribute binary versions. Also it is only the "Business Edition" that supports AIX and it is clearly sold with a proprietary license. So I doubt they would share their patches related to AIX, but I can always ask. Regards -- Sébastien Sablé

On 2010-09-20, at 7:41 AM, Sébastien Sablé wrote:
Hi Sébastien, We definitely like to share our core Python patches for AIX 5.1/5.2 and other platforms. We have, in the past, submitted a few of them already in the Python bug tracker. As not all of them have been submitted (for lack of time), I uploaded them all here for your access, http://gist.github.com/628519 Please do send us any patches you may have for building Python 3 on AIX. -srid

Hi Sridhar, Le 15/10/2010 18:50, Sridhar Ratnakumar a écrit :
We definitely like to share our core Python patches for AIX 5.1/5.2 and other platforms.
Great to hear that ActiveState shares their improvements for Python on AIX! Thanks for the patch in git, I will check it and try to open some issues in the Python bug tracker in order to get those modifications integrated.
Please do send us any patches you may have for building Python 3 on AIX.
Here is a list of issues concerning AIX on which I have been working: http://bugs.python.org/issue678250 = OK (test_mmap error) http://bugs.python.org/issue678264 = WAIT (test_resource error) http://bugs.python.org/issue730467 = WAIT (AIX_GENUINE_CPLUSPLUS error) http://bugs.python.org/issue713169 (test_pty error) http://bugs.python.org/issue941346 = OK (shared library) http://bugs.python.org/issue1542544 = OK (shared library - duplicate) http://bugs.python.org/issue1563807 (compilation ctypes) http://bugs.python.org/issue1633863 = OK (bad default compiler) http://bugs.python.org/issue1745108 (curses error) http://bugs.python.org/issue3526 (custom memory allocation system - dlmalloc) http://bugs.python.org/issue4026 = OK (fcntl extension) http://bugs.python.org/issue4499 = OK (tilde macro) http://bugs.python.org/issue5718 (ctypes compilation) http://bugs.python.org/issue6006 (ctypes compilation) http://bugs.python.org/issue6164 (flag -blibpath) http://bugs.python.org/issue7657 (test_ctypes error) http://bugs.python.org/issue8695 = OK (crash on Python 2.6.4 at compilation) http://bugs.python.org/issue8882 (pb getsockaddrarg) http://bugs.python.org/issue9700 = OK (semwait) http://bugs.python.org/issue9799 = (xlc requires --without-computed-gotos) => OK add some doc http://bugs.python.org/issue9862 = OK (broken PIPE_BUF on AIX) The 2 most interesting patches I think are: * in issue 941346, a patch to be able to compile Python as a shared library (already applied in svn) * in issue 3526, a patch to use a custom memory allocation system (dlmalloc) in Python in order to greatly reduce memory consumption on AIX => not applied yet; I need to formally discuss it on Python-dev, but I am waiting for the AIX buildbot to be setup before that. best regards -- Sébastien Sablé

Hi Sebastien,
I still think this would be the best thing, and I feel that from a security point of view, it doesn't really differ from what you are doing now already - see below.
Neither can I expose the buildbot master web interface.
That shouldn't be necessary.
If you are having the build slave compile Python, I'd like to point out that you *already* run arbitrary shell commands provided by some external source: if somebody would check some commands into Python's configure.in, you would unconditionally execute them. So if it's ok that you run the Python build process at all, it should (IMO) also be acceptable to run a build slave. If there are concerns that running it under your Unix account gives it too much power, you should create a separate, locked-down account.
I'd be hesitant to support this as a special case. If the results are not in the standard locations, people won't look at them, anyway. Given that one often also needs access to the hardware in order to fix problems, it might be sufficient if only you look at the buildslave results, and create bug reports whenever you notice a problem. Regards, Martin

Hi Martin, Le 17/09/2010 14:42, "Martin v. Löwis" a écrit :
Someone messing with the configure script in python svn would probably get noticed very quickly, but I agree this is also a security risk, and the buildbot slave runs with a user with limited privileges. I will try to convince the IT Team that this is an acceptable risk and setup a chroot or something like that for the buildbot slave. That may take some time. Also could you provide me the master.cfg file (with obfuscated passwords) that is used by the Python buildbot master or tell me if it is in subversion somewhere? I would like to make my script as close as possible to yours, in order to propose a patch for the AIX specific flags that have to be used for compilation on this platform when everything will be stable enough. Regards -- Sébastien Sablé

Hi Martin, I finally got the authorization to run some buildbot slaves on our AIX servers. I made some tests with a buildbot script as close as possible to what you already use. Here is a patch that show the kind of modifications I had to do in order to get the buildbot slave to correctly run on AIX. It is also necessary to apply the patch provided in issue 9862 so that the tests won't hang forever. It also would help if the corrections provided in the following issues could be commited: 4499, 678250, 730467. Could you please take a look at those modifications in master.cfg, provide me some password for the bot slaves and apply the corrections in those issues? Once this is done, I can run 2 buildbot slaves for AIX 6.1 and AIX 5.3 with build plans for gcc and xlc. I can't guarantee that those bots will run forever, and I may have to ask to schedule them only at night if the activity it generates on the server appears to be too high. But I think it could greatly help to improve the state of Python on AIX (which is far from perfect at the moment). regards -- Sébastien Sablé Le 20/09/2010 21:21, "Martin v. Löwis" a écrit :

On Fri, 15 Oct 2010 17:38:47 +0200, =?UTF-8?B?U8OpYmFzdGllbiBTYWJsw6k=?= <sable@users.sourceforge.net> wrote:
I've committed #9862, #4499 and #678250. I'd like a confirming opinion on #730467, though it sounds reasonable to me. -- R. David Murray www.bitdance.com

Hi Martin, I got no reply concerning those modifications to the buildbot script so that I could run 2 slaves on AIX. I am able to spend some time at the moment on Python for AIX because we are migrating the product in my company from AIX 5.3 to AIX 6.1. However once this is done, by the end of November, it will be more difficult for me to justify spending too much time on this builbot installation and on Python AIX bugs. So it would be nice if I could get this slave running as soon as possible, so that I could spend a few more days solving some of the remaining issues on this platform. regards -- Sébastien Sablé Le 15/10/2010 17:38, Sébastien Sablé a écrit :

I got no reply concerning those modifications to the buildbot script so that I could run 2 slaves on AIX.
I'm really really reluctant here. The proposed/requested changes are fairly intrusive, even though AIX is just a minority platform. So I need to find some time to rewrite them, which I haven't been able to, yet. Regards, Martin

On Fri, 15 Oct 2010 17:38:47 +0200 Sébastien Sablé <sable@users.sourceforge.net> wrote:
About the master.cfg modifications: there should be no need for separate environment variables. Instead, you should be able to specify them as command-line arguments to ./configure, e.g.: ["--with-pydebug", "--without-computed-gotos", "CC=xlc", 'OPT="-O2 -qmaxmem=18000"'] Can you check this works for you? Also, there's no need to complicate the buildbot naming procedure. You should be able to run several buildslaves on a single machine, provided we give you separate credentials: one per compiler type. Regards Antoine.

Hi Antoine, I tried to provide command lines arguments to configure instead of environment variables with: configureFlags = ["--with-pydebug", "--without-computed-gotos", "CC=xlc", 'OPT="-O2 -qmaxmem=18000"'] But that would fail: on the slave, configure would run like that: ./configure --with-pydebug --without-computed-gotos CC=xlc OPT="-O2 -qmaxmem=18000" And the compilation would give some error like that: xlc -c "-O2 -qmaxmem=18000" -O2 -O2 -I. -IInclude -I./Include -I/home/cis/.buildbot/support-buildbot/include -I/home/cis/.buildbot/support-buildbot/include/ncurses -I/home/cis/.buildbot/support-buildbot/include -I/home/cis/.buildbot/support-buildbot/include/ncurses -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c xlc: 1501-216 (W) command option - -qmaxmem=18000 is not recognized - passed to ld However running 2 different slaves per host in order to distinguish xlc and gcc would be OK; though I would appreciate if they could run sequentially rather than in parallel as that would limit the host load. regards -- Sébastien Sablé Le 28/10/2010 16:45, Antoine Pitrou a écrit :

Le lundi 08 novembre 2010 à 18:46 +0100, Sébastien Sablé a écrit :
xlc: 1501-216 (W) command option - -qmaxmem=18000 is not recognized - passed to ld
Is -qmaxmem really necessary to build Python? If so, you could try passing it in CFLAGS.
If there are two separate slaves, I can't think of any simple way to run builds sequentially. Perhaps you can assign both of them to a single CPU (assuming AIX allows that). Regards Antoine.

On Mon, Nov 08, 2010 at 06:50:32PM +0100, Antoine Pitrou wrote:
You can specify a slave lock to do this, in buildbot: http://buildbot.net/buildbot/docs/0.8.1/Interlocks.html One the neat things that a master/slave system like buildbot provides... cheers, --titus -- C. Titus Brown, ctb@msu.edu

On Fri, 17 Sep 2010 11:40:12 +0200 Sébastien Sablé <sable@users.sourceforge.net> wrote:
Following on Martin's comments, you might also want to share things with the ActiveState guys who, AFAIK, maintain an AIX version of Python (but you have been the most active AIX user on the bug tracker lately; perhaps they are keeping their patches to themselves). (see http://www.activestate.com/activepython ) Regards Antoine.

Le 17/09/2010 15:05, Antoine Pitrou a écrit :
I tried to find some source package or subversion repository on their web site, but they only distribute binary versions. Also it is only the "Business Edition" that supports AIX and it is clearly sold with a proprietary license. So I doubt they would share their patches related to AIX, but I can always ask. Regards -- Sébastien Sablé

On 2010-09-20, at 7:41 AM, Sébastien Sablé wrote:
Hi Sébastien, We definitely like to share our core Python patches for AIX 5.1/5.2 and other platforms. We have, in the past, submitted a few of them already in the Python bug tracker. As not all of them have been submitted (for lack of time), I uploaded them all here for your access, http://gist.github.com/628519 Please do send us any patches you may have for building Python 3 on AIX. -srid

Hi Sridhar, Le 15/10/2010 18:50, Sridhar Ratnakumar a écrit :
We definitely like to share our core Python patches for AIX 5.1/5.2 and other platforms.
Great to hear that ActiveState shares their improvements for Python on AIX! Thanks for the patch in git, I will check it and try to open some issues in the Python bug tracker in order to get those modifications integrated.
Please do send us any patches you may have for building Python 3 on AIX.
Here is a list of issues concerning AIX on which I have been working: http://bugs.python.org/issue678250 = OK (test_mmap error) http://bugs.python.org/issue678264 = WAIT (test_resource error) http://bugs.python.org/issue730467 = WAIT (AIX_GENUINE_CPLUSPLUS error) http://bugs.python.org/issue713169 (test_pty error) http://bugs.python.org/issue941346 = OK (shared library) http://bugs.python.org/issue1542544 = OK (shared library - duplicate) http://bugs.python.org/issue1563807 (compilation ctypes) http://bugs.python.org/issue1633863 = OK (bad default compiler) http://bugs.python.org/issue1745108 (curses error) http://bugs.python.org/issue3526 (custom memory allocation system - dlmalloc) http://bugs.python.org/issue4026 = OK (fcntl extension) http://bugs.python.org/issue4499 = OK (tilde macro) http://bugs.python.org/issue5718 (ctypes compilation) http://bugs.python.org/issue6006 (ctypes compilation) http://bugs.python.org/issue6164 (flag -blibpath) http://bugs.python.org/issue7657 (test_ctypes error) http://bugs.python.org/issue8695 = OK (crash on Python 2.6.4 at compilation) http://bugs.python.org/issue8882 (pb getsockaddrarg) http://bugs.python.org/issue9700 = OK (semwait) http://bugs.python.org/issue9799 = (xlc requires --without-computed-gotos) => OK add some doc http://bugs.python.org/issue9862 = OK (broken PIPE_BUF on AIX) The 2 most interesting patches I think are: * in issue 941346, a patch to be able to compile Python as a shared library (already applied in svn) * in issue 3526, a patch to use a custom memory allocation system (dlmalloc) in Python in order to greatly reduce memory consumption on AIX => not applied yet; I need to formally discuss it on Python-dev, but I am waiting for the AIX buildbot to be setup before that. best regards -- Sébastien Sablé
participants (7)
-
"Martin v. Löwis"
-
Antoine Pitrou
-
C. Titus Brown
-
exarkun@twistedmatrix.com
-
R. David Murray
-
Sridhar Ratnakumar
-
Sébastien Sablé