[Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

[Ned Deily]

On Nov 11, 2018, at 15:27, Michael <aixtools at felt.demon.nl> wrote:
> Anyway, my bot went from having 9 to 10 fails (and 10k lines of output)
> to 20 fails and roughly 22k lines of test.
> 
> The last "near normal FAIL" is:
> https://buildbot.python.org/all/#/builders/161/builds/325/steps/4/logs/stdio
> and the first mega fail is
> https://buildbot.python.org/all/#/builders/161/builds/326/steps/4/logs/stdio

Michael,

Thanks for tracking down where the additional failures began.  I took a quick look at the buildbot listings for builds 325 and 325 and it seems like all of the additional failures that appear in 326 vs 325 are a result of a new build failure for the pyexpat module.  That's visible at the end of the "compile" stage output:

Failed to build these modules:
pyexpat                                                        

and near the top of the test output when the Makefile tries to rebuild any failed modules.

Looking at the output of the git stage of 326, the git branch HEAD of the build is 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d.  Running git show to get the commit message:

$ git show 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d
commit 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d
Author: Gregory P. Smith <greg at krypto.org>
Date:   Wed Oct 17 18:10:46 2018 -0700

    bpo-35011: Restore use of pyexpatns.h in libexpat (GH-9939)

    Restores the use of pyexpatns.h to isolate our embedded copy of the expat C
    library so that its symbols do not conflict at link or dynamic loading time
    with an embedding application or other extension modules with their own
    version of libexpat.

    https://github.com/python/cpython/commit/5dc3f23b5fb0b510926012cb3732dae63cddea60#diff-3afaf7274c90ce1b7405f75ad825f545 inadvertently removed it when upgrading expat.

That looks rather suspicious!

So I guess the next step would be to try to figure out why that change (or some other in the range between buildbot run 325 and 326) causes the pyexpact module to stop building.  There are a bunch of ld: 0711-317 ERROR: Undefined symbol errors in the link step of the "building 'pyexpat' extension" step.  In the "configure" step, it is determined that it is not using an operating system provided version of expat:

     checking for --with-system-expat... no

Which is to be expected.  So it should be trying to pick up a version of expat that is provided elsewhere outside of the build.  Perhaps there is an issue with it.  Or the above change doesn't take something into account.  To proceed further, it would be most helpful to have someone with AIX experience and/or familiarity with that buildbot's setup.


--
  Ned Deily
  nad at python.org -- []