Problem with libgfortran installed with pip install numpy
Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue. I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen. If I download the numpy source and do python setup.py install, I don't have this problem. After this long description, my question is why is such an old version of gcc used to build the distribution of numpy that gets installed from pypi? gcc version 4.9.0 is from 2014. Can a newer version be used? Thanks! Dave
On Wed, Sep 5, 2018 at 5:38 PM David Grote <dpgrote@lbl.gov> wrote:
Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue.
I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen.
If I download the numpy source and do python setup.py install, I don't have this problem.
After this long description, my question is why is such an old version of gcc used to build the distribution of numpy that gets installed from pypi? gcc version 4.9.0 is from 2014. Can a newer version be used?
The library came in with the use of OpenBLAS, I don't think there is a fundamental reason that a newer version of gfortran couldn't be used, but I have little experience with the Mac. Note that we have also given up on 32 bit Python on Mac for library related reasons. Matthew Brett would be the guy to discuss this with. Chuck
Hi Matthew - Do you have any comment in this? Thanks! Dave On Wed, Sep 5, 2018 at 5:01 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
On Wed, Sep 5, 2018 at 5:38 PM David Grote <dpgrote@lbl.gov> wrote:
Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue.
I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen.
If I download the numpy source and do python setup.py install, I don't have this problem.
After this long description, my question is why is such an old version of gcc used to build the distribution of numpy that gets installed from pypi? gcc version 4.9.0 is from 2014. Can a newer version be used?
The library came in with the use of OpenBLAS, I don't think there is a fundamental reason that a newer version of gfortran couldn't be used, but I have little experience with the Mac. Note that we have also given up on 32 bit Python on Mac for library related reasons. Matthew Brett would be the guy to discuss this with.
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On Wed, Sep 5, 2018 at 4:37 PM, David Grote <dpgrote@lbl.gov> wrote:
Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue.
I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen.
Normally on MacOS, it's fine to have multiple versions of the same library used at the same time, because the linker looks up symbols using a (source library, symbol name) pair. (This is called the "two-level namespace".) So it's strange that these two libgfortrans would interfere with each other. Does gfortran not use the two-level namespace when linking fortran code? -n -- Nathaniel J. Smith -- https://vorpus.org
Yes, thanks, that's something I hadn't looked into. For legacy reasons, my code was being built with the option -flat_namespace. I don't remember the reason why, but many years ago that option was needed for the code to run on the mac. The code is made up of several shared objects that have dependencies on each other and apparently there was a problem getting it all linked together properly without that option. But, I tried it out now without flat_namespace and it seemed to work fine. Whatever the problem was, it seems to been fixed some other way. It works Ok having the pip version of numpy (with its old libgfortran). I'm still curious about why such an old version of gfortran is still being used. Dave On Fri, Nov 9, 2018 at 3:09 PM Nathaniel Smith <njs@pobox.com> wrote:
On Wed, Sep 5, 2018 at 4:37 PM, David Grote <dpgrote@lbl.gov> wrote:
Hi - I have recently come across this problem. On my mac, I build a
code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I
Fortran tracked
this down to a gfortran version issue.
I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy
(/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib).
Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen.
Normally on MacOS, it's fine to have multiple versions of the same library used at the same time, because the linker looks up symbols using a (source library, symbol name) pair. (This is called the "two-level namespace".) So it's strange that these two libgfortrans would interfere with each other. Does gfortran not use the two-level namespace when linking fortran code?
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
participants (3)
-
Charles R Harris -
David Grote -
Nathaniel Smith