Does numpy depend upon a Fortran library?
Hello,
I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib.
While investigating the source of this problem, I downloaded the numpy source (https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce...), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation.
I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS.
I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing?
thanks Marc
On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno paterno@fnal.gov wrote:
Hello,
I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib.
While investigating the source of this problem, I downloaded the numpy source ( https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce... ), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation.
I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS.
I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing?
That's interesting, it looks like the wheel includes four libraries:
-rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 libopenblasp-r0.3.5.dev.dylib -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib
I thought we only needed the openblas, but that in turn probably depends on libgcc. But why we have the fortran library and quadmath escapes me. Perhaps someone else knows.
Note that compiling from source is different and will generally use different libraries. We don't use Accelerate because it is buggy, not thread safe, and it appears Apple is not interested in doing anything about that.
Chuck
On Wed, Jan 30, 2019 at 5:19 PM Charles R Harris charlesr.harris@gmail.com wrote:
On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno paterno@fnal.gov wrote:
Hello,
I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib.
While investigating the source of this problem, I downloaded the numpy source ( https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce... ), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation.
I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS.
I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing?
That's interesting, it looks like the wheel includes four libraries:
-rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 libopenblasp-r0.3.5.dev.dylib -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib
I thought we only needed the openblas, but that in turn probably depends on libgcc. But why we have the fortran library and quadmath escapes me. Perhaps someone else knows.
I suspect it's a leftover from when we were using ATLAS, we did need a Fortran runtime library at some point. The cause will be somewhere in the numpy-wheel build scripts, there is a gfortran-install git submodule: https://github.com/MacPython/numpy-wheels
Note that compiling from source is different and will generally use different libraries. We don't use Accelerate because it is buggy, not thread safe, and it appears Apple is not interested in doing anything about that.
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On Wed, Jan 30, 2019 at 6:32 PM Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jan 30, 2019 at 5:19 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno paterno@fnal.gov wrote:
Hello,
I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib.
While investigating the source of this problem, I downloaded the numpy source ( https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce... ), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation.
I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS.
I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing?
That's interesting, it looks like the wheel includes four libraries:
-rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 libopenblasp-r0.3.5.dev.dylib -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib
I thought we only needed the openblas, but that in turn probably depends on libgcc. But why we have the fortran library and quadmath escapes me. Perhaps someone else knows.
I suspect it's a leftover from when we were using ATLAS, we did need a Fortran runtime library at some point. The cause will be somewhere in the numpy-wheel build scripts, there is a gfortran-install git submodule: https://github.com/MacPython/numpy-wheels
And fortran is probably why the quadmath is there. Hmm, if we fix it we will need to test it...
Chuck
Note that compiling from source is different and will generally use different libraries. We don't use Accelerate because it is buggy, not thread safe, and it appears Apple is not interested in doing anything about that.
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On Wed, Jan 30, 2019 at 6:03 PM Charles R Harris charlesr.harris@gmail.com wrote:
On Wed, Jan 30, 2019 at 6:32 PM Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jan 30, 2019 at 5:19 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno paterno@fnal.gov wrote:
Hello,
I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib.
While investigating the source of this problem, I downloaded the numpy source ( https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce... ), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation.
I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS.
I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing?
That's interesting, it looks like the wheel includes four libraries:
-rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 libopenblasp-r0.3.5.dev.dylib -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib
I thought we only needed the openblas, but that in turn probably depends on libgcc. But why we have the fortran library and quadmath escapes me. Perhaps someone else knows.
I suspect it's a leftover from when we were using ATLAS, we did need a Fortran runtime library at some point. The cause will be somewhere in the numpy-wheel build scripts, there is a gfortran-install git submodule: https://github.com/MacPython/numpy-wheels
And fortran is probably why the quadmath is there. Hmm, if we fix it we will need to test it...
I opened an issue: https://github.com/MacPython/numpy-wheels/issues/42
Ralf
Chuck
Note that compiling from source is different and will generally use different libraries. We don't use Accelerate because it is buggy, not thread safe, and it appears Apple is not interested in doing anything about that.
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
I have a different view on the assessments in this thread, having built OpenBLAS manually a number of times on different platforms now.
Those shared objects packed in the wheel, including libgfortran and libquadmath, are proper runtime dependencies for the OpenBLAS library we ship with wheels, not artifacts of an old ATLAS dependency structure. Another way to achieve compliance with the wheels standard is to statically link them in when we build OpenBLAS on macpython / manually. This seems to be relatively doable on some platforms, and harder on others.
There is a demonstration for Mac OS + NumPy available: https://github.com/numpy/numpy/pull/13191
On Linux, it is much harder, we would need a custom build of gcc from source with -fPIC compiler flag used to build libgfortran.a. The Julia language also faced this challenge on static links: https://github.com/JuliaLang/julia/issues/326#issuecomment-191781005
I'm not saying we should jump on static linking of the GCC runtime into OpenBLAS right away, but removing those shared objects from the wheels without a linking change doesn't seem quite right unless I'm missing something major. If we do eventually embrace the static link of the GCC runtime into the OpenBLAS wheels, this also makes our daily CI infrastructure less complex because we don't get pinned to specific runtime library versions of libgfortran / libquadmath & could likely just remove the gfortran-install submodule from our wheels workflow as well.
But we don't get that gain for nothing--we do transfer some non-trivial burden to "upstream" builds of OpenBLAS & things do mostly tend to work the way they are now. The PEPs surrounding the wheel ecosystem also contain some cautions about the complexities of trying to static link with default OS lib availabilities.
Best wishes, Tyler
On Thu, 31 Jan 2019 at 15:58, Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jan 30, 2019 at 6:03 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Wed, Jan 30, 2019 at 6:32 PM Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jan 30, 2019 at 5:19 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno paterno@fnal.gov wrote:
Hello,
I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib.
While investigating the source of this problem, I downloaded the numpy source ( https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce... ), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation.
I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS.
I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing?
That's interesting, it looks like the wheel includes four libraries:
-rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 libopenblasp-r0.3.5.dev.dylib -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib
I thought we only needed the openblas, but that in turn probably depends on libgcc. But why we have the fortran library and quadmath escapes me. Perhaps someone else knows.
I suspect it's a leftover from when we were using ATLAS, we did need a Fortran runtime library at some point. The cause will be somewhere in the numpy-wheel build scripts, there is a gfortran-install git submodule: https://github.com/MacPython/numpy-wheels
And fortran is probably why the quadmath is there. Hmm, if we fix it we will need to test it...
I opened an issue: https://github.com/MacPython/numpy-wheels/issues/42
Ralf
Chuck
Note that compiling from source is different and will generally use different libraries. We don't use Accelerate because it is buggy, not thread safe, and it appears Apple is not interested in doing anything about that.
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Yeah, the libgfortran dependency is expected. The question is why this should cause any problems. The wheel building infrastructure goes to great lengths to make sure that the dynamic libraries in the wheels shouldn't interfere with other packages. The original complaint in this thread is very vague...
On Wed, Apr 10, 2019, 22:16 Tyler Reddy tyler.je.reddy@gmail.com wrote:
I have a different view on the assessments in this thread, having built OpenBLAS manually a number of times on different platforms now.
Those shared objects packed in the wheel, including libgfortran and libquadmath, are proper runtime dependencies for the OpenBLAS library we ship with wheels, not artifacts of an old ATLAS dependency structure. Another way to achieve compliance with the wheels standard is to statically link them in when we build OpenBLAS on macpython / manually. This seems to be relatively doable on some platforms, and harder on others.
There is a demonstration for Mac OS + NumPy available: https://github.com/numpy/numpy/pull/13191
On Linux, it is much harder, we would need a custom build of gcc from source with -fPIC compiler flag used to build libgfortran.a. The Julia language also faced this challenge on static links: https://github.com/JuliaLang/julia/issues/326#issuecomment-191781005
I'm not saying we should jump on static linking of the GCC runtime into OpenBLAS right away, but removing those shared objects from the wheels without a linking change doesn't seem quite right unless I'm missing something major. If we do eventually embrace the static link of the GCC runtime into the OpenBLAS wheels, this also makes our daily CI infrastructure less complex because we don't get pinned to specific runtime library versions of libgfortran / libquadmath & could likely just remove the gfortran-install submodule from our wheels workflow as well.
But we don't get that gain for nothing--we do transfer some non-trivial burden to "upstream" builds of OpenBLAS & things do mostly tend to work the way they are now. The PEPs surrounding the wheel ecosystem also contain some cautions about the complexities of trying to static link with default OS lib availabilities.
Best wishes, Tyler
On Thu, 31 Jan 2019 at 15:58, Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jan 30, 2019 at 6:03 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Wed, Jan 30, 2019 at 6:32 PM Ralf Gommers ralf.gommers@gmail.com wrote:
On Wed, Jan 30, 2019 at 5:19 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno paterno@fnal.gov wrote:
Hello,
I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib.
While investigating the source of this problem, I downloaded the numpy source ( https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce... ), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation.
I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS.
I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing?
That's interesting, it looks like the wheel includes four libraries:
-rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 libopenblasp-r0.3.5.dev.dylib -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib
I thought we only needed the openblas, but that in turn probably depends on libgcc. But why we have the fortran library and quadmath escapes me. Perhaps someone else knows.
I suspect it's a leftover from when we were using ATLAS, we did need a Fortran runtime library at some point. The cause will be somewhere in the numpy-wheel build scripts, there is a gfortran-install git submodule: https://github.com/MacPython/numpy-wheels
And fortran is probably why the quadmath is there. Hmm, if we fix it we will need to test it...
I opened an issue: https://github.com/MacPython/numpy-wheels/issues/42
Ralf
Chuck
Note that compiling from source is different and will generally use different libraries. We don't use Accelerate because it is buggy, not thread safe, and it appears Apple is not interested in doing anything about that.
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
participants (5)
-
Charles R Harris -
Marc F Paterno -
Nathaniel Smith -
Ralf Gommers -
Tyler Reddy