Hi - I have recently come across this problem. On my mac, I build a Fortran
code, producing a shared object, that I import into Python along with
numpy. This had been working fine until recently when I started seeing sag
faults deep inside the Fortran code, usually in Fortran print statements. I
tracked this down to a gfortran version issue.
I use the brew installation of Python and gcc (using the most recent
version, 8.2.0). gcc of course installs a version of libgfortran.dylib.
Doing a lsof of a running Python, I see that it finds that copy of
libgfortran, and also a copy that was downloaded with numpy
Looking at numpy's copy of libgfortran, I see that it is version 4.9.0,
much older. Since my code is importing numpy first, the OS seems be using
numpy's version of libgfortran to link when importing my code. I know from
other experience that older versions of libgfortran are not compatible with
code compiled using a new version of gfortran and so therefore segfaults
If I download the numpy source and do python setup.py install, I don't have
After this long description, my question is why is such an old version of
gcc used to build the distribution of numpy that gets installed from pypi?
gcc version 4.9.0 is from 2014. Can a newer version be used?
TL;DR - should we revert the attribute-hiding constructs in
ndarraytypes.h and unify PyArrayObject_fields with PyArrayObject?
NumPy 1.8 deprecated direct access to PyArrayObject fields. It made
PyArrayObject "opaque", and hid the fields behind a PyArrayObject_fields
with a comment about moving this to a private header. In order to access
the fields, users are supposed to use PyArray_FIELDNAME functions, like
PyArray_DATA and PyArray_NDIM. It seems there were thoughts at the time
that numpy might move away from a C-struct based
underlying data structure. Other changes were also made to enum names,
but those are relatively painless to find-and-replace.
NumPy has a mechanism to manage deprecating APIs, C users define
NPY_NO_DEPRICATED_API to a desired level, say NPY_1_8_API_VERSION, and
can then access the API "as if" they were using NumPy 1.8. Users who do
not define NPY_NO_DEPRICATED_API get a warning when compiling, and
default to the pre-1.8 API (aliasing of PyArrayObject to
PyArrayObject_fields and direct access to the C struct fields). This is
convenient for downstream users, both since the new API does not provide
much added value, and it is much easier to write a->nd than
PyArray_NDIM(a). For instance, pandas uses direct assignment to the data
field for fast json parsing
via chunks. Working around the new API in pandas would require more
engineering. Also, for example, cython has a mechanism to transpile
python code into C, mapping slow python attribute lookup to fast C
struct field access
In a parallel but not really related universe, cython recently upgraded
the object mapping so that we can quiet the annoying "size changed"
runtime warning https://github.com/numpy/numpy/issues/11788 without
requiring warning filters, but that requires updating the numpy.pxd file
provided with cython, and it was proposed that NumPy actually vendor its
own file rather than depending on the cython one
We have now made further changes to our API. In NumPy 1.14 we changed
UPDATEIFCOPY to WRITEBACKIFCOPY, and in 1.16 we would like to deprecate
PyArray_SetNumericOps and PyArray_GetNumericOps. The strange warning
when NPY_NO_DEPRICATED_API is annoying. The new API cannot be supported
by cython without some deep surgery
(https://github.com/cython/cython/pull/2640). When I tried dogfooding an
updated numpy.pxd for the only cython code in NumPy, mtrand.pxy, I came
across some of these issues (https://github.com/numpy/numpy/pull/12284).
Forcing the new API will require downstream users to refactor code or
re-engineer constructs, as in the pandas example above.
Is the attribute-hiding effort worth it? Should we give up, revert the
PyArrayObject/PyArrayObject_fields division and allow direct access from
C to the numpy internals? Is there another path forward that is less
Just a heads up that I am planning on making a 1.15.4 release this coming
weekend. The only fixes planned at this point are
- BUG: Fix fill value in masked array '==' and '!=' ops, #12257
- BUG: clear buffer_info_cache on scalar dealloc, #12249
If there are other fixes that you think needed, please let me know.
The team at BIDS meets once a week to discuss progress, priorities, and
roadblocks. While our priorities are broadly determined by the project
roadmap , we would like to provide an opportunity for the community
to give more regular and detailed feedback on our work.
We therefore invite you to join us for our weekly calls,
each **Wednesday from 12:00 to 13:00 Pacific Time**.
Detail of the next meeting (2018-10-24) is given in the agenda ,
which is a living document. Feel free to add topics you wish to discuss.
We hope to see you there!
Stéfan, Tyler, Matti