Detecting C API mismatch (was Managing Python with NumPy and many external libraries on multiple Windows machines)
Hi,
I worked on some code to detect C API mismatches both for developers and for users:
http://github.com/cournape/numpy/tree/runtime_feature
It adds the following: - if a numpy or ufunc function is added in the C API without the NPY_FEATURE_VERSION to be updated, a warning is generated at built time (the warning is turned into an exception for release) - I added a function PyArray_GetNDArrayCFeatureVersion which returns the C API version, and the version is checked in import_array. If the compile-time version > import-time version, an import error is raised, so the following happens (assuming the ABI is not changed).
So we keep backward compatibility (building an extension with say numpy 1.2.1 will still work after installing numpy 1.3), and forward incompatibility is detected (building an extension with numpy 1.3.0 and importing it with installed numpy 1.2.1 will fail).
Ironically, adding the function means that we have to add one function to the C API, so this will not be useful for numpy < 1.4, but I don't think it is possible to do it without modifying the C API.
cheers,
David
2009/5/10 David Cournapeau cournape@gmail.com:
I worked on some code to detect C API mismatches both for developers and for users:
Great, thanks for taking care of this!
I think the message "ABI version %%x of C-API" is unclear, maybe simply use "ABI version %%x" on its own.
The hash file can be loaded in one line with
np.loadtxt('/tmp/dat.dat', usecols=(0, 2), dtype=[('api', 'S10'), ('hash', 'S32')])
The rest looks good.
Cheers Stéfan
Stéfan van der Walt wrote:
2009/5/10 David Cournapeau cournape@gmail.com:
I worked on some code to detect C API mismatches both for developers and for users:
Great, thanks for taking care of this!
I think the message "ABI version %%x of C-API" is unclear, maybe simply use "ABI version %%x" on its own.
Ok, I changed it.
The hash file can be loaded in one line with
np.loadtxt('/tmp/dat.dat', usecols=(0, 2), dtype=[('api', 'S10'), ('hash', 'S32')])
Well, we need to do this at build time, and we can't assume numpy is already installed when building numpy :)
David
2009/5/10 Stéfan van der Walt stefan@sun.ac.za:
I think the message "ABI version %%x of C-API" is unclear, maybe simply use "ABI version %%x" on its own.
The hash file can be loaded in one line with
np.loadtxt('/tmp/dat.dat', usecols=(0, 2), dtype=[('api', 'S10'), ('hash', 'S32')])
The rest looks good.
Ok, I committed the branch to numpy trunk.
thanks for the review,
David
On Sat, May 9, 2009 at 11:45 PM, David Cournapeau cournape@gmail.comwrote:
Hi,
I worked on some code to detect C API mismatches both for developers and for users:
http://github.com/cournape/numpy/tree/runtime_feature
It adds the following:
- if a numpy or ufunc function is added in the C API without the
NPY_FEATURE_VERSION to be updated, a warning is generated at built time (the warning is turned into an exception for release)
- I added a function PyArray_GetNDArrayCFeatureVersion which returns
the C API version, and the version is checked in import_array. If the compile-time version > import-time version, an import error is raised, so the following happens (assuming the ABI is not changed).
So we keep backward compatibility (building an extension with say numpy 1.2.1 will still work after installing numpy 1.3), and forward incompatibility is detected (building an extension with numpy 1.3.0 and importing it with installed numpy 1.2.1 will fail).
Ironically, adding the function means that we have to add one function to the C API, so this will not be useful for numpy < 1.4, but I don't think it is possible to do it without modifying the C API.
Why not just use the current API to get the number? That looks easy to do, what is the problem? I'll fix it up if you want.
Chuck
On Sun, May 10, 2009 at 10:13 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sat, May 9, 2009 at 11:45 PM, David Cournapeau cournape@gmail.comwrote:
Hi,
I worked on some code to detect C API mismatches both for developers and for users:
http://github.com/cournape/numpy/tree/runtime_feature
It adds the following:
- if a numpy or ufunc function is added in the C API without the
NPY_FEATURE_VERSION to be updated, a warning is generated at built time (the warning is turned into an exception for release)
- I added a function PyArray_GetNDArrayCFeatureVersion which returns
the C API version, and the version is checked in import_array. If the compile-time version > import-time version, an import error is raised, so the following happens (assuming the ABI is not changed).
So we keep backward compatibility (building an extension with say numpy 1.2.1 will still work after installing numpy 1.3), and forward incompatibility is detected (building an extension with numpy 1.3.0 and importing it with installed numpy 1.2.1 will fail).
Ironically, adding the function means that we have to add one function to the C API, so this will not be useful for numpy < 1.4, but I don't think it is possible to do it without modifying the C API.
Why not just use the current API to get the number? That looks easy to do, what is the problem? I'll fix it up if you want.
As you may have noticed, I really, really, don't like adding functions to the API ;) Especially unneeded ones or ones that could be done at the python level. So I think the thing to do here is split the version into two 16 bit parts, then start with API version 0x000A and ABI version 0x0100 with the version number being 0x01000009. There is nothing sacred about the numbers as long as they remain ordered.
Chuck
Charles R Harris wrote:
As you may have noticed, I really, really, don't like adding functions to the API ;)
Me neither :)
Especially unneeded ones or ones that could be done at the python level. So I think the thing to do here is split the version into two 16 bit parts, then start with API version 0x000A and ABI version 0x0100 with the version number being 0x01000009.
I don't think you can do that: you will break a lot of code if you do so. The currently built extensions will fail to load if the number is any different with a new numpy. We need two independent numbers: one which should prevents loading if the number is any different (the ABI part) and one which should prevents loading if the compile-time number is strictly greater than the runtime one. As the ABI part is already checked in since at least numpy 1.2 and maybe lower, you can't change it easily.
They could be part of the same underlying int if we did that from the beginning, but that's not the case.
There is nothing sacred about the numbers as long as they remain ordered.
if you change the number NPY_VERSION, you break every single extension already built. With my scheme, you needs one more function, but you don't break backward compatibility.
cheers,
David
-
Charles R Harris -
David Cournapeau -
David Cournapeau -
Stéfan van der Walt