[Numpy-discussion] Help - numpy / scipy binary compatibility

[Matthew Brett]

Hi,

On Sun, Aug 10, 2014 at 11:41 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Mon, Aug 11, 2014 at 8:39 AM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>>
>>
>>
>>
>> On Sat, Aug 9, 2014 at 5:04 AM, David Cournapeau <cournape at gmail.com>
>> wrote:
>>>
>>>
>>>
>>>
>>> On Sat, Aug 9, 2014 at 10:41 AM, Matthew Brett <matthew.brett at gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I would be very happy of some help trying to work out a numpy package
>>>> binary incompatibility.
>>>>
>>>> I'm trying to work out what's happening for this ticket:
>>>>
>>>> https://github.com/scipy/scipy/issues/3863
>>>>
>>>> which I summarized at the end:
>>>>
>>>> https://github.com/scipy/scipy/issues/3863#issuecomment-51669861
>>>>
>>>> but basically, we're getting these errors:
>>>>
>>>> RuntimeWarning: numpy.dtype size changed, may indicate binary
>>>> incompatibility
>>>>
>>>> I now realize I am lost in the world of numpy / scipy etc binary
>>>> compatibility, I'd really like some advice.    In this case
>>>>
>>>> numpy == 1.8.1
>>>> scipy == 0.14.0 - compiled against numpy 1.5.1
>>>> scikit-learn == 0.15.1 compiled against numpy 1.6.0
>>>>
>>>> Can y'all see any potential problem with those dependencies in binary
>>>> builds?
>>>>
>>>> The relevant scipy Cython c files seem to guard against raising this
>>>> error by doing not-strict checks of the e.g. numpy dtype, so I am
>>>> confused how these errors come about.  Can anyone give any pointers?
>>>
>>>
>>> Assuming the message is not bogus, I would try import von_mises with a
>>> venv containing numpy 1.5.1, then 1.6.0, etc... to detect when the change
>>> happened.
>>
>>
>> That should be a recent change in 1.8.1, either because the dtype size did
>> actually change or because the silencing of this message in NoseTester
>> (numpy.testing) is not effective anymore.
>>
>> Note that the warning is too agressive, because it triggers both on ABI
>> breaks and on backwards-compatible extensions to dtype. That's why it's
>> filtered in numpy.testing. This was reported to Cython a while ago, not sure
>> they've fixed the issue in the meantime or not.
>
>
> Never mind, saw that you already figured out that it's due to scikit-learn:
> https://github.com/scipy/scipy/issues/3863

Yes, sorry, I should have reported back to the list - the problem was
that sklearn is removing all the warnings filters that numpy installs.

For reference, no, Cython left the warnings are they were.

As you implied, Cython raises these warnings (if not filtered by
numpy) if the various numpy C structs have increased size since
compile time, including numpy dtype ufuncs and ndarrays.  The structs
get bigger when we add to the entries in the struct; this is backwards
compatible but not forwards compatible.

Because I got very confused I wrote this little piece of code to show
current compiled and in-memory sizes of dtypes and ufuncs:

https://github.com/matthew-brett/npsizes

Giving (result of ./npreport):

Numpy version: 1.5.1
dtype: static size 80; memory size 80
ndarray: static size 80; memory size 80
ufunc: static size 144; memory size 144

Numpy version: 1.6.0
dtype: static size 80; memory size 80
ndarray: static size 80; memory size 80
ufunc: static size 144; memory size 144

Numpy version: 1.7.1
dtype: static size 88; memory size 88
ndarray: static size 80; memory size 80
ufunc: static size 176; memory size 176

Numpy version: 1.8.1
dtype: static size 88; memory size 88
ndarray: static size 80; memory size 80
ufunc: static size 192; memory size 192

on OSX....

Cheers,

Matthew