[SciPy-Dev] pyproject.toml for scipy dependent project

[Roman Yurchak]

Thank you for the detailed explanations! I see you have also opened a PR 
to document it https://github.com/scipy/scipy/pull/11792

Thanks, that's very useful!

-- 
Roman

On 04/04/2020 13:33, Pauli Virtanen wrote:
> Hi,
> 
> pe, 2020-04-03 kello 20:23 +0200, Ralf Gommers kirjoitti:
>> Great find, thanks! That issue says that the ABI is backwards but not
>> forwards compatible I believe, given Pauli's comment about being free
>> to add extra signatures. So go with (b) I'd say.
> 
> The situation with the Cython APIs is to my understanding that
> currently the ABI is both forward and backward compatible, so projects
> built with newer Scipy versions work also with older versions.
> 
> However, whether this remains true depends on what we do in the future.
> In particular, the situation is not necessarily the same as with Numpy:
> 
> - For plain functions (e.g. cython_blas/lapack): Cython checks function
> signatures at import time, so if we change signatures later on (or you
> use functions not available in older Scipy), you'll get an import error
> at import time. This works on a per-function basis, so you don't need
> to care about functions you don't use. Any signature change will break
> both API and ABI compatibility, so there's no big separate ABI issue
> (binaries will work with any Scipy versions they can be compiled with).
> 
> - With structs and types defined in our .pxd files: if we make changes
> to them, the ABI can become **both** forward and backward incompatible,
> depending on who allocates them and what changes were made and how they
> are used. AFAIK, Cython does not check type/struct signatures (only
> names), so errors made result to crashes at runtime. So the situation
> here is basically the same as in plain C.
> 
> E.g. in cython_optimize the structs are supposed to be allocated on the
> caller side --- if we make changes to the structs by adding things at
> the end, code compiled with *new* Scipy versions is compatible with old
> versions, but not vice versa. So exactly opposite of the Numpy
> situation. If stuff is allocated on the callee side, it's then
> different.
> 
> To avoid mess, it probably is simplest if we will **not allow any
> changes to be made in any public structs and types in the Cython API**.
> If changes have to be made, the names of the types/structs must be
> changed (and we maybe could decide on a naming scheme). Then the
> situation can be dealt similarly as with adding new APIs and
> deprecating/removing old ones.
> 
> Even simpler is if we additionally will **not allow any non-opaque
> structs in the public Cython API**. I.e. no struct members may be
> listed in public .pxd files. All data structures must be allocated by a
> separate public allocation function, and all attributes accessed by
> accessor functions. (Types, defines, etc. would still have to be
> considered final, as above.)
> 
> Maintaining a Numpy-style forward-compatible ABI situation requires a
> bit more care in managing who allocates what data structures and how
> they are used. However, because unlike Numpy we don't have big legacy
> API here, I think we would be able to maintain the simpler "binaries
> will work with any Scipy versions they can be compiled with" situation.
> 
> 	Pauli
> 
> 
>> Would be good to know what happens if you built SciPy master with
>>> latest
>>>> Cython, built your project against it, then ran tests against an
>>> older
>>>> SciPy binary - have you tried?
>>>
>>> You are right, it seems to work fine regardless of the scipy
>>> version for
>>> the build.
>>>
>>> Built scikit-learn in develop mode (to bypass pyproject.toml)
>>> with:
>>> Python 3.6, numpy 1.14.5, cython 0.29.16 and scipy master. Run
>>> scikit-learn test suite with scipy 0.19.1 -> no errors. Only a
>>> (small)
>>> part of the scipy Cython API is used there though.
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
> 
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>