[Wheel-builders] pinned-down dependencies for building wheel versus loose dependencies for experimentation

[Matthew Brett]

Hi,

On Thu, Sep 8, 2016 at 2:21 PM, Vitaly Kruglikov <vkruglikov at numenta.com> wrote:
> On 9/8/16, 1:35 AM, "Philippe Ombredanne" <pombredanne at nexb.com> wrote:
>
>>On Thu, Sep 8, 2016 at 10:23 AM, Philippe Ombredanne
>><pombredanne at nexb.com> wrote:
>>> On Wed, Sep 7, 2016 at 11:00 PM, vitaly numenta
>>> <vitaly.krugl.numenta at gmail.com> wrote:
>>>> I would like to structure my project such that when I build a wheel for
>>>> deployment to PyPi, the wheel uses the pinned-down versions of
>>>>dependencies,
>>>> but during development/experimental builds uses the
>>>> abstract/loosely-versioned dependencies.
>>>>
>>>> I want developers to be able to use the abstract dependencies for
>>>> experimenting with my package against other versions of the
>>>>dependencies,
>>>> but installs of my wheel from PyPi need to reference the concrete
>>>> (pinned-down) dependencies with which my build system actually tested
>>>>my
>>>> wheel.
>>>>
>>>> Users that install my wheel from PyPi will not (and should not need
>>>>to) have
>>>> access to my package's requirements.txt.
>>>
>>> Vitaly:
>>> you cannot treat setup.py as a both a flexible set of version ranges
>>> and a pinned set at the same time.
>>> For a detailed explanation please read Donald's post here:
>>> https://caremad.io/2013/07/setup-vs-requirement/
>>>
>>>> However, when building a wheel, I don't see a way to build one that
>>>>incorporates the pinned-down dependencies from requirements.txt instead
>>>>of the loosely-versioned requirements from `extras_require`.
>>>
>>> The requirements are not used by setup.py and THEY SHOULD NOT, IMHO.
>>> Use requirements for pinned versions, not setup.py.
>>
>>In particular if based on your email this is some code you are talking
>>about I would never do this:
>>https://github.com/numenta/nupic/blob/d2aad354226bba6b554c706e74e18b3ed441
>>5a0f/setup.py#L84
>>e.g. I would NEVER source install_requires from a requirements.txt
>>because this would rob my users and flexibility from any flexibility
>>in versions.
>>
>>I find quite unfortunate that there are so many setup.py that do this
>>and so many snippets and SO posts that recommend this.  **sigh** This
>>is the source of many problems, like the one you are facing.
>>
>>--
>>Cordially
>>Philippe Ombredanne
>
> Dear Philippe, thank you for your detailed follow-up. I have received the
> same negative feedback concerning the population of install_requires with
> pinned-down versions (via requirements.txt or inline) from a number of
> developers, including several wheel members (and yes, good guess with the
> nupic link :) ). It's a problem that we run into occasionally with our
> users as well as internally, so I agree that something needs to be done to
> address it.
>
> My colleagues and I are struggling to reconcile this problem: if our CI
> system only tests our library with a specific set of pinned-down
> dependencies, how can we deploy a wheel to PyPi that references a looser
> set of dependencies? What guarantee would there be that it will work
> correctly with the looser set of dependencies, since we can¹t possibly
> test with **all** of their combinations?
>
> The suggestion from your previous post for solving my problem by deploying
> library A with loose dependencies to PyPy as well as another ³codeless²
> library B with only setup.py containing the pinned-down dependencies would
> likely work. However, the additional complexity of the solution (more
> moving parts) is telling me that the problem I am trying to solve may be
> going against the grain of what python packaging and deployment intended.
>
> If it¹s common practice for wheels deployed to PyPi to incorporate
> loosely-versioned dependencies, then how do developers mitigate the risk
> that their wheel might fail with some combination of those dependencies?

I believe the most common route is to accept that risk, and mitigate
by adding travis-ci matrix entries with different combinations of
dependencies to test the common ones.  For example:

https://github.com/nipy/nibabel/blob/master/.travis.yml

Best,

Matthew