[Wheel-builders] pinned-down dependencies for building wheel versus loose dependencies for experimentation

[Philippe Ombredanne]

On Wed, Sep 7, 2016 at 11:00 PM, vitaly numenta
<vitaly.krugl.numenta at gmail.com> wrote:
> I would like to structure my project such that when I build a wheel for
> deployment to PyPi, the wheel uses the pinned-down versions of dependencies,
> but during development/experimental builds uses the
> abstract/loosely-versioned dependencies.
>
> I want developers to be able to use the abstract dependencies for
> experimenting with my package against other versions of the dependencies,
> but installs of my wheel from PyPi need to reference the concrete
> (pinned-down) dependencies with which my build system actually tested my
> wheel.
>
> Users that install my wheel from PyPi will not (and should not need to) have
> access to my package's requirements.txt.

Vitaly:
you cannot treat setup.py as a both a flexible set of version ranges
and a pinned set at the same time.
For a detailed explanation please read Donald's post here:
https://caremad.io/2013/07/setup-vs-requirement/

> However, when building a wheel, I don't see a way to build one that incorporates the pinned-down dependencies from requirements.txt instead of the loosely-versioned requirements from `extras_require`.

The requirements are not used by setup.py and THEY SHOULD NOT, IMHO.
Use requirements for pinned versions, not setup.py.

That said, here is what I would do if I had your requirements to
fulfill, and still keeping up with the spirit and intent of what a
setup.py (abstract) and requirements.txt (concrete) use should be:

1. for my base library (that I will call library A), create a setup.py
with install_requires with either no version ranges or a minimum
version or a permissive version range and push that to Pypi
2. for that same library A, create a pinned requirements.txt with pip
freeze for development usage
3. create a new library B that will have no code except for a setup.py
. That setup.py will have install_requires for all the pinned deps
with exact version of library A and the pinned version of library A
too.

With this config:
- you users can install library B and get exactly specified versions
for Library A and its deps.
- your developers can install library A instead and have flexibility
if which dep versions they want to use and explore, and can also use
exactly pinned down requirements if needed.

I amot using this kinda of approach but here are some pointers:

For an (old) example of a setup.py-only project for library B you can
check this simple
https://github.com/pombredanne/anyreadline/blob/master/setup.py
(just as an example it is outdated, and predates extras_require in particular)

For an example of a setup.py with some extras_requires you can check.
There I use lax enough version ranges and in some case very specific
ones to work around some bugs in upstream lxml:
https://github.com/nexB/scancode-toolkit/blob/develop/setup.py

When built as an application, I eventually freeze requirements to get
exact versions but I rarely if ever freeze versions in setup.py.

-- 
Cordially
Philippe Ombredanne