install_recommends proposal
Hi there,
Currently setup.py allows install_requires to specify the dependencies, possibly including exact version numbers of a package.
It is however beneficial to only specify the *minimum* of expected requirements in there, to retain maximum flexibility of use. This means you'd typically say:
install_requires = [ 'foo', 'bar >= 1.3', ]
if you know you need some version of foo, and at least version 1.3 of your package. This allows developers who use your package to choose themselves which version of foo and bar they want to use, without getting version conflicts as long as you stay within the stated constraints.
If you write a framework, or a library that uses other libraries, you sometimes want to do something more. You'd like to ensure that if someone uses version X of your framework, they use a list of *exact* dependencies, that is, exactly foo 1.2 and bar 1.4.3. This makes all installations of framework X bug for bug compatible. New releases of packages you depend on don't break your framework install, as the framework install will ensure those versions.
I'd therefore like to propose a new setup.py section where these can be specified:
install_recommends = [ 'foo == 1.2', 'bar == 1.4.3', ]
The person or tool installing the package is entirely free to ignore this information; it's a recommendation, not a requirement. The only specifications that can be made are *exact*; there is no support for anything but '=='. Distutils at most needs to be taught that install_recommends cannot directly conflict with install_requires. This would be a conflict:
install_requires = [ 'bar > 1.3' ], install_recommends = [ 'bar == 1.2', ]
By default, distutils and setuptools would otherwise ignore this information. I do presume setuptools would include this information in egg-info so that external tools can make use of it.
This information can then be (optionally) used by a tool like buildout to ensure exact dependencies are installed (once zc.buildout is taught to use this information, of course). This means I can release a framework that allows flexibility to knowledgable developers, but also, in its default install, doesn't have the chance of breaking the moment someone releases a new version of some dependency.
For an extensive article where I describe what led me to look for this read this:
http://faassen.n--tree.net/blog/view/weblog/2007/09/26/0
It goes into much detail on the concerns and reasoning behind this in the context of the development of Zope 3, and Grok, a framework based on many Zope 3 packages. This isn't a theoretical problem but is currently biting us regularly. I'll also note that it looks like zc.buildout doesn't appear to need much work to start using this information (but Jim will correct me if I'm wrong).
Regards,
Martijn
The allmydata.org "Tahoe" project does something similar in a non- machine-readable way. We specify a minimum version if one is known, and we also mention specific versions that have been tested and are known to work:
http://allmydata.org/trac/tahoe/browser/README
The fact that this kind of practice is common suggests that a machine- readable implementation of it might be useful.
Regards,
Zooko
Martijn Faassen wrote:
Hi there,
Currently setup.py allows install_requires to specify the dependencies, possibly including exact version numbers of a package.
It is however beneficial to only specify the *minimum* of expected requirements in there, to retain maximum flexibility of use. This means you'd typically say:
install_requires = [ 'foo', 'bar >= 1.3', ]
if you know you need some version of foo, and at least version 1.3 of your package. This allows developers who use your package to choose themselves which version of foo and bar they want to use, without getting version conflicts as long as you stay within the stated constraints.
Can't you use the extras_require feature for this?
Michael Hoffman wrote:
Martijn Faassen wrote:
Hi there,
Currently setup.py allows install_requires to specify the dependencies, possibly including exact version numbers of a package.
It is however beneficial to only specify the *minimum* of expected requirements in there, to retain maximum flexibility of use. This means you'd typically say:
install_requires = [ 'foo', 'bar >= 1.3', ]
if you know you need some version of foo, and at least version 1.3 of your package. This allows developers who use your package to choose themselves which version of foo and bar they want to use, without getting version conflicts as long as you stay within the stated constraints.
Can't you use the extras_require feature for this?
It is my understanding that extra_requires is used to express *extra*, optional requirements beyond those in install_requires. That's not at all what I'm trying to talk about, so my apologies for being so unclear.
install_recommends as I proposed it would be used to offer recommendations for the core requirements (or, I guess, also recommendations of versions for the extra requirements if desired). So, the exact same requirements are expressed, just with an additional tightening of version numbers.
I understand how the name "install_recommends" could be confusing terminology given the way package managers use the term 'recommended packages', which means extra that you could install too to get more features. "install_prefers" instead?
An alternative would be to expand the syntax of install_requires and extra_require to allow the recommended version number hint. Something like this:
install_requires = [ 'foo (1.2.1)', 'bar >= 1.3 (1.3.2)', ]
Installation tools are free to ignore the latter number and just install something that fits the basic requirement:
install_requires = [ 'foo', 'bar >= 1.3', ]
But an installation could be taught to take this into account and get those versions, and do the equivalent of this:
install_requires = [ 'foo == 1.2.1', 'bar == 1.3.2', ]
The install tool could choose in the case of conflicting version numbers to pick the one in the outer packages. So, if foo depends on bar, and bar says it would prefer qux 1.3, while foo actually prefers 1.3.2, foo would trump bar.
Again, all this would be entirely optional behavior of the installation tool. I'm just asking for a way to express this information in the package metadata at all, so that install tools could be taught to use it.
Regards,
Martijn
At 10:38 PM 9/26/2007 +0200, Martijn Faassen wrote:
An alternative would be to expand the syntax of install_requires and extra_require to allow the recommended version number hint. Something like this:
install_requires = [ 'foo (1.2.1)', 'bar >= 1.3 (1.3.2)', ]
Installation tools are free to ignore the latter number and just install something that fits the basic requirement:
install_requires = [ 'foo', 'bar >= 1.3', ]
But an installation could be taught to take this into account and get those versions, and do the equivalent of this:
install_requires = [ 'foo == 1.2.1', 'bar == 1.3.2', ]
The install tool could choose in the case of conflicting version numbers to pick the one in the outer packages. So, if foo depends on bar, and bar says it would prefer qux 1.3, while foo actually prefers 1.3.2, foo would trump bar.
Note that the previous proposal for having a way to specify "or'ed" conditions would allow this, but it isn't going to happen until I get an 0.6 final out so I can focus on refactoring and new features in 0.7.
Phillip J. Eby wrote:
At 10:38 PM 9/26/2007 +0200, Martijn Faassen wrote:
[snip]
Note that the previous proposal for having a way to specify "or'ed" conditions would allow this, but it isn't going to happen until I get an 0.6 final out so I can focus on refactoring and new features in 0.7.
I will look for this proposal, any hints as to where to look?
My first response, not based on actually looking at the proposal is that a simple 'or' condition would not be sufficient, as the package manager doesnt' know which clause in the condition to look at to find the preferred version. My second response upon pondering a bit more is that the package manager could be told to always prefer the or-ed condition that is most specific.
Do you have any (vague) estimates on timeline before a 0.7 with such a feature would become available? I'm asking as ... oh, the pain, the pain! :)
Regards,
Martijn
Martijn Faassen wrote: [snip]
My first response, not based on actually looking at the proposal is that a simple 'or' condition would not be sufficient, as the package manager doesnt' know which clause in the condition to look at to find the preferred version. My second response upon pondering a bit more is that the package manager could be told to always prefer the or-ed condition that is most specific.
This assumes of course that setuptools has a way to let installation tools:
* access the or clauses
* determine which ones are the most specific
Or has a resolution algorithm for this itself.
We also have to resolve situations where we could have multiple specific versions:
Package A says (hypothetical syntax):
install_requires = [ 'B or B == 1.3', 'C or C == 1.7', ]
and then a package B which says:
install_requires = [ 'C or C == 1.7.1', ]
which one to pick? C will do, but if we want to be specific, should we pick C 1.7 or C 1.7.1?
I propose we let the outer package (A) break the contention and thus decide on C 1.7. The inner package winning would otherwise block framework packages from having the ability to make informed decisions to diverge from recommendations lower down the dependency tree.
Regards,
Martijn
On Wed, Sep 26, 2007 at 10:38:49PM +0200, Martijn Faassen wrote:
I understand how the name "install_recommends" could be confusing terminology
Yes it is
given the way package managers use the term 'recommended packages', which means extra that you could install too to get more features. "install_prefers" instead?
An alternative would be to expand the syntax of install_requires and extra_require to allow the recommended version number hint. Something like this:
install_requires = [ 'foo (1.2.1)', 'bar >= 1.3 (1.3.2)', ]
A really human-readable version would be:
install_requires = [ 'foo (prefer 1.2.1)', 'bar >= 1.3 (prefer 1.3.2)', ]
Marius Gedminas
Marius Gedminas wrote:
On Wed, Sep 26, 2007 at 10:38:49PM +0200, Martijn Faassen wrote:
I understand how the name "install_recommends" could be confusing terminology
Yes it is
given the way package managers use the term 'recommended packages', which means extra that you could install too to get more features. "install_prefers" instead?
An alternative would be to expand the syntax of install_requires and extra_require to allow the recommended version number hint. Something like this:
install_requires = [ 'foo (1.2.1)', 'bar >= 1.3 (1.3.2)', ]
A really human-readable version would be:
install_requires = [ 'foo (prefer 1.2.1)', 'bar >= 1.3 (prefer 1.3.2)', ]
Or make 'prefer' an actual version operator like == or >=:
install_requires = [ 'foo prefer 1.2.1', 'bar >=1.3, prefer 1.3.2', ]
The advantage of including it into install_requires is that you avoid redudancy by not having to repeat the list of required packages twice.
Another advantage might be that the information would be written to a place that we already know (EGG-INFO/requires.txt) and have extraction tools for. The only problem with that would be that eggs with a "modern" EGG-INFO like that would not be installable on older setuptools versions that know nothing about the 'prefer' operator. Then again, that would also be the case with Martijn's proposed install_recommends parameter.
Marius Gedminas wrote:
On Wed, Sep 26, 2007 at 10:38:49PM +0200, Martijn Faassen wrote:
I understand how the name "install_recommends" could be confusing terminology
Yes it is
given the way package managers use the term 'recommended packages', which means extra that you could install too to get more features. "install_prefers" instead?
An alternative would be to expand the syntax of install_requires and extra_require to allow the recommended version number hint. Something like this:
install_requires = [ 'foo (1.2.1)', 'bar >= 1.3 (1.3.2)', ]
A really human-readable version would be:
install_requires = [ 'foo (prefer 1.2.1)', 'bar >= 1.3 (prefer 1.3.2)', ]
This is kind of an old thread, but I personally would really like install_recommends. One use case is to recommend packages that are not strictly required. E.g., I have WSGI applications that work well with PasteScript for serving the application, but PasteScript isn't actually required. This leads to an awkward situation where I have to choose whether to require a package that isn't actually needed (e.g., it's never imported anywhere), or leave it out and confuse people who install the application. If preferred versions were part of install_requires, I don't see how I could achieve this.
The workingenv requirements, and now poach-eggs, try to solve this by installing a description of packages instead of just a single package, but this would make that mostly unnecessary.
At 02:45 PM 10/8/2007 -0500, Ian Bicking wrote:
Marius Gedminas wrote:
On Wed, Sep 26, 2007 at 10:38:49PM +0200, Martijn Faassen wrote:
I understand how the name "install_recommends" could be confusing terminology
Yes it is
given the way package managers use the term 'recommended packages', which means extra that you could install too to get more features. "install_prefers" instead?
An alternative would be to expand the syntax of install_requires and extra_require to allow the recommended version number hint. Something like this:
install_requires = [ 'foo (1.2.1)', 'bar >= 1.3 (1.3.2)', ]
A really human-readable version would be:
install_requires = [ 'foo (prefer 1.2.1)', 'bar >= 1.3 (prefer 1.3.2)', ]
This is kind of an old thread, but I personally would really like install_recommends. One use case is to recommend packages that are not strictly required. E.g., I have WSGI applications that work well with PasteScript for serving the application, but PasteScript isn't actually required. This leads to an awkward situation where I have to choose whether to require a package that isn't actually needed (e.g., it's never imported anywhere), or leave it out and confuse people who install the application. If preferred versions were part of install_requires, I don't see how I could achieve this.
That's what 'extras_require' is for; you can always have an extra named 'recommended', and then install 'Foo[recommended]' to get all the recommended stuff.
This is a very different use case than the original request of this thread (which is really just preferred packages/versions within requirement specs).
participants (7)
-
Ian Bicking -
Marius Gedminas -
Martijn Faassen -
Michael Hoffman -
Philipp von Weitershausen -
Phillip J. Eby -
zooko