Feature request: Enable index shadowing via configuration or option
I submitted this pull request <https://bitbucket.org/hpk42/devpi/pull-request/139/only-return-python-packag...> to devpi, which would cause devpi to "shadow" package versions across indexes. I will quote my description of the pull request here: Say I have package 'a' on index 'x' which lists index 'y' as a base index. That is, 'x' inherits from 'y'. A common use case is to want to install package 'a' from 'x' and 'y'. If I wanted the version of 'a' listed on in the 'y' index, I would have listed that index in my pip configuration. But I want the version of 'a' listed in index 'x', not 'y'. Currently, if I as a user point pip to index 'x', pip will only install 'a' from 'x' if the package 'a' is of a later version than the package 'a' listed in 'y'. But in our scenario, pip should always install the package 'a' listed in index 'x', not 'y', else there would be no need for multiple repositories at all. This patch fixes that. With this patch, if devpi is queried for a package 'a' from index 'x', it will get package 'a' from 'x' every time, even if a newer version of 'a' is listed in index 'y'. The pull request in its current form would have changed the way devpi pulls packages, and would have broken important workflows. However, for other workflows, I feel this feature is relevant and handy. I therefore request that: 1. The "shadow indexes" feature be added to devpi, although by default it be *turned off*. 2. The feature should/could be enabled either by command line option or configuration file. I think that index shadowing isn't something for everyone, but I think a lot of people would find it useful. We certainly would in our shop. Thanks
APT handles this by priorities (000 to 999), an implementation based on prios is not hard, more flexible, and IMHO more understandable than "shadowing" (i.e. in the end, a binary prio 0/1). Why is implementation (likely) easy? Just use the prio as an epoch to the version, i.e. prepend it to the versions. All other processing stays the same. Default prio would be 500, and prios would be an index attribute.
That sounds like it would suit our shop nicely. I also remember that YUM also had a priorities plugin that I've used in the past. I second the idea. On October 11, 2014 4:49:24 AM MDT, "Jürgen Hermann" <j...@web.de> wrote:
APT handles this by priorities (000 to 999), an implementation based on
prios is not hard, more flexible, and IMHO more understandable than "shadowing" (i.e. in the end, a binary prio 0/1).
Why is implementation (likely) easy? Just use the prio as an epoch to the version, i.e. prepend it to the versions. All other processing stays the same. Default prio would be 500, and prios would be an index attribute.
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.
Thanks -- Daniel Jay Haskin http://djhaskin987.github.io/
On Sat, Oct 11, 2014 at 03:49 -0700, Jürgen Hermann wrote:
APT handles this by priorities (000 to 999), an implementation based on prios is not hard, more flexible, and IMHO more understandable than "shadowing" (i.e. in the end, a binary prio 0/1).
Why is implementation (likely) easy? Just use the prio as an epoch to the version, i.e. prepend it to the versions. All other processing stays the same. Default prio would be 500, and prios would be an index attribute.
I am not worried about the implementation but rather about semantics. And i am not sure how exactly to relate the apt "priorities" concept to devpi. (I just read up on it at http://debian-handbook.info/browse/wheezy/sect.apt-get.html at bit). Juergen, could you detail your thoughts a bit? Below i offer mine. Basically if we query links for a given NAME on an index INDEX, if INDEX has release files for NAME we may either want - to not consider base indexes of INDEX for NAME or to - merge release files for NAME from base indexes This might even be a per-NAME choice not a global one for an index. So i'd map it with an option, maybe: bases_whitelist=* so that by default we have the current behaviour but if you set devpi index bases_whitelist=NAME1 it would mean that only NAME1 would be merged with bases unconditionally whereas all other names would be not be merged from bases if INDEX contains it already. If the INDEX does not contain a name it would be looked up in the bases just as now. (This is recursive definition so the base indexes might have the same setting). Do these considerations make sense to you? I chose bases_whitelist as an option name because we already have pypi_whitelist. Maybe there is some clever more generalized option that we could do instead of having two, not sure. Something like: bases_whitelist= root/pypi=,other/base=NAME1 or so. (please don't use "," in index names btw :) best, holger
Thinking about it further, I agree, Holger. Priorities only make sense in situations where you stop at the first available found NAME. This is what happens with apt and YUM. They look for a package in the first available "index", and stop looking in other indexes once they've found it. Devpi simply doesn't work that way. The whitelist option you have described would create the desired effect. To that idea I would only add a special value, say "None", as in bases_whitelist=None So that the shadowing behavior can be turned off completely. Daniel Jay Haskin http://djhaskin987.github.io/ On Mon, Oct 13, 2014 at 9:50 AM, holger krekel <hol...@merlinux.eu> wrote:
On Sat, Oct 11, 2014 at 03:49 -0700, Jürgen Hermann wrote:
APT handles this by priorities (000 to 999), an implementation based on prios is not hard, more flexible, and IMHO more understandable than "shadowing" (i.e. in the end, a binary prio 0/1).
Why is implementation (likely) easy? Just use the prio as an epoch to the version, i.e. prepend it to the versions. All other processing stays the same. Default prio would be 500, and prios would be an index attribute.
I am not worried about the implementation but rather about semantics. And i am not sure how exactly to relate the apt "priorities" concept to devpi. (I just read up on it at http://debian-handbook.info/browse/wheezy/sect.apt-get.html at bit). Juergen, could you detail your thoughts a bit? Below i offer mine.
Basically if we query links for a given NAME on an index INDEX, if INDEX has release files for NAME we may either want
- to not consider base indexes of INDEX for NAME or to
- merge release files for NAME from base indexes
This might even be a per-NAME choice not a global one for an index. So i'd map it with an option, maybe:
bases_whitelist=*
so that by default we have the current behaviour but if you set
devpi index bases_whitelist=NAME1
it would mean that only NAME1 would be merged with bases unconditionally whereas all other names would be not be merged from bases if INDEX contains it already. If the INDEX does not contain a name it would be looked up in the bases just as now. (This is recursive definition so the base indexes might have the same setting).
Do these considerations make sense to you? I chose bases_whitelist as an option name because we already have pypi_whitelist. Maybe there is some clever more generalized option that we could do instead of having two, not sure. Something like:
bases_whitelist= root/pypi=,other/base=NAME1
or so. (please don't use "," in index names btw :)
best, holger
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.
On Tue, Oct 14, 2014 at 07:35 -0600, Dan Haskin wrote:
Thinking about it further, I agree, Holger. Priorities only make sense in situations where you stop at the first available found NAME. This is what happens with apt and YUM. They look for a package in the first available "index", and stop looking in other indexes once they've found it. Devpi simply doesn't work that way.
The whitelist option you have described would create the desired effect. To that idea I would only add a special value, say "None", as in
bases_whitelist=None
So that the shadowing behavior can be turned off completely.
I'd just say you can do: devpi index bases_whitelist= to turn of shadowing, i.e. get the behaviour you are asking for. Still curious to hear what/how Juergen thinks if priorities/apt-concepts could be made useful here, alternatively. best, holger
Daniel Jay Haskin http://djhaskin987.github.io/
On Mon, Oct 13, 2014 at 9:50 AM, holger krekel <hol...@merlinux.eu> wrote:
On Sat, Oct 11, 2014 at 03:49 -0700, Jürgen Hermann wrote:
APT handles this by priorities (000 to 999), an implementation based on prios is not hard, more flexible, and IMHO more understandable than "shadowing" (i.e. in the end, a binary prio 0/1).
Why is implementation (likely) easy? Just use the prio as an epoch to the version, i.e. prepend it to the versions. All other processing stays the same. Default prio would be 500, and prios would be an index attribute.
I am not worried about the implementation but rather about semantics. And i am not sure how exactly to relate the apt "priorities" concept to devpi. (I just read up on it at http://debian-handbook.info/browse/wheezy/sect.apt-get.html at bit). Juergen, could you detail your thoughts a bit? Below i offer mine.
Basically if we query links for a given NAME on an index INDEX, if INDEX has release files for NAME we may either want
- to not consider base indexes of INDEX for NAME or to
- merge release files for NAME from base indexes
This might even be a per-NAME choice not a global one for an index. So i'd map it with an option, maybe:
bases_whitelist=*
so that by default we have the current behaviour but if you set
devpi index bases_whitelist=NAME1
it would mean that only NAME1 would be merged with bases unconditionally whereas all other names would be not be merged from bases if INDEX contains it already. If the INDEX does not contain a name it would be looked up in the bases just as now. (This is recursive definition so the base indexes might have the same setting).
Do these considerations make sense to you? I chose bases_whitelist as an option name because we already have pypi_whitelist. Maybe there is some clever more generalized option that we could do instead of having two, not sure. Something like:
bases_whitelist= root/pypi=,other/base=NAME1
or so. (please don't use "," in index names btw :)
best, holger
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.
participants (3)
-
Dan Haskin -
holger krekel -
Jürgen Hermann