Re: [Distutils] buildout in offline mode - install receipts also from cache?
On Mon Dec 17 15:35:47 CET 2012, Anton Koval' psihonavt at gmail.com wrote:
in our project we've decided to completely move buildout process to offline mode. according to documentation (http://pypi.python.org/pypi/zc.buildout/2.0.0a5) I made updates in our main buildout configuration file, like: download-cache = ... install-from-cache = true offline = true But when buildout tries to install any part with recipe specified, it is (obviously) failing, e.g.: Error: Couldn't find a distribution for 'iw.recipe.cmd'. So, my question is: are there ways to fetch recipes also from some cache (or whatever) on local disk?
I have experienced the same issue See https://github.com/buildout/buildout/issues/41 This is a bug and also per Jim Fulton a lack of specification/documentation of what offline really means. IMHO it is not possible today to bootstrap and buildout offline. Anyone with a better story there? That said the fixes are rather easy ... And I have an ugly but nicely working monkey patching extension that I will publish on pypi in a couple days... Which ironically enough needs to be installed with pip before a boostrap. Now Jim suggested I should bring the discussion here. In general installing packages without a network connection should be straight forward, simple and easy. That would be the common understanding of what offline means. It works fine with setuptools/distribute and pip albeit a tad circumvoluted and under documented. It does not with buildout and fails because with the offline flag/config, buildout calls its install function with a None dest argument making setuptools/distribute fail downstream with a weird exception as you reported. This affects all current and recent versions of buildout. -- Philippe Ombredanne nexB
On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne <pombredanne@nexb.com> wrote:
On Mon Dec 17 15:35:47 CET 2012, Anton Koval' psihonavt at gmail.com wrote:
in our project we've decided to completely move buildout process to offline mode. according to documentation (http://pypi.python.org/pypi/zc.buildout/2.0.0a5) I made updates in our main buildout configuration file, like: download-cache = ... install-from-cache = true offline = true But when buildout tries to install any part with recipe specified, it is (obviously) failing, e.g.: Error: Couldn't find a distribution for 'iw.recipe.cmd'. So, my question is: are there ways to fetch recipes also from some cache (or whatever) on local disk?
I have experienced the same issue See https://github.com/buildout/buildout/issues/41 This is a bug and also per Jim Fulton a lack of specification/documentation of what offline really means.
IMHO it is not possible today to bootstrap and buildout offline. Anyone with a better story there?
Try: install-from-cache = true without specifying offline mode. install-from-cache was intended to support exactly this use case.
That said the fixes are rather easy ... And I have an ugly but nicely working monkey patching extension that I will publish on pypi in a couple days... Which ironically enough needs to be installed with pip before a boostrap.
Now Jim suggested I should bring the discussion here.
In general installing packages without a network connection should be straight forward, simple and easy. That would be the common understanding of what offline means.
It works fine with setuptools/distribute and pip albeit a tad circumvoluted and under documented.
It does not with buildout and fails because with the offline flag/config, buildout calls its install function with a None dest argument making setuptools/distribute fail downstream with a weird exception as you reported.
This affects all current and recent versions of buildout.
Again, I think this works by setting install-from-cache to true. The original semantics of offline mode didn't fit it's name. It origibally meant "don't install anything". That's how it was implemented. The original goal was both not to download anything and not to **install** anything. Note that this was added before there was a cache. Later, offline mode was extended to avoid network access. In particular offline mode was changed to disallow extending remote configuration files. This change, not made by me, broke some of my companies buildouts. I didn't argue with it at the time as the change was consistent with the option name, if not the original intent. Offline mode is largely a relic of buildout's early evolution. I almost never use it and, when I do, I use it in a way that depend on it's current semantics. In particular, I use buildout to create configurations for installed software. In this context, buildout is running as root and it is not OK to install new software. I believe that install-from-cache (without offline mode) would address Philippe's use case. (It's used by zc.sourcerelease.) That leaves the ambiguous meaning of offline mode. In particular, buildout 2 potentially provides an opportunity to fix things in a backward- incompatible way. OTOH, I'd like to avoid any major changes in 2.0 at this point because it really needs to get finished as soon as possible. Among the options I see at the present: - Do nothing. And just document the somewhat weird semantics of the name. - Add a new option: install, such that install=false has the same meaning as offline=true. Deprecate the offline option and accompanying command-line option. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm
On Sun, Jan 13, 2013 at 9:49 AM, Jim Fulton <jim@zope.com> wrote:
On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne <pombredanne@nexb.com> wrote:
On Mon Dec 17 15:35:47 CET 2012, Anton Koval' psihonavt at gmail.com wrote:
in our project we've decided to completely move buildout process to offline mode. according to documentation (http://pypi.python.org/pypi/zc.buildout/2.0.0a5) I made updates in our main buildout configuration file, like: download-cache = ... install-from-cache = true offline = true But when buildout tries to install any part with recipe specified, it is (obviously) failing, e.g.: Error: Couldn't find a distribution for 'iw.recipe.cmd'. So, my question is: are there ways to fetch recipes also from some cache (or whatever) on local disk?
I have experienced the same issue See https://github.com/buildout/buildout/issues/41 This is a bug and also per Jim Fulton a lack of specification/documentation of what offline really means.
IMHO it is not possible today to bootstrap and buildout offline. Anyone with a better story there?
Try:
install-from-cache = true
without specifying offline mode.
install-from-cache was intended to support exactly this use case.
That said the fixes are rather easy ... And I have an ugly but nicely working monkey patching extension that I will publish on pypi in a couple days... Which ironically enough needs to be installed with pip before a boostrap.
Now Jim suggested I should bring the discussion here.
In general installing packages without a network connection should be straight forward, simple and easy. That would be the common understanding of what offline means.
It works fine with setuptools/distribute and pip albeit a tad circumvoluted and under documented.
It does not with buildout and fails because with the offline flag/config, buildout calls its install function with a None dest argument making setuptools/distribute fail downstream with a weird exception as you reported.
This affects all current and recent versions of buildout.
Again, I think this works by setting install-from-cache to true.
The original semantics of offline mode didn't fit it's name. It origibally meant "don't install anything". That's how it was implemented. The original goal was both not to download anything and not to **install** anything. Note that this was added before there was a cache.
Later, offline mode was extended to avoid network access. In particular offline mode was changed to disallow extending remote configuration files. This change, not made by me, broke some of my companies buildouts. I didn't argue with it at the time as the change was consistent with the option name, if not the original intent.
Offline mode is largely a relic of buildout's early evolution. I almost never use it and, when I do, I use it in a way that depend on it's current semantics. In particular, I use buildout to create configurations for installed software. In this context, buildout is running as root and it is not OK to install new software.
I believe that install-from-cache (without offline mode) would address Philippe's use case. (It's used by zc.sourcerelease.)
That leaves the ambiguous meaning of offline mode. In particular, buildout 2 potentially provides an opportunity to fix things in a backward- incompatible way.
OTOH, I'd like to avoid any major changes in 2.0 at this point because it really needs to get finished as soon as possible.
Among the options I see at the present:
- Do nothing. And just document the somewhat weird semantics of the name.
I think this is the best option for now. In addition: - Close some holes in the code where offline mode can lead to weird errors - Change some offline-related error messages. - Error in a hopefully helpful way when offline mode and install-from-cache are used together.
- Add a new option: install, such that install=false has the same meaning as offline=true. Deprecate the offline option and accompanying command-line option.
Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm
On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne <pombredanne@nexb.com> wrote:
IMHO it is not possible today to bootstrap and buildout offline. Anyone with a better story there? Try: install-from-cache = true without specifying offline mode. install-from-cache was intended to support exactly this use case. I works! My bad, and I can now get something to work with for instance:
On Sun, Jan 13, 2013 at 3:49 PM, Jim Fulton <jim@zope.com> wrote: python bootstrap.py --version 1.7.0 \ --distribute --setup-source=./thirdparty/dist/distribute_setup.py \ --download-base=./thirdparty/dist/ and a [buildout] with: include-site-packages = false install-from-cache = true newest = false allow-picked-versions = false # we need this in buidlout 1 too zc.buildout-version = 1.7.0 distribute-version = 0.6.34 # Weirdly enough buildout demands the files to be cached in a dist subdir # See https://github.com/buildout/buildout/issues/40 download-cache = thirdparty find-links = file:///thirdparty/ file:///thirdparty/dist/
Again, I think this works by setting install-from-cache to true. The original semantics of offline mode didn't fit it's name. It originally meant "don't install anything". That's how it was implemented. The original goal was both not to download anything and not to **install** anything. Note that this was added before there was a cache. My naive understanding of offline is "without a network connection"
Later, offline mode was extended to avoid network access. In particular offline mode was changed to disallow extending remote configuration files. This change, not made by me, broke some of my companies buildouts. I didn't argue with it at the time as the change was consistent with the option name, if not the original intent. Offline mode is largely a relic of buildout's early evolution. I almost never use it and, when I do, I use it in a way that depend on it's current semantics. In particular, I use buildout to create configurations for installed software. In this context, buildout is running as root and it is not OK to install new software.
If anything the option name is as you said incorrect, and the "do not install" use case somewhat arcane and undocumented. But that is another story.
I believe that install-from-cache (without offline mode) would address Philippe's use case. (It's used by zc.sourcerelease.) Yes!
That leaves the ambiguous meaning of offline mode. In particular, buildout 2 potentially provides an opportunity to fix things in a backward- incompatible way. OTOH, I'd like to avoid any major changes in 2.0 at this point because it really needs to get finished as soon as possible. Among the options I see at the present: - Do nothing. And just document the somewhat weird semantics of the name.
Leaving this ambiguity in offline longer would be unfortunate IMHO
- Add a new option: install, such that install=false has the same meaning as offline=true. Deprecate the offline option and accompanying command-line option.
That would be the preferred approach IMHO Do you want a patch? Though I am not sure how to proceed to deprecate the option.... -- Philippe Ombredanne +1 650 799 0949 | pombredanne@nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com
On Mon, Jan 21, 2013 at 8:36 AM, Philippe Ombredanne <pombredanne@nexb.com> wrote:
On Sun, Jan 13, 2013 at 3:49 PM, Jim Fulton <jim@zope.com> wrote: ...
- Do nothing. And just document the somewhat weird semantics of the name.
Leaving this ambiguity in offline longer would be unfortunate IMHO
It is, but some recipes check this option. I'm just not up for changing this now.
- Add a new option: install, such that install=false has the same meaning as offline=true. Deprecate the offline option and accompanying command-line option.
That would be the preferred approach IMHO Do you want a patch?
No. I decided to use the other option. I now error if install-from-cache is used with offline, and I've improved the error message when requirements aren't satisfied in offline mode (for buildout 2). Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm
On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne <pombredanne@nexb.com> wrote: ...
IMHO it is not possible today to bootstrap and buildout offline. Anyone with a better story there?
To peal this a bit further. - It's definitely possible to buildout in offline mode if you already have the distributions you need installed in your eggs or develop-eggs directory. - It's not possible to bootstrap buildout with the regular bootstrap script without doing network access. You could certainly arrange a custom bootstrap script that installed from a local cache, but you can't use offline mode, which is really no-install mode. You could conceivably arrange to bootstrap in offline mode if you pre-populated your eggs directory with buildout and setuptools/distribute eggs, but this is splitting hairs. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm
On Sun, Jan 20, 2013 at 7:36 PM, Jim Fulton <jim@zope.com> wrote:
On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne <pombredanne@nexb.com> wrote: ...
IMHO it is not possible today to bootstrap and buildout offline. Anyone with a better story there?
To peal this a bit further. - It's not possible to bootstrap buildout with the regular bootstrap script without doing network access.
You could certainly arrange a custom bootstrap script that installed from a local cache, but you can't use offline mode, which is really no-install mode.
You could conceivably arrange to bootstrap in offline mode if you pre-populated your eggs directory with buildout and setuptools/distribute eggs, but this is splitting hairs. Actually it is perfectly possible to boostrap in a mode where no network connection is available whcih is my naive assumption for the "offline" option More details to come in another reply in the same thread
-- Philippe Ombredanne +1 650 799 0949 | pombredanne@nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com
participants (2)
-
Jim Fulton -
Philippe Ombredanne