[Distutils] buildout in offline mode - install receipts also from cache?

Jim Fulton jim at zope.com
Sun Jan 20 19:43:54 CET 2013

On Sun, Jan 13, 2013 at 9:49 AM, Jim Fulton <jim at zope.com> wrote:
> On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne
> <pombredanne at nexb.com> wrote:
>> On Mon Dec 17 15:35:47 CET 2012, Anton Koval' psihonavt at gmail.com wrote:
>>> in our project we've decided to completely move
>>> buildout process to offline mode.
>>> according to documentation
>>> (http://pypi.python.org/pypi/zc.buildout/2.0.0a5)
>>> I made updates in our main buildout configuration file, like:
>>> download-cache = ...
>>> install-from-cache = true
>>> offline = true
>>> But when buildout tries to install any part with recipe specified,
>>> it is (obviously) failing, e.g.:
>>> Error: Couldn't find a distribution for 'iw.recipe.cmd'.
>>> So, my question is: are there ways to fetch recipes also from
>>> some cache (or whatever) on local disk?
>> I have experienced the same issue
>> See https://github.com/buildout/buildout/issues/41
>> This is a bug and also per Jim Fulton a lack of specification/documentation
>> of what offline really means.
>> IMHO it is not possible today to bootstrap and buildout offline. Anyone with
>> a better story there?
> Try:
> install-from-cache = true
> without specifying offline mode.
> install-from-cache was intended to support exactly this use case.
>> That said the fixes are rather easy ... And I have an ugly but nicely
>> working monkey patching extension that I will publish on pypi in a couple
>> days... Which ironically enough needs to be installed with pip before a
>> boostrap.
>> Now Jim suggested I should bring the discussion here.
>> In general installing packages without a network connection should be
>> straight forward, simple and easy. That would be the common understanding of
>> what offline means.
>> It works fine with setuptools/distribute and pip albeit a tad circumvoluted
>> and under documented.
>> It does not with buildout and fails because with the offline flag/config,
>> buildout calls its install function with a None dest argument making
>> setuptools/distribute fail downstream with a weird exception as you
>> reported.
>> This affects all current and recent versions of buildout.
> Again, I think this works by setting install-from-cache to true.
> The original semantics of offline mode didn't fit it's name. It origibally
> meant "don't install anything".  That's how it was implemented.  The original
> goal was both not to download anything and not to **install** anything.
> Note that this was added before there was a cache.
> Later, offline mode was extended to avoid network access. In particular
> offline mode was changed to disallow extending remote configuration
> files.  This change,
> not made by me, broke some of my companies buildouts.  I didn't argue with it at
> the time as the change was consistent with the option name, if not the original
> intent.
> Offline mode is largely a relic of buildout's early evolution.  I almost
> never use it and, when I do, I use it in a way that depend on it's
> current semantics.  In particular, I use buildout to create configurations
> for installed software.  In this context, buildout is running as root and it
> is not OK to install new software.
> I believe that install-from-cache (without offline mode) would address
> Philippe's use case.  (It's used by zc.sourcerelease.)
> That leaves the ambiguous meaning of offline mode.  In particular,
> buildout 2 potentially provides an opportunity to fix things in a backward-
> incompatible way.
> OTOH, I'd like to avoid any major changes in 2.0 at this point
> because it really needs to get finished as soon as possible.
> Among the options I see at the present:
> - Do nothing.  And just document the somewhat weird semantics
>   of the name.

I think this is the best option for now.

In addition:

- Close some holes in the code where offline mode can lead to
  weird errors

- Change some offline-related error messages.

- Error in a hopefully helpful way when offline mode
  and install-from-cache are used together.

> - Add a new option: install, such that install=false has the same
> meaning as offline=true.
>   Deprecate the offline option and accompanying command-line option.


Jim Fulton
Jerky is better than bacon! http://zo.pe/Kqm

More information about the Distutils-SIG mailing list