Buildout: caching of base configuration ('extends')
Hi, I've been implementing a download API for buildout and have reached the point where I'd like to try using it for downloading the base config files referenced from a buildout config by the 'extends' option. This would have the benefit of allowing to cache those config files and being able to run a buildout using that feature while being offline. However, I'd like to discuss two points about doing this: - I think buildout should not unconditionally use a cached copy of a base configuration file. While it is sensible in other use cases to never access the network if a file is found in the cache, base configs can be expected to change and should IMHO be taken from the cache only when offline mode is active or an attempt at downloading failed. - The download cache to be used is configured as one of the options that are read from either ~/.buildout/default.cfg or buildout.cfg and their respective bases. In order to keep things simple, I'd suggest to use a download cache only if specified directly inside any of these two files, and ignore download-cache options in any downloaded base configs. This would be an exception to how options read from config files are combined, but one I deem worthwhile. I'd allow for configuring and using different download caches in those two files, though. That way, bases of the user's default config can be cached centrally while allowing to keep project-related stuff in a different place. Opinions? -- Thomas
On Wed, May 27, 2009 at 14:56, Thomas Lotze <thomas@thomas-lotze.de> wrote:
Hi,
Hi Thomas :) First at all, thank you for your work.
I've been implementing a download API for buildout and have reached the point where I'd like to try using it for downloading the base config files referenced from a buildout config by the 'extends' option. This would have the benefit of allowing to cache those config files and being able to run a buildout using that feature while being offline.
Do you talk about index and find-links too? And for the bootstrap? I search a scenario to deploy a buildout fully offline.
- I think buildout should not unconditionally use a cached copy of a base configuration file. While it is sensible in other use cases to never access the network if a file is found in the cache, base configs can be expected to change and should IMHO be taken from the cache only when offline mode is active or an attempt at downloading failed.
Right. You think about "offline" option or "install-from-cache" option? offline seems to be a good candidate.
- The download cache to be used is configured as one of the options that are read from either ~/.buildout/default.cfg or buildout.cfg and their respective bases. In order to keep things simple, I'd suggest to use a download cache only if specified directly inside any of these two files, and ignore download-cache options in any downloaded base configs. This would be an exception to how options read from config files are combined, but one I deem worthwhile.
Good for me.
Opinions?
Hmm, just an idea on the same area: caching only some packages (not only "all" or "none"). Surely out of your scope... -- Sebastien Douche <sdouche@gmail.com>
On Thu, May 28, 2009 at 11:44, Sebastien Douche <sdouche@gmail.com> wrote:
Opinions?
Hmm, just an idea on the same area: caching only some packages (not only "all" or "none"). Surely out of your scope...
Oups, missing primary needs: show the location of a download (cache, url of the website,...). -- Sebastien Douche <sdouche@gmail.com>
On May 27, 2009, at 8:56 AM, Thomas Lotze wrote:
Hi,
I've been implementing a download API for buildout and have reached the point where I'd like to try using it for downloading the base config files referenced from a buildout config by the 'extends' option. This would have the benefit of allowing to cache those config files and being able to run a buildout using that feature while being offline.
However, I'd like to discuss two points about doing this:
- I think buildout should not unconditionally use a cached copy of a base configuration file. While it is sensible in other use cases to never access the network if a file is found in the cache, base configs can be expected to change and should IMHO be taken from the cache only when offline mode is active or an attempt at downloading failed.
Absolutely.
- The download cache to be used is configured as one of the options that are read from either ~/.buildout/default.cfg or buildout.cfg and their respective bases. In order to keep things simple, I'd suggest to use a download cache only if specified directly inside any of these two files, and ignore download-cache options in any downloaded base configs. This would be an exception to how options read from config files are combined, but one I deem worthwhile.
-1 on allowing the settings in only these 2 files.
I'd allow for configuring and using different download caches in those two files, though. That way, bases of the user's default config can be cached centrally while allowing to keep project-related stuff in a different place.
Opinions?
The solution seems too complicated to me. Too many special rules. Maybe there should be a separate settinf for this, like extends-cache or something. Jim -- Jim Fulton Zope Corporation
Jim Fulton <jim@zope.com> schrieb:
- I think buildout should not unconditionally use a cached copy of a base configuration file. While it is sensible in other use cases to never access the network if a file is found in the cache, base configs can be expected to change and should IMHO be taken from the cache only when offline mode is active or an attempt at downloading failed.
Absolutely.
I've added the ability to use the cache in this way to the download API in the meantime.
- The download cache to be used is configured as one of the options that are read from either ~/.buildout/default.cfg or buildout.cfg and their respective bases. In order to keep things simple, I'd suggest to use a download cache only if specified directly inside any of these two files, and ignore download-cache options in any downloaded base configs. This would be an exception to how options read from config files are combined, but one I deem worthwhile.
-1 on allowing the settings in only these 2 files.
OK, avoiding these particular special rules is fine with me.
The solution seems too complicated to me. Too many special rules.
Maybe there should be a separate settinf for this, like extends-cache or something.
Would it be OK for you to special-case that new option? The problem that needs to be solved when downloading base configs is that in order for a particular config file to be taken from the cache if necessary, the cache location must be known beforehand, so it shouldn't be possible for the cache location to depend on the contents of the config file in question. I cannot see another way to achieve this but to read the extends-cache location from the root config file (either ~/.buildout/default.cfg or buildout.cfg) exclusively. This wouldn't account for an root config file that is specified to buildout on the command line and needs to be downloaded itself, but I think it's OK to assume being online when specifying an online resource in such an explicit way. -- Viele Grüße, Thomas
Sebastien Douche <sdouche@gmail.com> schrieb:
I've been implementing a download API for buildout and have reached the point where I'd like to try using it for downloading the base config files referenced from a buildout config by the 'extends' option. This would have the benefit of allowing to cache those config files and being able to run a buildout using that feature while being offline.
Do you talk about index and find-links too? And for the bootstrap? I search a scenario to deploy a buildout fully offline.
Those would be useful indeed. However, afaics the respective code doesn't have access to the zc.buildout package (easy-install downloads use setuptools and at bootstrap time, zc.buildout hasn't even been downloaded yet).
Right. You think about "offline" option or "install-from-cache" option? offline seems to be a good candidate.
I've implemented three ways of handling the cache: - don't use it at all, fail if offline mode is on or the resource cannot be accessed - use a cached copy of a file unconditionally if one is found in the cache - try to download a resource if not in offline mode, only fall back to using the cache if this fails I was talking about using the third option when fetching configuration. Other client code (i.e. recipes) may choose freely how to use the cache and whether to make that decision based on a configuration option.
Hmm, just an idea on the same area: caching only some packages (not only "all" or "none"). Surely out of your scope...
This is out of the scope of zc.buildout's core IMO. A particular recipe might implement any algorithm to decide how to use the cache when downloading one package or another. I even think this would be too complex for a core recipe such as zc.recipe.egg. -- Viele Grüße, Thomas
Sebastien Douche <sdouche@gmail.com> schrieb:
Oups, missing primary needs: show the location of a download (cache, url of the website,...).
Good idea, I'm going to add loggin anyway. However, I think this particular information should only be printed in verbose mode. -- Viele Grüße, Thomas
On May 28, 2009, at 11:51 AM, Thomas Lotze wrote:
Jim Fulton <jim@zope.com> schrieb:
- I think buildout should not unconditionally use a cached copy of a base configuration file. While it is sensible in other use cases to never access the network if a file is found in the cache, base configs can be expected to change and should IMHO be taken from the cache only when offline mode is active or an attempt at downloading failed.
Absolutely.
I've added the ability to use the cache in this way to the download API in the meantime.
- The download cache to be used is configured as one of the options that are read from either ~/.buildout/default.cfg or buildout.cfg and their respective bases. In order to keep things simple, I'd suggest to use a download cache only if specified directly inside any of these two files, and ignore download-cache options in any downloaded base configs. This would be an exception to how options read from config files are combined, but one I deem worthwhile.
-1 on allowing the settings in only these 2 files.
OK, avoiding these particular special rules is fine with me.
The solution seems too complicated to me. Too many special rules.
Maybe there should be a separate settinf for this, like extends-cache or something.
Would it be OK for you to special-case that new option?
Yup
The problem that needs to be solved when downloading base configs is that in order for a particular config file to be taken from the cache if necessary, the cache location must be known beforehand, so it shouldn't be possible for the cache location to depend on the contents of the config file in question. I cannot see another way to achieve this but to read the extends-cache location from the root config file (either ~/.buildout/default.cfg or buildout.cfg) exclusively.
This wouldn't account for an root config file that is specified to buildout on the command line and needs to be downloaded itself, but I think it's OK to assume being online when specifying an online resource in such an explicit way.
Agreed. Jim -- Jim Fulton Zope Corporation
participants (3)
-
Jim Fulton -
Sebastien Douche -
Thomas Lotze