Mailman 3 August 2009 - Distutils-SIG

problem with zc.buildout download-cache setting
by Sydney Weidman 04 Aug '09

04 Aug '09

I have a buildout recipe (zc.buildout 1.3.0) for a Plone site I'm building and it contains the line: download-cache = downloads but I find that when I run bin/buildout, the directory does not get created and I get the following traceback: [educommons@vrs OCWBuildout]$ bin/buildout -Uv While: Initializing. Error: The specified download cache: '/home/educommons/OCWBuildout/downloads' Doesn't exist. The code from zc.buildout.buildout looks like: download_cache = options.get('download-cache') if download_cache: download_cache = os.path.join(options['directory'], download_cache) if not os.path.isdir(download_cache): raise zc.buildout.UserError( 'The specified download cache:\n' '%r\n' "Doesn't exist.\n" % download_cache) download_cache = os.path.join(download_cache, 'dist') if not os.path.isdir(download_cache): os.mkdir(download_cache) zc.buildout.easy_install.download_cache(download_cache) It seems odd to me that if the download_cache directory doesn't exist, that UserError is raised. Shouldn't the directory be created? What would be the harm in that? Or am I missing something? I realize that I could just mkdir downloads and it would work, but it seems to defeat the simplicity of "python bootstrap.py; bin/buidout; bin/instance start". I'm sure this is some failure of understanding on my part, otherwise I'm sure someone else would have encountered this issue before. Regards, Syd

2 1

Update on recent distutils (and related projects) progress
by Ben Finney 03 Aug '09

03 Aug '09

Howdy all, Second only to doing work on improvement is to *make it known* what improvements are going on. So, thank you very much to Tarek Ziadé for spending the time on status updates for recent distutils-related work <URL:http://tarekziade.wordpress.com/2009/07/24/words-on-distribute-distutil…>. -- \ “I have the simplest tastes. I am always satisfied with the | `\ best.” —Oscar Wilde | _o__) | Ben Finney

1 0

Re: [Distutils] Cache PYTHONPATH? (Re: make unzipped eggs be the default)
by P.J. Eby 02 Aug '09

02 Aug '09

At 06:52 PM 8/2/2009 +0200, Tarek Ziadé wrote: >On Wed, Jul 29, 2009 at 6:44 AM, P.J. Eby<pje(a)telecommunity.com> wrote: > > At 10:35 PM 7/28/2009 -0500, Ian Bicking wrote: > >> > >> On Tue, Jul 28, 2009 at 9:40 PM, P.J. Eby<pje(a)telecommunity.com> wrote: > >> > At 09:22 PM 7/28/2009 -0500, Ian Bicking wrote: > >> >> > >> >> I can see how this could go quite wrong, but maybe if installers touch > >> >> some file in the library directory anytime a package is > >> >> installed/reinstalled/removed/etc, > >> > > >> > You mean, like, the mtime of the directory itself? Â ;-) > >> > >> Do directory mtimes get recursively updated? I don't think they do. > > > > That's not necessary; if imports use a cached listdir, then the children > > will get handled recursively. > > > >> So if you have a layout: > >> > >> site-packages/ > >> zope/ > >> interface/ > >> __init__.py > >> > >> And you update the package and update __init__.py, the mtime of > >> site-packages doesn't change, does it? > > > > Nope, but at the top level, the fact that 'zope' is present is > unchanged, as > > is the presence of an 'interface' subdirectory. > > > > > >> I'm saying if there was a file in site-packages/last_updated that gets > >> touched everytime an installer does anything in site-packages, then > >> you could cache (between processes) the lookups. > > > > Since each invocation of the interpreter can have a different PYTHONPATH, > > the cache has to be per-directory, not global. If it's per-directory, then > > there's no real benefit over runtime caching, since you now have > to open and > > read a file (instead of just reading the directory). And as I said, it's > > not realistic to think that opening and reading a file is going to beat > > opening and reading a directory for speed. > >But opening and reading one file should beat opening hundreds of directories : >In the PEP 376 prototype, after thinking about a per-directory cache >like you are >describing, I was thinking about having a global index file to replace >the global dictionnary that keeps track of the distributions per >directory (currently the directory path >is the key in the dictionnary and the value the distribution objects). > >That can even be a simple shelve of the dictionary, that become a >global index of directories >that [are/were once] in the path. This works as long as the index file >is per-user. >Or even better : per-application. I don't know how this could be >managed/done, but >a simple cache file created alongside the script the application is >launched with, could >speed up the lookups at the second launch. You'd still have to stat the directories to know if they changed - in which case the logic I've already laid out still applies. I think, however, we are discussing different nominal scenarios. I'm assuming a post-PEP 376 world where the only use for .egg files or directories are for *non-default* versions of packages, that only get added to sys.path for apps or libraries that need them, rather than being in a default .pth file. However, if you're discussing speeding up an environment where we use .egg directories and they're on sys.path, then a per-user global cache might speed things up. For security reasons, however, that cache would need to be ignored by Python when running secure scripts. (e.g. -s and -E options, and definitely anything setuid.) In contrast, directory stat caching with a modest number of (non-egg) PYTHONPATH entries would speed things nicely in the hopefully-future-default case.

1 0

idea for Distribute: make unzipped eggs be the default
by Zooko Wilcox-O'Hearn 02 Aug '09

02 Aug '09

Folks: I have been keeping a running tally of people who have trouble with zipped eggs and the trouble goes away when they have unzipped eggs instead. I just added the newest entry: someone was trying to build a package with py2exe, and configuring their distutils config file to always unzip eggs fixed the problem: http://bugs.python.org/setuptools/issue33 # zipped eggs cause various problems -- perhaps set default setting to unzipped? Perhaps the Distribute project should consider installing eggs unzipped by default. It would avoid a slew of problems (see that ticket for a catalog of problems), would be fully backwards compatible (it is what many people already do, per the suggested workaround, and I've never heard a report of a problem due to an *unzipped* egg), and I think it might improve performance a bit. (Yes, that's right -- we haven't done a real measurement of performance, but the few times that people briefly glanced at performance it seemed like zipping the eggs made them slower to load, not faster.) Regards, Zooko

13 21

Re: [Distutils] Cache PYTHONPATH? (Re: make unzipped eggs be the default)
by P.J. Eby 02 Aug '09

02 Aug '09

At 10:35 PM 7/28/2009 -0500, Ian Bicking wrote: >On Tue, Jul 28, 2009 at 9:40 PM, P.J. Eby<pje(a)telecommunity.com> wrote: > > At 09:22 PM 7/28/2009 -0500, Ian Bicking wrote: > >> > >> I can see how this could go quite wrong, but maybe if installers touch > >> some file in the library directory anytime a package is > >> installed/reinstalled/removed/etc, > > > > You mean, like, the mtime of the directory itself? Â ;-) > >Do directory mtimes get recursively updated? I don't think they do. That's not necessary; if imports use a cached listdir, then the children will get handled recursively. >So if you have a layout: > >site-packages/ > zope/ > interface/ > __init__.py > >And you update the package and update __init__.py, the mtime of >site-packages doesn't change, does it? Nope, but at the top level, the fact that 'zope' is present is unchanged, as is the presence of an 'interface' subdirectory. >I'm saying if there was a file in site-packages/last_updated that gets >touched everytime an installer does anything in site-packages, then >you could cache (between processes) the lookups. Since each invocation of the interpreter can have a different PYTHONPATH, the cache has to be per-directory, not global. If it's per-directory, then there's no real benefit over runtime caching, since you now have to open and read a file (instead of just reading the directory). And as I said, it's not realistic to think that opening and reading a file is going to beat opening and reading a directory for speed.

2 1