
Hi, In some of my projects, buildout takes a looooooong time to complete. Sometimes the same problem occurs on the server or on another developer's laptop, sometimes not. The difference is something like "15 seconds if there's no problem" and "5 minutes if the problem occurs". It is terribly hard to pinpoint the exact problem and/or cause. An error in the setup.py or or MANIFEST.in is unlikely, as the very same project might take 5 minutes locally and 15 seconds on the server or vice versa... Anyway, if I run "bin/buildout" as "strace -f bin/buildout", I see a lot of "stat" calls. Setuptools walks through all the files inside my project dir. Including parts/omelette/* and, if available, a bower_compontents/ directory full of thousands of javascript files. Removing some of these directories (which running the buildout re-creates) fixes the speed issue for one run. I modified my local buildout copy to run "setup.py develop" with a -v instead of a -q option. This way I found out where it approximately happens: /usr/bin/python /tmp/tmp6UdsMl -v develop -mxN -d /vagrant/sso/develop-eggs/tmpfioc1Ibuild running develop running egg_info writing requirements to sso.egg-info/requires.txt writing sso.egg-info/PKG-INFO writing top-level names to sso.egg-info/top_level.txt writing dependency_links to sso.egg-info/dependency_links.txt writing entry points to sso.egg-info/entry_points.txt ### This is where the process seems to stop a couple of minutes to scan all the files. reading manifest file 'sso.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sso.egg-info/SOURCES.txt' running build_ext Creating /vagrant/sso/develop-eggs/tmpfioc1Ibuild/sso.egg-link (link to .) The MANIFEST.in loooks like this: # Include docs in the root. include *.rst # Include everything in our project directory (sso/views.py, sso/static/some.js, etc) graft sso It is a git project. The setup.py looks like this: from setuptools import setup version = '1.1.dev0' long_description = '\n\n'.join([ open('README.rst').read(), open('CREDITS.rst').read(), open('CHANGES.rst').read(), ]) install_requires = [ 'Django >= 1.4.2, < 1.7', 'django-nose', 'lizard-auth-server', 'gunicorn', 'raven', 'werkzeug', 'south', 'django-auth-ldap', 'django-mama-cas', ], setup(name='sso', version=version, description="Single sign on server (and more) for lizard", long_description=long_description, # Get strings from http://www.python.org/pypi?%3Aaction=list_classifiers classifiers=['Programming Language :: Python', 'Framework :: Django', ], keywords=[], author='Do not blame Reinout', author_email='reinout.vanrees@nelen-schuurmans.nl', url='', license='GPL', packages=['sso'], zip_safe=False, install_requires=install_requires, entry_points={ 'console_scripts': [ ]}, ) Conclusion for me: something somewhere in setuptools is reading my whole project folder. It does it after "writing entry points to sso.egg-info/entry_points.txt" and before "reading manifest file 'sso.egg-info/SOURCES.txt'". The SOURCES.txt itself is small: CHANGES.rst CREDITS.rst LICENSE.rst MANIFEST.in README.rst setup.cfg setup.py sso/__init__.py sso/__init__.pyc sso/admin.py sso/developmentsettings.py sso/developmentsettings.pyc sso/models.py sso/models.pyc sso/settings.py sso/settings.pyc sso/stagingsettings.py sso/tests.py sso/urls.py sso/views.py sso.egg-info/PKG-INFO sso.egg-info/SOURCES.txt sso.egg-info/dependency_links.txt sso.egg-info/entry_points.txt sso.egg-info/not-zip-safe sso.egg-info/requires.txt Hm. I see some .pyc files in there. Something that needs fixing, but not the cause, I think. Is there something I'm missing? Where should I look? I cannot find "writing entry points to..." in the setuptools source code right away. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"