[Distutils] buildout/setuptools slow as it scans the whole project dir

[Ionel Cristian Mărieș]

Could this be the same problem
https://bitbucket.org/pypa/setuptools/issue/249/have-a-way-to-ingore-specific-dirs-when
?

I frequently have this issue (.tox dir can sometimes get quite large) and I
resorted to monkeypatching setuptools (I know, right?) in
~/.local/lib/python2.7/site-packages/usercustomize.py:

import setuptools
from distutils import filelist
import os
def findall(dir=os.curdir, original=filelist.findall, listdir=os.listdir,
basename=os.path.basename):
    os.listdir = lambda path: [i for i in listdir(path) if '/.tox/' not in
i and not i.startswith('.tox/')]
    try:
        return original(dir)
    finally:
        os.listdir = listdir
filelist.findall = findall



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

On Wed, Apr 15, 2015 at 5:01 PM, Reinout van Rees <reinout at vanrees.org>
wrote:

> Hi,
>
> In some of my projects, buildout takes a looooooong time to complete.
> Sometimes the same problem occurs on the server or on another developer's
> laptop, sometimes not. The difference is something like "15 seconds if
> there's no problem" and "5 minutes if the problem occurs".
>
> It is terribly hard to pinpoint the exact problem and/or cause. An error
> in the setup.py or or MANIFEST.in is unlikely, as the very same project
> might take 5 minutes locally and 15 seconds on the server or vice versa...
>
> Anyway, if I run "bin/buildout" as "strace -f bin/buildout", I see a lot
> of "stat" calls. Setuptools walks through all the files inside my project
> dir. Including parts/omelette/* and, if available, a bower_compontents/
> directory full of thousands of javascript files. Removing some of these
> directories (which running the buildout re-creates) fixes the speed issue
> for one run.
>
> I modified my local buildout copy to run "setup.py develop" with a -v
> instead of a -q option. This way I found out where it approximately happens:
>
>
> /usr/bin/python /tmp/tmp6UdsMl -v develop -mxN -d
> /vagrant/sso/develop-eggs/tmpfioc1Ibuild
> running develop
> running egg_info
> writing requirements to sso.egg-info/requires.txt
> writing sso.egg-info/PKG-INFO
> writing top-level names to sso.egg-info/top_level.txt
> writing dependency_links to sso.egg-info/dependency_links.txt
> writing entry points to sso.egg-info/entry_points.txt
>
> ### This is where the process seems to stop a couple of minutes to scan
> all the files.
>
> reading manifest file 'sso.egg-info/SOURCES.txt'
> reading manifest template 'MANIFEST.in'
> writing manifest file 'sso.egg-info/SOURCES.txt'
> running build_ext
> Creating /vagrant/sso/develop-eggs/tmpfioc1Ibuild/sso.egg-link (link to .)
>
>
>
> The MANIFEST.in loooks like this:
>
> # Include docs in the root.
> include *.rst
> # Include everything in our project directory (sso/views.py,
> sso/static/some.js, etc)
> graft sso
>
>
> It is a git project. The setup.py looks like this:
>
>
> from setuptools import setup
>
> version = '1.1.dev0'
>
> long_description = '\n\n'.join([
>     open('README.rst').read(),
>     open('CREDITS.rst').read(),
>     open('CHANGES.rst').read(),
>     ])
>
> install_requires = [
>     'Django >= 1.4.2, < 1.7',
>     'django-nose',
>     'lizard-auth-server',
>     'gunicorn',
>     'raven',
>     'werkzeug',
>     'south',
>     'django-auth-ldap',
>     'django-mama-cas',
>     ],
>
> setup(name='sso',
>       version=version,
>       description="Single sign on server (and more) for lizard",
>       long_description=long_description,
>       # Get strings from http://www.python.org/pypi?%
> 3Aaction=list_classifiers
>       classifiers=['Programming Language :: Python',
>                    'Framework :: Django',
>                    ],
>       keywords=[],
>       author='Do not blame Reinout',
>       author_email='reinout.vanrees at nelen-schuurmans.nl',
>       url='',
>       license='GPL',
>       packages=['sso'],
>       zip_safe=False,
>       install_requires=install_requires,
>       entry_points={
>           'console_scripts': [
>           ]},
>       )
>
>
>
> Conclusion for me: something somewhere in setuptools is reading my whole
> project folder. It does it after "writing entry points to
> sso.egg-info/entry_points.txt" and before "reading manifest file
> 'sso.egg-info/SOURCES.txt'". The SOURCES.txt itself is small:
>
>
> CHANGES.rst
> CREDITS.rst
> LICENSE.rst
> MANIFEST.in
> README.rst
> setup.cfg
> setup.py
> sso/__init__.py
> sso/__init__.pyc
> sso/admin.py
> sso/developmentsettings.py
> sso/developmentsettings.pyc
> sso/models.py
> sso/models.pyc
> sso/settings.py
> sso/settings.pyc
> sso/stagingsettings.py
> sso/tests.py
> sso/urls.py
> sso/views.py
> sso.egg-info/PKG-INFO
> sso.egg-info/SOURCES.txt
> sso.egg-info/dependency_links.txt
> sso.egg-info/entry_points.txt
> sso.egg-info/not-zip-safe
> sso.egg-info/requires.txt
>
>
> Hm. I see some .pyc files in there. Something that needs fixing, but not
> the cause, I think.
>
> Is there something I'm missing? Where should I look? I cannot find
> "writing entry points to..." in the setuptools source code right away.
>
>
>
> Reinout
>
> --
> Reinout van Rees                          http://reinout.vanrees.org/
> reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
> "Learning history by destroying artifacts is a time-honored atrocity"
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150415/212822fe/attachment-0001.html>