Could this be the same problem https://bitbucket.org/pypa/setuptools/issue/249/have-a-way-to-ingore-specific-dirs-when ?

I frequently have this issue (.tox dir can sometimes get quite large) and I resorted to monkeypatching setuptools (I know, right?) in ~/.local/lib/python2.7/site-packages/usercustomize.py:

import setuptools
from distutils import filelist
import os
def findall(dir=os.curdir, original=filelist.findall, listdir=os.listdir, basename=os.path.basename):
    os.listdir = lambda path: [i for i in listdir(path) if '/.tox/' not in i and not i.startswith('.tox/')]
    try:
        return original(dir)
    finally:
        os.listdir = listdir
filelist.findall = findall



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

On Wed, Apr 15, 2015 at 5:01 PM, Reinout van Rees <reinout@vanrees.org> wrote:
Hi,

In some of my projects, buildout takes a looooooong time to complete. Sometimes the same problem occurs on the server or on another developer's laptop, sometimes not. The difference is something like "15 seconds if there's no problem" and "5 minutes if the problem occurs".

It is terribly hard to pinpoint the exact problem and/or cause. An error in the setup.py or or MANIFEST.in is unlikely, as the very same project might take 5 minutes locally and 15 seconds on the server or vice versa...

Anyway, if I run "bin/buildout" as "strace -f bin/buildout", I see a lot of "stat" calls. Setuptools walks through all the files inside my project dir. Including parts/omelette/* and, if available, a bower_compontents/ directory full of thousands of javascript files. Removing some of these directories (which running the buildout re-creates) fixes the speed issue for one run.

I modified my local buildout copy to run "setup.py develop" with a -v instead of a -q option. This way I found out where it approximately happens:


/usr/bin/python /tmp/tmp6UdsMl -v develop -mxN -d /vagrant/sso/develop-eggs/tmpfioc1Ibuild
running develop
running egg_info
writing requirements to sso.egg-info/requires.txt
writing sso.egg-info/PKG-INFO
writing top-level names to sso.egg-info/top_level.txt
writing dependency_links to sso.egg-info/dependency_links.txt
writing entry points to sso.egg-info/entry_points.txt

### This is where the process seems to stop a couple of minutes to scan all the files.

reading manifest file 'sso.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sso.egg-info/SOURCES.txt'
running build_ext
Creating /vagrant/sso/develop-eggs/tmpfioc1Ibuild/sso.egg-link (link to .)



The MANIFEST.in loooks like this:

# Include docs in the root.
include *.rst
# Include everything in our project directory (sso/views.py, sso/static/some.js, etc)
graft sso


It is a git project. The setup.py looks like this:


from setuptools import setup

version = '1.1.dev0'

long_description = '\n\n'.join([
    open('README.rst').read(),
    open('CREDITS.rst').read(),
    open('CHANGES.rst').read(),
    ])

install_requires = [
    'Django >= 1.4.2, < 1.7',
    'django-nose',
    'lizard-auth-server',
    'gunicorn',
    'raven',
    'werkzeug',
    'south',
    'django-auth-ldap',
    'django-mama-cas',
    ],

setup(name='sso',
      version=version,
      description="Single sign on server (and more) for lizard",
      long_description=long_description,
      # Get strings from http://www.python.org/pypi?%3Aaction=list_classifiers
      classifiers=['Programming Language :: Python',
                   'Framework :: Django',
                   ],
      keywords=[],
      author='Do not blame Reinout',
      author_email='reinout.vanrees@nelen-schuurmans.nl',
      url='',
      license='GPL',
      packages=['sso'],
      zip_safe=False,
      install_requires=install_requires,
      entry_points={
          'console_scripts': [
          ]},
      )



Conclusion for me: something somewhere in setuptools is reading my whole project folder. It does it after "writing entry points to sso.egg-info/entry_points.txt" and before "reading manifest file 'sso.egg-info/SOURCES.txt'". The SOURCES.txt itself is small:


CHANGES.rst
CREDITS.rst
LICENSE.rst
MANIFEST.in
README.rst
setup.cfg
setup.py
sso/__init__.py
sso/__init__.pyc
sso/admin.py
sso/developmentsettings.py
sso/developmentsettings.pyc
sso/models.py
sso/models.pyc
sso/settings.py
sso/settings.pyc
sso/stagingsettings.py
sso/tests.py
sso/urls.py
sso/views.py
sso.egg-info/PKG-INFO
sso.egg-info/SOURCES.txt
sso.egg-info/dependency_links.txt
sso.egg-info/entry_points.txt
sso.egg-info/not-zip-safe
sso.egg-info/requires.txt


Hm. I see some .pyc files in there. Something that needs fixing, but not the cause, I think.

Is there something I'm missing? Where should I look? I cannot find "writing entry points to..." in the setuptools source code right away.



Reinout

--
Reinout van Rees                          http://reinout.vanrees.org/
reinout@vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig