[Distutils] buildout/setuptools slow as it scans the whole project dir

Reinout van Rees reinout at vanrees.org
Wed Apr 15 19:02:51 CEST 2015

Reinout van Rees schreef op 15-04-15 om 17:01:
> So.... it almost seems as if there's no solution to this?
> Or can someone give a hint on os.walk performance relative to VMs?
Bingo! I think I've found the real problem: symlinks between filesystems.

The project is on the host file system and is mounted via vmware/vagrant 
inside the VM as /vagrant.
So my project is on /vagrant/projectname.

I have buildout set up with a download and egg cache, in 
~/.buildout/downloads and ~/.buildout/eggs respectively.

The omelette recipy makes symlinks in 
/vagrant/projectname/parts/omelette/ for all the eggs, pointing at 

So: the symlink is FROM the mounted host filesystem at /vagrant to the 
VM's own ubuntu filesystem.
And apparently symlinks from such a filesystem to another aren't effective.

As a test, I've set the egg-cache directory to /vagrant/.buildout/eggs/
And yes: buildout performance is great again!

I'll try to write it up a bit more eloquently and clearer later on. 
Dinner is calling now.
(Let me indulge in a quote by my brother Maurits, not unknown on this 
mailinglist: "When dinner is calling, you need a rifle.")


Reinout van Rees                          http://reinout.vanrees.org/
reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"

More information about the Distutils-SIG mailing list