slow Python startup with a couple of easy_install-ed packages
Well, easy_install makes it snap to install packages that I may or may not use. Nothing is easier than to 'easy_install something-recently-announced', even if I never will use it (after a preliminary look). Often I forget to uninstall the packages again, or promise to do that later. Disk space is cheap. However, I found that with the ~20 eggs that I have installed the startup time of the Python interpreter (Python2.5, WinXP, measured with 'timeit python -c ""') rises considerably - 2.5 times slower. This only occurs when the eggs are listed in the easy_install.pth file; if they are moved into a 'normal' pth file this slowdown does not occur, although I can still import the eggs fine. I assume the additional time is spent by setuptools looking up entry points or whatever? And that this additional processing is not needed for the eggs that do not provide entry points? Or are not *using* setuptools themselves? If this is true, wouldn't it be possible to skip the processing for those eggs? IMO this is a critical problem of easy_install... (Imaging how the installed and possibly never used eggs pile up). Thomas
At 08:44 PM 2/23/2007 +0100, Thomas Heller wrote:
Well, easy_install makes it snap to install packages that I may or may not use. Nothing is easier than to 'easy_install something-recently-announced', even if I never will use it (after a preliminary look). Often I forget to uninstall the packages again, or promise to do that later. Disk space is cheap.
However, I found that with the ~20 eggs that I have installed the startup time of the Python interpreter (Python2.5, WinXP, measured with 'timeit python -c ""') rises considerably - 2.5 times slower.
This only occurs when the eggs are listed in the easy_install.pth file; if they are moved into a 'normal' pth file this slowdown does not occur, although I can still import the eggs fine.
I assume the additional time is spent by setuptools looking up entry points or whatever?
No. What's happening is that easy-install.pth puts the eggs at the *beginning* of sys.path, in order to ensure that eggs override non-egg packages; this prevents a *huge* amount of complication in sys.path management with respect to detecting conflicts, as well as allowing eggs to be installed that override the standard library (e.g. to allow hotfixing it). The downside is that this means eggs are searched before the stdlib, slowing down the import of stdlib modules at startup. The easy fix is to use the "-m" option, which doesn't put the eggs in any .pth file at all -- and removes them if they're there.
IMO this is a critical problem of easy_install... (Imaging how the installed and possibly never used eggs pile up).
How many of these eggs are directories, vs. how many are zipfiles? Zipfiles shouldn't be having so much effect. Meanwhile, in setuptools 0.7 there will be a completely different way of managing installation locations, and it will not depend on .pth files or on putting eggs first. Instead, it will use direct old-style installation, combined with .egg-info directories for metadata and for installation manifests to support automated uninstallation, upgrades, and integrity verification.
Phillip J. Eby wrote:
Meanwhile, in setuptools 0.7 there will be a completely different way of managing installation locations, and it will not depend on .pth files or on putting eggs first. Instead, it will use direct old-style installation, combined with .egg-info directories for metadata and for installation manifests to support automated uninstallation, upgrades, and integrity verification.
Sorry for following up to a very old thread, but I'm intrigued by the above. I couldn't find anything in the setuptools trunk, so I assume this isn't implemented yet. Are there any docs that I could read to find out what exactly you guys have planned there? -- http://worldcookery.com -- Professional Zope documentation and training
At 08:01 PM 6/17/2007 +0200, Philipp von Weitershausen wrote:
Phillip J. Eby wrote:
Meanwhile, in setuptools 0.7 there will be a completely different way of managing installation locations, and it will not depend on .pth files or on putting eggs first. Instead, it will use direct old-style installation, combined with .egg-info directories for metadata and for installation manifests to support automated uninstallation, upgrades, and integrity verification.
Sorry for following up to a very old thread, but I'm intrigued by the above. I couldn't find anything in the setuptools trunk, so I assume this isn't implemented yet.
Nope.
Are there any docs that I could read to find out what exactly you guys have planned there?
Also nope, although googling "setuptools nest" (without the quotes) will probably find you some of my previous posts on the subject.
participants (3)
-
Philipp von Weitershausen -
Phillip J. Eby -
Thomas Heller