<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=NL-BE link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hi list,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>A bit of context: we’re running buildout 2.0.1, building zope and plone applications, zope 2.13.10, combined with plone.recipe.zope2instance. This question only impacts the performance of the startup of an instance, the the performance of a running instance. But starting an instance is something developpers do quite often on a days development.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>While debugging some instance startup performance issues, I came across the following.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>The buildout Scripts prepends all the eggs to the system path, before the python path.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>In our setup this causes quite some delays, because imports from standard python modules, also try to find that module in every eggs directory, before it can find in in the default python location (because the eggs are prepended). The eggs even go before the local folder, so even for importing a local module, all eggs are poked first. If you are only working on local disks, the starup performance difference is neglectable (speaking seconds), but if the eggs are located on network disks, there is a performance difference of about 30% in startup time (speaking minutes), with the path prepended vs appended (where appended is the faster startup).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Some numbers from strace: with eggs appended to sys.path:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>% time seconds usecs/call calls errors syscall<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>------ ----------- ----------- --------- --------- ----------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>48.96 0.014658 0 142555 134820 open<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>And with eggs prepended to sys.path<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>% time seconds usecs/call calls errors syscall<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>------ ----------- ----------- --------- --------- ----------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>58.82 0.035995 0 199897 192150 open<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>As you can see the amount of calls, and consequently the amount of errors, is noticeable higher when prepending the eggs to the sys.path. On local disks, like the above numbers, the difference is noticeable, but still fairly small (timewise, not perdcentage wise), add the bit of extra delay of a network disk, and the differences become really noticable.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>As far as I’ve always understood, the default procedure working with paths should be to append, unless you have a good reason. The good reason in this case, that I see, could be that you want to prepend certain packages and that way make sure you use your version instead of what’s present in site-packages. Now my expectation would be that that is a fairly limited set of packages that need to be prepended. If there are many, options like virtualenv exist to avoid taking site-packages at all (that’s what we do btw).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Am I missing a use case for the sys.path prepending, or has this never been an issue before? Because if there is only the site-packages, or a like issue, I’m happy to have a look into splitting the python path up in what should be prepended (whole python path except site-packages?), and making append the default (site-packages + eggs + …), or looking how I could provide a buildout syntax, where append would be default and one could explicitly prepend some packages with buildout.cfg.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Another option I’ve investigated is to use the meta_path hook, providing my own find_module and load_module, keeping a dictionary of module locations. I don’t have similar timings about the numbers of syscalls like the ones above for this scenario. I’m reconstructing that setup for the moment and see if that makes a difference. Timing wise it’s a bit slower (but still acceptable, instance starts in 30sec instead of 15/20), but I have no measurements yet, that indicate something usefull.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Thanks for your feedback,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Jan<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>