Re: [Python-Dev] urllib2 EP + decr. startup time
At 07:29 PM 2/16/2007 +0200, KoDer wrote:
2007/2/16, Phillip J. Eby
: At 04:38 PM 2/16/2007 +0200, KoDer wrote: .....
Also, are you aware that putting a zipped version of the standard library on sys.path already speeds up startup considerably? Python since 2.3 automatically includes an appropriate entry in sys.path:
zipped version has one weakness - you can't put .so(or dll) files inside. In my system 19 from 25 installed egg add directories ,not archives (because it's contain dll ).
There's something wrong there. You can put .so, .dll, or .pyd files in eggs - they get extracted to a cache directory automatically, but those directories *don't* get added to sys.path. The mere presence of such files is not enough to cause an egg to be installed unzipped. The package has to be using things like __file__, the package author has to have marked it zip-unsafe, or easy_install was invoked with a request to install unzipped. As long as none of these conditions apply, the egg should be installed zipped, with dynamic libraries automatically extracted on-demand to the $PYTHON_EGG_CACHE.
But even without egg directories >> ['', 'C:\\Python25\\Scripts', 'C:\\WINDOWS\\system32\\python25.zip', 'C:\\Python25\\DLLs', 'C:\\Python25\\lib', 'C:\\Python25\\lib\\plat-win', ......... 'C:\\Python25\\lib\\site-packages\\wx-2.8-msw-unicode'] len(sys.path) == 18 (without eggs) near 18 / 2 = 9 'file not found' errors for every first module import.
That depends on what the module is. In the path you've shown, having a python25.zip would allow only 2 failures before each stdlib import. (Note that an failed zipfile imports are roughly equivalent to a dictionary lookup in time - they don't access the filesystem once the zipfile index is loaded).
So improvement of setuptools will help, but not solve this problem .
Right -- most of your problem will be solved by creating 'C:\\WINDOWS\\system32\\python25.zip', containing the contents of C:\\Python25\\lib\\. Hm. Interesting, actually. Does anybody know why it's looking for 'C:\\WINDOWS\\system32\\python25.zip'? That seems wrong to me.
On 16/02/07, Phillip J. Eby
Hm. Interesting, actually. Does anybody know why it's looking for 'C:\\WINDOWS\\system32\\python25.zip'? That seems wrong to me.
It looks alongside python25.dll, which is installed in windows\system32 by default. If you then ask why the DLL goes in the system directory, I believe this is to help with Python COM objects, which don't otherwise have the Python directory on their PATH (and so wouldn't find python25.dll if it were there). Paul.
Right -- most of your problem will be solved by creating 'C:\\WINDOWS\\system32\\python25.zip', containing the contents of C:\\Python25\\lib\\.
C:\\Python25\\lib\\. contain *many* packages with .dll files - i can't just zip it. wxPython,pyOpenGL,PIL,tk and so on. On Fedora 6 more than 40% dirs of /usr/lib/site-packages contained .so files. Some of them add dirs to path (wx,PIL,Gtk,...). yum,apt and other will bee very angry if i zip site-packages directory. I don't known any package manager which can properly work with packages installed in archive. Are setuptools can work properly with packages packed in one big zip archive (i really don't known)? And finally - if it's so easy why this don't done already? Python widely used in many linux distros and i don't known any one which can install even standard library in zip archive. Most of users can't done it(because they don't known about python at all). Or this just because lack of time? Yesterday i test some programs with strace and receive follow results: command num of sys_calls num of FILE_NOT_FOUND python -c "pass" 2807 619 ~20% yum 20263 11282 >50% pychecker 6181 2527 ~40% meld(nice GUI merge util) 16075 8024 50% ipython < exit.txt 16448 8957 >50% (exit txt contain "exit()\n") (if filter some of FILE_NOT_FOUND which are not produced by python modules search) BTW. In trace results many call chain like this: open("/usr/lib/python2.4/site-packages/Durus-3.6-py2.4-linux-i686.egg", O_RDONLY|O_LARGEFILE) = 6 ...... _llseek(6, 98304, [98304], SEEK_SET) = 0 read(6, "\340\377\224\322\373C\200\177.\245\367\205\0\307x\207\r"..., 4096) = 4096 _llseek(6, 102400, [102400], SEEK_SET) = 0 _llseek(6, 102400, [102400], SEEK_SET) = 0 _llseek(6, 102400, [102400], SEEK_SET) = 0 ..... and so on. As i understand all _llseek(6, 102400, [102400], SEEK_SET) = 0 calls after first are just heating air. -- K.Danilov aka KoDer
KoDer schrieb:
open("/usr/lib/python2.4/site-packages/Durus-3.6-py2.4-linux-i686.egg", O_RDONLY|O_LARGEFILE) = 6 ...... _llseek(6, 98304, [98304], SEEK_SET) = 0 read(6, "\340\377\224\322\373C\200\177.\245\367\205\0\307x\207\r"..., 4096) = 4096 _llseek(6, 102400, [102400], SEEK_SET) = 0 _llseek(6, 102400, [102400], SEEK_SET) = 0 _llseek(6, 102400, [102400], SEEK_SET) = 0 ..... and so on. As i understand all _llseek(6, 102400, [102400], SEEK_SET) = 0 calls after first are just heating air.
If you want to implement a patch to eliminate unnecessary system calls, please submit it to sf.net/projects/python. Regards, Martin
participants (4)
-
"Martin v. Löwis"
-
KoDer
-
Paul Moore
-
Phillip J. Eby