Windows installer and .pyc files mtime
I was watching file modification times on my Windows box (strange hobby, I know :-), and I noticed that after a fresh install of Python, the .pyc files seem to be written when the first code that imports the corresponding module runs, rather than all of the .pyc files being compiled at once by the installer. Wasn't there code in the installer that precompiles all modules? I know the Unix install does this, and I vaguely remember that the Windows installer did this too -- or was it only the Win32all installer??? If there's code to do that in the Windows installer now, it seems it's not working. If there isn't such code, perhaps there should be? --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido]
I was watching file modification times on my Windows box (strange hobby, I know :-), and I noticed that after a fresh install of Python, the .pyc files seem to be written when the first code that imports the corresponding module runs, rather than all of the .pyc files being compiled at once by the installer. Wasn't there code in the installer that precompiles all modules?
Not in the PLabs Windows installer, no.
I know the Unix install does this, and I vaguely remember that the Windows installer did this too -- or was it only the Win32all installer???
Probably the latter, then.
If there's code to do that in the Windows installer now, it seems it's not working.
There isn't, so it's working fine <wink>.
If there isn't such code, perhaps there should be?
Why? Not that increasing installation time and disk consumption aren't worthy goals ...
[Guido]
I was watching file modification times on my Windows box (strange hobby, I know :-), and I noticed that after a fresh install of Python, the .pyc files seem to be written when the first code that imports the corresponding module runs, rather than all of the .pyc files being compiled at once by the installer. Wasn't there code in the installer that precompiles all modules?
[Tim]
Not in the PLabs Windows installer, no.
I know the Unix install does this, and I vaguely remember that the Windows installer did this too -- or was it only the Win32all installer???
Probably the latter, then.
If there's code to do that in the Windows installer now, it seems it's not working.
There isn't, so it's working fine <wink>.
OK, so maybe I was hallucinating.
If there isn't such code, perhaps there should be?
Why? Not that increasing installation time and disk consumption aren't worthy goals ...
If this isn't done, a problem might be (and this is why this is always done on Unix) that if the user who installs Python has more privileges than the user who uses Python, the user who uses Python may not be able to write the directory containing .pyc files, so they end up recompiling all modules each time they are loaded. I expect this will be more of a problem as typical Windows users and installations (e.g. XP) become more security aware, software is installed by Administrator, and users don't have Administrator privileges. I guess the way to implement it (and I believe Mark Hammond did indeed do this for win32all) is to run Python near the end of the installer with the compileall.py script as an argument. Feeling-quite-the-Windows-XP-expert-lately, --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido]
If this isn't done, a problem might be (and this is why this is always done on Unix) that if the user who installs Python has more privileges than the user who uses Python, the user who uses Python may not be able to write the directory containing .pyc files, so they end up recompiling all modules each time they are loaded.
I expect this will be more of a problem as typical Windows users and installations (e.g. XP) become more security aware, software is installed by Administrator, and users don't have Administrator privileges.
I guess the way to implement it (and I believe Mark Hammond did indeed do this for win32all) is to run Python near the end of the installer with the compileall.py script as an argument.
Feeling-quite-the-Windows-XP-expert-lately,
Cool. If an organization has enough money to afford an administrator who installs stuff for unprivileged masses, they have enough money to pay Thomas to make this change <0.5 wink>. Mark once told me he compiles stuff at the end of the win32all install because generating Windows type libraries can take a long time, and users griped about feeling the pain of that on first use otherwise. That's a reason I can understand <wink>. If you can too, then it's more important to precompile everything needed for IDLE to start up the first time.
"Tim Peters"
[Guido]
If this isn't done, a problem might be (and this is why this is always done on Unix) that if the user who installs Python has more privileges than the user who uses Python, the user who uses Python may not be able to write the directory containing .pyc files, so they end up recompiling all modules each time they are loaded.
I expect this will be more of a problem as typical Windows users and installations (e.g. XP) become more security aware, software is installed by Administrator, and users don't have Administrator privileges.
I guess the way to implement it (and I believe Mark Hammond did indeed do this for win32all) is to run Python near the end of the installer with the compileall.py script as an argument.
Feeling-quite-the-Windows-XP-expert-lately,
Cool. If an organization has enough money to afford an administrator who installs stuff for unprivileged masses, they have enough money to pay Thomas to make this change <0.5 wink>.
While I'm still waiting for the money to come in, I have changed the wise script (but this is still uncommitted) to fire up compileall.py on the standard library, exluding the Lib/test subdir, in both normal and optimized mode. This can be disabled in the advanced install options dialog. It fires up this ugly dos box, but I don't have time nor motivation to write a wise extension only for a progress bar instead - the only other possibility would be to run the compilation with pythonw.exe, and so totally hide it from the user, and run it in the background without waiting for it. If I find time before my vacation, I can try to build an installer beta and upload it to starship (hopefully this works again), if anyone wants to try it out.
Mark once told me he compiles stuff at the end of the win32all install because generating Windows type libraries can take a long time, and users griped about feeling the pain of that on first use otherwise.
That's a reason I can understand <wink>. If you can too, then it's more important to precompile everything needed for IDLE to start up the first time.
Thomas
While I'm still waiting for the money to come in, I have changed the wise script (but this is still uncommitted) to fire up compileall.py on the standard library, exluding the Lib/test subdir, in both normal and optimized mode. This can be disabled in the advanced install options dialog. It fires up this ugly dos box, but I don't have time nor motivation to write a wise extension only for a progress bar instead - the only other possibility would be to run the compilation with pythonw.exe, and so totally hide it from the user, and run it in the background without waiting for it.
That would be bad I think. Personally, I can live with the DOS box, although I wonder if it would scare Tim's sisters. Maybe you can make this another compilation option, one that is off by default?
If I find time before my vacation, I can try to build an installer beta and upload it to starship (hopefully this works again), if anyone wants to try it out.
I'd love to, so please do. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum
If I find time before my vacation, I can try to build an installer beta and upload it to starship (hopefully this works again), if anyone wants to try it out.
I'd love to, so please do.
Temporary download url: http://starship.python.net/crew/theller/python-2.3.1-beta1.exe This is built from CVS 23maint branch, some hours ago. The uncommited changes I made are here, together with additional comments: http://www.python.org/sf/796919 Only slightly tested so far. I'm nearly off for vacation, until Sept, 15. Thomas
[Thomas Heller]
Temporary download url:
http://starship.python.net/crew/theller/python-2.3.1-beta1.exe
This is built from CVS 23maint branch, some hours ago. The uncommited changes I made are here, together with additional comments:
http://www.python.org/sf/796919
Only slightly tested so far.
Slightly more now, on Win98SE. Cool! I thought something was wrong because the install on Win98SE went faster than I was used to. Turns out that's just because we're installing 1,200+ fewer files than we used to (thanks to shipping all the HTML docs in a single .chm file). Everything I tried worked fine. I think the *default* should be not to compile .pyc files. Regardless of the default, popping up a DOS box during compilation doesn't bother me at all.
"Tim Peters"
[Thomas Heller]
Temporary download url:
http://starship.python.net/crew/theller/python-2.3.1-beta1.exe
This is built from CVS 23maint branch, some hours ago. The uncommited changes I made are here, together with additional comments:
http://www.python.org/sf/796919
Only slightly tested so far.
Slightly more now, on Win98SE. Cool! I thought something was wrong because the install on Win98SE went faster than I was used to. Turns out that's just because we're installing 1,200+ fewer files than we used to (thanks to shipping all the HTML docs in a single .chm file).
Everything I tried worked fine. I think the *default* should be not to compile .pyc files. Regardless of the default, popping up a DOS box during compilation doesn't bother me at all.
Great, I'll change it. But when there's an option to compile .pyc, shouldn't there also be an option for .pyo (and should it use -O or -OO)? Thomas
[Tim]
Everything I tried worked fine. I think the *default* should be not to compile .pyc files. Regardless of the default, popping up a DOS box during compilation doesn't bother me at all.
[Thomas Heller]
Great, I'll change it. But when there's an option to compile .pyc, shouldn't there also be an option for .pyo (and should it use -O or -OO)?
Sounds like YAGNI. In the years I did the Windows installer, nobody asked for compilation. I haven't seen anyone here ask for it either, just a few people saying they think other people might want it. Since I wasn't one of them, you're going to have to ask someone who thinks they can channel these oddly unassertive users <wink>.
[Guido]
I expect this will be more of a problem as typical Windows users and installations (e.g. XP) become more security aware, software is
I expect you are correct. My reason was a little more practical - certain h2py.py generated scripts would takes *ages* to compile (way back when sub GHz machines were the norm!) and this made the first pywin32 program they used (often Pythonwin) very very slow to start. I didn't want to give the wrong impression about how slow it really was <wink>.
I guess the way to implement it (and I believe Mark Hammond did indeed do this for win32all) is to run Python near the end of the installer with the compileall.py script as an argument.
Quite frankly, I would find a DOS window executing "compileall" a deadly sin - so I got a little carried away and wrote a WISE extension, so that I got a cute progress bar during the compilation :) However, I haven't advocated or implemented this for the main installer I would prefer to see that effort spent moving from WISE to Inno. The next Inno version will include the previously optional "scripting extensions", and this should make it practical to use with Python. Obviously, a WISE extension wouldn't be heading in this direction. FYI, the new version of inno is at http://www.jrsoftware.org/is4.php, and its license (http://www.jrsoftware.org/files/is/license.txt) would seem a better fit for Python than WISE. Could make a cute little PSA sponsored project <wink>. Pity it uses pascal <frown>. Mark
The newest Zope installs schlep along a whole Python installation (including win32all), and precompile all the pyc/pyos at the end of the install process. This includes all py files shipped as part of the Python stdlib, the win32all extension py files, and all Zope py files. It does take a bit, but it's not intolerable, even on the creaky 500MHz Celeron on which I build the stuff. I think this was probably just an overgeneralization carried over from the install process for RPMs. Not compiling .py files after installation on UNIX typically results in needing to recompile them every time Pytyhon is invoked because the directories they're typically installed in are generally not normal-user-writable. This probably isn't the case for the majority of Windows installs, but we ignore that fact and do it anyway. nothing-quite-like-building-python-using-make-bash-and-pascal-ly y'rs, - C On Wed, 2003-08-27 at 00:20, Mark Hammond wrote:
[Guido]
I expect this will be more of a problem as typical Windows users and installations (e.g. XP) become more security aware, software is
I expect you are correct. My reason was a little more practical - certain h2py.py generated scripts would takes *ages* to compile (way back when sub GHz machines were the norm!) and this made the first pywin32 program they used (often Pythonwin) very very slow to start. I didn't want to give the wrong impression about how slow it really was <wink>.
I guess the way to implement it (and I believe Mark Hammond did indeed do this for win32all) is to run Python near the end of the installer with the compileall.py script as an argument.
Quite frankly, I would find a DOS window executing "compileall" a deadly sin - so I got a little carried away and wrote a WISE extension, so that I got a cute progress bar during the compilation :)
However, I haven't advocated or implemented this for the main installer I would prefer to see that effort spent moving from WISE to Inno. The next Inno version will include the previously optional "scripting extensions", and this should make it practical to use with Python. Obviously, a WISE extension wouldn't be heading in this direction.
FYI, the new version of inno is at http://www.jrsoftware.org/is4.php, and its license (http://www.jrsoftware.org/files/is/license.txt) would seem a better fit for Python than WISE. Could make a cute little PSA sponsored project <wink>. Pity it uses pascal <frown>.
Mark
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev
participants (6)
-
Chris McDonough -
Guido van Rossum -
Mark Hammond -
Thomas Heller -
Tim Peters -
Tim Peters