[Python-Dev] Inconsistent behaviour in import/zipimport hooks
ulrich.berning at desys.de
Fri Nov 11 14:32:21 CET 2005
Phillip J. Eby schrieb:
>At 04:33 PM 11/9/2005 -0800, Guido van Rossum wrote:
>>On 11/9/05, Phillip J. Eby <pje at telecommunity.com> wrote:
>>>By the way, while we're on this subject, can we make the optimization
>>>options be part of the compile() interface? Right now the distutils has to
>>>actually exec another Python process whenever you want to compile
>>>a different optimization level than what's currently in effect, whereas if
>>>it could pass the desired level to compile(), this wouldn't be necessary.
>>Makes sense to me; we need a patch of course.
>But before we can do that, it's not clear to me if it should be part of the
>existing "flags" argument, or whether it should be separate. Similarly,
>whether it's just going to be a level or an optimization bitmask in its own
>right might be relevant too.
>For the current use case, obviously, a level argument suffices, with 'None'
>meaning "whatever the command-line level was" for backward
>compatibility. And I guess we could go with that for now easily enough,
>I'd just like to know whether any of the AST or optimization mavens had
>anything they were planning in the immediate future that might affect how
>the API addition should be structured.
I'm using a totally different approach for the above problem. I have
implemented two functions in the sys module, that make the startup flags
accessible at runtime. This also solves some other problems I had, as
you will see in the examples below:
The first function makes most of the flags readable (I have ommited the
flags, that are documented as deprecated in the code):
sys.getrunflag(name) -> integer
Return one of the interpreter run flags. Possible names are 'Optimize',
'Verbose', 'Interactive', 'IgnoreEnvironment', 'Debug',
'DivisionWarning', 'NoSite', 'NoZipImport', 'UseClassExceptions',
'Unicode', 'Frozen', 'Tabcheck'. getrunflag('Optimize') for example
returns the current value of Py_OptimizeFlag.
The second function makes a few flags writable:
sys.setrunflag(name, value) -> integer
Set an interpreter run flag. The only flags that can be changed at
runtime are Py_VerboseFlag ('Verbose') and Py_OptimizeFlag ('Optimize').
Returns the previous value of the flag.
As you can see, I have also introduced the new flag Py_NoZipImport that
can be activated with -Z at startup. This bypasses the activation of
zipimport and is very handy, if you edit modules stored in the
filesystem, that are normally imported from a zip archive and you want
to test your modifications. With this flag, there is no need to delete,
rename or update the zip archive or to modify sys.path to ensure that
your changed modules are imported from the filesystem and not from the
And here are a few usable examples for the new functions:
1.) You have an application, that does a huge amount of imports and
some of them are mysterious, so you want to track them in verbose mode.
You could start python with -v or -vv, but then you get hundreds or
thousands of lines of output. Instead, you can do the following:
oldval = sys.setrunflag('Verbose', 1) # -v, use 2 for -vv
Now, you get only verbose messages for the imports that you want to track.
2.) You need to generate optimized byte code (without assertions and
docstrings) from a source code, no matter how the interpreter was started:
source = ...
oldval = sys.setrunflag('Optimize', 2) # -OO, use 1 for -O
bytecode = compile(source, ...)
3.) You have to build a command line for the running application (e.g.
for registration in the registry) and need to check, if you are running
a script or a frozen executable (this assumes, that your freeze tool
sets the Py_FrozenFlag):
commandline = sys.executable
commandline = '%s %s' % (sys.executable, sys.argv)
NOTE: My own freeze tool sib.py, which is part of the VendorID package
(www.riverbankcomputing.co.uk/vendorid) doesn't set the Py_FrozenFlag
yet. I will provide an update soon.
And now back to the original subject:
I have done nearly the same changes, that Osvaldo provided with his
patch and I would highly appreciate if this patch goes into the next
release. The main reason why I changed the import behavior was
pythonservice.exe from the win32 extensions. pythonservice.exe imports
the module that contains the service class, but because
pythonservice.exe doesn't run in optimized mode, it will only import a
.py or a .pyc file, not a .pyo file. Because we always generate bytecode
with -OO at distribution time, we either had to change the behavior of
pythonservice.exe or change the import behavior of Python.
It is essential for us to remove assertions and docstrings in our
commercial Python applications at distribution time, because assertions
are meaningful only at development time and docstrings may contain
implementation details, that our customers should never see (this makes
reverse engineering a little bit harder, but not impossible).
Another important reason is, that the number of files in the standard
library is reduced dramatically. I can have .py and .pyc files in
lib/pythonX.Y containing assertions and docstrings and put optimized
code in lib/pythonXY.zip. Then, if I need docstrings, I just bypass
zipimport as described above with -Z. On the customer site, we only need
to install the zip archive (our customers are not developers, just end
users; they may not even recognize that we use Python for application
Guido, if it was intentional to separate slightly different generated
bytecode into different files and if you have good reasons for doing
this, why have I never seen a .pyoo file :-)
For instance, nobody would give the output of a C compiler a different
extension when different compiler flags are used.
I would appreciate to see the generation of .pyo files completely
removed in the next release. Just create them as .pyc files, no matter
if -O or -OO is used or not. At runtime, Python should just ignore (jump
over?) assert statements when started with -O or ignore assert
statements and docstrings when started with -OO if they are in the .pyc
There are two reasons, why I haven't started a discussion about those
issues earlier on this list:
1.) I have done those changes (and a lot of other minor and major
changes) in Python 2.3.x at a time when Python 2.4.x came up and we
still use Python 2.3.5. I just wanted to wait until I have the next
release of our Python runtime environment (RTE) ready that will contain
the most recent Python version.
2.) In the last months, my time was limited, because I first had to
stabilize the current RTE (developers and customers were waiting on it).
A few notes about this Python runtime environment:
The RTE that I maintain contains everything (mainly, but not only
Python) that is needed to run our applications on the customer site. The
applications are distributed as frozen binaries either containing the
whole application or only the frozen main script together with an
additional zip archive holding the application specific modules and
packages. Tools like py2exe or cx_Freeze follow a different approach:
they always package a kind of runtime environment together with the
application. There is nothing bad with this approach, but if you
provide more than one application, you waste resources and it may be
harder to maintain (bugfixes, updates, etc.).
Our RTE currently runs on AIX, HP-UX, IRIX, Windows and Linux,
SunOS/Solaris will follow in the near future. It gets installed together
with the application(s) on the customer site if it is not already there
in the appropriate version. It is vendor specific (more in the sense of
a provider, not necessarily in the sense of a seller) and shouldn't
interfere with any other software already installed on the target site.
This RTE is the result of years of experience of commercial application
development with Python (we started with Python-1.3.x a long long time
ago). My guideline is: Application developers and/or customers should
not worry about the requirements needed to get the applications running
on the target site. Developers should spend all their time for
application development and customers can expect a complete installation
with only a very small set of system requirements/dependencies.
In general, I would like to discuss those things that increase Python's
usability especially in a commercial environment and things that make
Python more consistent across platforms (e.g. unifying the filesystem
layout on Windows and UNIX/Linux), but I don't know if this is the right
More information about the Python-Dev