[Patches] [ python-Patches-416704 ] More robust freeze
noreply@sourceforge.net
noreply@sourceforge.net
Mon, 25 Nov 2002 12:15:49 -0800
Patches item #416704, was opened at 2001-04-17 10:24
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=416704&group_id=5470
Category: None
Group: None
Status: Open
Resolution: Accepted
Priority: 7
Submitted By: Toby Dickenson (htrd)
>Assigned to: Just van Rossum (jvr)
Summary: More robust freeze
Initial Comment:
This patch addresses three issues, all relating to
robustness of frozen programs.
Specifically, this patch allows explicit and complete
control over which modules may be loaded from source
on the filesystem of the host system where the frozen
program is run, and which may not.
Without this patch it is impossible to create a non-
trivial frozen program which will *never* load a
module from source on the filesystem.
1. A patch to correct bug #404545 (frozen package
import uses wrong files). Under this change,
submodules of a frozen package must themselves be
frozen modules. Previously, the import machinery may
also try to import submodules from curiously named
files (packagename.modulename.py) from directories in
sys.path
2. A patch to add an extra command line option -E to
freeze.py, which forces freeze to terminate with an
error message if there are modules that it can not
locate.
If this switch is not specified then the default
behaviour is unchanged: modules which can not be found
by freeze will not be included in the frozen program,
and the import machinery will try to load them from
source on sys.path when the frozen program is run.
In practice we have found that a missing module is
probably an error (and it is a fairly frequent error
too!). The -E switch can be used to detect this error;
any missing modules will cause freeze.py to fail.
In the rare case of a frozen module importing a non-
frozen one (ie one which should be loaded from source
when the program is run), the non-frozen module must
be excluded from the freeze using the -x option.
3. A patch to add an extra command line option -X to
freeze.py, which indicates that a specified module is
excluded from the freeze, and also that the frozen
program should not try to load the module from
sys.path when it is imported. Importing the specified
module will always trigger an ImportError.
This is useful if a module used by a frozen program
can optionally use a submodule...
try:
import optional_submodule
except ImportError:
pass
It may be preferable for the frozen program's
behaviour to not depend on whether optional_submodule
happens to be installed on the host system, and that
the 'import optional_submodule' should always fail
with an ImportError. This can be achieved using the '-
X optional_submodule' command line switch to freeze.py
This is implemented by including the excluded module
in the frozen imports table (_PyImport_FrozenModules),
with the code pointer set to NULL.
----------------------------------------------------------------------
>Comment By: Guido van Rossum (gvanrossum)
Date: 2002-11-25 15:15
Message:
Logged In: YES
user_id=6380
Please leave me out of this. I have unfortunately no time
left for patch triage.
----------------------------------------------------------------------
Comment By: Just van Rossum (jvr)
Date: 2002-11-25 05:41
Message:
Logged In: YES
user_id=92689
You write "yes this really happened". I would like to
believe you, but I'd like to understand *how* it can happen
(I tried hard to _make_ it happen an failed ;-). If a.b
exists as a frozen module, I really don't see how a.b.py
could ever be found before it. Can you elaborate?
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2002-11-25 03:21
Message:
Logged In: YES
user_id=46460
Changing this behaviour would make it difficult to write frozen programs
whose behaviour is predictable - where the behaviour can not be subverted
by files with funny names. This is important for:
1. suid frozen programs (although there are other ways around this)
2. vendors who ship frozen programs, who dont want it to break if the
user happens to have a funny named file in the current directory (yes, this
really happened.)
It is currently possible to work around this new problem in pure python, by
importing the extension module packagelessly, and twiddling sys.modules to
add it to the package. I think this solution is well known in the python/COM
world, where there are many extension modules in the win32com package.
The following example hack is taken from the top of one of our frozen
products:
if pythoncom.frozen:
# A bug in freeze causes problems for extension modules that originally
appeared in packages.
# They can only be imported packagelessly.
def _fix_frozen_extensions(package_name,extension_name):
extension = sys.modules[package_name+'.'+extension_name] =
__import__(extension_name)
__import__(package_name)
setattr(sys.modules[package_name],extension_name,extension)
_fix_frozen_extensions('GeminiDataLoggers.Cedar','SimplePort')
_fix_frozen_extensions('GeminiDataLoggers.Cedar','SimpleMAPI')
_fix_frozen_extensions('GeminiDataLoggers.Cedar','WinINet')
_fix_frozen_extensions('win32com.axcontrol','axcontrol')
Is there a reason why the extension needs to be dynamically loaded, rather
----------------------------------------------------------------------
Comment By: Just van Rossum (jvr)
Date: 2002-11-24 16:41
Message:
Logged In: YES
user_id=92689
Reopening this patch, adding a counterpatch. (Now I'm sorry
I didn't monitor this issue when it came up!)
I'd like to undo part of T.Dickensons' patch for 2.3 as I
think it's a mistake. For one, it breaks a patch of my own
from 1998, which Guido back then checked in as rev. 2.116.
The reasons I want it out again are twofold:
1) it removes a useful (if not *needed*) feature
2) it fixes a problem that is not reproducible.
Currently it is not possible to use a builtin module or a
dynamically loaded extension as a submodule of a frozen
package. While this is perhaps a dubious habit, it _is_
allowed in a non-frozen world and there _are_ packages out
there that employ this. My counterpatch makes this possible
again.
Guido's checkin msg of rev. 2.116 of import.c contains this:
"""(I *think* this means that we can now have a built-in
module bar that's a submodule of a frozen package foo, by
registering the built-in module with a name "foo.bar" in the
table of builtin modules.)"""
He is correct ;-) But the effect is broader: it works with
_any_ module type. So yes, .py files with dotted names are
also found, but _only_ if no frozen module by that same name
exists. I don't even think it's a wart, let alone a bug:
it's a feature.
See also the comment I just added to bug #404545
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-10-18 15:15
Message:
Logged In: YES
user_id=6380
Thanks. All checked in!
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-10-18 14:54
Message:
Logged In: YES
user_id=6380
I checked in the first part (import.c).
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-10-16 17:35
Message:
Logged In: YES
user_id=6380
I'll try to look at this now!
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2001-10-03 06:39
Message:
Logged In: YES
user_id=46460
Still no patch there. Maybe this time?
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2001-10-03 06:37
Message:
Logged In: YES
user_id=46460
Once again, including the patch this time.
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2001-10-03 06:34
Message:
Logged In: YES
user_id=46460
Attached is an updated patch against the current CVS.
I have verified each aspect of this patch with
Py_VerboseFlag set to 2, so that import.c traces out which
files it is checking during the import process.
(Im not aware of an easy way to set Py_VerboseFlag to 2 in
a frozen program.... Ive added a new patch #467455 to
enhance the PYTHONVERBOSE environment variable to support
this)
I can confirm that the current CVS (rougly 2.2a4) still
demonstrates the problem from bug #404545. The details are
slightly different to my original bug report; Im not sure
if this is due to an incidental change in python since
1.5.2, or if I messed up that bug report. Anyway, using
PyVerbose=2 clearly shows that before this patch it is
looking for files it shouldnt (and it uses those files if
they exist). After this patch, it definitely only looks for
the right files.
The other aspect of this patch, adding -E and -X to
freeze.py, is exactly as before.
----------------------------------------------------------------------
Comment By: Toby Dickenson (htrd)
Date: 2001-09-10 10:20
Message:
Logged In: YES
user_id=46460
OK, I should get round to this soon.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-05 11:50
Message:
Logged In: YES
user_id=6380
I'm willing to look at this, but right now the patch doesn't
apply cleanly. From the headers in the diff file it looks
like you're patching an early beta for 2.0.
If you can rework this for current CVS or Python 2.2a2 or
2.2a3, that would be great.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-06-07 18:26
Message:
Logged In: YES
user_id=21627
Why is this assigned to Mark? I cannot see anything
windows-specific in it. Mark, if you are not interested in
reviewing this patch, I recommend to unassign this from
yourself.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=416704&group_id=5470