We do this after running every test: # Unload the newly imported modules (best effort finalization) for module in sys.modules.keys(): if module not in save_modules and module.startswith("test."): test_support.unload(module) Unfortunately, that doesn't nuke whatever damaged non-test module objects may have been left behind. If I change it to make a truly <wink> best effort: if module not in save_modules: test_support.unload(module) then about a dozen std tests fail in mysterious ways, but only when running more than one test. Like just running test_email test_string causes test_string to die like so: python ../lib/test/regrtest.py -v test_email test_string ... test test_string crashed -- exceptions.AttributeError: 'NoneType' object has no attribute 'get' Traceback (most recent call last): File "../lib/test/regrtest.py", line 305, in runtest the_module = __import__(test, globals(), locals(), []) File "../lib/test\test_string.py", line 34, in ? string_tests.run_method_tests(test) File "../lib/test\string_tests.py", line 233, in run_method_tests verify('hello world'.encode('zlib') == data) File "C:\CODE\PYTHON\lib\encodings\__init__.py", line 43, in search_function entry = _cache.get(encoding,_unknown) AttributeError: 'NoneType' object has no attribute 'get' It seems generally the case that these failures are due to various module-level names getting rebound to None. A simpler example: python ../lib/test/regrtest.py -v test_string test_string ... test test_string crashed -- exceptions.AttributeError: 'NoneType' object has no attribute 'compress' Traceback (most recent call last): File "../lib/test/regrtest.py", line 305, in runtest the_module = __import__(test, globals(), locals(), []) File "../lib/test\test_string.py", line 34, in ? string_tests.run_method_tests(test) File "../lib/test\string_tests.py", line 233, in run_method_tests verify('hello world'.encode('zlib') == data) File "C:\CODE\PYTHON\lib\encodings\zlib_codec.py", line 25, in zlib_encode output = zlib.compress(input) AttributeError: 'NoneType' object has no attribute 'compress' What's up with that? So far, they all seem to involve the encodings directory ...
What's up with that? So far, they all seem to involve the encodings directory ...
Python/codecs.c holds search functions in _PyCodec_SearchPath, among them encodings.__init__.search_function. It also imports "encodings" the first time somebody invokes .encode/.decode on some kind of string, but doesn't hold onto the module. When the last reference to encodings.__init__ goes away, Python will clear all globals - but codecs still holds a reference to search_function, so that won't go away. Invoking search_function later will cause the problem you see. One solution would be to bind all globals used inside encodings.__init__.search_function to the search_function, e.g. by means of a class: class GlobalSearchFunction: _cache = _unknown = class CodecRegistryError(...): def search_function(self,encoding): codecs.register(GlobalSearchFunction().search_function) Of course, if this gets cleaned from sys.modules, nobody will be able to catch CodecRegistryError anymore (since re-importing the module will produce a different class). Regards, Martin
Martin v. Loewis writes:
Perhaps it should hold onto the module as well? That would avoid it getting cleaned up before possibly dependent modules. For the test case problem, perhaps there should be a list of exceptional modules that don't get cleaned up by regrtest once they appear. encodings and warnings would probably both need to be on the list. This list should probably be in the sys module; it would also be useful for the RollbackImporter in the unittestgui.py script that comes with the separate PyUnit package. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
[Fred]
I expect the problem is intractable. Here's a revealing experiment: add this block to the start of test___all__.check_all(): if modname in sys.modules: del sys.modules[modname] That is, just prior to test___all__ trying to import a module, remove *only* that module from sys.modules if it's already there. Sounds pretty harmless, right? But on Windows, it causes 28 standard tests to fail (possibly more on Linux): 28 tests failed: test___all__ test_difflib test_email test_fileinput test_fnmatch test_future test_gettext test_glob test_global test_htmllib test_htmlparser test_inspect test_mimetypes test_mmap test_os test_pyclbr test_re test_regex test_repr test_scope test_sre test_strftime test_strop test_sundry test_tokenize test_urllib2 test_xmllib test_xmlrpc Trying to figure which modules "hurt" turns out to be a nightmare. Here's the minimal set that allows the test suite to pass again on Windows, using the default test order: if modname in sys.modules: if modname not in ( "copy_reg", "os", "sgmllib", "sre", "tempfile", "tokenize", "warnings", ): del sys.modules[modname] Am I gonna check that in? Fat chance <wink>.
Re-importing encodings would still do funny things, then: It would register itself once more, resulting in two search functions being registered. codecs would hold one of the encodings modules, but not the other. So if the second search function is ever invoked, you get the same error. Regards, Martin
"Martin v. Loewis" wrote:
Rather than "fixing" the encodings package, I'd rather suggest to fix the unload procedure in the test suite. Some modules simply should not be unloaded because it is known that the interpreter needs them for internal purposes. In the case of the encodings package it is not only the search function that gets kept alive, it's also the codecs themselves which are cached by the implementation in _codecs.c. How about following Fred's suggestion to run each test in a separate process (possibly as an option to regrtest.py) ?! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
M.-A. Lemburg writes:
How about following Fred's suggestion to run each test in a separate process (possibly as an option to regrtest.py) ?!
I wasn't really suggesting it, but it would solve this particular problem. Note that the effectiveness of this to solve this particular problem depends on the set of modules regrtest itself imports; if it imports something and catches an ImportError, using a simple fork() to create the child is not sufficient. Using fork()/exec() would be extremely slow and should be avoided if we can. Frankly, I like that we find interactions between the tests, so I'm not convinced that there are any real problems with the test framework. I do use to a C unit test package that uses fork() to make sure individual tests can't core the framework, but that's not what we're protecting against here. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
"M" == M <mal@lemburg.com> writes:
M> Rather than "fixing" the encodings package, I'd rather suggest M> to fix the unload procedure in the test suite. Featurette request for Python 2.3: a builtin unload() which Does The Right Thing to unload a module, or raise an exception if the module cannot be unloaded. Lots and lots of applications would find this useful. -Barry
Barry A. Warsaw writes:
Useful or not, isn't the basic problem that we haven't been able to define The Right Thing? I'd certainly expect a module to be able to declare in some way that it could not be unloaded, or control how it gets unloaded. Perhaps allow Python modules to define an __unload__() function; when unload() is called on a module, the __unload__() is called; if that raises an exception, it gets propogated, otherwise the unload continues if the module still exists in sys.modules. Not clear that this would be useful for extensions. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
"Fred" == Fred L Drake, Jr <fdrake@acm.org> writes:
Fred> Useful or not, isn't the basic problem that we haven't Fred> been able to define The Right Thing? Probably so. Fred> I'd certainly expect a module to be able to declare in some Fred> way that it could not be unloaded, or control how it gets Fred> unloaded. Perhaps allow Python modules to define an Fred> __unload__() function; when unload() is called on a module, Fred> the __unload__() is called; if that raises an exception, it Fred> gets propogated, otherwise the unload continues if the Fred> module still exists in sys.modules. Not clear that this Fred> would be useful for extensions. The "unload protocol" would be I think. I.e. extension modules ought to be unloadable if they can cooperate. From the Linux dl*() manpages: dlclose decrements the reference count on the dynamic library handle handle. If the reference count drops to zero and no other loaded libraries use symbols in it, then the dynamic library is unloaded. If the dynamic library exports a routine named _fini, then that routine is called just before the library is unloaded. season-to-tasste-on-your-favorite-OS-bread-l y'rs, -Barry
I haven't been following this discussion. Can you give a clear definition of The Right Thing? --Guido van Rossum (home page: http://www.python.org/~guido/)
Then how can you call it a featurette? Who knows if it's even a tractable problem? --Guido van Rossum (home page: http://www.python.org/~guido/)
Okay, so it's a PEPable feature.
Not until you've thought about what it should do. I bet you lunch that you can't come up with a working definition. --Guido van Rossum (home page: http://www.python.org/~guido/)
Without diving into the unloading pit, the endless problems with -r on non-Unix boxes would go away if a module that failed to import didn't leave behind a damaged module object in sys.modules ("damaged" == it may not fulfill some or any of the module's contracts, and it's impossible to guess which ones it won't, or to guess how the unfufilled ones may fail). I'm tempted to add a pile of Windows-specific modules that import each other in incestuous ways, so everyone can enjoy this chronic pain <wink>. "A fix" seems to amount to treating import M as if it were try: import M except: if M in sys.modules: del sys.modules[M] raise How would that hurt?
On Mon, 17 Dec 2001, Tim Peters wrote:
I remember suggesting exactly that some time ago (i think it was motivated at the time by the extreme pain that broken modules were causing for webserver-mode pydoc in its attempt to update loaded modules if the source files had changed on disk). Guido rejected it because you can't guarantee that the refcount on M is 1 at the point where you attempt to 'del sys.modules[M]' above. (For example, some other module imported by M could have imported M again, and so hold a reference to it while M is running its startup commands. This is why the entry is added to sys.modules before the body of M starts to run.) He deemed the situation where M is loaded-but-missing-from-sys.modules to be even worse than for M to be left loaded-but-broken therein. If you allow M to stay in sys.modules, then you can at least maintain the guarantee that there is a one-to-one association between loaded module names and loaded module objects. If you remove M from sys.modules but it lingers in memory, referenced elsewhere, you lose even that -- there can be many modules loaded all with the same name and it's a nightmare. The argument is compelling and i am forced to agree, but i still think that we should look for a better solution. I'll ponder this and post any ideas i come up with. -- ?!ng
Ka-Ping Yee wrote:
How about only deleting it from sys.modules iff the ref count is 1 ?! Note that you cannot currently remove extension modules that have failed to import correctly. Would be nice if we could come up with a reasonable way of unloading modules, but I guess this requires some hard PEP work... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
[MAL]
Yes, I believe it would. There are different kinds of unloading too: a module may well want to do different things depending on whether, e.g., it's really a reload, or it's because Py_Finalize was called ... hmm! That reminds me: Guido sent me this msg in March; it's still sitting in my inbox waiting for a rainy day: """ OK, so we need to define a way for modules to specify a function to be called at cleanup time. I propose that we bring this out in the open, at least in python-dev. Note that there are three or four potential cleanup times: - when the module object is deleted from sys.modules - when reload() is used on an extension - when an interpreter created with PyInterpreter_New() is finalized - when Py_Finalize() is called - when Python exits - when the Python DLL is unloaded (Windows only) """ There. Somebody double the word count and call it a PEP <wink>. Note that this is orthogonal to regrtest -r's woes.
Tim Peters wrote:
I think we already have this one: I usually register a Py_AtExit (or whatever the name is) function with Python to implement this.
- when Python exits
- when the Python DLL is unloaded (Windows only)
Why Windows only ? I believe we could support this on other platforms as well (e.g. Linux).
"""
There. Somebody double the word count and call it a PEP <wink>.
Since Barry brought this up, I guess we already have a volunteer ;-)
Note that this is orthogonal to regrtest -r's woes.
True and it won't help the suite with the encodings package either (how would you know what to cleanup given the fact that the codec search functions decide how encoding names are mapped to codecs ?). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"M" == M <mal@lemburg.com> writes:
>> """ There. Somebody double the word count and call it a PEP >> <wink>. M> Since Barry brought this up, I guess we already have a M> volunteer ;-) Sure MAL, I'll assign you a PEP number! <wink> -Barry
Ka-Ping Yee <ping@lfw.org>:
He deemed the situation where M is loaded-but-missing-from-sys.modules to be even worse than for M to be left loaded-but-broken therein.
Maybe modules should have a this-module-is-broken flag which is set when the module fails to import properly, and which causes an exception on any further attempt to reference a name in it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
Fredrik Lundh <fredrik@pythonware.com>:
If an exception propagates back through an import statement, the module that it's importing is marked as broken. This may result in a single error causing more than one module to be marked as broken, but I think that's the right thing to do anyway. E.g. if you import module A which imports module B which fails, you can't be sure that either A or B is fit to use, so both should be marked as broken. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
[/F]
Presumably Greg's flag would be initialized to "not broken", and set to "broken" when and only when an uncaught exception is raised. So in import A A imports B which imports A back to B B completes back to A A blows up before completing A's "broken" flag isn't set until the end, so B's recursive import of A isn't harmed.
"Martin v. Loewis" <martin@v.loewis.de>:
I would have said "does not throw ImportError".
Or any other error. If a module fails to complete all its initialisation code for any reason, it can't be trusted. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
[Ka-Ping Yee] Hi, Ping! If we don't hear from you again, Merry Christmas too <wink>.
If somebody wants to hold on to a reference to a damaged module object, that's OK by me. That's an unusual case, though, so shouldn't wag the dog.
Understood, but it's rare.
He deemed the situation where M is loaded-but-missing-from-sys.modules to be even worse than for M to be left loaded-but-broken therein.
Why? The module is broken no mater what. I'm much keener on seeing that the *next* attempt to import the module doesn't falsely appear to succeed.
Whether there's one copy or a million, it's broken. Given that, I don't see value in maintaining the one_name<->one_module invariant: of what practical use is it if the module is an insane state regardless? If there is some value to it I'm missing, it would be easy to give it a unique random (re)name <wink>.
Greg's "broken module" flag would do it for me too, if an attempt to import a known-to-be-broken module raised ImportError (in addition to Greg's suggestion that attribute references complain). That scheme has holes too, but it gets increasingly harder to fall into them. The current scheme has so many pits it's proved impossible not to fall into them when using regrtest -r, just doing utterly vanilla non-recursive imports. compelling-my-foot<wink>-ly y'rs - tim
What's up with that? So far, they all seem to involve the encodings directory ...
Python/codecs.c holds search functions in _PyCodec_SearchPath, among them encodings.__init__.search_function. It also imports "encodings" the first time somebody invokes .encode/.decode on some kind of string, but doesn't hold onto the module. When the last reference to encodings.__init__ goes away, Python will clear all globals - but codecs still holds a reference to search_function, so that won't go away. Invoking search_function later will cause the problem you see. One solution would be to bind all globals used inside encodings.__init__.search_function to the search_function, e.g. by means of a class: class GlobalSearchFunction: _cache = _unknown = class CodecRegistryError(...): def search_function(self,encoding): codecs.register(GlobalSearchFunction().search_function) Of course, if this gets cleaned from sys.modules, nobody will be able to catch CodecRegistryError anymore (since re-importing the module will produce a different class). Regards, Martin
Martin v. Loewis writes:
Perhaps it should hold onto the module as well? That would avoid it getting cleaned up before possibly dependent modules. For the test case problem, perhaps there should be a list of exceptional modules that don't get cleaned up by regrtest once they appear. encodings and warnings would probably both need to be on the list. This list should probably be in the sys module; it would also be useful for the RollbackImporter in the unittestgui.py script that comes with the separate PyUnit package. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
[Fred]
I expect the problem is intractable. Here's a revealing experiment: add this block to the start of test___all__.check_all(): if modname in sys.modules: del sys.modules[modname] That is, just prior to test___all__ trying to import a module, remove *only* that module from sys.modules if it's already there. Sounds pretty harmless, right? But on Windows, it causes 28 standard tests to fail (possibly more on Linux): 28 tests failed: test___all__ test_difflib test_email test_fileinput test_fnmatch test_future test_gettext test_glob test_global test_htmllib test_htmlparser test_inspect test_mimetypes test_mmap test_os test_pyclbr test_re test_regex test_repr test_scope test_sre test_strftime test_strop test_sundry test_tokenize test_urllib2 test_xmllib test_xmlrpc Trying to figure which modules "hurt" turns out to be a nightmare. Here's the minimal set that allows the test suite to pass again on Windows, using the default test order: if modname in sys.modules: if modname not in ( "copy_reg", "os", "sgmllib", "sre", "tempfile", "tokenize", "warnings", ): del sys.modules[modname] Am I gonna check that in? Fat chance <wink>.
Re-importing encodings would still do funny things, then: It would register itself once more, resulting in two search functions being registered. codecs would hold one of the encodings modules, but not the other. So if the second search function is ever invoked, you get the same error. Regards, Martin
"Martin v. Loewis" wrote:
Rather than "fixing" the encodings package, I'd rather suggest to fix the unload procedure in the test suite. Some modules simply should not be unloaded because it is known that the interpreter needs them for internal purposes. In the case of the encodings package it is not only the search function that gets kept alive, it's also the codecs themselves which are cached by the implementation in _codecs.c. How about following Fred's suggestion to run each test in a separate process (possibly as an option to regrtest.py) ?! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
M.-A. Lemburg writes:
How about following Fred's suggestion to run each test in a separate process (possibly as an option to regrtest.py) ?!
I wasn't really suggesting it, but it would solve this particular problem. Note that the effectiveness of this to solve this particular problem depends on the set of modules regrtest itself imports; if it imports something and catches an ImportError, using a simple fork() to create the child is not sufficient. Using fork()/exec() would be extremely slow and should be avoided if we can. Frankly, I like that we find interactions between the tests, so I'm not convinced that there are any real problems with the test framework. I do use to a C unit test package that uses fork() to make sure individual tests can't core the framework, but that's not what we're protecting against here. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
"M" == M <mal@lemburg.com> writes:
M> Rather than "fixing" the encodings package, I'd rather suggest M> to fix the unload procedure in the test suite. Featurette request for Python 2.3: a builtin unload() which Does The Right Thing to unload a module, or raise an exception if the module cannot be unloaded. Lots and lots of applications would find this useful. -Barry
Barry A. Warsaw writes:
Useful or not, isn't the basic problem that we haven't been able to define The Right Thing? I'd certainly expect a module to be able to declare in some way that it could not be unloaded, or control how it gets unloaded. Perhaps allow Python modules to define an __unload__() function; when unload() is called on a module, the __unload__() is called; if that raises an exception, it gets propogated, otherwise the unload continues if the module still exists in sys.modules. Not clear that this would be useful for extensions. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
"Fred" == Fred L Drake, Jr <fdrake@acm.org> writes:
Fred> Useful or not, isn't the basic problem that we haven't Fred> been able to define The Right Thing? Probably so. Fred> I'd certainly expect a module to be able to declare in some Fred> way that it could not be unloaded, or control how it gets Fred> unloaded. Perhaps allow Python modules to define an Fred> __unload__() function; when unload() is called on a module, Fred> the __unload__() is called; if that raises an exception, it Fred> gets propogated, otherwise the unload continues if the Fred> module still exists in sys.modules. Not clear that this Fred> would be useful for extensions. The "unload protocol" would be I think. I.e. extension modules ought to be unloadable if they can cooperate. From the Linux dl*() manpages: dlclose decrements the reference count on the dynamic library handle handle. If the reference count drops to zero and no other loaded libraries use symbols in it, then the dynamic library is unloaded. If the dynamic library exports a routine named _fini, then that routine is called just before the library is unloaded. season-to-tasste-on-your-favorite-OS-bread-l y'rs, -Barry
I haven't been following this discussion. Can you give a clear definition of The Right Thing? --Guido van Rossum (home page: http://www.python.org/~guido/)
Then how can you call it a featurette? Who knows if it's even a tractable problem? --Guido van Rossum (home page: http://www.python.org/~guido/)
Okay, so it's a PEPable feature.
Not until you've thought about what it should do. I bet you lunch that you can't come up with a working definition. --Guido van Rossum (home page: http://www.python.org/~guido/)
Without diving into the unloading pit, the endless problems with -r on non-Unix boxes would go away if a module that failed to import didn't leave behind a damaged module object in sys.modules ("damaged" == it may not fulfill some or any of the module's contracts, and it's impossible to guess which ones it won't, or to guess how the unfufilled ones may fail). I'm tempted to add a pile of Windows-specific modules that import each other in incestuous ways, so everyone can enjoy this chronic pain <wink>. "A fix" seems to amount to treating import M as if it were try: import M except: if M in sys.modules: del sys.modules[M] raise How would that hurt?
On Mon, 17 Dec 2001, Tim Peters wrote:
I remember suggesting exactly that some time ago (i think it was motivated at the time by the extreme pain that broken modules were causing for webserver-mode pydoc in its attempt to update loaded modules if the source files had changed on disk). Guido rejected it because you can't guarantee that the refcount on M is 1 at the point where you attempt to 'del sys.modules[M]' above. (For example, some other module imported by M could have imported M again, and so hold a reference to it while M is running its startup commands. This is why the entry is added to sys.modules before the body of M starts to run.) He deemed the situation where M is loaded-but-missing-from-sys.modules to be even worse than for M to be left loaded-but-broken therein. If you allow M to stay in sys.modules, then you can at least maintain the guarantee that there is a one-to-one association between loaded module names and loaded module objects. If you remove M from sys.modules but it lingers in memory, referenced elsewhere, you lose even that -- there can be many modules loaded all with the same name and it's a nightmare. The argument is compelling and i am forced to agree, but i still think that we should look for a better solution. I'll ponder this and post any ideas i come up with. -- ?!ng
Ka-Ping Yee wrote:
How about only deleting it from sys.modules iff the ref count is 1 ?! Note that you cannot currently remove extension modules that have failed to import correctly. Would be nice if we could come up with a reasonable way of unloading modules, but I guess this requires some hard PEP work... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
[MAL]
Yes, I believe it would. There are different kinds of unloading too: a module may well want to do different things depending on whether, e.g., it's really a reload, or it's because Py_Finalize was called ... hmm! That reminds me: Guido sent me this msg in March; it's still sitting in my inbox waiting for a rainy day: """ OK, so we need to define a way for modules to specify a function to be called at cleanup time. I propose that we bring this out in the open, at least in python-dev. Note that there are three or four potential cleanup times: - when the module object is deleted from sys.modules - when reload() is used on an extension - when an interpreter created with PyInterpreter_New() is finalized - when Py_Finalize() is called - when Python exits - when the Python DLL is unloaded (Windows only) """ There. Somebody double the word count and call it a PEP <wink>. Note that this is orthogonal to regrtest -r's woes.
Tim Peters wrote:
I think we already have this one: I usually register a Py_AtExit (or whatever the name is) function with Python to implement this.
- when Python exits
- when the Python DLL is unloaded (Windows only)
Why Windows only ? I believe we could support this on other platforms as well (e.g. Linux).
"""
There. Somebody double the word count and call it a PEP <wink>.
Since Barry brought this up, I guess we already have a volunteer ;-)
Note that this is orthogonal to regrtest -r's woes.
True and it won't help the suite with the encodings package either (how would you know what to cleanup given the fact that the codec search functions decide how encoding names are mapped to codecs ?). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"M" == M <mal@lemburg.com> writes:
>> """ There. Somebody double the word count and call it a PEP >> <wink>. M> Since Barry brought this up, I guess we already have a M> volunteer ;-) Sure MAL, I'll assign you a PEP number! <wink> -Barry
Ka-Ping Yee <ping@lfw.org>:
He deemed the situation where M is loaded-but-missing-from-sys.modules to be even worse than for M to be left loaded-but-broken therein.
Maybe modules should have a this-module-is-broken flag which is set when the module fails to import properly, and which causes an exception on any further attempt to reference a name in it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
Fredrik Lundh <fredrik@pythonware.com>:
If an exception propagates back through an import statement, the module that it's importing is marked as broken. This may result in a single error causing more than one module to be marked as broken, but I think that's the right thing to do anyway. E.g. if you import module A which imports module B which fails, you can't be sure that either A or B is fit to use, so both should be marked as broken. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
[/F]
Presumably Greg's flag would be initialized to "not broken", and set to "broken" when and only when an uncaught exception is raised. So in import A A imports B which imports A back to B B completes back to A A blows up before completing A's "broken" flag isn't set until the end, so B's recursive import of A isn't harmed.
"Martin v. Loewis" <martin@v.loewis.de>:
I would have said "does not throw ImportError".
Or any other error. If a module fails to complete all its initialisation code for any reason, it can't be trusted. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
[Ka-Ping Yee] Hi, Ping! If we don't hear from you again, Merry Christmas too <wink>.
If somebody wants to hold on to a reference to a damaged module object, that's OK by me. That's an unusual case, though, so shouldn't wag the dog.
Understood, but it's rare.
He deemed the situation where M is loaded-but-missing-from-sys.modules to be even worse than for M to be left loaded-but-broken therein.
Why? The module is broken no mater what. I'm much keener on seeing that the *next* attempt to import the module doesn't falsely appear to succeed.
Whether there's one copy or a million, it's broken. Given that, I don't see value in maintaining the one_name<->one_module invariant: of what practical use is it if the module is an insane state regardless? If there is some value to it I'm missing, it would be easy to give it a unique random (re)name <wink>.
Greg's "broken module" flag would do it for me too, if an attempt to import a known-to-be-broken module raised ImportError (in addition to Greg's suggestion that attribute references complain). That scheme has holes too, but it gets increasingly harder to fall into them. The current scheme has so many pits it's proved impossible not to fall into them when using regrtest -r, just doing utterly vanilla non-recursive imports. compelling-my-foot<wink>-ly y'rs - tim
participants (9)
-
barry＠zope.com -
Fred L. Drake, Jr. -
Fredrik Lundh -
Greg Ewing -
Guido van Rossum -
Ka-Ping Yee -
M.-A. Lemburg -
Martin v. Loewis -
Tim Peters