Mailman 3 Fix import errors to have data - Python-Dev

newer
RE: [Python-Dev] Re: 2.4a2, and...

Fix import errors to have data

older
[Python-Dev] Re: [Python-checkins]...

Jim Fulton

27 Jul 2004 27 Jul '04

12:52 p.m.

A common idiom to optionally support some module if it is present is to use ImportError handlers: try: import foo except ImportError: # Configure for absense of foo ... else: # Configure for presense of foo ... Unfortunately, this is a bug trap. The module foo might be present and yet it's import could fail with an import error. This can happen if one of *its* imports fails. Code like that above will hide such bugs. Unfortunately, it's hard to do this correctly, because ImportErrors don't have the module name as data. When the import error is raised, it is raised with an error message rather than data. This is because most standard exception classes share a common __str__ that simply prints their initialization arguments. At present, to conditionally support a module, you have to use code like: try: import foo except ImportError, v: if not str(v).endsswith(' foo'): raise # Configure for absense of foo ... else: # Configure for presense of foo ... which is ugly and brittle. I'd like to get this fixed. I propose to: - Provide ImportError with an __init__ that takes a module name and sets a module_name attribute - Provide ImportError with an __str__ that produces the message we have now - Change standard code that raises import errors to provide just the module name. With this change, one could write careful conditional import code like this: try: import foo except ImportError, v: if v.module_name != 'foo': raise # Configure for absense of foo ... else: # Configure for presense of foo ... which is much cleaner IMO. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Show replies by thread

Guido van Rossum

27 Jul 27 Jul

2:45 p.m.

...

A common idiom to optionally support some module if it is present is to use ImportError handlers:

try: import foo except ImportError: # Configure for absense of foo ... else: # Configure for presense of foo ...

Unfortunately, this is a bug trap. The module foo might be present and yet it's import could fail with an import error. This can happen if one of *its* imports fails. Code like that above will hide such bugs.

Why is it important to catch the bug at this point? If the module imported by foo has a bug, importing it directly will reveal it. The code importing foo will work without foo. In some very real sense, foo is unusable, and its importer is making the right choice.

...

Unfortunately, it's hard to do this correctly, because ImportErrors don't have the module name as data. When the import error is raised, it is raised with an error message rather than data. This is because most standard exception classes share a common __str__ that simply prints their initialization arguments.

At present, to conditionally support a module, you have to use code like:

try: import foo except ImportError, v: if not str(v).endsswith(' foo'): raise # Configure for absense of foo ... else: # Configure for presense of foo ...

which is ugly and brittle.

I'd like to get this fixed.

I propose to:

- Provide ImportError with an __init__ that takes a module name and sets a module_name attribute

- Provide ImportError with an __str__ that produces the message we have now

- Change standard code that raises import errors to provide just the module name.

With this change, one could write careful conditional import code like this:

try: import foo except ImportError, v: if v.module_name != 'foo': raise # Configure for absense of foo ... else: # Configure for presense of foo ...

which is much cleaner IMO.

If you provide a working patch, I have no objection against its introduction. I'm just not going out of my way for it. --Guido van Rossum (home page: http://www.python.org/~guido/)

Jim Fulton

3:36 p.m.

Guido van Rossum wrote:

...

...
A common idiom to optionally support some module if it is present is to use ImportError handlers:

try: import foo except ImportError: # Configure for absense of foo ... else: # Configure for presense of foo ...

Unfortunately, this is a bug trap. The module foo might be present and yet it's import could fail with an import error. This can happen if one of *its* imports fails. Code like that above will hide such bugs.

Why is it important to catch the bug at this point?

Because the masked import failure could leave modules in a a broken state that causes very hard to diagnose problems later.

...

If the module imported by foo has a bug, importing it directly will reveal it.

No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable. Actually, both foo and B can be imported without errors, even though they are broken.

...

The code importing foo will work without foo. In some very real sense, foo is unusable, and its importer is making the right choice.

The intent is to ignore the absense of foo, not the brokenness of foo. If foo is around, but broken, I want to know about it. As a bonus, if I ignore the error in importing foo, I may leave a broken, but importable foo around. If any other code tries to conditionally import foo, it will think that foo is imported and usable. ...

...

If you provide a working patch, I have no objection against its introduction. I'm just not going out of my way for it.

Cool. I expected to do this myself anyway. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Tim Peters

3:57 p.m.

[Jim Fulton] ...

...

No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable. Actually, both foo and B can be imported without errors, even though they are broken.

Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules. Wouldn't it be better to change Python to stop leaving insane module objects in sys.modules to begin with? That's harder, but seems to me it would do a lot more good for a lot more people. Indeed, Python's own test suite fails on Windows often when run with -r (randomize test order) because of broken modules left behind by trying to run Unix tests. Typically one test imports pty, which fails because it can't import termios, but then a later test importing pty succeeds (getting an insane pty) -- and despite this insanity in pty.py already: # Absurd: import termios and then delete it. This is to force an attempt # to import pty to raise an ImportError on platforms that lack termios. # Without this explicit import of termios here, some other module may # import tty first, which in turn imports termios and dies with an # ImportError then. But since tty *does* exist across platforms, that # leaves a damaged module object for tty in sys.modules, and the import # of tty here then appears to work despite that the tty imported is junk. import termios del termios import tty That's a good example of the crazy brittleness we endure now. I really don't want to see more brittle code worming around a problem that "shouldn't" be there at all. That said, I have no objection to giving ImportError exceptions a module_name attribute. My objection is to the idea that it solves the real problem.

Jim Fulton

4:39 p.m.

Tim Peters wrote:

...

[Jim Fulton] ...

...
No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable. Actually, both foo and B can be imported without errors, even though they are broken.

Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules.

No. I'm proposing a way for a Python developer to detect the presence or absence of a module. Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present. AFAIK, the only way to do it is by trying the import. Unfortunately, the error type alone isn't enough to tell whether the error you caught is due to the error condition you are trying to test. Perhaps providing an API is a better way to go. In general, however, I hate the approach of formatting errors as they are raised, rather than as they are converted to strings. Formatting them earlier is a bit more convenient for the exception-class author, but: - It makes the exception data inaccessible without ugly brittle error parsing hacks, and - It makes error raising more costly, which sometimes matters (e.g. for attribute errors). - It makes I18n harder, as it's hard (impossible?) to translate error messages with embedded data.

...

Wouldn't it be better to change Python to stop leaving insane module objects in sys.modules to begin with?

It would make the current situation less icky, but it wouldn't solve the problem I posed. You still wouldn't be able to tell the difference between an absent module and a broken one without a error-parsing hack. I think that not ignoring errors is a good thing even if the errors ignored didn't have side effects. ...

...

That said, I have no objection to giving ImportError exceptions a module_name attribute.

Cool. I could also live with an API that tests whether a module exists.

...

My objection is to the idea that it solves the real problem.

It isn't intended to solve broken-module problem. It solves (or, at least addresses) a different problem, but it is certainly exacerbated by the broken-module problem. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Neal Norwitz

4:49 p.m.

On Tue, Jul 27, 2004 at 12:39:22PM -0400, Jim Fulton wrote:

...

Tim Peters wrote:

...
[Jim Fulton]

...
No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable. Actually, both foo and B can be imported without errors, even though they are broken.

...
Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules.

No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present.

Another possibility could be to add a ChainedImportError (or SubImportError?) which would derive from ImportError. The module which imports a broken module would get the normal ImportError. If the module doesn't catch the exception, it would get converted to this new type of ImportError. I don't know how easy this would be to implement, but I think it would solve Jim's problem and perhaps be easier to deal with? Neal

Tim Peters

4:59 p.m.

[Jim Fulton]

...

... No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Seems to me a module is present if and only if you try to import it, and the import succeeds, or the import fails and the module name is in sys.modules after.

...

...
...
import cupcake # no such thing Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named cupcake 'cupcake' in sys.modules False

...

...
...
import pty # exists, but is broken on Windows Traceback (most recent call last): File "<stdin>", line 1, in ? File "C:\Program Files\Zope-2.7.2-0\bin\Lib\pty.py", line 19, in ? import termios ImportError: No module named termios 'pty' in sys.modules True

Not that I like this -- it turns broken modules into "a feature". A function that answered the question directly would be mounds better.

Jim Fulton

5:07 p.m.

Tim Peters wrote:

...

[Jim Fulton]

...
... No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Seems to me a module is present if and only if you try to import it, and the import succeeds, or the import fails and the module name is in sys.modules after.

That's a definition I don't really care for at all. It's like saying that an object has an attribute if trying to get the attribute doesn't raise an error. :) Seems to me a module isn't present if it's implementation isn't in an import path. To me a module is broken if it has an implementation that fails. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Guido van Rossum

5:07 p.m.

...

Another possibility could be to add a ChainedImportError (or SubImportError?) which would derive from ImportError. The module which imports a broken module would get the normal ImportError. If the module doesn't catch the exception, it would get converted to this new type of ImportError.

I don't know how easy this would be to implement, but I think it would solve Jim's problem and perhaps be easier to deal with?

Having had the recent opportunity to root around in Java code that chains exceptions like this a lot, I think I'm against that approach. --Guido van Rossum (home page: http://www.python.org/~guido/)

Jim Fulton

5:09 p.m.

Neal Norwitz wrote:

...

On Tue, Jul 27, 2004 at 12:39:22PM -0400, Jim Fulton wrote:

...
Tim Peters wrote:

...
[Jim Fulton]

...
No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable. Actually, both foo and B can be imported without errors, even though they are broken.

...
Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules.

No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present.

Another possibility could be to add a ChainedImportError (or SubImportError?) which would derive from ImportError. The module which imports a broken module would get the normal ImportError. If the module doesn't catch the exception, it would get converted to this new type of ImportError.

I don't know how easy this would be to implement, but I think it would solve Jim's problem and perhaps be easier to deal with?

An even simpler aproach would be to have a ModuleNotFound error that extends ImportError and gets raised if the module can't be found. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Casey Duncan

5:17 p.m.

On Tue, 27 Jul 2004 12:39:22 -0400 Jim Fulton wrote:

...

Tim Peters wrote:

...
[Jim Fulton] ...

...
No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable. Actually,>both foo and B can be imported without errors, even though they are>broken.

Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules.

No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present.

If such an api were to exist, what question exactly would it answer? 1. That there is a module by a particular name that could possibly be imported, but the import may not succeed when actually tried. 2. There is a module by a particular name which can be imported and the import will not fail. I would vote for the former, but it might be surprising to run into behavior like this:

...

...
...
sys.hasmodule('foo') True import foo ImportError: foo blew

What if there was a new exception ModuleNotFoundError which subclassed ImportError. This would be raised by the import machinery when the module could not be found. Errors during import would continue to raise a standard ImportError. I think this, coupled with Jim's original suggestion and Tim's to prevent broken modules being retained in os.modules would help a lot. -Casey

Charles Cazabon

5:20 p.m.

Tim Peters wrote:

...

Seems to me a module is present if and only if you try to import it, and the import succeeds, or the import fails and the module name is in sys.modules after.

This is a bit I've never been quite sure of. If a module knows it imports something not available everywhere, and can handle that bit missing, shouldn't it already wrap its import of that module in a try: ... except ImportError: block?

...

...
...
...
import pty # exists, but is broken on Windows Traceback (most recent call last): File "<stdin>", line 1, in ? File "C:\Program Files\Zope-2.7.2-0\bin\Lib\pty.py", line 19, in ? import termios ImportError: No module named termios 'pty' in sys.modules True

To me, it would seem more sensible (and would address the Windows brittleness Tim mentions) if the above did /not/ leave pty in sys.modules -- i.e. the import failed either completely, or not at all. The above is a "partial failure" which seems (to me) to be the root cause of the confusion. Instead, it should fail completely, and then the pty module can (if it wants) try to import termios and handle it gracefully at that level if necessary. Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------

Jim Fulton

5:21 p.m.

Casey Duncan wrote:

...

On Tue, 27 Jul 2004 12:39:22 -0400 Jim Fulton wrote:

...
Tim Peters wrote:

...
[Jim Fulton] ...

...
No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable.

Actually,>both foo and B can be imported without errors, even though they are>broken.

Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules.

No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present.

If such an api were to exist, what question exactly would it answer?

1. That there is a module by a particular name that could possibly be imported, but the import may not succeed when actually tried.

Yup

...

2. There is a module by a particular name which can be imported and the import will not fail.

I would vote for the former, but it might be surprising to run into behavior like this:

...
...
...
sys.hasmodule('foo')

True

...
...
...
import foo

ImportError: foo blew

Of course it would be surprising, because foo would be broken. That's why they call them exceptions. :)

...

What if there was a new exception ModuleNotFoundError which subclassed ImportError. This would be raised by the import machinery when the module could not be found. Errors during import would continue to raise a standard ImportError.

Yup.

...

I think this, coupled with Jim's original suggestion and Tim's to prevent broken modules being retained in os.modules would help a lot.

Yup. It would certainly address the imediate problem. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Tim Peters

5:25 p.m.

[Charles Cazabon]

...

This is a bit I've never been quite sure of. If a module knows it imports something not available everywhere, and can handle that bit missing, shouldn't it already wrap its import of that module in a try: ... except ImportError: block?

Sure! Jim wants to distinguish "absenceness" from "brokenness" for some reason. It's not clear to my why (in response to Guido, he mentioned only the "broken module in sys.modules" problem, but when I suggested we address that directly he just repeated "you still wouldn't be able to tell the difference between an absent module and a broken one").

...

To me, it would seem more sensible (and would address the Windows brittleness Tim mentions) if the above did /not/ leave pty in sys.modules -- i.e. the import failed either completely, or not at all. The above is a "partial failure" which seems (to me) to be the root cause of the confusion. Instead, it should fail completely, and then the pty module can (if it wants) try to import termios and handle it gracefully at that level if necessary.

Leaving insane module objects behind is harmful, and I'd like to repair that. But you still wouldn't be able to tell the difference between an absent module and a broken one <wink>.

Jim Fulton

5:29 p.m.

Tim Peters wrote:

...

[Charles Cazabon]

...
This is a bit I've never been quite sure of. If a module knows it imports something not available everywhere, and can handle that bit missing, shouldn't it already wrap its import of that module in a try: ... except ImportError: block?

Sure! Jim wants to distinguish "absenceness" from "brokenness" for some reason. It's not clear to my why

Because a broken module is something that should get fixed. Hiding errors is a bad thing. It's like using hasattr. :) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Thomas Heller

5:35 p.m.

Jim Fulton writes:

...

Tim Peters wrote:

...
[Jim Fulton] ...

...
No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable. Actually, both foo and B can be imported without errors, even though they are broken. Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules.

No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present.

Doesn't imp.find_module do this? On Windows:

...

...
...
import imp imp.find_module("termios") Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named termios imp.find_module("pty") (, 'c:\\python23\\lib\\pty.py', ('.py', ^Z

Thomas

Barry Warsaw

5:37 p.m.

On Tue, 2004-07-27 at 13:25, Tim Peters wrote:

...

Sure! Jim wants to distinguish "absenceness" from "brokenness" for some reason.

So, why doesn't imp.find_module() fit the bill?

...

Leaving insane module objects behind is harmful, and I'd like to repair that.

+1 (wishing I could vote with a higher score). Leaving broken modules in sys.modules is a recurring headache. -Barry

Jim Fulton

5:45 p.m.

Thomas Heller wrote:

...

Jim Fulton writes:

...

...
Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present.

Doesn't imp.find_module do this? On Windows:

Nope. It doesn't handle packages.

...

...
...
import compiler.ast import imp imp.find_module('compiler.ast') Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named compiler.ast

Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Jim Fulton

5:45 p.m.

Barry Warsaw wrote:

...

On Tue, 2004-07-27 at 13:25, Tim Peters wrote:

...
Sure! Jim wants to distinguish "absenceness" from "brokenness" for some reason.

So, why doesn't imp.find_module() fit the bill?

It doesn't handle packages.

...

...
...
import compiler.ast import imp imp.find_module('compiler.ast') Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named compiler.ast

Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Phillip J. Eby

5:54 p.m.

At 01:45 PM 7/27/04 -0400, Jim Fulton wrote:

...

Barry Warsaw wrote:

...
On Tue, 2004-07-27 at 13:25, Tim Peters wrote:

...
Sure! Jim wants to distinguish "absenceness" from "brokenness" for some reason.

So, why doesn't imp.find_module() fit the bill?

It doesn't handle packages.

...
...
...
import compiler.ast import imp imp.find_module('compiler.ast') Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named compiler.ast

Sure it does; you just have to ask it nicely: :) Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. IDLE 0.8 -- press F1 for help

...

...
...
import imp imp.find_module('compiler') (None, 'C:\\PYTHON22\\lib\\compiler', ('', '', 5)) import compiler imp.find_module('ast',compiler.__path__) (, 'C:\\PYTHON22\\lib\\compiler\\ast.py', ('.py', 'r', 1))

Jim Fulton

5:57 p.m.

Phillip J. Eby wrote:

...

At 01:45 PM 7/27/04 -0400, Jim Fulton wrote:

...
Barry Warsaw wrote:

...
On Tue, 2004-07-27 at 13:25, Tim Peters wrote:

...
Sure! Jim wants to distinguish "absenceness" from "brokenness" for some reason.

So, why doesn't imp.find_module() fit the bill?

It doesn't handle packages.

...
...
...
import compiler.ast import imp imp.find_module('compiler.ast') Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named compiler.ast

Sure it does; you just have to ask it nicely: :)

Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. IDLE 0.8 -- press F1 for help

...
...
...
import imp imp.find_module('compiler') (None, 'C:\\PYTHON22\\lib\\compiler', ('', '', 5)) import compiler imp.find_module('ast',compiler.__path__) (, 'C:\\PYTHON22\\lib\\compiler\\ast.py', ('.py', 'r', 1))

Right. That's not exactly the API I had in mind though. You could certainly create a usable API on top of find_module. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Barry Warsaw

5:58 p.m.

On Tue, 2004-07-27 at 13:45, Jim Fulton wrote:

...

...
So, why doesn't imp.find_module() fit the bill?

It doesn't handle packages.

Sounds like a bug to me. ;) -Barry

Jim Fulton

6:02 p.m.

Barry Warsaw wrote:

...

On Tue, 2004-07-27 at 13:45, Jim Fulton wrote:

...
...
So, why doesn't imp.find_module() fit the bill?

It doesn't handle packages.

Sounds like a bug to me. ;)

It's documented: http://docs.python.org/lib/module-imp.html#l2h-688 Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Tim Peters

6:03 p.m.

[Tim]

...

...
Sure! Jim wants to distinguish "absenceness" from "brokenness" for some reason. It's not clear to my why

[Jim]

...

Because a broken module is something that should get fixed.

Maybe I don't know what "broken" means to you then. To me it means things like syntax errors, and module initialization code that raises runtime exceptions. If I try to import a module with problems like those, I don't get ImportError, I get the exception (like SyntaxError or ZereDivisionError) raised by the broken code in the module, and catching ImportError in the importer doesn't hide those. Is the only case you're concerned about of the "A imports B imports C, and B's attempt to import C raises an ImportError that's passed on to A" flavor? I have one nasty example of that on hand: Python's site.py tries to import sitecustomize.py, but if the latter contains a bogus import then site.py suppresses the error, because it treats the ImportError as meaning "I guess sitecustomize.py doesn't exist -- that's fine". That one cost me an hour yesterday! So I'm becoming a believer <wink>.

...

Hiding errors is a bad thing. It's like using hasattr. :)

No argument there.

Jim Fulton

6:07 p.m.

Tim Peters wrote:

...

[Tim]

...
...
Sure! Jim wants to distinguish "absenceness" from "brokenness" for some reason. It's not clear to my why

[Jim]

...
Because a broken module is something that should get fixed.

Maybe I don't know what "broken" means to you then. To me it means things like syntax errors, and module initialization code that raises runtime exceptions. If I try to import a module with problems like those, I don't get ImportError,

Yup ...

...

Is the only case you're concerned about of the "A imports B imports C, and B's attempt to import C raises an ImportError that's passed on to A" flavor?

Yes

...

I have one nasty example of that on hand: Python's site.py tries to import sitecustomize.py, but if the latter contains a bogus import then site.py suppresses the error, because it treats the ImportError as meaning "I guess sitecustomize.py doesn't exist -- that's fine". That one cost me an hour yesterday! So I'm becoming a believer <wink>.

:) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Guido van Rossum

8:43 p.m.

...

Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules. Wouldn't it be better to change Python to stop leaving insane module objects in sys.modules to begin with? That's harder, but seems to me it would do a lot more good for a lot more people.

+1 --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum

8:44 p.m.

...

No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present.

AFAIK, the only way to do it is by trying the import. Unfortunately, the error type alone isn't enough to tell whether the error you caught is due to the error condition you are trying to test.

Perhaps providing an API is a better way to go.

This exists as imp.find_module.

...

In general, however, I hate the approach of formatting errors as they are raised, rather than as they are converted to strings.

Agreed. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum

8:47 p.m.

...

Because a broken module is something that should get fixed.

But nine out of ten times, a module's absence is caused by breakage somewhere (e.g. a bad path). --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum

8:48 p.m.

...

Nope. It doesn't handle packages.

...
...
...
import compiler.ast import imp imp.find_module('compiler.ast') Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named compiler.ast

The docs explain how to handle that though. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum

8:50 p.m.

...

An even simpler aproach would be to have a ModuleNotFound error that extends ImportError and gets raised if the module can't be found.

That wouldn't solve anything, because if foo brokenly tries to import bar and that raises ModuleNotFound, so will "import foo". --Guido van Rossum (home page: http://www.python.org/~guido/)

Jim Fulton

9:55 p.m.

Guido van Rossum wrote:

...

...
Nope. It doesn't handle packages.

...
...
...
import compiler.ast import imp imp.find_module('compiler.ast') Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named compiler.ast

The docs explain how to handle that though.

Sure. But it would be better to have an API that you could just call to find out if a module is present. This could be implemented on top of find_module. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Barry Warsaw

10 p.m.

On Tue, 2004-07-27 at 17:55, Jim Fulton wrote:

...

Sure. But it would be better to have an API that you could just call to find out if a module is present. This could be implemented on top of find_module.

If a convenience function is desired, I'd recommend adding it to the imp module. Call it something like imp.package_or_module_exists() <wink>. -Barry

Brett C.

11:58 p.m.

Neal Norwitz wrote:

...

On Tue, Jul 27, 2004 at 12:39:22PM -0400, Jim Fulton wrote:

...
Tim Peters wrote:

...
[Jim Fulton]

...
No, it won't. For example, suppose foo imports B. B tries to import C, but fails. B is now broken, but it is still importable. Actually, both foo and B can be imported without errors, even though they are broken.

...
Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules.

No. I'm proposing a way for a Python developer to detect the presence or absence of a module.

Hm, perhaps it would be better to provide an API (if there isn't one already) to test whether a module is present.

Another possibility could be to add a ChainedImportError (or SubImportError?) which would derive from ImportError. The module which imports a broken module would get the normal ImportError. If the module doesn't catch the exception, it would get converted to this new type of ImportError.

I don't know how easy this would be to implement, but I think it would solve Jim's problem and perhaps be easier to deal with?

Having exceptions keeping track of previous exceptions might also work. I know that idea was brought up before last year (see http://www.python.org/dev/summary/2003-01-16_2003-01-31.html#idea-for-avoidi... for the historically accurate summary of the thread). Idea died when no one could come up with a good way of storing previous exception info. -Brett

Jim Fulton

28 Jul 28 Jul

10:56 a.m.

New subject: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)

Tim Peters wrote:

...

[Jim Fulton] ...

...

Then you're proposing a way for a highly knowledgable user to anticipate, and partially worm around, that Python leaves behind insane module objects in sys.modules. Wouldn't it be better to change Python to stop leaving insane module objects in sys.modules to begin with? That's harder, but seems to me it would do a lot more good for a lot more people.

I sympathize with your frustration with this problem, but I think that the problem is bigger that just sys.modules. For better or worse, importing a module may have side effects that extend beyond sys.modules. For example, In some applications, objects get registered into registries that exist in already-imported modules. Perhaps we want to declare this to be a poor style. If a module has an impact beyond new modules added to sys.modules, then removing all modules imported into sys.modules as a result of attempting the import would produce bugs even more subtle than what we have now. Do you think it's practical to limit the effects of module import to sys.modules, even by convention? Could we say that it is a bug for code executed during module import to mutate other modules, including mutating objects contained in those other modules? I would support this myself. If it is possible to limit the effects of import (even by convention), then I think it would be practical to roll-back changes to sys.modules. If it's not practical to limit the effects of module import then I think the problem is effectively unsolveable, short of making Python transactional. Personably I'm inclined to consider errors that occur while executing a module to be pretty much fatal. If a module has begun executing, I really don't know what state it's in or what state it might have left other modules in. I'd rather report the error and get some human to fix it. OTOH, I'm happy to recover from the inability to find a module as long as not module code has been executed. FWIW, In Zope, we generally generally limit non-transactional state changes to program startup. For that reason, we make little or no attempt to survive startup errors. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Jim Fulton

10:59 a.m.

Guido van Rossum wrote:

...

...
An even simpler aproach would be to have a ModuleNotFound error that extends ImportError and gets raised if the module can't be found.

That wouldn't solve anything, because if foo brokenly tries to import bar and that raises ModuleNotFound, so will "import foo".

Doh. Right. A possible way to make this work is to have an exception that is raised of a module can't be found *and* if no module code has executed. <shrug> Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Barry Warsaw

3:52 p.m.

New subject: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)

On Wed, 2004-07-28 at 06:56, Jim Fulton wrote:

...

Do you think it's practical to limit the effects of module import to sys.modules, even by convention? Could we say that it is a bug for code executed during module import to mutate other modules, including mutating objects contained in those other modules?

It's fairly common practice to execute code at module scope, as a side-effect of importation. I've always been uncomfortable with that idiom for many of the reasons you point out, but it is a darn convenient place to do such things. I wonder if we can't define a protocol for executing code at module scope, but only if the module is successfully imported. The idea is that we'd define something like an __init__() for modules that would only get executed after the module was imported, and if there were any failures in importing, it wouldn't get called. We'd have to handle chained imports though. I haven't thought through all the implications of that but I can imagine it will get messy. You probably don't want to execute any __init__()'s until all imports are finished, you probably want to execute them in the order you saw them, and you don't want any exceptions raised in __init__()'s to be deemed an ImportError worthy of sys.modules deletion. Maybe this isn't feasible, but I throw it out there as a thought. -Barry

Guido van Rossum

4:53 p.m.

[Jim]

...

A possible way to make this work is to have an exception that is raised of a module can't be found *and* if no module code has executed. <shrug>

Huh? That would only make a difference if the *first* import in a module failed.

...

From: Barry Warsaw On Wed, 2004-07-28 at 06:56, Jim Fulton wrote:

...
Do you think it's practical to limit the effects of module import to sys.modules, even by convention? Could we say that it is a bug for code executed during module import to mutate other modules, including mutating objects contained in those other modules?

I'm uncomfortable with that; while it's indeed conventional for modules to mostly define classes and functions (and constants and global variables and ...), there are many other useful things one could do at module level, and I hesitate to declare those in any way, shape or sense frowned upon. You can have stricter rules within a project, of course, but I don't want to make this part of the general Python culture; your concerns are mostly relevant only for very large projects. [Barry again]

...

It's fairly common practice to execute code at module scope, as a side-effect of importation. I've always been uncomfortable with that idiom for many of the reasons you point out, but it is a darn convenient place to do such things.

Um, importing a module is executing code. Defining a class or method is executing code. Defining a module-global (or class) constant is executing code. Initializing a module-global variable. Etc. Where do you draw the line?

...

I wonder if we can't define a protocol for executing code at module scope, but only if the module is successfully imported. The idea is that we'd define something like an __init__() for modules that would only get executed after the module was imported, and if there were any failures in importing, it wouldn't get called.

I don't think this can reasonably done. It would have to contain everything that's currently at module scope, which would defeat the purpose.

...

We'd have to handle chained imports though. I haven't thought through all the implications of that but I can imagine it will get messy. You probably don't want to execute any __init__()'s until all imports are finished, you probably want to execute them in the order you saw them, and you don't want any exceptions raised in __init__()'s to be deemed an ImportError worthy of sys.modules deletion.

Maybe this isn't feasible, but I throw it out there as a thought.

Let me throw it out again. You'll agree after you've had your coffee. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

Barry Warsaw

5:26 p.m.

On Wed, 2004-07-28 at 12:53, Guido van Rossum wrote:

...

Um, importing a module is executing code.

Oh sure! (I didn't mean it in that sense ;).

...

Defining a class or method is executing code. Defining a module-global (or class) constant is executing code. Initializing a module-global variable. Etc. Where do you draw the line?

Actually, I wouldn't draw the line implicitly, but instead explicitly, but...

...

I don't think this can reasonably done. It would have to contain everything that's currently at module scope, which would defeat the purpose.

yah, you're probably right. Bleh. -Barry

Phillip J. Eby

5:40 p.m.

At 09:53 AM 7/28/04 -0700, Guido van Rossum wrote:

...

...
On Wed, 2004-07-28 at 06:56, Jim Fulton wrote:

...
Do you think it's practical to limit the effects of module import to sys.modules, even by convention? Could we say that it is a bug for code executed during module import to mutate other modules, including mutating objects contained in those other modules?

I'm uncomfortable with that; while it's indeed conventional for modules to mostly define classes and functions (and constants and global variables and ...), there are many other useful things one could do at module level, and I hesitate to declare those in any way, shape or sense frowned upon. You can have stricter rules within a project, of course, but I don't want to make this part of the general Python culture; your concerns are mostly relevant only for very large projects.

And for those very large projects, I'm wondering if what is *really* wanted is actually to be able to isolate plug-ins or extensions into their own "module space", similar to the way the Java "classloader" hierarchy works. In effect, Java can have multiple 'sys' modules, arranged in a hierarchy. Each classloader has its own sys.path equivalent, and can also delegate imports to its parent classloader(s), in order to have some modules shared between child classloaders. Classloaders are typically used in application servers or other systems that wrap around multiple third-party application modules, so that each application unit appears to have the entire VM to itself, even though side-by-side there may be other applications seeing the same or different versions of various modules. This effect is achieved by creating a classloader for each segregated application unit, and then using it to load that application's main class. If a mechanism like this were available for Python, then large systems like Zope could simply (well, maybe *simply* is the wrong word!) load extensions in a separate "module space". If there's a problem, just throw the whole thing away, with no side effects in the current module space such as "insane" half-empty modules. As a side benefit, individual extensions can depend on/use different versions of common packages, as long as they get loaded into separate module spaces. Heck, you can give extensions their own private copy of the 'logging' module, or anything else that has module-level configuration and doesn't "play nice" when multiple extensions want it configured differently. I don't know for certain that such a thing is possible using just the standard import hooks, but it seems to me it might be. If anybody is interested in such an animal, have a look at: http://www.eby-sarna.com/pipermail/peak/2004-July/001652.html and drop me a line. I presume this is offtopic for further discussion on python-dev unless it's proposed as a stdlib addition, and I don't think it should be so proposed without first having some field experience in at least a couple of those "very large projects". My personal interest in this is mainly aimed at integrating Python with Eclipse, but it seems to me that systems like Zope and Chandler that want to support large numbers of third-party plugins, probably want to be able to do things like this. It would be nice to have a common implementation, or failing that, at least understand what prevents a common implementation from being practical.

Guido van Rossum

8:44 p.m.

...

And for those very large projects, I'm wondering if what is *really* wanted is actually to be able to isolate plug-ins or extensions into their own "module space", similar to the way the Java "classloader" hierarchy works. [...]

You can certainly build all that using import hooks. But perhaps that's too low a level, and a higher-level infrastructure would be helpful? Import hooks have been around for ages and many improvements have been proposed but few of those have actually found much use (or even found to be real improvements). So perhaps some interested parties (count me out for lack of time :-( ) should get together and design something, learning from Java? --Guido van Rossum (home page: http://www.python.org/~guido/)

Phillip J. Eby

9:14 p.m.

At 01:44 PM 7/28/04 -0700, Guido van Rossum wrote:

...

...
And for those very large projects, I'm wondering if what is *really* wanted is actually to be able to isolate plug-ins or extensions into their own "module space", similar to the way the Java "classloader" hierarchy works. [...]

You can certainly build all that using import hooks. But perhaps that's too low a level, and a higher-level infrastructure would be helpful? Import hooks have been around for ages and many improvements have been proposed but few of those have actually found much use (or even found to be real improvements).

Sorry, I'm confused. By "higher-level infrastructure", are you meaning a way to co-ordinate import hooks, or something higher-level than the "module spaces" concept? As far as I can tell, a properly implemented module space should allow arbitrary import hooks to be used within that module space, so long as the hooks obtain their copy of 'sys' using the previous value of '__import__'. For example, if you create a hook using 'import ihooks' within a given module space, then that space's copy of 'ihooks' will see the space's copy of 'sys', so it will thus do all its magic using the right 'sys'. The weak spot is 'imp', and any other C-coded modules that access the "real" 'sys' module directly (e.g. '__builtin__'). These modules would have to be replaced by "safe" versions. E.g., a 'safe_imp' in Python that explicitly passed in a path to the "real" imp functions, and so on. Finding all of the modules that do this and providing "safe" replacements (while making sure that truly global "sys" attributes are shared) would probably be the biggest task in creating a Java-like import framework. (Oh... and another infrastructure-level issue: Python classes know their __module__ name, but in a system using module spaces, the module name alone is not sufficient to identify the module where the class originated. In Java, a class is uniquely identified by the combination of its fully qualified name, and the classloader used to load it. This is not an issue for Python functions (which reference the module globals) or for modules themselves, which both indirectly refer to the loader used to load the module. But it could be an issue for classes.)

Guido van Rossum

10:13 p.m.

[Guido]

...

...
You can certainly build all that using import hooks. But perhaps that's too low a level, and a higher-level infrastructure would be helpful? Import hooks have been around for ages and many improvements have been proposed but few of those have actually found much use (or even found to be real improvements).

[Phillip]

...

Sorry, I'm confused. By "higher-level infrastructure", are you meaning a way to co-ordinate import hooks, or something higher-level than the "module spaces" concept?

I meant whatever you want. I am not familiar with the need for this functionality, all I know is import hooks and you were clear that those are too low-level.

...

As far as I can tell, a properly implemented module space should allow arbitrary import hooks to be used within that module space, so long as the hooks obtain their copy of 'sys' using the previous value of '__import__'. For example, if you create a hook using 'import ihooks' within a given module space, then that space's copy of 'ihooks' will see the space's copy of 'sys', so it will thus do all its magic using the right 'sys'.

Today's import hooks aren't written with recursive import hooks in mind.

...

The weak spot is 'imp', and any other C-coded modules that access the "real" 'sys' module directly (e.g. '__builtin__'). These modules would have to be replaced by "safe" versions. E.g., a 'safe_imp' in Python that explicitly passed in a path to the "real" imp functions, and so on. Finding all of the modules that do this and providing "safe" replacements (while making sure that truly global "sys" attributes are shared) would probably be the biggest task in creating a Java-like import framework.

Sure. Feel free to build it and propose it as a standard library addition, if you think it will be useful for others. I might even find a use for it -- but right now I'm +0 because I really don't have a need for this stuff in anything I'm doing.

...

(Oh... and another infrastructure-level issue: Python classes know their __module__ name, but in a system using module spaces, the module name alone is not sufficient to identify the module where the class originated. In Java, a class is uniquely identified by the combination of its fully qualified name, and the classloader used to load it. This is not an issue for Python functions (which reference the module globals) or for modules themselves, which both indirectly refer to the loader used to load the module. But it could be an issue for classes.)

But who ever uses __module__ for anything else than printing it? --Guido van Rossum (home page: http://www.python.org/~guido/)

Bob Ippolito

10:40 p.m.

On Jul 28, 2004, at 6:13 PM, Guido van Rossum wrote:

...

...
(Oh... and another infrastructure-level issue: Python classes know their __module__ name, but in a system using module spaces, the module name alone is not sufficient to identify the module where the class originated. In Java, a class is uniquely identified by the combination of its fully qualified name, and the classloader used to load it. This is not an issue for Python functions (which reference the module globals) or for modules themselves, which both indirectly refer to the loader used to load the module. But it could be an issue for classes.)

But who ever uses __module__ for anything else than printing it?

I've used sys.modules[__module__] to get a reference to the current module object. Mostly for running doctests I guess, but I've probably used it in other places. -bob

Phillip J. Eby

10:42 p.m.

At 03:13 PM 7/28/04 -0700, Guido van Rossum wrote:

...

But who ever uses __module__ for anything else than printing it?

Well, I've done some rather esoteric things with it in the past, but nothing that would be needed here. I was just thinking that applications like Zope, Chandler et al would sometimes want to know, "what plugin is this class/object from?" But I suppose that they could just as easily build that concept explicitly into their APIs. So never mind that bit. :)

Fred L. Drake, Jr.

29 Jul 29 Jul

3:44 a.m.

On Wednesday 28 July 2004 06:40 pm, Bob Ippolito wrote:

...

I've used sys.modules[__module__] to get a reference to the current module object. Mostly for running doctests I guess, but I've probably used it in other places.

Did you mean __name__? __module__ is an attribute of classes that provides the __name__ of the defining module. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

Jim Fulton

9:02 p.m.

Guido van Rossum wrote:

...

[Guido]

...

But who ever uses __module__ for anything else than printing it?

It's crucial for pickling. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Phillip J. Eby

9:26 p.m.

At 05:02 PM 7/29/04 -0400, Jim Fulton wrote:

...

Guido van Rossum wrote:

...
But who ever uses __module__ for anything else than printing it?

It's crucial for pickling.

Interesting point. That means that cPickle wouldn't work in a "module space" without modification. The pure-Python pickle module would unpickle into the current module space.

Greg Ewing

30 Jul 30 Jul

2:40 a.m.

Jim Fulton :

...

try: import foo except ImportError: # Configure for absense of foo ... else: # Configure for presense of foo ...

Unfortunately, this is a bug trap.

Maybe there's room for another creative use of 'else' here... import: foo else: # Configure for absense of foo Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Jim Fulton

12:28 p.m.

Phillip J. Eby wrote:

...

At 05:02 PM 7/29/04 -0400, Jim Fulton wrote:

...
Guido van Rossum wrote:

...
But who ever uses __module__ for anything else than printing it?

It's crucial for pickling.

Interesting point. That means that cPickle wouldn't work in a "module space" without modification. The pure-Python pickle module would unpickle into the current module space.

And that might be the wrong module space. If one was going to introduce a concept of module spaces, one would need to extend the pickling machinery as well. I'll note that Python already has module spaces. They're called "packages". :) The problem with Python's current package system is that it's not possible, in general, to write packages that can be moved around within the package system, because relative imports aren't reobust or flexible enough. PEP 328 would fix this. PEP 328 would allow creation of packages that worked regardless of where there were places or that worked relative to a containing package regardless of where that was placed. You could then have different module spaces expressed by adding a new layer of top-level modules. You wouldn't need any new machinery beyond PEP 328. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Phillip J. Eby

3:06 p.m.

At 08:28 AM 7/30/04 -0400, Jim Fulton wrote:

...

The problem with Python's current package system is that it's not possible, in general, to write packages that can be moved around within the package system, because relative imports aren't reobust or flexible enough. PEP 328 would fix this. PEP 328 would allow creation of packages that worked regardless of where there were places or that worked relative to a containing package regardless of where that was placed. You could then have different module spaces expressed by adding a new layer of top-level modules. You wouldn't need any new machinery beyond PEP 328.

Hm. The concept I'm proposing would allow code to be used in a module space without needing to know about module spaces or be written to co-operate in such a fashion. If you can make everybody conform to some sort of coding standard, you wouldn't even need PEP 328 to accomplish your goal. ;) As a counterexample, consider seven Zope products that depend on two different versions of PIL. Using PEP 328, you can't make this work without either loading seven copies of PIL, or else requiring the version number to be part of the package name. You actually have it somewhat better right now because you can at least fool the "this package or absolute" mechanism into doing what you want. If I understand correctly, PEP 328 will ultimately remove the possibility of doing it that way.

Tim Peters

31 Jul 31 Jul

4:17 a.m.

New subject: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)

[Tim, wants to keep insane modules out of sys.modules] [Jim Fulton]

...

I sympathize with your frustration with this problem, but I think that the problem is bigger that just sys.modules. For better or worse, importing a module may have side effects that extend beyond sys.modules. For example, In some applications, objects get registered into registries that exist in already-imported modules. Perhaps we want to declare this to be a poor style. If a module has an impact beyond new modules added to sys.modules, then removing all modules imported into sys.modules as a result of attempting the import would produce bugs even more subtle than what we have now.

I wouldn't want to remove all, just the modules that failed. For example, A imports B B imports C # no problem B imports D # and that raises an exception not caught by B C is fine, I only want to nuke D and B. As to style, in my own code I strive to make modules "reload safe". So, for example, I wouldn't even consider doing one of these things as a side effect of merely importing a module: + Create a lock file. + Start a thread. + Open a socket. + Register with a registry indeed. Now that said, I've only seen imports wrapped in a try in two ways: 1. try: import X except ImportError: something That's invariably trying to check for the availability of X, though, not also trying to check for whether something X imports doesn't exist. If you pursue a saner way to write that, I'll always use it. 2. try: import X except: something That one is almost always a mistake, as a bare "except" is almost always a mistake in any context. The author almost always intended the same thing as #1, but was too lazy or inexperienced to write that. Bug in, bugs out. That a later attempt to import X doesn't also fail is a bug magnifier. I've never seen something like 3. try: import X except ZeroDivisionError: something If, as I suspect, nobody (and "almost nobody" is the same to me <wink>) *intends* to catch an error from an import other than ImportError, then import errors other than ImportError are fatal soon after in practice, and then there's nothing much to worry about. Catching ImportError still leaves insane modules around, though, and that does cause real problems. You've convinced me I'd rather have a better way to spell "does X exist?" than catching an ImportError from an attempt to import X.

...

Do you think it's practical to limit the effects of module import to sys.modules, even by convention?

I'm sure you didn't intend that to be *so* extreme -- like surely a module is allowed to initialize its own module-level variables. If I read "effects" as "effects visible outside the module", then that's what you said <wink>.

...

Could we say that it is a bug for code executed during module import to mutate other modules, including mutating objects contained in those other modules? I would support this myself.

It's hard to spell the intent precisely. "reload safe" covers a world of non-local (wrt the module in question) that both are and aren't problematic. For example, calling random.random() during module initialization should be fine, but it certainly mutates state, and irrevocably so, inside the random module. Because it's hard to be precise here, best practice is likely to remain more a matter of good judgment than of legislation.

...

If it is possible to limit the effects of import (even by convention), then I think it would be practical to roll-back changes to sys.modules. If it's not practical to limit the effects of module import then I think the problem is effectively unsolveable, short of making Python transactional.

There we don't agree -- I think it's already practical, based on that virtually no Python application *intends* to catch errors from imports other than ImportError, so that almost all "real bugs" in module initialization are intended to stop execution. In turn, in the cases where ImportErrors are intentionally caught now, they generally occur in "import blocks" near the starts of all modules in the failing import chain, and so none of the modules involved have yet *done* any non-trivial initialization -- they're all still trying to import the stuff they need to *start* doing the meat of their initialization. If some modules happen to import successfully along the way, fine, they should stay in sys.modules, and then importing them again later won't run their initialization code again. IOW, once a module has announced its sanity by importing successfully, I want that to "stick" no matter what happens later.

...

Personably I'm inclined to consider errors that occur while executing a module to be pretty much fatal. If a module has begun executing, I really don't know what state it's in or what state it might have left other modules in. I'd rather report the error and get some human to fix it.

I think that's widespread belief too. Heck, if Zope doesn't violate it, who else would be so perverse <wink>?

...

OTOH, I'm happy to recover from the inability to find a module as long as no module code has been executed.

Having a clearer way to determine module availability/existence would be a real help.

...

FWIW, In Zope, we generally generally limit non-transactional state changes to program startup. For that reason, we make little or no attempt to survive startup errors.

I've never tried to survive a startup error myself either, nor have any Python projects I'm aware of attached to any of my previous employers. Anyone else?

Jim Fulton

2 Aug 2 Aug

2:01 p.m.

Phillip J. Eby wrote:

...

At 08:28 AM 7/30/04 -0400, Jim Fulton wrote:

...
The problem with Python's current package system is that it's not possible, in general, to write packages that can be moved around within the package system, because relative imports aren't reobust or flexible enough. PEP 328 would fix this. PEP 328 would allow creation of packages that worked regardless of where there were places or that worked relative to a containing package regardless of where that was placed. You could then have different module spaces expressed by adding a new layer of top-level modules. You wouldn't need any new machinery beyond PEP 328.

Hm. The concept I'm proposing would allow code to be used in a module space without needing to know about module spaces or be written to co-operate in such a fashion. If you can make everybody conform to some sort of coding standard, you wouldn't even need PEP 328 to accomplish your goal. ;)

I don't think that's true. If your package has sub-packages, there's no way for a subackage to import from the containing package without using an absolute import. Similarly, a package currentlly can't import from a sibling package without using an absolute import. With PEP 328, relative imports from parent or sibling (or cousin ...) packages will be possible.

...

As a counterexample, consider seven Zope products that depend on two different versions of PIL. Using PEP 328, you can't make this work without either loading seven copies of PIL, or else requiring the version number to be part of the package name.

No, I don't think this is right. PEP 328 should allow you to create module spaces using container packages. For your example, we create two top-level packages, space1 and space 2. You put version x of PIL in space1. That package's absolute name is space1.PIL. You put version y of PIL in space2, creating space2.PIL. Now you put the products that depend on version x of PIL in space1. You put the products that depend on version y of PIL on space 2. The products mist use relative imports to import from PIL: from ..PIL import somePILmodule For this to work, PIL also has to use relative imports to import it's own modules.

...

You actually have it somewhat better right now because you can at least fool the "this package or absolute" mechanism into doing what you want.

I assume you are fering to letting local modules hide global ones. This only works if the modules are included directly in the package. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Jim Fulton

2:30 p.m.

New subject: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)

Tim Peters wrote:

...

[Tim, wants to keep insane modules out of sys.modules]

[Jim Fulton]

...

...
Do you think it's practical to limit the effects of module import to sys.modules, even by convention?

I'm sure you didn't intend that to be *so* extreme -- like surely a module is allowed to initialize its own module-level variables. If I read "effects" as "effects visible outside the module", then that's what you said <wink>.

Yup. :)

...

...
If it is possible to limit the effects of import (even by convention), then I think it would be practical to roll-back changes to sys.modules. If it's not practical to limit the effects of module import then I think the problem is effectively unsolveable, short of making Python transactional.

There we don't agree -- I think it's already practical, based on that virtually no Python application *intends* to catch errors from imports other than ImportError, so that almost all "real bugs" in module initialization are intended to stop execution. In turn, in the cases where ImportErrors are intentionally caught now, they generally occur in "import blocks" near the starts of all modules in the failing import chain, and so none of the modules involved have yet *done* any non-trivial initialization -- they're all still trying to import the stuff they need to *start* doing the meat of their initialization.

Except in cases where imports are places later in a module to make circular imports work. It would be nice to disallow circular imports, although I don't know if that's possible. (Some time, when I have time, I'd like to see if we can get rid of them in Zope 3. I'd like to have a tool to report circular imports, to keep them from creeping in, which they do so easily.) Having said that, you make a good point. If all modules did their imports before making any changes outside the modules, then it would be pretty safe to clean up the imports.

...

If some modules happen to import successfully along the way, fine, they should stay in sys.modules, and then importing them again later won't run their initialization code again.

This only works if all of the modules do all of their imports before doing other work. If there are circular imports, you could have: A defines class C A imports B B imports C from A A imports D and gets an import error If we clean up A and D, then B has a class from a module that doesn't exist. Hm. Suppose we have: A imports B B imports A A imports D and gets an import error We clean up A and D. What about the A that was imported into B? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Tim Peters

4:42 p.m.

New subject: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)

[Tim Peters]

...

...
...I think it's already practical, based on that virtually no Python application *intends* to catch errors from imports other than ImportError, so that almost all "real bugs" in module initialization are intended to stop execution. In turn, in the cases where ImportErrors are intentionally caught now, they generally occur in "import blocks" near the starts of all modules in the failing import chain, and so none of the modules involved have yet *done* any non-trivial initialization -- they're all still trying to import the stuff they need to *start* doing the meat of their initialization.

[Jim Fulton]

...

Except in cases where imports are places later in a module to make circular imports work.

That's being discussed in "the other" thread (the now-massive "Fix import errors to have data" thread). I never do circular imports intentionally, so my interest in that is unmeasurable -- on the low end <wink>. Note that I checked in changes last night so that a failing import in 2.4 *will* delete the failing importee from sys.modules. This breaks a couple Zope3 tests, BTW (explained in the other thread).

...

It would be nice to disallow circular imports, although I don't know if that's possible.

I doubt it, because it would break existing code "gratuitously". Maybe in Python 3. IMO the behavior of circular imports in CPython today is about half implementation accident.

...

(Some time, when I have time, I'd like to see if we can get rid of them in Zope 3.

...

I'd like to have a tool to report circular imports, to keep them from creeping in, which they do so easily.)

Wasn't Fred working on such a tool?

...

Having said that, you make a good point. If all modules did their imports before making any changes outside the modules, then it would be pretty safe to clean up the imports.

I think most code already does.

...

...
If some modules happen to import successfully along the way, fine, they should stay in sys.modules, and then importing them again later won't run their initialization code again.

...

This only works if all of the modules do all of their imports before doing other work.

It works in many more cases than just that. do a ton of work import A do more work blow up A is almost certainly still fine in that case. I understand we can concoct pathological cases in which it isn't.

...

If there are circular imports, you could have:

A defines class C A imports B B imports C from A A imports D and gets an import error

If we clean up A and D, then B has a class from a module that doesn't exist.

Yes, and that's unpleasant. If C defines any methods, A.__dict__ will still be alive (because methods of C refer to it via their func_globals), but when the module object itself goes away its __dict__ values get set to None. So any method of C that refers to a module global is likely to suffer a mysterious None-related exception. It's not going to segfault, though. OTOH, to get into trouble, someone has to suppress the ImportError raised by trying to import A. You have to work pretty hard to get in trouble.

...

Hm. Suppose we have:

A imports B B imports A A imports D and gets an import error

We clean up A and D. What about the A that was imported into B?

I believe that works the same as before the patch, except that 'A' and 'D' are no longer in sys.modules. The only "clean up" done by the patch is, in case of error, to remove the module name from sys.modules. That removes a reference to the module object; it doesn't do anything more than that directly. The module object in this case remains alive, and its __dict__ remains intact, because B holds a reference to the module object. If someone suppresses the ImportError from importing A, then, then the primary difference is that *another* attempt to import A will again raise ImportError now. Over in the other thread, people are talking about schemes to do better in circular import cases that suffer errors. My patch ignored those issues (beyond convincing myself segfaults wouldn't result), because Guido's patch I built on ignored them <wink>. That was enough to allow removing some atrocious code in the Python libraries trying to worm around damaged modules left behind in sys.modules, and appears so far to be enough to allow "regrtest.py -r" (randomize test order) to pass on Windows reliably for the first time in Python's history. So I think it's a real improvement, despite that it doesn't solve all import-error problems.

Phillip J. Eby

4:57 p.m.

At 10:01 AM 8/2/04 -0400, Jim Fulton wrote:

...

Phillip J. Eby wrote:

...
At 08:28 AM 7/30/04 -0400, Jim Fulton wrote:

...
The problem with Python's current package system is that it's not possible, in general, to write packages that can be moved around within the package system, because relative imports aren't reobust or flexible enough. PEP 328 would fix this. PEP 328 would allow creation of packages that worked regardless of where there were places or that worked relative to a containing package regardless of where that was placed. You could then have different module spaces expressed by adding a new layer of top-level modules. You wouldn't need any new machinery beyond PEP 328.

Hm. The concept I'm proposing would allow code to be used in a module space without needing to know about module spaces or be written to co-operate in such a fashion. If you can make everybody conform to some sort of coding standard, you wouldn't even need PEP 328 to accomplish your goal. ;)

No, I don't think this is right. PEP 328 should allow you to create module spaces using container packages. For your example, we create two top-level packages, space1 and space 2. You put version x of PIL in space1. That package's absolute name is space1.PIL. You put version y of PIL in space2, creating space2.PIL. Now you put the products that depend on version x of PIL in space1. You put the products that depend on version y of PIL on space 2. The products mist use relative imports to import from PIL:

from ..PIL import somePILmodule

For this to work, PIL also has to use relative imports to import it's own modules.

As I said, if you get to make PIL and everybody else rewrite their imports, you can do whatever you want to already. :) The point of my proposal is to make it possible *without* rewriting imports, in versions of Python from 2.2 up.

Jim Fulton

5:32 p.m.

New subject: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)

Tim Peters wrote:

...

[Tim Peters]

...

...
I'd like to have a tool to report circular imports, to keep them from creeping in, which they do so easily.)

Wasn't Fred working on such a tool?

He did, but I can't make sense of the output, so I can't use it. :) I'm hoping that soon Fred and I can spend some time to make sense of the output. ... Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Jim Fulton

4 Aug 4 Aug

1:53 p.m.

Phillip J. Eby wrote:

...

At 10:01 AM 8/2/04 -0400, Jim Fulton wrote:

...

The point of my proposal is to make it possible *without* rewriting imports, in versions of Python from 2.2 up.

Your proposal would, at least, require canges to the pickle framework. I imagine it would affect other serialization and persistence frameworks as well. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Phillip J. Eby

4:16 p.m.

At 09:53 AM 8/4/04 -0400, Jim Fulton wrote:

...

Phillip J. Eby wrote:

...
At 10:01 AM 8/2/04 -0400, Jim Fulton wrote:

..

...
The point of my proposal is to make it possible *without* rewriting imports, in versions of Python from 2.2 up.

Your proposal would, at least, require canges to the pickle framework. I imagine it would affect other serialization and persistence frameworks as well.

Actually, in the updated interface I've been discussing off-list with a few people, all that would happen is that you'd need to be explicit about what interpreter you use to do the unpickling, if you don't want it to be the current one. E.g.: result = someInterpreter.call(somePickler.loads, data) In a framework like ZODB, however, I would assume that if supporting multiple interpreters were desirable, one would change the persistent object ID to include a reference to the {bundle|parcel|product} that originated the object. In any case, the implementation we're discussing (based on the C multiple interpreter API) should be completely transparent to end user code, to the extent that it should be possible to run two different versions of Zope side by side in the same process -- and that will probably make a nice "stress test" for the implementation when we're done with it. :) I hope that the end-product of our work will be an implementation of a Python-accessible API for interpreter management, and a PEP covering both that API and how to fix the most problematic aspects of the Python C API for multiple interpreters. (Specifically, additions to the API to allow extensions to manage selected static data as interpreter-specific rather than global, and for ensuring that imports stay interpreter-local when execution crosses interpreter boundaries.) I expect the fixes will be able to be used with versions of Python back to 2.2, which was the version that introduced the necessary base APIs for walking the interpreter and thread state chains.

7198

Age (days ago)

7206

Last active (days ago)

List overview

Download

57 comments

13 participants

participants (13)

Barry Warsaw
Bob Ippolito
Brett C.
Casey Duncan
Charles Cazabon
Fred L. Drake, Jr.
Greg Ewing
Guido van Rossum
Jim Fulton
Neal Norwitz
Phillip J. Eby
Thomas Heller
Tim Peters

Fix import errors to have data

Neal Norwitz

Charles Cazabon

tags

participants (13)