[Python-Dev] Re: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)

[Tim Peters]

[Tim Peters]
>> ...I think it's already practical, based on that
>> virtually no Python application *intends* to catch errors from imports
>> other than ImportError, so that almost all "real bugs" in module
>> initialization are intended to stop execution.  In turn, in the cases
>> where ImportErrors are intentionally caught now, they generally occur
>> in "import blocks" near the starts of all modules in the failing
>> import chain, and so none of the modules involved have yet *done* any
>> non-trivial initialization -- they're all still trying to import the
>> stuff they need to *start* doing the meat of their initialization.

[Jim Fulton]
> Except in cases where imports are places later in a module to make
> circular imports work.

That's being discussed in "the other" thread (the now-massive "Fix
import errors to have data" thread).  I never do circular imports
intentionally, so my interest in that is unmeasurable -- on the low
end <wink>.

Note that I checked in changes last night so that a failing import in
2.4 *will* delete the failing importee from sys.modules.  This breaks
a couple Zope3 tests, BTW (explained in the other thread).

> It would be nice to disallow circular imports, although I don't know if that's
> possible.

I doubt it, because it would break existing code "gratuitously". 
Maybe in Python 3.  IMO the behavior of circular imports in CPython
today is about half implementation accident.

>  (Some time, when I have time, I'd like to see if we can get rid of them in Zope 3.

+1

> I'd like to have a tool to report circular imports, to keep them from creeping in,
> which they do so easily.)

Wasn't Fred working on such a tool?

> Having said that, you make a good point.  If all modules did their imports
> before making any changes outside the modules, then it would be pretty safe
> to clean up the imports.

I think most code already does.

>> If some modules happen to import successfully along the way, fine, they
>> should stay in sys.modules, and then importing them again later won't
>> run their initialization code again.

> This only works if all of the modules do all of their imports before doing other
> work.

It works in many more cases than just that.

   do a ton of work
   import A
   do more work
   blow up

A is almost certainly still fine in that case.  I understand we can
concoct pathological cases in which it isn't.

> If there are circular imports, you could have:
>
>   A defines class C
>   A imports B
>   B imports C from A
>   A imports D and gets an import error
>
> If we clean up A and D, then B has a class from a module that doesn't exist.

Yes, and that's unpleasant.  If C defines any methods, A.__dict__ will
still be alive (because methods of C refer to it via their
func_globals), but when the module object itself goes away its
__dict__ values get set to None.  So any method of C that refers to a
module global is likely to suffer a mysterious None-related exception.
 It's not going to segfault, though.

OTOH, to get into trouble, someone has to suppress the ImportError
raised by trying to import A.  You have to work pretty hard to get in
trouble.

> Hm. Suppose we have:
>
>   A imports B
>   B imports A
>   A imports D and gets an import error
> 
> We clean up A and D. What about the A that was imported into B?

I believe that works the same as before the patch, except that 'A' and
'D' are no longer in sys.modules.  The only "clean up" done by the
patch is, in case of error, to remove the module name from
sys.modules.  That removes a reference to the module object; it
doesn't do anything more than that directly.  The module object in
this case remains alive, and its __dict__ remains intact, because B
holds a reference to the module object.  If someone suppresses the
ImportError from importing A, then, then the primary difference is
that *another* attempt to import A will again raise ImportError now.

Over in the other thread, people are talking about schemes to do
better in circular import cases that suffer errors.  My patch ignored
those issues (beyond convincing myself segfaults wouldn't result),
because Guido's patch I built on ignored them <wink>.  That was enough
to allow removing some atrocious code in the Python libraries trying
to worm around damaged modules left behind in sys.modules, and appears
so far to be enough to allow "regrtest.py -r" (randomize test order)
to pass on Windows reliably for the first time in Python's history. 
So I think it's a real improvement, despite that it doesn't solve all
import-error problems.