package import dangers

[Steven D'Aprano]

On Tue, 06 Oct 2009 17:01:41 -0700, Carl Banks wrote:

>> Why would a module need to import itself? Surely that's a very rare
>> occurrence -- I think I've used it twice, in 12 years or so. I don't
>> see why you need to disparage the idea of combining modules and scripts
>> in the one file because of one subtle gotcha.
> 
> I'm sorry, this can't reasonably be characterized as a "subtle gotcha". 
> I totally disagree, it's not a gotcha but a major time- killing
> head-scratcher, and it's too thoroughly convoluted to be called subtle
> (subtle is like one tricky detail that messes up an otherwise clean
> design, whereas this is like a dozen tricky details the mess the whole
> thing up).

Even if that were true, it's still rare for a module to import itself. If 
a major head-scratcher only bites you one time in a hundred combination 
module+scripts, that's hardly a reason to say don't write combos. It's a 
reason to not have scripts that import themselves, or a reason to learn 
how Python behaves in this case.

But I dispute it's a head-scratcher. You just need to think a bit about 
what's going on. (See below.)


> It's easily the most confusing thing commonly encountered in Python.

But it's not commonly encountered at all, in my opinion. I see no 
evidence for it being common.

I'll admit it might be surprising the first time you see it, but if you 
give it any thought it shouldn't be: when you run a module, you haven't 
imported it. Therefore it hasn't gone through the general import 
machinery. The import machinery needs to execute the code in a module, 
and it can't know that the module is already running. Therefore you get 
two independent executions of the code, which means the class accessible 
via the running code and the class accessible via the imported code will 
be different objects.

Fundamentally, it's no more mysterious than this:


>>> def factory():
...     class K:
...         pass
...     return K
...
>>> factory() is factory()
False



> I've seen experts struggle to grasp the details.

Perhaps they're trying to hard and ignoring the simple things:

$ cat test.py
class Foo(object):
    pass

if __name__ == "__main__":
    import test
    print Foo
    print test.Foo

$ python test.py
<class '__main__.Foo'>
<class 'test.Foo'>

All you have to do is look at the repr() of the class, and the answer is 
right there in your face.

Still too hard to grasp? Then make it really simple:

$ cat test2.py
print "hello"
if __name__ == "__main__":
    import test2
$ python test2.py
hello
hello


I don't see how it could be more obvious what's going on. You run the 
script, and the print line is executed. Then the script tries to import a 
module (which just happens to be the same script running). Since the 
module hasn't gone through the import machinery yet, it gets loaded, and 
executed.

Simple and straight-forward and not difficult at all.



> Newbies and intermediate programmers should be advised never to do it,
> use a file as either a script or a module, not both.

There's nothing wrong with having modules be runnable as scripts. There 
are at least 93 modules in the std library that do it (as of version 
2.5). It's a basic Pythonic technique that is ideal for simple scripts.

Of course, once you have a script complicated enough that it needs to be 
broken up into multiple modules, you run into all sorts of complications, 
including circular imports. A major command line app might need hundreds 
of lines just dealing with the UI. It's fundamentally good advice to 
split the UI (the front end, the script) away from the backend (the 
modules) once you reach that level of complexity. Your earlier suggestion 
of having a single executable script to act as a front end for your 
multiple modules and packages is a good idea. But that's because of the 
difficulty of managing complicated applications, not because there's 
something fundamentally wrong with having an importable module also be 
runnable from the command line.



-- 
Steven