Project organization and import

[Russell E. Owen]

In article <1173054099.721593.201870 at h3g2000cwc.googlegroups.com>,
 "Martin Unsal" <martinunsal at gmail.com> wrote:

> I'm using Python for what is becoming a sizeable project and I'm
> already running into problems organizing code and importing packages.
> I feel like the Python package system, in particular the isomorphism
> between filesystem and namespace, doesn't seem very well suited for
> big projects. However, I might not really understand the Pythonic way.
> I'm not sure if I have a specific question here, just a general plea
> for advice.
> 
> 1) Namespace. Python wants my namespace heirarchy to match my
> filesystem heirarchy. I find that a well organized filesystem
> heirarchy for a nontrivial project will be totally unwieldy as a
> namespace. I'm either forced to use long namespace prefixes, or I'm
> forced to use "from foo import *" and __all__, which has its own set
> of problems.

> 1a) Module/class collision. I like to use the primary class in a file
> as the name of the file. However this can lead to namespace collisions
> between the module name and the class name. Also it means that I'm
> going to be stuck with the odious and wasteful syntax foo.foo
> everywhere, or forced to use "from foo import *".

The issue of module names vs contained class names is one thing I find a 
bit frustrating about python. Fortunately it is fairly easy to work 
around.

My own solution has been to import up just one level. So for example:
pkg/subpkg/foo.py defines class foo and associated stuff
pkg/subpkg/bar.py defines class bar
pkt/subpkg/__init__.py contains:

from foo import *
from bar import *

To use this I then do:
import pkg.subpkg
myfoo = pkg.subpkg.foo(...)

But that's the only "from x import" that I do. I never raise stuff from 
a sub-package to a higher level.

Once you do this (or in some other way eliminate the foo.foo problem), I 
think you will find that python namespaces work very well for large 
projects.

Overall I personally like having the namespace follow the file structure 
(given that one has to use source files in the first place; my smalltalk 
roots are showing). Java reportedly does much the same thing and it is 
very helpful for finding code.

I'm sure it's partly what you're used to that counts. C++ experts 
probably enjoy the freedom of C++ namespaces, but to me it's just a pain 
that they are totally independent of file structure.

> 1b) The Pythonic way seems to be to put more stuff in one file, but I
> believe this is categorically the wrong thing to do in large projects.
> The moment you have more than one developer along with a revision
> control system, you're going to want files to contain the smallest
> practical functional blocks. I feel pretty confident saying that "put
> more stuff in one file" is the wrong answer, even if it is the
> Pythonic answer.

I don't personally find that python encourages lots of code per file. I 
think this perception only stems from (1a) and once you solve that 
you'll find it's fine to divide your code into small files.

> 2) Importing and reloading. I want to be able to reload changes
> without exiting the interpreter. This pretty much excludes "from foo
> import *", unless you resort to this sort of hack:
> 
> http://www.python.org/search/hypermail/python-1993/0448.html
> 
> Has anyone found a systematic way to solve the problem of reloading in
> an interactive interpreter when using "from foo import *"?

I totally agree here. This is a real weakness to python and makes it 
feel much more static than it ought to be. I know of no solution other 
than restarting. That tends to be fast, but it can be a pain to get back 
to where you were.

Smalltalk solved this problem long ago in a way that makes for very 
dynamic development and debugging. Unfortunately few languages have 
followed suit. The Smalltalk development environment is the one feature 
I really miss in all other languages I've used (I certainly don't miss 
its quirky syntax for control flow :)).

-- Russell