Python packages - problems, pitfalls.

[Paul Boddie]

Prabhu Ramachandran <prabhu at aero.iitm.ernet.in> wrote in message news:<mailman.1004897886.4368.python-list at python.org>...
> 
> I have an application that is shipped as part of a package.  I'd like
> the application to run right off the main directory or when the
> package is installed.  Also lets add to the problem by saying that the
> project is under CVS and its a pain putting everything into another
> directory.  Something like this:
> 
>    pkg_root/
> 	__init__.py
> 	app.py
> 	# app is the application that is not part of the package

There's something about this which intrigues me, too, but more on that
below.

> 	common_module.py
> 	setup.py
> 	common/
> 		__init__.py 
> 		foo.py  # more common stuff
> 	sub/
> 		__init__.py
> 		submodule.py
> 		a.py
> 	sub1/
> 		__init__.py
> 		b.py
> 
> pkg_root might either be installed as Package (in site-packages) or
> someone may decide to run app.py from within pkg_root.
> 
> Also, irrespective of whether people have run into such problems or
> not, don't you think it makes sense for modules to be searched the way
> I had suggested earlier??  Or is there some serious issue with this.

I've been reorganising some package structures recently, and I found a
number of issues which may be of interest to you. Firstly, that remark
I made above: let's say we run app.py from outside pkg_root...

  python pkg_root/app.py

What would you expect to happen if app.py contained the line given
below?

  import pkg_root.common_module

What seems to happen is that unless the current directory (the parent
of pkg_root) resides on your PYTHONPATH, this will not work - it's as
if app.py is part of the package and should be using...

  import common_module

I found this slightly confusing, but it is related to what you want to
do in your example, because I agree that it can make some sense to
include the test, demonstration or main application program within the
package directory, even though it isn't strictly part of the package -
it's a user of the package. Of course, in Java one uses an explicit
package keyword to exclude or include "modules" from/in packages,
whereas in Python it's implicit.

This is really an aside, though, because your main issue is with the
way importing of "super-package" members is done. What happens if
there's a package called common somewhere else on your system outside
(and unconnected with) this package, and then in a.py you do use the
following statement?

  import common.foo

In the scheme you propose, access to the external common module is
likely to be restricted in some way by the presence of the common
subpackage of the new package. With the existing mechanisms, however,
this ambiguity is avoided entirely in this case - you have to do the
following:

  import pkg_root.common.foo

Of course, you can argue that you might want to use the new package's
common subpackage without acknowledging that it is part of pkg_root,
but then you should ask yourself whether the common subpackage
wouldn't be better off outside pkg_root if it is to be used
independently of it. You might need to rename the newly independent
common package, though.

I do see your point, though. In one of my works, I use packages to
maintain a sort-of logical structure as follows:

  XMLForms
    __init__.py
    Accessor.py
    ...
    DOM
      __init__.py
      DOMAccessor.py  # Yes, I could have called this Accessor,
                      # but I did say that I just reorganised
                      # the package!
      ...
      DOMElementTypes
        __init__.py
        BaseTypes.py
    ElementTypes
      __init__.py
      BaseTypes.py
    test.py           # This just uses the package - it isn't
                      # supposed to be part of the services
                      # offered by the package, even though it
                      # logically belongs inside the package.

Now, DOMElementTypes is related to ElementTypes through their "nature"
- they perform similar activities. However, I have chosen to group
define subpackages according to implementation technology. As a
result, I would have to do the following kind of import from within
XMLForms.DOM.DOMElementTypes.BaseTypes:

  import XMLForms.ElementTypes.BaseTypes

You might want to know why I can't do this:

  import ElementTypes.BaseTypes

Well, I don't really see the need to do that unless I consider
ElementTypes to be independent of XMLForms. Moreover, if I decided to
give in and rename DOMElementTypes to ElementTypes, because it lives
in a separate namespace to the other ElementTypes, then I would still
be compelled to do a "full" import under any import scheme which isn't
"really clever".

  import DOM.ElementTypes.BaseTypes
  import ElementTypes.BaseTypes      # What does this import?

One could insist that the first statement always be used to access the
DOM version, and that the second statement always accesses the
higher-level version, but how is the import mechanism supposed to know
what you mean?

So, to summarise, perhaps there's no simple way of introducing what
you want without introducing some of the consequences outlined here.
Explicit package specifiers might help in certain cases (the app.py or
test.py cases), but they could promote a package structure which isn't
obvious from browsing the filesystem. Unions of packages (which could
be done using either explicit specifiers or import magic) could
potentially introduce side-effects as modules start to co-exist with
modules they were never meant to co-exist with.

Paul