[Python-Dev] Relative Package Imports

[M.-A. Lemburg]

Tim Peters wrote:
> 
> [Guido]
> > I'd much rather use absolute package names for anything that's not in
> > the same directory as the current module.
> 
> [M.-A. Lemburg]
> > Of course, you could do everything with absolute names, but then
> > the package author would dictate the complete absolute path which
> > is not always desirable since it can cause name collisions such
> > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be
> > released) and Mark's win32 stuff.
> >
> > As more and more packages appear, we run into this problem more
> > and more often.
> 
> I never understand package complaints.  Maybe it's the imprecision of the
> language people use, or maybe because it's because people don't give fully
> fleshed-out examples.  Whatever, in the end, I never have any idea what the
> complaint is, or in what way the solution is "solving" anything.

My original post contained an example package using relative
imports. The example uses intra-package imports across subpackage
levels which currently is only possible using absolute module
names (see below for a definition).

Note that I did not want to start a discussion about absolute vs.
relative names (I believe everybody agrees that realtive file names
are a Good Thing). The 'import __.module' thing is not new: ni.py
had support for this and my patch simply adds it back to the
implementation.

Think of the '__' as an additional feature that authors can use
at their own will. They don't *have* to, just as you don't have
to use relative file paths in your config files. It's
just a convenience that some authors may want to use in their code.
 
> In the above, "absolute" doesn't appear to mean "absolute" in any
> OS-sensible sense.  So what does it really mean?  Does it mean the same
> things to Guido and MAL?

"Absolute" means fully qualified name, i.e. the complete path to 
the modules from the top-level root via all subpackage down to the
module name itself, e.g. TextTools.Constants.TagTable.

> In MAL's hint of examples, I don't see any problem.  If mxDateTime unpacks
> itself into a directory named DateTime, then *of course* it's going to
> collide with other packages that want to do likewise.  Install it into
> mxDateTime instead, and take "absolute" to mean "any module that wants an
> mxDateTime service and does not itself live directly in mxDateTime/ must
> import the desired module via a path beginning 'mxDateTime.'", and
> everything looks straightforward to me (and that outcome makes me infer that
> this is thus probably what Guido has in mind too).

This is what I intend to do: move all my stuff under a 'mx'
package and then continue reinventing (faster, different,
doesn't-slip-when-wet) wheels ;-). BTW, the queue thingie is just
a proof of concept fun project.

> Similarly for
> win32.Queue vs mxQueue.Queue vs the Queue in the std library (it *would* be
> good to have an explicit way to say "std library" -- "Lib." comes to mind).

I'd prefer 'org.python.core.' or 'stdlib.' or just simply 'python.'
but that's a different issue.
 
> > I could then make all my packages self-contained and
> > distribute them in two forms without having to change
> > a single line for the added support:
> >
> > 1. under the new 'mx' package, e.g. mx.DateTime
> > 2. for backward compatibility under 'DateTime'
> 
> Ah, so that's what this is about.  I vote screw #2.  Releasing it that way
> was a mistake.

Not until Zope went Open Source ;-) ...

>  Better to live with the brief & finite pain of repairing it
> than complicating Python to cover up for it.

Screwing #2 is not possible unless I want all those already
stored DateTime pickles to fail loading... ok, I could probably
provide some kind of compatibility package which then redirects
the import to mx.DateTime.

> > Another major advantage is that I could take any other
> > self-contained package and install it under absolute paths
> > of my choice, e.g. put Zope under org.zope.core, Python under
> > org.python.core etc., without harming their functionality
> > or having to dive deep into their import structures to fix
> > them manually.
> 
> I view that not as an advantage but as harmful complication.  Zope etc add
> great value to a Python installation, and when I write a killer app full of
> "import zope.this" and "import zope.that", I don't want the possibility that
> it's not going to work on my client's machine just because their sysadmin
> installed Zope into some silly site-specific path named after his soon-to-be
> ex-girlfriend <wink>.  I don't want a way to work around him doing that,
> either:  I don't want him to be able to screw me to begin with.

Zope is not a package AFAIK (and probably never will be due to the
pickle complications), so 'import zope.this' won't work anyways
unless you add a Zope wrapping package of your own -- and this will
only work for Zope modules not relying on other Zope modules unless
they use relative imports.

> > To further enhance this mechanism I would like to have an
> > alias mechanism in import, pickle et al. so that changes
> > in the package structures become manageable without user
> > intervention: pickles are a major problem whenever import
> > structures change because they store absolute module names.
> 
> This is a different issue, and may have merits of its own.  WRT the relative
> import scheme, its advantage seems to lie in providing a way to partially
> recover from the damage the new scheme causes <0.5 wink>.

I'm not proposing a new scheme... only a convenience for package
authors.
 
> As is, the package name used by a release is part of its published
> interface.  You can't change it without causing pain, any more than you can
> make incompatible changes to public class methods or input-output behavior.
> In return, package clients are uniform, simple and portable, making life
> easiest for the people who know least.  The burden is on package authors to
> choose names wisely, and that's where the burden should be.

Sure, but we are heading into these kind of problems just now and
have no proper solution at hand. Simply coping out by blaiming the
package authors is not the right way to handle the situation; referring
them to some global name registry (the one at NIST) isn't either.

As the Zope example shows, it's not easy to move from half-packaged
to full packaged (relative imports will ease this move, though) due
to external references using absolute object type/class names. Changes
in the package structure produce the same problems.

It's not so much the burden of the programmer having to adjust his
code to the new layout I'm talking about here: it's the user with
all his data stored in object databases that worries me.

> if-100-pkgs-all-want-their-own-queue-they-already-can-ly y'rs  - tim

Not if just one of them decides to make it top-level... grabbing
the name until eternity ;-)

-- 
Marc-Andre Lemburg
______________________________________________________________________
Y2000:                                                   109 days left
Business:                                      http://www.lemburg.com/
Python Pages:                           http://www.lemburg.com/python/