[Python-Dev] PEP 366 - Relative imports from main modules

[Nick Coghlan]

A c.l.p discussion referenced from Python-URL just brought this topic 
back to my attention, and with the relatively low traffic on the 
development lists in the last few days, it seemed like a good time to 
repost this PEP (it vanished beneath the Unicode identifier discussion 
last time).

Cheers,
Nick.


PEP: 366
Title: Main module explicit relative imports
Version: $Revision: 56172 $
Last-Modified: $Date: 2007-07-04 22:47:13 +1000 (Wed, 04 Jul 2007) $
Author: Nick Coghlan <ncoghlan at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 1-May-2007
Python-Version: 2.6
Post-History: 1-May-2007


Abstract
========

This PEP proposes a backwards compatible mechanism that permits
the use of explicit relative imports from executable modules within
packages. Such imports currently fail due to an awkward interaction
between PEP 328 and PEP 338 - this behaviour is the subject of at
least one open SF bug report (#1510172)[1], and has most likely
been a factor in at least a few queries on comp.lang.python (such
as Alan Isaac's question in [2]).

With the proposed mechanism, relative imports will work automatically
if the module is executed using the ``-m`` switch. A small amount of
boilerplate will be needed in the module itself to allow the relative
imports to work when the file is executed by name.


Import Statements and the Main Module
=====================================

(This section is taken from the final revision of PEP 338)

The release of 2.5b1 showed a surprising  (although obvious in
retrospect) interaction between PEP 338 and PEP 328 - explicit
relative imports don't work from a main module. This is due to
the fact that relative imports rely on ``__name__`` to determine
the current module's position in the package hierarchy. In a main
module, the value of ``__name__`` is always ``'__main__'``, so
explicit relative imports will always fail (as they only work for
a module inside a package).

Investigation into why implicit relative imports *appear* to work when
a main module is executed directly but fail when executed using ``-m``
showed that such imports are actually always treated as absolute
imports. Because of the way direct execution works, the package
containing the executed module is added to sys.path, so its sibling
modules are actually imported as top level modules. This can easily
lead to multiple copies of the sibling modules in the application if
implicit relative imports are used in modules that may be directly
executed (e.g. test modules or utility scripts).

For the 2.5 release, the recommendation is to always use absolute
imports in any module that is intended to be used as a main module.
The ``-m`` switch already provides a benefit here, as it inserts the
current directory into ``sys.path``, instead of the directory containing
the main module. This means that it is possible to run a module from
inside a package using ``-m`` so long as the current directory contains
the top level directory for the package. Absolute imports will work
correctly even if the package isn't installed anywhere else on
sys.path. If the module is executed directly and uses absolute imports
to retrieve its sibling modules, then the top level package directory
needs to be installed somewhere on sys.path (since the current directory
won't be added automatically).

Here's an example file layout::

     devel/
         pkg/
             __init__.py
             moduleA.py
             moduleB.py
             test/
                 __init__.py
                 test_A.py
                 test_B.py

So long as the current directory is ``devel``, or ``devel`` is already
on ``sys.path`` and the test modules use absolute imports (such as
``import pkg.moduleA`` to retrieve the module under test, PEP 338
allows the tests to be run as::

     python -m pkg.test.test_A
     python -m pkg.test.test_B

Rationale for Change
====================

In rejecting PEP 3122 (which proposed a higher impact solution to this
problem), Guido has indicated that he still isn't particularly keen on
the idea of executing modules inside packages as scripts [2]. Despite
these misgivings he has previously approved the addition of the ``-m``
switch in Python 2.4, and the ``runpy`` module based enhancements
described in PEP 338 for Python 2.5.

The philosophy that motivated those previous additions (i.e. access to
utility or testing scripts without needing to worry about name clashes in
either the OS executable namespace or the top level Python namespace) is
also the motivation behind fixing what I see as a bug in the current
implementation.

This PEP is intended to provide a solution which permits explicit
relative imports from main modules, without incurring any significant
costs during interpreter startup or normal module import.


Proposed Solution
=================

The heart of the proposed solution is a new module attribute
``__package_name__``. This attribute will be defined only in
the main module (i.e. modules where ``__name__ == "__main__"``).

For a directly executed main module, this attribute will be set
to the empty string. For a module executed using
``runpy.run_module()`` with the ``run_name`` parameter set to
``"__main__"``, the attribute will be set to
``mod_name.rpartition('.')[0]`` (i.e., everything up to
but not including the last period).

In the import machinery there is an error handling path which
deals with the case where an explicit relative reference attempts
to go higher than the top level in the package hierarchy. This
error path would be changed to fall back on the ``__package_name__``
attribute for explicit relative imports when the importing module
is called ``"__main__"``.

With this change, explicit relative imports will work automatically
from a script executed with the ``-m`` switch. To allow direct
execution of the module, the following boilerplate would be needed at
the top of the script::

   if __name__ == "__main__" and not __package_name__:
       __package_name__ = "<expected_pkg_name>"

Note that this boilerplate is sufficient only if the top level package
is already accessible via sys.path. Additional code that manipulates
sys.path would be needed in order for direct execution to work
without the top level package already being on sys.path.

This approach also has the same disadvantage as the use of absolute
imports of sibling modules - if the script is moved to a different
package or subpackage, the boilerplate will need to be updated
manually.

With this feature in place, the test scripts in the package above
would be able to change their import lines to something along the
lines of ``import ..moduleA``. The scripts could then be
executed unmodified even if the name of the package was changed.

(Rev 47142 in SVN implemented an early variant of this proposal
which stored the main module's real module name in the
'__module_name__' attribute. It was reverted due to the fact
that 2.5 was already in beta by that time.)


Alternative Proposals
=====================

PEP 3122 proposed addressing this problem by changing the way
the main module is identified. That's a huge compatibility cost
to incur to fix something that is a pretty minor bug in the overall
scheme of things.

The advantage of the proposal in this PEP is that its only impact on
normal code is the tiny amount of time needed at startup to set the extra
attribute in the main module. The changes to the import machinery are all
in an existing error handling path, so normal imports don't incur any
performance penalty at all.


References
==========

.. [1] Absolute/relative import not working?
    (http://www.python.org/sf/1510172)

.. [2] Guido's rejection of PEP 3122
    (http://mail.python.org/pipermail/python-3000/2007-April/006793.html)

.. [3] c.l.p. question about modules and relative imports
 
(http://groups.google.com/group/comp.lang.python/browse_thread/thread/c44c769a72ca69fa/)

Copyright
=========

This document has been placed in the public domain.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org