A c.l.p discussion referenced from Python-URL just brought this topic back to my attention, and with the relatively low traffic on the development lists in the last few days, it seemed like a good time to repost this PEP (it vanished beneath the Unicode identifier discussion last time).

Cheers, Nick.

PEP: 366 Title: Main module explicit relative imports Version: $Revision: 56172 $ Last-Modified: $Date: 2007-07-04 22:47:13 +1000 (Wed, 04 Jul 2007) $ Author: Nick Coghlan ncoghlan@gmail.com Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 1-May-2007 Python-Version: 2.6 Post-History: 1-May-2007

Abstract

This PEP proposes a backwards compatible mechanism that permits the use of explicit relative imports from executable modules within packages. Such imports currently fail due to an awkward interaction between PEP 328 and PEP 338 - this behaviour is the subject of at least one open SF bug report (#1510172)[1], and has most likely been a factor in at least a few queries on comp.lang.python (such as Alan Isaac's question in [2]).

With the proposed mechanism, relative imports will work automatically if the module is executed using the -m switch. A small amount of boilerplate will be needed in the module itself to allow the relative imports to work when the file is executed by name.

Import Statements and the Main Module

(This section is taken from the final revision of PEP 338)

The release of 2.5b1 showed a surprising (although obvious in retrospect) interaction between PEP 338 and PEP 328 - explicit relative imports don't work from a main module. This is due to the fact that relative imports rely on __name__ to determine the current module's position in the package hierarchy. In a main module, the value of __name__ is always '__main__', so explicit relative imports will always fail (as they only work for a module inside a package).

Investigation into why implicit relative imports appear to work when a main module is executed directly but fail when executed using -m showed that such imports are actually always treated as absolute imports. Because of the way direct execution works, the package containing the executed module is added to sys.path, so its sibling modules are actually imported as top level modules. This can easily lead to multiple copies of the sibling modules in the application if implicit relative imports are used in modules that may be directly executed (e.g. test modules or utility scripts).

For the 2.5 release, the recommendation is to always use absolute imports in any module that is intended to be used as a main module. The -m switch already provides a benefit here, as it inserts the current directory into sys.path, instead of the directory containing the main module. This means that it is possible to run a module from inside a package using -m so long as the current directory contains the top level directory for the package. Absolute imports will work correctly even if the package isn't installed anywhere else on sys.path. If the module is executed directly and uses absolute imports to retrieve its sibling modules, then the top level package directory needs to be installed somewhere on sys.path (since the current directory won't be added automatically).

Here's an example file layout::

 devel/
     pkg/
         __init__.py
         moduleA.py
         moduleB.py
         test/
             __init__.py
             test_A.py
             test_B.py

So long as the current directory is devel, or devel is already on sys.path and the test modules use absolute imports (such as import pkg.moduleA to retrieve the module under test, PEP 338 allows the tests to be run as::

 python -m pkg.test.test_A
 python -m pkg.test.test_B

Rationale for Change

In rejecting PEP 3122 (which proposed a higher impact solution to this problem), Guido has indicated that he still isn't particularly keen on the idea of executing modules inside packages as scripts [2]. Despite these misgivings he has previously approved the addition of the -m switch in Python 2.4, and the runpy module based enhancements described in PEP 338 for Python 2.5.

The philosophy that motivated those previous additions (i.e. access to utility or testing scripts without needing to worry about name clashes in either the OS executable namespace or the top level Python namespace) is also the motivation behind fixing what I see as a bug in the current implementation.

This PEP is intended to provide a solution which permits explicit relative imports from main modules, without incurring any significant costs during interpreter startup or normal module import.

Proposed Solution

The heart of the proposed solution is a new module attribute __package_name__. This attribute will be defined only in the main module (i.e. modules where __name__ == "__main__").

For a directly executed main module, this attribute will be set to the empty string. For a module executed using runpy.run_module() with the run_name parameter set to "__main__", the attribute will be set to mod_name.rpartition('.')[0] (i.e., everything up to but not including the last period).

In the import machinery there is an error handling path which deals with the case where an explicit relative reference attempts to go higher than the top level in the package hierarchy. This error path would be changed to fall back on the __package_name__ attribute for explicit relative imports when the importing module is called "__main__".

With this change, explicit relative imports will work automatically from a script executed with the -m switch. To allow direct execution of the module, the following boilerplate would be needed at the top of the script::

if __name__ == "__main__" and not __package_name__: __package_name__ = "<expected_pkg_name>"

Note that this boilerplate is sufficient only if the top level package is already accessible via sys.path. Additional code that manipulates sys.path would be needed in order for direct execution to work without the top level package already being on sys.path.

This approach also has the same disadvantage as the use of absolute imports of sibling modules - if the script is moved to a different package or subpackage, the boilerplate will need to be updated manually.

With this feature in place, the test scripts in the package above would be able to change their import lines to something along the lines of import ..moduleA. The scripts could then be executed unmodified even if the name of the package was changed.

(Rev 47142 in SVN implemented an early variant of this proposal which stored the main module's real module name in the '__module_name__' attribute. It was reverted due to the fact that 2.5 was already in beta by that time.)

Alternative Proposals

PEP 3122 proposed addressing this problem by changing the way the main module is identified. That's a huge compatibility cost to incur to fix something that is a pretty minor bug in the overall scheme of things.

The advantage of the proposal in this PEP is that its only impact on normal code is the tiny amount of time needed at startup to set the extra attribute in the main module. The changes to the import machinery are all in an existing error handling path, so normal imports don't incur any performance penalty at all.

References

.. [1] Absolute/relative import not working? (http://www.python.org/sf/1510172)

.. [2] Guido's rejection of PEP 3122 (http://mail.python.org/pipermail/python-3000/2007-April/006793.html)

.. [3] c.l.p. question about modules and relative imports

(http://groups.google.com/group/comp.lang.python/browse_thread/thread/c44c769...)

Copyright

This document has been placed in the public domain.

--

Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

         http://www.boredomandlaziness.org

Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org