Draft: PEP for imports
Here's the draft PEP for imports. I'll wait a few days for comments before sending to the PEPmeister. PEP: XXX Title: Imports: multi-line and absolute/relative Version: Last-Modified: Author: Aahz <aahz@pythoncraft.com Status: Draft Type: Standards Track Python-Version: 2.4 Content-Type: text/x-rst Created: 21-Dec-2003 Post-History: Abstract ======== The ``import`` statement has two problems: * Long ``import`` statements can be difficult to write, requiring various contortions to fit Pythonic style guidelines. * Imports can be ambiguous in the face of packages; within a package, it's not clear whether ``import foo`` refers to a module within the package or some module outside the package. For the first problem, it is proposed that parentheses be permitted to surround names, thus allowing Python's standard mechanisms for multi-line values to apply. For the second problem, it is proposed that all ``import`` statements be absolute by default (more precisely, relative to ``sys.path``) with special syntax for accessing package-relative imports. Rationale for parentheses ========================= Currently, if you want to import a lot of names from a module or package, you have to choose one of several unpalatable options: * Write a long line with backslash continuations: :: from Tkinter import Tk, Frame, Button, Entry, Canvas, Text \ LEFT, DISABLED, NORMAL, RIDGE, END * Write multiple ``import`` statements: :: from Tkinter import Tk, Frame, Button, Entry, Canvas, Text from Tkinter import LEFT, DISABLED, NORMAL, RIDGE, END (``import *`` is *not* an option ;-) Instead, it should be possible to use Python's standard grouping mechanism (parentheses) to write the ``import`` statement:: from Tkinter import ( Tk, Frame, Button, Entry, Canvas, Text LEFT, DISABLED, NORMAL, RIDGE, END ) This part of the proposal already has BDFL approval. Rationale for absolute imports ============================== In current Python, if you're reading a module located inside a package, it is not clear whether :: import foo refers to a top-level module or another module inside the package. To resolve the ambiguity, it is proposed that ``foo`` will always be a module or package reachable from ``sys.path``. Because this represents a change in semantics, absolute imports will be optional in Python 2.4 through the use of :: from __future__ import absolute_import This PEP will be updated when it is decided to make absolute imports the default, probably Python 2.5 or 2.6. This part of the proposal already has BDFL approval. Rationale for relative imports ============================== With the shift to absolute imports, the question arose whether relative imports should be allowed at all. Several use cases were presented, the most important of which is being able to rearrange the structure of large packages without having to edit sub-packages. Guido approved of the idea of relative imports, but there has been a lot of disagreement on the spelling (syntax). There does seem to be agreement that relative imports will require listing specific names to import (that is, ``import foo`` as a bare term will always be an absolute import). Here are the contenders: * One from Guido: :: from .foo import and :: from ...foo import These two forms have a couple of different suggested semantics. One semantic is to make each dot represent one level. There have been many complaints about the difficulty of counting dots. Another option is to only allow one level of relative import. That misses a lot of functionality, and people still complained about missing the dot in the one-dot form. The final option is to define an algorithm for finding relative modules and packages; the objection here is "Explicit is better than implicit". * The next set of options is conflated from several posters: :: from __pkg__.__pkg__ import and :: from .__parent__.__parent__ import Many people (Guido included) think these look ugly, but they *are* clear and explicit. Overall, more people prefer ``__pkg__`` as the shorter option. * Finally, some people dislike the way you have to change ``import`` to ``from ... import`` when you want to dig inside a package. They suggest completely rewriting the ``import`` syntax: :: from MODULE import NAMES as RENAME searching HOW or :: import NAMES as RENAME from MODULE searching HOW [from NAMES] [in WHERE] import ... However, this most likely could not be implemented for Python 2.4 (too big a change), and allowing relative imports is sufficiently critical that we need something now (given that the standard ``import`` will change to absolute import). Open Issues =========== The BDFL needs to decide which of the various options for relative imports works best. Additional proposals are still welcome. As usual, Guido prefers reasons to histrionics. References ========== For more background, see the following python-dev threads: - `Re: Christmas Wishlist <http://mail.python.org/pipermail/python-dev/2003-December/040973.html>`_ - `Re: Python-Dev Digest, Vol 5, Issue 57 <http://mail.python.org/pipermail/python-dev/2003-December/041078.html>`_ - `Relative import <http://mail.python.org/pipermail/python-dev/2003-December/041065.html>`_ - `Another Strategy for Relative Import <http://mail.python.org/pipermail/python-dev/2003-December/041418.html>`_ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:
aahz> Here's the draft PEP for imports. I'll wait a few days for aahz> comments before sending to the PEPmeister. I don't know if it's relevant to the discussion, but at various times people have asked about pushing the current global module/package namespace into a package, so instead of import sys you might use import std.sys or from std import sys to reference the sys module as delivered with Python. It might be worthwhile to consider that as an alternative or addition to this capability. Because of all the code changes it would cause, it's not likely to be considered before Python 3, but it seems to me that it might solve many common module collision problems without resorting to major changes to the semantics of the import statement. Skip
I don't know if it's relevant to the discussion, but at various times people have asked about pushing the current global module/package namespace into a package, so instead of
import sys
you might use
import std.sys
or
from std import sys
to reference the sys module as delivered with Python.
It might be worthwhile to consider that as an alternative or addition to this capability. Because of all the code changes it would cause, it's not likely to be considered before Python 3, but it seems to me that it might solve many common module collision problems without resorting to major changes to the semantics of the import statement.
I strongly suggest to keep this out of Aahz' PEP; it's just going to confuse the issue in the short run, and in the long run it's pretty independent. I also don't think that it's ever going to be that way. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, 2004-01-19 at 17:49, Guido van Rossum wrote:
I strongly suggest to keep this out of Aahz' PEP; it's just going to confuse the issue in the short run, and in the long run it's pretty independent. I also don't think that it's ever going to be that way.
There was also the "import __me__" idea, which should also not go in Aahz's pep <wink>. Maybe we should just go ahead and reserve peps 800-899 for import related enhancements. -Barry
Here's the draft PEP for imports. I'll wait a few days for comments before sending to the PEPmeister.
I've seen a few suggestions of using slashes intead of dots, enabling the use of ../ for relative import. I believe these are misguided. This would blur between the filesystem and the package namespace; next people would be expecting to be able to say from /home/guido/lib/python import neatTricks and then all hell breaks loose: what would the package name for neatTricks be? Should it be a toplevel module, 'neatTricks'? Or should it be called 'home.guido.lib.python.neatTricks'? Neither is acceptable. So please don't include that in the PEP -- the idea is dead on arrival. [...]
Here are the contenders:
* One from Guido:
::
from .foo import
and
::
from ...foo import
These two forms have a couple of different suggested semantics. One semantic is to make each dot represent one level. There have been many complaints about the difficulty of counting dots. Another option is to only allow one level of relative import. That misses a lot of functionality, and people still complained about missing the dot in the one-dot form. The final option is to define an algorithm for finding relative modules and packages; the objection here is "Explicit is better than implicit".
The PEP should specify the algorithm proposed (which is to search up from the containing package until foo is found). [...]
* Finally, some people dislike the way you have to change ``import`` to ``from ... import`` when you want to dig inside a package. They suggest completely rewriting the ``import`` syntax:
::
from MODULE import NAMES as RENAME searching HOW
or
::
import NAMES as RENAME from MODULE searching HOW [from NAMES] [in WHERE] import ...
Nobody seems to have questioned this syntax. But what does NAMES stand for? The current renaming syntax lets you write e.g. import foo as bar, spam as ham It doesn't make sense to write import foo, spam as one_name and import foo, spam as bar, ham is definitely worse IMO (not to mention that this already has a meaning today). So then the two questions about the first example are: - how string does the "searching HOW" clause bind? Do you write import foo as bar searching XXX, spam as ham searching XXX or import foo as bar, spam as ham searching XXX ??? Also, the PEP should mention the choices for HOW (I don't recall what they are). And your syntax should show which parts are optional, e.g. [from MODULE] import NAME [as RENAME], NAME [as RENAME], ... [searching HOW] I'm totally lost with the second alternative. Maybe examples of both should be given, unless you want to abandon these right away (fine with me :-). Anyway, stick it in the PEP archive already. You can still edit it later. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Hello Guido, On Sun, Jan 25, 2004 at 01:59:16PM -0800, Guido van Rossum wrote:
from /home/guido/lib/python import neatTricks
Keeping the amount of magic behind import as low as possible seems very important, because they are not a minor feature but something that every beginner must reasonably understand; I've already seen it as an obstacle. The above statement has the advantage of looking obvious; but in addition to the package name problem there is the fact that directory names are not always valid Python identifiers. A last try: import neatTricks in "/home/guido/lib/python" # no package import package.module in "/home/guido/lib/python" # package import foo in "." # relative import from neatTricks in "../cmds" import a, b, c s=os.path.join("some", "where"); import foo in s # expression path where the semantics would be to search sys.path if and only if no 'in' clause is specified. ('in' doesn't sound quite right...) Armin
On Fri, Jan 30, 2004 at 01:32:53PM +0000, Armin Rigo wrote:
Hello Guido,
On Sun, Jan 25, 2004 at 01:59:16PM -0800, Guido van Rossum wrote:
from /home/guido/lib/python import neatTricks
Keeping the amount of magic behind import as low as possible seems very important, because they are not a minor feature but something that every beginner must reasonably understand; I've already seen it as an obstacle. The above statement has the advantage of looking obvious; but in addition to the package name problem there is the fact that directory names are not always valid Python identifiers. A last try:
import neatTricks in "/home/guido/lib/python" # no package import package.module in "/home/guido/lib/python" # package import foo in "." # relative import from neatTricks in "../cmds" import a, b, c s=os.path.join("some", "where"); import foo in s # expression path
An experienced Python programmer could probably use this to some advantage, but I think many beginners would use it without realizing it will very easily result in a piece of software which is completely undistributable. They may save some time up front by not having to understanding how the rest of the import system works, but will it be more time than they have to spend later trying to fix their app so it runs on systems other than the one it was developed on? Jp
Hello, On Fri, Jan 30, 2004 at 09:26:13AM -0500, Jp Calderone wrote:
import neatTricks in "/home/guido/lib/python" # no package import package.module in "/home/guido/lib/python" # package import foo in "." # relative import from neatTricks in "../cmds" import a, b, c s=os.path.join("some", "where"); import foo in s # expression path
An experienced Python programmer could probably use this to some advantage, but I think many beginners would use it without realizing it will very easily result in a piece of software which is completely undistributable.
Point taken. As far as I can see I believe that the whole module system problably needs some more in-depth rethinking at some point. For the beginner the package system is baffling, and for the advanced user it lacks precise control over what is imported from where, for the cases where it is really needed. [tentative development] In the multi-directory projects I am familiar with, I've seen two cases: * a stand-alone application that has to deal with file-system-level directories anyway to access data files; this uses sys.path hacks all around to make sure the needed directories are accessible when needed. It would have been immediately useful to be able to write "import gamesrv fromdir '../common'". It would also have been nice to be able to open a file relative to the current module. (In this case relative file and module names are good.) * a big package named after the application, in which we try to make all imports absolute. This is cleaner apart from the need to setup sys.path initially from user-runnable scripts that are not at the root (which is the case of all the test_*.py). The needed hackery can be nicely isolated in a file 'autopath.py' that is copied into each directory, so that each user script starts with "import autopath". I never find myself using "import package.module" but always "from package import module". Actually I still have to be convinced that packages are a Good Thing. As far as I am concerned, sys.path hacks are the only case in which I miss a Python feature sufficiently to have to copy similar custom code all around my projects. A more radical point of view is that these sys.path hacks are here not because of a missing feature, but on the contrary to work around the way Python tries to isolate me from the "messiness out there" (the file system) by mapping it into a neat language construct (packages). Trying to map a user-controlled structure over another, different one is a receipt for eventual confusion, however good the design is. I think I'd prefer to have just a reasonably natural way to talk about modules -- and files in general -- relative to the current module or the standard library. This is only a vague idea, but I believe it would fit better into (at least my view of) Python's philosophy: keep the language simple by avoiding abstraction layers that are not "natural", i.e. that do some kind of remapping of concepts, as opposed to just expose a higher-level view of the same concepts. A bientot, Armin.
At 11:40 PM 1/30/04 +0000, Armin Rigo wrote:
A more radical point of view is that these sys.path hacks are here not because of a missing feature, but on the contrary to work around the way Python tries to isolate me from the "messiness out there" (the file system) by mapping it into a neat language construct (packages).
Without that, various kinds of "hacks" such as importing from DLLs/.so's, zipfiles, frozen modules, py2exe, PYTHONPATH, and the like, would not be practical. These use cases alone are enough to make me -1 on adding explicit paths into the module system from within code. I'm *much* more interested in making non-module (but "read-only") files available through the package/module abstraction, than in breaking the abstraction to specify directories. For example, in PEAK I've implemented both a utility function ('fileNearModule("some.module","relative/path")') and a URL scheme ('pkgfile:some.module/relative/path') to support this.
On Fri, 2004-01-30 at 18:40, Armin Rigo wrote:
* a stand-alone application that has to deal with file-system-level directories anyway to access data files; this uses sys.path hacks all around to make sure the needed directories are accessible when needed. It would have been immediately useful to be able to write "import gamesrv fromdir '../common'". It would also have been nice to be able to open a file relative to the current module. (In this case relative file and module names are good.)
I don't use path hacks to find data files, but I have gotten into the habit of making data file directories packages when they live inside the source tree. Then I can just do from my.package import data myfile = os.path.join(os.path.dirname(data.__file__), 'myfile.xml')
* a big package named after the application, in which we try to make all imports absolute. This is cleaner apart from the need to setup sys.path initially from user-runnable scripts that are not at the root (which is the case of all the test_*.py). The needed hackery can be nicely isolated in a file 'autopath.py' that is copied into each directory, so that each user script starts with "import autopath".
This is definitely a common idiom. I call the file path.py and yes, every top level script must import this before it imports any of the application's modules. We've been doing this kind of thing for years, and while I very often wish there was something we could put in the standard library, or hack we could add to the executable, there have really never been any elegant-enough solutions to the problem.
I never find myself using "import package.module" but always "from package import module". Actually I still have to be convinced that packages are a Good Thing. As far as I am concerned, sys.path hacks are the only case in which I miss a Python feature sufficiently to have to copy similar custom code all around my projects.
Two different things. Packages are a huge win IMO because it allows for larger organizing principles. Flat import space was really painful before packages came along. Things like distutils wouldn't be feasible without them, and we'd be arguing about naming conventions much more if we didn't have them. That's aside from the path hackery necessary for top level application scripts. I use "from package import module" all the time and I think Python's rules here are just about right. Personally, my only complaint is the default-relative import rule.
A more radical point of view is that these sys.path hacks are here not because of a missing feature, but on the contrary to work around the way Python tries to isolate me from the "messiness out there" (the file system) by mapping it into a neat language construct (packages). Trying to map a user-controlled structure over another, different one is a receipt for eventual confusion, however good the design is.
I disagree. I really like the layer of abstraction Python imposes here and I'd hate to see that taken away. I'm -1 on any proposal to use (forward) slashes in import statements. To me, it's way to system dependent (wait 'til the Windows people push for backslashes). I also like the level of indirection, control, and flexibility that sys.path gives you. I agree it would be nice to not have to write autopath.py modules, but that only affects a handful of files in an otherwise large application. Other than relative imports, the only other thing I occasionally miss in Python's import semantics is the fact that more than one file system space can't be mapped into one package namespace without trickery. E.g. if in "import my.package.sub1" and "import my.package.sub2", the directories sub1 and sub2 didn't have to live under a directory called my/package.
I think I'd prefer to have just a reasonably natural way to talk about modules -- and files in general -- relative to the current module or the standard library. This is only a vague idea, but I believe it would fit better into (at least my view of) Python's philosophy: keep the language simple by avoiding abstraction layers that are not "natural", i.e. that do some kind of remapping of concepts, as opposed to just expose a higher-level view of the same concepts.
I disagree that Python's current rules are not natural. I think they're very natural, modulo a few nits here and there. I definitely don't want to live closer to the metal in the import statement. -Barry
At 08:15 PM 1/30/04 -0500, Barry Warsaw wrote:
Other than relative imports, the only other thing I occasionally miss in Python's import semantics is the fact that more than one file system space can't be mapped into one package namespace without trickery. E.g. if in "import my.package.sub1" and "import my.package.sub2", the directories sub1 and sub2 didn't have to live under a directory called my/package.
Didn't Guido write a 'pkgutil' module for this? Or am I confusing what he wrote with what you want? :)
On Fri, 2004-01-30 at 20:25, Phillip J. Eby wrote:
At 08:15 PM 1/30/04 -0500, Barry Warsaw wrote:
Other than relative imports, the only other thing I occasionally miss in Python's import semantics is the fact that more than one file system space can't be mapped into one package namespace without trickery. E.g. if in "import my.package.sub1" and "import my.package.sub2", the directories sub1 and sub2 didn't have to live under a directory called my/package.
Didn't Guido write a 'pkgutil' module for this? Or am I confusing what he wrote with what you want? :)
He did! I totally spaced on that. Okay, Python packages are more close to perfect than I thought. :) Maybe, though, it's time to get rid of this warning from http://www.python.org/doc/current/lib/module-pkgutil.html Warning: This is an experimental module. It may be withdrawn or completely changed up to an including the release of Python 2.3 beta 1. :) -Barry
At 08:39 PM 1/30/04 -0500, Barry Warsaw wrote:
On Fri, 2004-01-30 at 20:25, Phillip J. Eby wrote:
At 08:15 PM 1/30/04 -0500, Barry Warsaw wrote:
Other than relative imports, the only other thing I occasionally miss in Python's import semantics is the fact that more than one file system space can't be mapped into one package namespace without trickery. E.g. if in "import my.package.sub1" and "import my.package.sub2", the directories sub1 and sub2 didn't have to live under a directory called my/package.
Didn't Guido write a 'pkgutil' module for this? Or am I confusing what he wrote with what you want? :)
He did! I totally spaced on that. Okay, Python packages are more close to perfect than I thought. :)
Obviously, the time machine still works in California, but now with added cross-continent spatial reloaction, allowing him to write 'pkgutil' not only some time ago, but on the opposite coast as well. :)
Armin Rigo <arigo@tunes.org> wrote:
import neatTricks in "/home/guido/lib/python" # no package import package.module in "/home/guido/lib/python" # package
[...]
where the semantics would be to search sys.path if and only if no 'in' clause is specified. ('in' doesn't sound quite right...)
"at" more closely matches the apparent usage above, i.e.: from neatTricks at "/path/to/dir" import bar It's a little less rosy as: import neatTricks at "/home/guido/lib/python" but not too ugly. Unfortunately the best grammatical match would be "from", but I'm positive that would be objectionable to someone: from neatTricks from "/path/to/dir" import TrickyDick Charles -- ----------------------------------------------------------------------- Charles Cazabon <python@discworld.dyndns.org> GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------
Charles Cazabon wrote:
"at" more closely matches the apparent usage above, i.e.:
from neatTricks at "/path/to/dir" import bar
It's a little less rosy as:
import neatTricks at "/home/guido/lib/python"
In my opinion, "import foo at bar" reads too much like "import foo as bar". just my 2 mucents, Gerrit. -- 31. If he hire it out for one year and then return, the house, garden, and field shall be given back to him, and he shall take it over again. -- 1780 BC, Hammurabi, Code of Law -- PrePEP: Builtin path type http://people.nl.linux.org/~gerrit/creaties/path/pep-xxxx.html Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/
Gerrit Holl <gerrit@nl.linux.org> wrote:
import neatTricks at "/home/guido/lib/python"
In my opinion, "import foo at bar" reads too much like "import foo as bar".
Agreed. I'm -0 on the whole idea, but if it's wanted badly enough, "fromdir" might be less confusing than "in" or "at". Charles -- ----------------------------------------------------------------------- Charles Cazabon <python@discworld.dyndns.org> GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------
Hello Guido,
Hello Armin,
from /home/guido/lib/python import neatTricks
Keeping the amount of magic behind import as low as possible seems very important, because they are not a minor feature but something that every beginner must reasonably understand; I've already seen it as an obstacle. The above statement has the advantage of looking obvious; but in addition to the package name problem there is the fact that directory names are not always valid Python identifiers. A last try:
import neatTricks in "/home/guido/lib/python" # no package import package.module in "/home/guido/lib/python" # package import foo in "." # relative import from neatTricks in "../cmds" import a, b, c s=os.path.join("some", "where"); import foo in s # expression path
where the semantics would be to search sys.path if and only if no 'in' clause is specified. ('in' doesn't sound quite right...)
The 'from package at <string> import module' syntax proposed later reads better, and the new keyword 'at' could be handled syntactically the same way as 'as'. I'm still -0 on this feature. You say that every beginner must learn import. This is indeed true. But shouldn't they learn how to set the path right rather than learn how to import by filename? Setting the path is one of the things that everybody must learn about import anyway. In any software endeavor, there comes a time when a piece of code is released with a reference to a file name that is only valid on the author's machine. I think we shouldn't encourage beginners to make such mistakes! --Guido van Rossum (home page: http://www.python.org/~guido/)
Armin Rigo wrote:
import neatTricks in "/home/guido/lib/python" # no package import package.module in "/home/guido/lib/python" # package import foo in "." # relative import from neatTricks in "../cmds" import a, b, c s=os.path.join("some", "where"); import foo in s # expression path
where the semantics would be to search sys.path if and only if no 'in' clause is specified. ('in' doesn't sound quite right...)
Armin - Thinking about the semantics of the above, including corner cases, I find myself wanting to make the semantics of: <some_kind_of_import_statement> in "<path>" to be equivalent to: sys.path.insert(0, "<path>") <some_kind_of_import_statement> del sys.path[0] # assuming the import doesn't modify sys.path That doesn't seem to be quite equivalent to what you've proposed. On the other hand, that seems to provide at least semi-reasonable answers to questions about what happens to sys.modules, what happens if sys.modules['package'] is defined when importing package.module, what's the correct search path when a module being imported does an import, etc. Having said that, that makes me feel negatively about the proposal, since you could do it yourself directly. Chris
Chris Reedy <chrisandannreedy@comcast.net> writes:
Thinking about the semantics of the above, including corner cases, I find myself wanting to make the semantics of:
<some_kind_of_import_statement> in "<path>"
to be equivalent to:
sys.path.insert(0, "<path>") <some_kind_of_import_statement> del sys.path[0] # assuming the import doesn't modify sys.path
This seems to me to be another example of the sort of setup/teardown idiom that PEP 310 was designed for: with path_entry("<path">): import whatever Here, we have class path_entry def __init(self, path): self.path = path def __enter__(self): sys.path.insert(0, path) def __exit__(self): del sys.path[0] ... and of course, path_entry might be a suitable candidate for some sort of standard library module of useful "with" idioms. Hmm, maybe it's time to look at PEP 310 again. Paul -- This signature intentionally left blank
Summary of response: Yay! Oh, I mean, +1. I am one of those who likes Python's package system quite a bit, except for the issues addressed by this PEP. I like the simple dot-prefix proposal. We use this now in Zope 3's configuration system to refer to modules within a package. A single leading dot refers to the current package, so: ".interfaces.IFoo" refers to the IFoo object in the interfaces module within the current package. Additional dots refer to containing packages, so: "..interfaces.IFoo" refers to the IFoo object in the interfaces module within the containing package of the current package. We've never used more than two dots. I don't really think that I've ever had problems counting up to two. I imagine that I could make it to three if pressed. ;) I've never had problems seeing the leading dot (or dots). I find this syntax to be simple and unobtrusive. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
participants (12)
-
Aahz -
Armin Rigo -
Barry Warsaw -
Charles Cazabon -
Chris Reedy -
Gerrit Holl -
Guido van Rossum -
Jim Fulton -
Jp Calderone -
Paul Moore -
Phillip J. Eby -
Skip Montanaro