[issue16891] Fix docs about module search order
New submission from Dmitry Mugtasimov: http://docs.python.org/2/tutorial/modules.html should be rewritten. AS IS 6.1.2. The Module Search Path When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations: TO BE 6.1.2. The Module Search Path When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it looks in the containing package (the package of which the current module is a submodule). If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations: ------ Note that now "6.1.2. The Module Search Path" and "6.4.2. Intra-package References" are contradictary since in 6.4.2 it is said: "In fact, such references are so common that the import statement first looks in the containing package before looking in the standard module search path.", but this is not reflected in 6.1.2. ------ EXAMPLE (for more information see http://stackoverflow.com/questions/14183541/why-python-finds-module-instead-... ): /home/dmugtasimov/tmp/name-res3/xyz __init__.py a.py b.py t.py xyz.py Files init.py, b.py and xyz.py are empty File a.py: import os, sys ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) if not sys.path or ROOT_DIRECTORY not in sys.path: print 'sys.path is modified in a.py' sys.path.insert(0, ROOT_DIRECTORY) else: print 'sys.path is NOT modified in a.py' print 'sys.path:', sys.path print 'BEFORE import xyz.b' import xyz.b print 'AFTER import xyz.b' File t.py: import os, sys ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) if not sys.path or ROOT_DIRECTORY not in sys.path: print 'sys.path is modified in t.py' sys.path.insert(0, ROOT_DIRECTORY) else: print 'sys.path is NOT modified in t.py' import xyz.a Run: python a.py Output: sys.path is modified in a.py sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] BEFORE import xyz.b AFTER import xyz.b Run: python -vv a.py Output: import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/b.so # trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/b.py # /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc Run: python t.py Output: sys.path is modified in t.py sys.path is NOT modified in a.py sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] BEFORE import xyz.b Traceback (most recent call last): File "t.py", line 9, in <module> import xyz.a File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module> import xyz.b ImportError: No module named b Run: python -vv t.py Output: import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/a.so # trying /home/dmugtasimov/tmp/name-res3/xyz/amodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/a.py # /home/dmugtasimov/tmp/name-res3/xyz/a.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/a.py import xyz.a # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/a.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/os.so # trying /home/dmugtasimov/tmp/name-res3/xyz/osmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/os.py # trying /home/dmugtasimov/tmp/name-res3/xyz/os.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.so # trying /home/dmugtasimov/tmp/name-res3/xyz/sysmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.py # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.so # trying /home/dmugtasimov/tmp/name-res3/xyz/xyzmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.py # /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/xyz.py import xyz.xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc # clear[2] __file__ # clear[2] __package__ # clear[2] sys # clear[2] ROOT_DIRECTORY # clear[2] __name__ # clear[2] os sys.path is modified in t.py sys.path is NOT modified in a.py sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] BEFORE import xyz.b Traceback (most recent call last): File "t.py", line 9, in <module> import xyz.a File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module> import xyz.b ImportError: No module named b As you see sys.path is the same for both cases: sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client'] But the behaviour is different. For a.py python searches for package xyz first, and them for module b in it: import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc # trying /home/dmugtasimov/tmp/name-res3/xyz/b.so # trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/b.py # /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc In other words: Search PACKAGE xyz in directory sys.path[0] -> FOUND Search module b in PACKAGE xyz -> FOUND Continue execution For t.py it searches for moduel xyz in the same directory as a.py itself and then fails to find module b in module xyz: # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.so # trying /home/dmugtasimov/tmp/name-res3/xyz/xyzmodule.so # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.py # /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/xyz.py import xyz.xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc In other words: Search MODULE xyz in directory in the same directory as a.py (or sys.path[1] ?) -> FOUND Search MODULE b in MODULE xyz -> NOT FOUND ImportError So it looks like if "import xyz.b" bahaves different depending on how a.py was initially loaded as a script or imported from another module. ---------- assignee: docs@python components: Documentation messages: 179321 nosy: dmugtasimov, docs@python priority: normal severity: normal status: open title: Fix docs about module search order type: behavior versions: Python 2.7 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
Dmitry Mugtasimov added the comment: UPDATE: CHANGE http://stackoverflow.com/questions/14183541/why-python-finds-module-instead-... TO http://stackoverflow.com/questions/14183541/why-python-finds-module-instead-... Because the whole question and replies are important. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
Changes by Éric Araujo <merwok@netwok.org>: ---------- nosy: +brett.cannon, ncoghlan _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
Dmitry Mugtasimov added the comment: As I investigate it a little closer it seems to me that it is not a documentation issue, but an implementation issue. http://docs.python.org/2/reference/simple_stmts.html#import "A package can contain other packages and modules while modules cannot contain other modules or packages." The only why to import name from module is from xyz import b where "b" is name defined inside xyz module. Issuing import xyz.b means that "b" is module or package. Therefore xyz cannot be a module, since "...modules cannot contain other modules or packages." This means that xyz is package. The problem is that it is considered as module for case: python t.py ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
R. David Murray added the comment:
So it looks like if "import xyz.b" bahaves different depending on how a.py was initially loaded as a script or imported from another module.
There are several differences between importing a module and running a script, one of which is what is on sys.path. You constructed your example to mask the path difference. You are getting hit by the difference between absolute and relative imports. How implicit relative imports behave are one of the other things that are different between running a file as a script and importing it. This kind of confusion is one of the reasons implicit relative imports were dropped in Python3. If you add from __future__ import absolute_import to the top of your a and t files, t will no longer produce an import error. It could be that the documentation could be improved, but I'm not sure it is worth the effort for 2.7. If there are statements in the 3.x docs that are incorrect now that implicit relative imports are gone, those would definitely be worth fixing. ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
Dmitry Mugtasimov added the comment: A lot of people are still using python 2.7, even 2.6. For me it would be a nice fix in docs since I spent a plenty of time, trying to figure out what is going on. In my previous comment I also pointed out that implementation probably should be fixed too, since interpreter tries to import module from module, which should not be possible according to docs. It may be an issue for Python 3 too, but I did not check. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
Nick Coghlan added the comment: The docs sometimes try to draw a sharp distinction between modules and packages, but it's essentially a lie - a package is really just a module with a __path__ attribute, and if you know what you are doing, you make even an ordinary module behave like a package (e.g. os.path) Until 3.3, the import system had too many internal inconsistencies for sane documentation. That's fixed in 3.3, but no comprehensive docs will be happening for earlier versions. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
Nick Coghlan added the comment: That said, the originally proposed docs change looks like a solid improvement to me. It's still a lie, but an appropriate one for the tutorial. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
Dmitry Mugtasimov added the comment: Further investigation led me to the conclusion that "TO BE" should look like this: 6.1.2. The Module Search Path When a module named spam is imported, the interpreter first searches in the containing package (the package of which the current module is a submodule) if applicable (a module is not required to be a submodule of a package). If not found or module is not a part of any package it searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations: ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16891> _______________________________________
Change by Brett Cannon <brett@python.org>: ---------- nosy: -brett.cannon _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue16891> _______________________________________
Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment: Python 2.7 is no longer supported. ---------- nosy: +serhiy.storchaka resolution: -> out of date stage: -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue16891> _______________________________________
participants (6)
-
Brett Cannon -
Dmitry Mugtasimov
-
Nick Coghlan
-
R. David Murray
-
Serhiy Storchaka
-
Éric Araujo