In a recent discussion in a SF patch, I noticed that PEP 328* only seems to support relative imports within packages, while bare import statements use the entirety of sys.path, not solving the shadowing of standard library module names. I have certainly forgotten bits of discussion from last spring, but I would offer that Python could offer standard library shadowing protection through the use of an extended PEP 328 semantic. More specifically; after a 'from __future__ import absolute_import' statement, any import in the module performing "import foo" will only check for foo in the standard library, and the use of the leading period, "from . import foo", the will signify relative to the current path. ** The lack of a 'from __future__ import absolute_import' statement in a module will not change the import semantic of that module. This allows current code to continue to work, and for those who want to choose names which shadow the standard library modules, a way of dealing with their choices. Further, in the case of PEP 328, the package relative imports were to become the default in 2.6 (with deprecation in 2.5, availability in 2.4), but with the lack of an implementation, perhaps those numbers should be incremented. If the behavior I describe is desireable, it would subsume PEP 328, and perhaps should also become the default behavior at some point in time (perhaps in the same adjusted timeline as PEP 328). Alternatively, PEP 328 could be implemented as-is, and a second future import could be defined which offers this functionality, being permanently optional (or on a different timeline) via the future import. Essentially, it would ignore the empty path "" in sys.path when the functionality has been enabled via the proper future import in the current module. - Josiah * PEP 328 first describes the use of parenthesis in import statements so that long import listings do not require backslash-escaping of newlines. It then describes a semantic for not checking sys.path when performing an import, as well as allowing parent, cousin, uncle, etc., imports via additional leading periods. "from . import foo" for sibling imports, "from .. import foo" for parent imports, etc. ** I describe the semantic as being a per-module option, as this is the only backwards-compatible mechanism in the near future (Python 2.5). An import implementation would merely check the existance of the proper name binding -> object pair in the importer's global namespace. The standard library would need to be modified if or when current behavior is deprecated (this would extend the modules needing to be modified due to PEP 328).
In a recent discussion in a SF patch, I noticed that PEP 328* only seems to support relative imports within packages, while bare import statements use the entirety of sys.path, not solving the shadowing of standard library module names.
Hm. I'm not convinced that there is a *problem* with shadowing of standard library module names. You shouldn't pick a module name that shadows a standard library module, or if you do you shouldn't want to be able to still use those modules that you're shadowing. Anything else is just asking for trouble.
I have certainly forgotten bits of discussion from last spring, but I would offer that Python could offer standard library shadowing protection through the use of an extended PEP 328 semantic.
More specifically; after a 'from __future__ import absolute_import' statement, any import in the module performing "import foo" will only check for foo in the standard library, and the use of the leading period, "from . import foo", the will signify relative to the current path. **
And how exactly do you define "the standard library"? Anything that's on sys.path? That would seem the only reasonable interpretation to me. So I take it that you want the "script directory" off that path. (Let's for the sake of argument call it ".".)
The lack of a 'from __future__ import absolute_import' statement in a module will not change the import semantic of that module.
It's hard to imagine how this would work. sys.path is global, so either "." is on it, or it isn't. So things in "." are either considered part of the standard library, or they are not; this can't be made dependent on the module's importation of something from __future__.
This allows current code to continue to work, and for those who want to choose names which shadow the standard library modules, a way of dealing with their choices.
My suggested way of dealing with their choices is summarized in the first paragraph of my reply above.
Further, in the case of PEP 328, the package relative imports were to become the default in 2.6 (with deprecation in 2.5, availability in 2.4), but with the lack of an implementation, perhaps those numbers should be incremented. If the behavior I describe is desireable, it would subsume PEP 328, and perhaps should also become the default behavior at some point in time (perhaps in the same adjusted timeline as PEP 328).
That's a separate issue; the absolute/relative import part of PEP 328 didn't make it into 2.4 so I suppose we should ++ all those version numbers.
Alternatively, PEP 328 could be implemented as-is, and a second future import could be defined which offers this functionality, being permanently optional (or on a different timeline) via the future import.
I don't like permanently optional language features; that causes too much confusion. I'd much rather settle on clear semantics that everyone can understand (even if they may disagree). But I certainly would prefer that the proposed feature becomes a separate PEP which can be discussed, accepted or rejected, and implemented separately from PEP 328, which is complete and accepted and just awaiting someone to implement it.
Essentially, it would ignore the empty path "" in sys.path when the functionality has been enabled via the proper future import in the current module.
But it's not always "" -- it's the directory where the "main" script was found. Let me explain the biggest problem I see for your proposal: what would be the canonical name for a module imported using your "new relative semantics"? Remember, the canonical name of a module is its __name__ attribute, and the key that finds it in the sys.modules dict. Because there's only one sys.modules dict (barring restricted execution sandboxes), the canonical name must be unique. So if there's a standard library module string, its canonical name is "string". Now suppose you have your own non-standard-linrary module read from a file string.py. What should its canonical name be? It can't be "string" because that's already reserved for the standard library module name. The best solution I can think of for this off the top of my head is to somehow allow for the arrangement of a pseudo-package named __main__ and to make all these non-standard-library modules reside (logically) in that module. If you can write a PEP along those lines you may be on to something -- but I expect that the way to turn it on is not to import something from __future__ but perhaps from __main__. I'm not exactly sure how to get "." off sys.path, but maybe you can think about that for your PEP proposal. What do you say?
- Josiah
* PEP 328 first describes the use of parenthesis in import statements so that long import listings do not require backslash-escaping of newlines. It then describes a semantic for not checking sys.path when performing an import, as well as allowing parent, cousin, uncle, etc., imports via additional leading periods. "from . import foo" for sibling imports, "from .. import foo" for parent imports, etc.
** I describe the semantic as being a per-module option, as this is the only backwards-compatible mechanism in the near future (Python 2.5). An import implementation would merely check the existance of the proper name binding -> object pair in the importer's global namespace. The standard library would need to be modified if or when current behavior is deprecated (this would extend the modules needing to be modified due to PEP 328).
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <gvanrossum@gmail.com> wrote:
In a recent discussion in a SF patch, I noticed that PEP 328* only seems to support relative imports within packages, while bare import statements use the entirety of sys.path, not solving the shadowing of standard library module names.
Hm. I'm not convinced that there is a *problem* with shadowing of standard library module names. You shouldn't pick a module name that shadows a standard library module, or if you do you shouldn't want to be able to still use those modules that you're shadowing. Anything else is just asking for trouble.
While I personally don't tend to use names previously existing in the standard library, seemingly a large number of people do, hence the not-so-rare threads on comp.lang.python which ask about such things.
More specifically; after a 'from __future__ import absolute_import' statement, any import in the module performing "import foo" will only check for foo in the standard library, and the use of the leading period, "from . import foo", the will signify relative to the current path. **
And how exactly do you define "the standard library"? Anything that's on sys.path? That would seem the only reasonable interpretation to me. So I take it that you want the "script directory" off that path. (Let's for the sake of argument call it ".".)
Sounds reasonable to me, with one caveat; if one were to consider everything on sys.path to be in the standard library, then every script ever written for Python, which doesn't remove the standard ''/'.' from sys.path, would be part of the standard library. I would suggest, as a replacement definition (probably with a caveat or two), that any module with a reference in the documentation, that also lies on the default sys.path, which is shipped with Python that is distributed at python.org, is part of the standard library.
The lack of a 'from __future__ import absolute_import' statement in a module will not change the import semantic of that module.
It's hard to imagine how this would work. sys.path is global, so either "." is on it, or it isn't. So things in "." are either considered part of the standard library, or they are not; this can't be made dependent on the module's importation of something from __future__.
Perhaps not, but in the process of importing a module into a namespace, one can check for the existance of the object imported from __future__, and ignore or not the "." entry in sys.path.
This allows current code to continue to work, and for those who want to choose names which shadow the standard library modules, a way of dealing with their choices.
My suggested way of dealing with their choices is summarized in the first paragraph of my reply above.
Perfectly reasonable. I can think of examples where it would not be reasonable, but they are quite cooked *wink*.
Alternatively, PEP 328 could be implemented as-is, and a second future import could be defined which offers this functionality, being permanently optional (or on a different timeline) via the future import.
I don't like permanently optional language features; that causes too much confusion. I'd much rather settle on clear semantics that everyone can understand (even if they may disagree).
But I certainly would prefer that the proposed feature becomes a separate PEP which can be discussed, accepted or rejected, and implemented separately from PEP 328, which is complete and accepted and just awaiting someone to implement it.
Sounds good. I'll see what I can do this weekend about getting a proto-pep started.
Essentially, it would ignore the empty path "" in sys.path when the functionality has been enabled via the proper future import in the current module.
But it's not always "" -- it's the directory where the "main" script was found.
Let me explain the biggest problem I see for your proposal: what would be the canonical name for a module imported using your "new relative semantics"? Remember, the canonical name of a module is its __name__ attribute, and the key that finds it in the sys.modules dict. Because there's only one sys.modules dict (barring restricted execution sandboxes), the canonical name must be unique. So if there's a standard library module string, its canonical name is "string". Now suppose you have your own non-standard-linrary module read from a file string.py. What should its canonical name be? It can't be "string" because that's already reserved for the standard library module name.
The best solution I can think of for this off the top of my head is to somehow allow for the arrangement of a pseudo-package named __main__ and to make all these non-standard-library modules reside (logically) in that module. If you can write a PEP along those lines you may be on to something -- but I expect that the way to turn it on is not to import something from __future__ but perhaps from __main__. I'm not exactly sure how to get "." off sys.path, but maybe you can think about that for your PEP proposal. What do you say?
You make very good points about naming in sys.modules. I will think about this and place some options in the proto-pep. Thank you, - Josiah
While I personally don't tend to use names previously existing in the standard library, seemingly a large number of people do, hence the not-so-rare threads on comp.lang.python which ask about such things.
Sure. There are lots of FAQs whose answer is not "Python will have to change".
And how exactly do you define "the standard library"? Anything that's on sys.path? That would seem the only reasonable interpretation to me. So I take it that you want the "script directory" off that path. (Let's for the sake of argument call it ".".)
Sounds reasonable to me, with one caveat; if one were to consider everything on sys.path to be in the standard library, then every script ever written for Python, which doesn't remove the standard ''/'.' from sys.path, would be part of the standard library.
I would suggest, as a replacement definition (probably with a caveat or two), that any module with a reference in the documentation, that also lies on the default sys.path, which is shipped with Python that is distributed at python.org, is part of the standard library.
This is not an operational definition for the sake of what you're trying to accomplish; the import statement cannot possibly know which modules are considered standard according to this definition.
The lack of a 'from __future__ import absolute_import' statement in a module will not change the import semantic of that module.
It's hard to imagine how this would work. sys.path is global, so either "." is on it, or it isn't. So things in "." are either considered part of the standard library, or they are not; this can't be made dependent on the module's importation of something from __future__.
Perhaps not, but in the process of importing a module into a namespace, one can check for the existance of the object imported from __future__, and ignore or not the "." entry in sys.path.
How would you recognize the "." entry in sys.path? (Remember "." is just the name we give it, not its actual value.) It is not a fixed string; it is not in a fixed position; it may even have been added explicitly to sys.path in which case it should act as if its contents was part of the standard library.
This allows current code to continue to work, and for those who want to choose names which shadow the standard library modules, a way of dealing with their choices.
My suggested way of dealing with their choices is summarized in the first paragraph of my reply above.
Perfectly reasonable. I can think of examples where it would not be reasonable, but they are quite cooked *wink*.
OK, then let's drop the issue -- my initial point was "this is not broken so there's no need to fix it."
Alternatively, PEP 328 could be implemented as-is, and a second future import could be defined which offers this functionality, being permanently optional (or on a different timeline) via the future import.
I don't like permanently optional language features; that causes too much confusion. I'd much rather settle on clear semantics that everyone can understand (even if they may disagree).
But I certainly would prefer that the proposed feature becomes a separate PEP which can be discussed, accepted or rejected, and implemented separately from PEP 328, which is complete and accepted and just awaiting someone to implement it.
Sounds good. I'll see what I can do this weekend about getting a proto-pep started. [...] You make very good points about naming in sys.modules. I will think about this and place some options in the proto-pep.
I recommend that you try to produce a working implementation (using import hooks so you can write the whole thing in Python). This is likely to clarify many of the semantic problems that I have tried to hint at. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <gvanrossum@gmail.com> wrote:
Sure. There are lots of FAQs whose answer is not "Python will have to change".
And I'm not saying Python has to change either, hence the initial query and planned PEP. Boiling it down; if we could change import in such a way that made standard library imports different from standard library imports, we could fix module shadowing and perhaps gain a "bonus feature" or two. [...]
Sounds good. I'll see what I can do this weekend about getting a proto-pep started. [...] You make very good points about naming in sys.modules. I will think about this and place some options in the proto-pep.
I recommend that you try to produce a working implementation (using import hooks so you can write the whole thing in Python). This is likely to clarify many of the semantic problems that I have tried to hint at.
Will do. Thank you, - Josiah
Josiah Carlson wrote:
if we could change import in such a way that made standard library imports different from standard library imports, we could
...go on to prove that black is white and get ourselves killed by a python on the next zebra crossing. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
[Josiah Carlson]
if we could change import in such a way that made standard library imports different from standard library imports, we could
[Greg Ewing]
...go on to prove that black is white and get ourselves killed by a python on the next zebra crossing.
I was hoping that Josiah would find out for himself while doing the assigned homework of writing an implementation, saving him the effort of writing a PEP. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <gvanrossum@gmail.com> wrote:
[Josiah Carlson]
if we could change import in such a way that made standard library imports different from standard library imports, we could
[Greg Ewing]
...go on to prove that black is white and get ourselves killed by a python on the next zebra crossing.
[Guido van Rossum] I was hoping that Josiah would find out for himself while doing the assigned homework of writing an implementation, saving him the effort of writing a PEP. :-)
Gah that should have been "... in such a way that made standard library imports different from user module/package imports ...". I started on the implementation, and now know why PEP 328 probably wasn't implemented in time for 2.4; import hooks can be ugly. - Josiah
Josiah Carlson wrote:
Guido van Rossum <gvanrossum@gmail.com> wrote:
[Josiah Carlson]
if we could change import in such a way that made standard library imports different from standard library imports, we could
[Greg Ewing]
...go on to prove that black is white and get ourselves killed by a python on the next zebra crossing.
[Guido van Rossum] I was hoping that Josiah would find out for himself while doing the assigned homework of writing an implementation, saving him the effort of writing a PEP. :-)
Gah that should have been "... in such a way that made standard library imports different from user module/package imports ...".
I started on the implementation, and now know why PEP 328 probably wasn't implemented in time for 2.4; import hooks can be ugly.
- Josiah
Hear, hear. If you'd like a working database import module to play with I can oblige, thanks to the assistance of many willing hands. The import arena is full of obsolete hooks and other cruft and, while I can't help feeling that it would benefit from a complete redefinition I suspect that breakage-avoidance might require that waits until 3.0. regards Steve
participants (4)
-
Greg Ewing -
Guido van Rossum -
Josiah Carlson -
Steve Holden