If you shadow a module in the standard library that IDLE depends on, bad things happen
see the following: lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py lac@smartwheels:~/junk$ idle-python3.5 hello there Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.5/idlelib/run.py", line 10, in <module> from idlelib import CallTips File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in <module> from idlelib.HyperParser import HyperParser File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in <module> _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_") AttributeError: module 'string' has no attribute 'ascii_letters' IDLE then produces a popup that says: IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess por personal firewall software is blocking the connection. <ok> -------- I think that life would be a whole lot easier for people if instead we got a message: Warning: local file /u/lac/junk/string.py shadows module named string in the Standard Library I think that it is python exec that would have to do this -- though of course the popup could also warn about shadowing in general, instead of sending people on wild goose chases over their firewalls. Would this be hard to do? Laura
On Thu, 29 Oct 2015 at 09:00 Laura Creighton <lac@openend.se> wrote:
see the following: lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py lac@smartwheels:~/junk$ idle-python3.5 hello there Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.5/idlelib/run.py", line 10, in <module> from idlelib import CallTips File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in <module> from idlelib.HyperParser import HyperParser File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in <module> _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_") AttributeError: module 'string' has no attribute 'ascii_letters'
IDLE then produces a popup that says:
IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess por personal firewall software is blocking the connection. <ok>
--------
I think that life would be a whole lot easier for people if instead we got a message:
Warning: local file /u/lac/junk/string.py shadows module named string in the Standard Library
I think that it is python exec that would have to do this -- though of course the popup could also warn about shadowing in general, instead of sending people on wild goose chases over their firewalls.
Would this be hard to do?
It would require a custom importer or overriding __import__ but it's doable.
On Oct 29, 2015 10:22 AM, "Brett Cannon" <brett@python.org> wrote:
On Thu, 29 Oct 2015 at 09:00 Laura Creighton <lac@openend.se> wrote:
see the following: lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py lac@smartwheels:~/junk$ idle-python3.5 hello there Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.5/idlelib/run.py", line 10, in <module> from idlelib import CallTips File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in <module> from idlelib.HyperParser import HyperParser File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in <module> _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits +
AttributeError: module 'string' has no attribute 'ascii_letters'
IDLE then produces a popup that says:
IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess por personal firewall software is blocking the connection. <ok>
--------
I think that life would be a whole lot easier for people if instead we got a message:
Warning: local file /u/lac/junk/string.py shadows module named string in
"_") the
Standard Library
I think that it is python exec that would have to do this -- though of course the popup could also warn about shadowing in general, instead of sending people on wild goose chases over their firewalls.
Would this be hard to do?
It would require a custom importer or overriding __import__ but it's doable.
Is there any reason not to issue such warnings by default in the standard importer? The issue of accidentally shadowing stdlib modules isn't restricted to idle, it's difficult for idle to handle correctly (how do you define a custom importer if you don't yet have access to the stdlib?), and it's not like there's any legitimate reason to want string.py in the working directory to auto-monkeypatch stdlib string. (I know saying that last part out loud will probably just cause someone to pop out of the woodwork and explain how shadowing the sys module is a great idea and they do it all the time or whatever, but I guess I'll take that risk :-).) -n
On October 29, 2015 at 1:32:31 PM, Nathaniel Smith (njs@pobox.com) wrote:
(I know saying that last part out loud will probably just cause someone to pop out of the woodwork and explain how shadowing the sys module is a great idea and they do it all the time or whatever, but I guess I'll take that risk :-).)
How about someone saying that I wish the standard library was more easily shadow-able? ;) In 2.x it’s easy to do it by accident because of the implicit relative imports which is of course, crummy. I think it’d be nice if a package could override the standard library in a sane way though. Like how pdb++ does. This is already possible if you install stuff as .eggs and the world hasn’t burned down, it’s just not easily possible if you don’t install as eggs. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 October 2015 at 17:36, Donald Stufft <donald@stufft.io> wrote:
On October 29, 2015 at 1:32:31 PM, Nathaniel Smith (njs@pobox.com) wrote:
(I know saying that last part out loud will probably just cause someone to pop out of the woodwork and explain how shadowing the sys module is a great idea and they do it all the time or whatever, but I guess I'll take that risk :-).)
How about someone saying that I wish the standard library was more easily shadow-able? ;) In 2.x it’s easy to do it by accident because of the implicit relative imports which is of course, crummy. I think it’d be nice if a package could override the standard library in a sane way though. Like how pdb++ does.
This is already possible if you install stuff as .eggs and the world hasn’t burned down, it’s just not easily possible if you don’t install as eggs.
The idle issues seem to me to demonstrate that shadowing the stdlib is a bad idea. Of course, consenting adults, and if you override you're responsible for correctly replacing the functionality, and all that, but honestly, I don't think it needs to be *easy* to shadow the stdlib - there's nothing wrong with it being an "advanced" technique that people have to understand in order to use. And I don't see pdb++ as a good motivating use case. When are you going to find code with pdb calls in, that you can't modify to add "import pdbpp as pdb" at the top, if you want the advanced functionality? (And if you can't modify the code, what's the point of debugging it anyway?) I think the reason the world hasn't burned down because you can shadow the stdlib with eggs, is because people *don't*. Or to rephrase, because people don't actually need to shadow the stdlib. But I have no hard data to support that contention, so if you do, I'll concede the point (but I don't think there are many packages that *have* to be shipped as eggs these days, if they did they wouldn't work with pip, so doesn't that mean that nobody is using the ability of eggs to shadow the stdlib?) Paul
On October 29, 2015 at 2:28:02 PM, Paul Moore (p.f.moore=40gmail.com) wro= te: > On 29 October 2015 at 17:36, Donald Stufft wrote: > > On October 29, 2015 at 1:32:31 PM, Nathaniel Smith (njs=40pobox.com) = wrote: > >> > (I know saying that last part out loud will probably just cause > >> someone to pop out of the woodwork and explain how shadowing the > >> sys module is a great idea and they do it all the time or whatever, > >> but I guess I'll take that risk :-).) > > > > How about someone saying that I wish the standard library was more ea= sily shadow-able=3F =20 > ;) In 2.x it=E2=80=99s easy to do it by accident because of the implici= t relative imports which =20 > is of course, crummy. I think it=E2=80=99d be nice if a package could o= verride the standard library =20 > in a sane way though. Like how pdb++ does. > > > > This is already possible if you install stuff as .eggs and the world = hasn=E2=80=99t burned down, =20 > it=E2=80=99s just not easily possible if you don=E2=80=99t install as e= ggs. > =20 > The idle issues seem to me to demonstrate that shadowing the stdlib is > a bad idea. Of course, consenting adults, and if you override you're > responsible for correctly replacing the functionality, and all that, > but honestly, I don't think it needs to be *easy* to shadow the stdlib > - there's nothing wrong with it being an =22advanced=22 technique that > people have to understand in order to use. I think the idle problem demonstrates that shadowing the stdlib accidenta= lly is a bad idea, and the old implicit relative imports made it extremel= y trivial to do so. Having =60.=60 first on the path still makes it trivi= al to do so, but less so than the old implicit relative imports did. > =20 > And I don't see pdb++ as a good motivating use case. When are you > going to find code with pdb calls in, that you can't modify to add > =22import pdbpp as pdb=22 at the top, if you want the advanced > functionality=3F (And if you can't modify the code, what's the point of= > debugging it anyway=3F) Every test runner that includes a =E2=80=94pdb flag that will automatical= ly invoke pdb at the point of failure. If pdb++ didn=E2=80=99t force the = shadowing of stdlib, then every single test runner would need an option l= ike =E2=80=94pdb++ instead of =E2=80=94pdb (or it would need to unconditi= onally prefer pdb++ over pdb if you had =E2=80=94pdb). But what if I don=E2= =80=99t like pdb++ and I want to use some other pdb replacement=3F Possib= ly one I=E2=80=99m writing myself=3F Either I can convince every test run= ner to support every pdb replacement or I need some sort of central regis= try where I can insert my pdb as a replacement to the =E2=80=9Creal=E2=80= =9D pdb. The sys.path and import system is one such registry that already= exists and is (my opinion) the logical choice. An example close to home for me where it would have been immensely useful= is for adding pip to the standard library. I was opposed to doing that b= ecause I wanted people to be able to upgrade their pip without having to = upgrade their Python (because we=E2=80=99ve seen how well it worked for d= istutils). Because there=E2=80=99s not a great way to do that, we ended u= p having to go with ensurepip instead which is more complicated and cause= d policy problems for a lot of downstream in a way that just adding pip t= o the standard library wouldn=E2=80=99t have. Instead we could have simpl= y added pip to the standard library, and then upgrading pip would have dr= opped it into site-packages and that could have taken precedence, allowin= g people to upgrade easily. > =20 > I think the reason the world hasn't burned down because you can shadow > the stdlib with eggs, is because people *don't*. Or to rephrase, > because people don't actually need to shadow the stdlib. But I have no > hard data to support that contention, so if you do, I'll concede the > point (but I don't think there are many packages that *have* to be > shipped as eggs these days, if they did they wouldn't work with pip, > so doesn't that mean that nobody is using the ability of eggs to > shadow the stdlib=3F) > =20 People route around technical limitations. In pdb++ case they used to bre= ak the ability of pip to prevent installation of egg by munging sys.argv = to remove the =E2=80=94single-version-externally-managed. Now they instal= l a .pth hack to force it to the front of sys.path.=C2=A0Other examples o= f ways people route around this is by using monkey patching (either expli= citly invoked or implicitly via shenagins), an example of this is setupto= ols monkeypatching distutils on import (which pip uses to override what d= istutils is). Another example is a more explicit shadowing which is done = by having some slightly differently named thing available and expecting a= ll users to do something like try: import unittest2 as unittest except Im= portError: import unittest. So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad= ow the standard library, they just have various ways to do it that have s= everal gotchas and require people to generally hack around the limitation= .=C2=A0 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DC=46A // 7C6B 7C5D 5E2B 6356 A926 =4604=46 6E3C BCE9 = 3372 DC=46A
On Oct 29, 2015 11:45 AM, "Donald Stufft" <donald@stufft.io> wrote:
Every test runner that includes a =E2=80=94pdb flag that will automatical= ly invoke pdb at the point of failure. If pdb++ didn=E2=80=99t force the = shadowing of stdlib, then every single test runner would need an option l= ike =E2=80=94pdb++ instead of =E2=80=94pdb (or it would need to unconditi= onally prefer pdb++ over pdb if you had =E2=80=94pdb). But what if I
don=E2=
=80=99t like pdb++ and I want to use some other pdb replacement=3F Possib= ly one I=E2=80=99m writing myself=3F Either I can convince every test run= ner to support every pdb replacement or I need some sort of central regis= try where I can insert my pdb as a replacement to the =E2=80=9Creal=E2=80= =9D pdb. The sys.path and import system is one such registry that already= exists and is (my opinion) the logical choice.
This strikes me as an argument for adding some sort of explicit plugin support to pdb, or defining a standard interface that test tools could use to query for installed debuggers. (Presumably via some entry point that alternative debugger packages would export?) In any case it's orthogonal to the issue of accidental shadowing. It seems unlikely that at this late date we'll be swapping the order of sys.path so site-packages comes before the stdlib, and I doubt there are a lot of people who are using pdb++ via the mechanism of making sure that it's always present in their working directory... -n
On 29 October 2015 at 18:45, Donald Stufft <donald@stufft.io> wrote:
So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad= ow the standard library, they just have various ways to do it that have s= everal gotchas and require people to generally hack around the limitation= .=C2=A0
(Your mailer or mine seems to have gone weird with encoding...) There's a difference between the opting into using (say) pdb++ at runtime, and at install time. The problem with the sort of shadowing eggs did (and people try to emulate) is that simply *installing* a package changes the behaviour of programs that don't use that package, by virtue of the fact that it's there at all. I'd much rather pdb++ or anything similar allowed the user to opt in on demand. For example, it could put a file pdb.py containing "from pdbpp import *" into a subdirectory, and direct users who want to replace pdb with pdbpp to add that directory to PYTHONPATH. Just don't do that at install time, leaving the user needing to uninstall pdbpp to opt out of the behaviour. As I said, I'm not against people being *able* to shadow the stdlib, I just don't want to see it being *easy*, nor do I want it to be the norm. Paul PS My experience with a similar case is pyreadline on Windows, which hooks into the stdlib readline functionality when installed. That's a PITA, because mostly I don't want pyreadline (I prefer the default Windows command line editing, despite it having less functionality) but I want pyreadline when using ipython. There's no way for me to get that behaviour because (py)readline is enabled by installing it, not by user choice at runtime.
In a message of Thu, 29 Oct 2015 19:30:09 +0000, Paul Moore writes:
On 29 October 2015 at 18:45, Donald Stufft <donald@stufft.io> wrote:
So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad= ow the standard library, they just have various ways to do it that have s= everal gotchas and require people to generally hack around the limitation= .=C2=A0
(Your mailer or mine seems to have gone weird with encoding...)
Dstuffts. I see this problem too Laura
On 10/29/2015 12:30 PM, Paul Moore wrote:
On 29 October 2015 at 18:45, Donald Stufft <donald@stufft.io> wrote:
So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad= ow the standard library, they just have various ways to do it that have s= everal gotchas and require people to generally hack around the limitation= .=C2=A0 (Your mailer or mine seems to have gone weird with encoding...)
There's a difference between the opting into using (say) pdb++ at runtime, and at install time. The problem with the sort of shadowing eggs did (and people try to emulate) is that simply *installing* a package changes the behaviour of programs that don't use that package, by virtue of the fact that it's there at all.
I'd much rather pdb++ or anything similar allowed the user to opt in on demand. For example, it could put a file pdb.py containing "from pdbpp import *" into a subdirectory, and direct users who want to replace pdb with pdbpp to add that directory to PYTHONPATH. Just don't do that at install time, leaving the user needing to uninstall pdbpp to opt out of the behaviour.
As I said, I'm not against people being *able* to shadow the stdlib, I just don't want to see it being *easy*, nor do I want it to be the norm.
Paul
PS My experience with a similar case is pyreadline on Windows, which hooks into the stdlib readline functionality when installed. That's a PITA, because mostly I don't want pyreadline (I prefer the default Windows command line editing, despite it having less functionality) but I want pyreadline when using ipython. There's no way for me to get that behaviour because (py)readline is enabled by installing it, not by user choice at runtime.
+1 Although, I wouldn't mind if it were easy & documented, like say having both shadow-packages before stdlib and site-packages after it, so that multiple, super-tricky and annoying ways of doing the shadowing were not needed. But, say, packages shadow-packages would only be used if there were an extra directive somewhere to enable it, or a subset of the packages that reside there. Glenn
In a message of Thu, 29 Oct 2015 18:27:59 +0000, Paul Moore writes:
The idle issues seem to me to demonstrate that shadowing the stdlib is a bad idea. Of course, consenting adults, and if you override you're responsible for correctly replacing the functionality, and all that, but honestly, I don't think it needs to be *easy* to shadow the stdlib - there's nothing wrong with it being an "advanced" technique that people have to understand in order to use.
I am actually sick of the 'consenting adults' argument. I am dealing with '11 year old children trying to write their first, third and tenth python programs'. For the life of me I cannot see how convenience for the sort of person who has a legitimate reason to shadow the syslib should get a higher priority over these mites who are doing their damndest to write python despite natural language barriers and the fact that their peers and parents think they are nuts to want to do so. (a grumpy comment from a teacher at a Swedish 'coding for kids' club. Disregard if too grumpy.) Laura
On Oct 29, 2015, at 11:46, Laura Creighton <lac@openend.se> wrote:
In a message of Thu, 29 Oct 2015 18:27:59 +0000, Paul Moore writes:
The idle issues seem to me to demonstrate that shadowing the stdlib is a bad idea. Of course, consenting adults, and if you override you're responsible for correctly replacing the functionality, and all that, but honestly, I don't think it needs to be *easy* to shadow the stdlib - there's nothing wrong with it being an "advanced" technique that people have to understand in order to use.
I am actually sick of the 'consenting adults' argument. I am dealing with '11 year old children trying to write their first, third and tenth python programs'. For the life of me I cannot see how convenience for the sort of person who has a legitimate reason to shadow the syslib should get a higher priority over these mites who are doing their damndest to write python despite natural language barriers and the fact that their peers and parents think they are nuts to want to do so.
(a grumpy comment from a teacher at a Swedish 'coding for kids' club. Disregard if too grumpy.)
+1 on **warning**, warning still allow people to shadow stdlib, and for people who have **legitimate** reasons to shadow, we can always find a solution to to tag a module as “yes I know I am shadowing, I am doing that on purpose”. StdlibShadowWarning and warning filter ? Also a warning would be useful for people to discover that some Stdlib modules exist, and maybe explore them. -- M
On 29 October 2015 at 18:46, Laura Creighton <lac@openend.se> wrote:
In a message of Thu, 29 Oct 2015 18:27:59 +0000, Paul Moore writes:
The idle issues seem to me to demonstrate that shadowing the stdlib is a bad idea. Of course, consenting adults, and if you override you're responsible for correctly replacing the functionality, and all that, but honestly, I don't think it needs to be *easy* to shadow the stdlib - there's nothing wrong with it being an "advanced" technique that people have to understand in order to use.
I am actually sick of the 'consenting adults' argument. I am dealing with '11 year old children trying to write their first, third and tenth python programs'. For the life of me I cannot see how convenience for the sort of person who has a legitimate reason to shadow the syslib should get a higher priority over these mites who are doing their damndest to write python despite natural language barriers and the fact that their peers and parents think they are nuts to want to do so.
That's actually a very good point, and I agree totally. To my mind, the point about "consenting adults" (and when I referred to that I was anticipating others using that argument, not proposing it myself) is that we don't *prevent* people from doing weird and wonderful things. But conversely, it's not a reason for making it *easy* to do such things. Quite the opposite - a "consenting adult" should be assumed to be capable of writing an import hook, or manipulating sys.path, or whatever. Paul
In a message of Thu, 29 Oct 2015 19:13:08 +0000, Paul Moore writes:
I am actually sick of the 'consenting adults' argument. I am dealing with '11 year old children trying to write their first, third and tenth python programs'. For the life of me I cannot see how convenience for the sort of person who has a legitimate reason to shadow the syslib should get a higher priority over these mites who are doing their damndest to write python despite natural language barriers and the fact that their peers and parents think they are nuts to want to do so.
That's actually a very good point, and I agree totally. To my mind, the point about "consenting adults" (and when I referred to that I was anticipating others using that argument, not proposing it myself) is that we don't *prevent* people from doing weird and wonderful things. But conversely, it's not a reason for making it *easy* to do such things. Quite the opposite - a "consenting adult" should be assumed to be capable of writing an import hook, or manipulating sys.path, or whatever.
Paul
Hmmm, I think the set of 'consenting adults who cannot write an import hook' is rather large. But all I am asking for is a warning -- and it would be good if Idle noticed the warning and before it fell over dead with its message of firewalls it would mention the warning again, as a problem to look for. It will bugger up doctests for people who legitimately shadow the stdlib and now get a new warning. Anybody else be seriously inconvenienced if we do this? I cannot think of any, but then legitimately shadowing the stdlib in not on the list of things I have done. Perhaps Dstufft has ideas on this line. Laura
On 10/29/2015 11:59 AM, Laura Creighton wrote:
see the following: lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py lac@smartwheels:~/junk$ idle-python3.5 hello there Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.5/idlelib/run.py", line 10, in <module> from idlelib import CallTips File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in <module> from idlelib.HyperParser import HyperParser File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in <module> _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_") AttributeError: module 'string' has no attribute 'ascii_letters'
IDLE then produces a popup that says:
IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess por personal firewall software is blocking the connection. <ok>
--------
I think that life would be a whole lot easier for people if instead we got a message:
Warning: local file /u/lac/junk/string.py shadows module named string in the Standard Library
I think that it is python exec that would have to do this -- though of course the popup could also warn about shadowing in general, instead of sending people on wild goose chases over their firewalls.
Would this be hard to do?
Leaving IDLE aside, the reason '' is added to sys.path is so that people can import their own modules. This is very useful. Shadowing is the result of putting it at the front. I have long thought this a dubious choice. If '' were instead appended, people could still import modules that did not duplicate stdlib names. Anyone who wanted shadowing could move '' to the front. But then shadowing would be intentional, not an accident. -- Terry Jan Reedy
On 30.10.15 03:22, Terry Reedy wrote:
Leaving IDLE aside, the reason '' is added to sys.path is so that people can import their own modules. This is very useful. Shadowing is the result of putting it at the front. I have long thought this a dubious choice. If '' were instead appended, people could still import modules that did not duplicate stdlib names. Anyone who wanted shadowing could move '' to the front. But then shadowing would be intentional, not an accident.
LGTM. AFAIK the sys module can't be shadowed.
On Fri, 30 Oct 2015 at 21:42 Terry Reedy <tjreedy@udel.edu> wrote:
On 10/30/2015 3:21 AM, Serhiy Storchaka wrote:
AFAIK the sys module can't be shadowed.
I tried it and it seems to be true of builtins in general.
The importer on sys.meta_path that handles built-ins is earlier than the one that handles sys.path (technically you can shadow them if you changed sys.meta_path).
On Oct 29, 2015 6:26 PM, "Terry Reedy" <tjreedy@udel.edu> wrote:
On 10/29/2015 11:59 AM, Laura Creighton wrote:
see the following: lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py lac@smartwheels:~/junk$ idle-python3.5 hello there Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.5/idlelib/run.py", line 10, in <module> from idlelib import CallTips File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in <module> from idlelib.HyperParser import HyperParser File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in <module> _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits +
AttributeError: module 'string' has no attribute 'ascii_letters'
IDLE then produces a popup that says:
IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess por personal firewall software is blocking the connection. <ok>
--------
I think that life would be a whole lot easier for people if instead we got a message:
Warning: local file /u/lac/junk/string.py shadows module named string in
Standard Library
I think that it is python exec that would have to do this -- though of course the popup could also warn about shadowing in general, instead of sending people on wild goose chases over their firewalls.
Would this be hard to do?
Leaving IDLE aside, the reason '' is added to sys.path is so that people can import their own modules. This is very useful. Shadowing is the result of putting it at the front. I have long thought this a dubious choice. If '' were instead appended, people could still import modules
"_") the that did not duplicate stdlib names. Anyone who wanted shadowing could move '' to the front. But then shadowing would be intentional, not an accident. Unfortunately I think that (among other things) there are a lot of scripts out there that blindly do sys.path.pop(0) to remove the "" entry, so the backcompat costs of changing this would probably be catastrophic. -n
On 30.10.15 09:57, Nathaniel Smith wrote:
Unfortunately I think that (among other things) there are a lot of scripts out there that blindly do sys.path.pop(0) to remove the "" entry, so the backcompat costs of changing this would probably be catastrophic.
You are right. There are too much occurrences even in public libraries. https://code.openhub.net/search?s=%22sys.path.pop(0)%22&p=0 Possible workaround is to add fake path (or a duplicate of system path) at the start of sys.path. Then dropping first element will not break the script.
CC'ing Python-Ideas. Follow-ups to Python-Ideas please. On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote:
Leaving IDLE aside, the reason '' is added to sys.path is so that people can import their own modules. This is very useful. Shadowing is the result of putting it at the front. I have long thought this a dubious choice. If '' were instead appended, people could still import modules that did not duplicate stdlib names. Anyone who wanted shadowing could move '' to the front. But then shadowing would be intentional, not an accident.
Terry is right. Shadowing should be possible, and it should require a deliberate decision on the part of the programmer. Consider the shell, say, bash or similar. My understanding is that the shell PATH deliberately excludes the current directory because of the possibility of malicious software shadowing usual commands in /bin etc. If you want to run an executable in the current directory, you have to explicitly provide the path to it: ./myscript rather than just myscript. Now Python isn't exactly the shell, and so I'm not proposing that Python does the same thing. But surely we can agree on the following? - Shadowing explicitly installed packages, including the stdlib, is *occasionally* useful. - But when shadowing occurs, it is *nearly always* accidental. - Such accidental shadowing often causes problems. - And further more, debugging shadowing problems is sometimes tricky even for experienced coders, and almost impossible for beginners. (It's not until you've been burned once or thrice by shadowing that you recognise the symptoms, at which point it is then usually easy to debug.) - Hence, we should put the onus on those who want to shadow installed packages) to do so *explicitly*, or at least make it easier to avoid accidental shadowing. I propose the following two changes: (1) Beginning with Python 3.6, the default is that the current directory is put at the end of sys.path rather than the beginning. Instead of: >>> print(sys.path) ['', '/this', '/that', '/another'] we will have this instead: >>> print(sys.path) ['/this', '/that', '/another', ''] Those who don't shadow installed packages won't notice any difference. Scripts which deliberately or unintentionally shadow installed packages will break from this change. I don't have a problem with this. You can't fix harmful behaviour without breaking code that depends on that harmful behaviour. Additionally, I expect that those who rely on the current behaviour will be in a small minority, much fewer than those who will be bitten by accidental shadowing into the indefinite future. And if you want the old behaviour back, it is easy to do so, by changing the path before doing your imports: import sys if sys.path[-1] == "": sys.path = [""] + sys.path[:-1] or equivalent. I do not belive that it is onerous for those who want shadowing to have to take steps to do so explicitly. That can be added to your scripts on a case-by-case basis, or your PYTHONSTARTUP file, by modifying your site.py, or (I think) by putting the code into the sitecustomize or usercustomize modules. (2) IDLE doesn't need to wait for Python 3.6 to make this change. I believe that IDLE is permitted to make backwards incompatible changes in minor releases, so there is no reason why it can't change the path effective immediately. That's a simpler fix than scanning the entire path, raising warnings (which beginners won't understand and will either ignore or panic over) or other complex solutions. It may not prevent *every* shadowing incident, but it will improve the situation immeasurably. Thoughts? -- Steve
participants (10)
-
Brett Cannon -
Donald Stufft -
Glenn Linderman -
Laura Creighton -
Matthias Bussonnier -
Nathaniel Smith -
Paul Moore -
Serhiy Storchaka -
Steven D'Aprano -
Terry Reedy