[Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen

Donald Stufft donald at stufft.io
Thu Oct 29 14:45:09 EDT 2015

On October 29, 2015 at 2:28:02 PM, Paul Moore (p.f.moore=40gmail.com) wro=
> On 29 October 2015 at 17:36, Donald Stufft wrote:
> > On October 29, 2015 at 1:32:31 PM, Nathaniel Smith (njs=40pobox.com) =
> >> > (I know saying that last part out loud will probably just cause
> >> someone to pop out of the woodwork and explain how shadowing the
> >> sys module is a great idea and they do it all the time or whatever,
> >> but I guess I'll take that risk :-).)
> >
> > How about someone saying that I wish the standard library was more ea=
sily shadow-able=3F =20
> ;) In 2.x it=E2=80=99s easy to do it by accident because of the implici=
t relative imports which =20
> is of course, crummy. I think it=E2=80=99d be nice if a package could o=
verride the standard library =20
> in a sane way though. Like how pdb++ does.
> >
> > This is already possible if you install stuff as .eggs and the world =
hasn=E2=80=99t burned down, =20
> it=E2=80=99s just not easily possible if you don=E2=80=99t install as e=
> =20
> The idle issues seem to me to demonstrate that shadowing the stdlib is
> a bad idea. Of course, consenting adults, and if you override you're
> responsible for correctly replacing the functionality, and all that,
> but honestly, I don't think it needs to be *easy* to shadow the stdlib
> - there's nothing wrong with it being an =22advanced=22 technique that
> people have to understand in order to use.

I think the idle problem demonstrates that shadowing the stdlib accidenta=
lly is a bad idea, and the old implicit relative imports made it extremel=
y trivial to do so. Having =60.=60 first on the path still makes it trivi=
al to do so, but less so than the old implicit relative imports did.

> =20
> And I don't see pdb++ as a good motivating use case. When are you
> going to find code with pdb calls in, that you can't modify to add
> =22import pdbpp as pdb=22 at the top, if you want the advanced
> functionality=3F (And if you can't modify the code, what's the point of=

> debugging it anyway=3F)

Every test runner that includes a =E2=80=94pdb flag that will automatical=
ly invoke pdb at the point of failure. If pdb++ didn=E2=80=99t force the =
shadowing of stdlib, then every single test runner would need an option l=
ike =E2=80=94pdb++ instead of =E2=80=94pdb (or it would need to unconditi=
onally prefer pdb++ over pdb if you had =E2=80=94pdb). But what if I don=E2=
=80=99t like pdb++ and I want to use some other pdb replacement=3F Possib=
ly one I=E2=80=99m writing myself=3F Either I can convince every test run=
ner to support every pdb replacement or I need some sort of central regis=
try where I can insert my pdb as a replacement to the =E2=80=9Creal=E2=80=
=9D pdb. The sys.path and import system is one such registry that already=
 exists and is (my opinion) the logical choice.

An example close to home for me where it would have been immensely useful=
 is for adding pip to the standard library. I was opposed to doing that b=
ecause I wanted people to be able to upgrade their pip without having to =
upgrade their Python (because we=E2=80=99ve seen how well it worked for d=
istutils). Because there=E2=80=99s not a great way to do that, we ended u=
p having to go with ensurepip instead which is more complicated and cause=
d policy problems for a lot of downstream in a way that just adding pip t=
o the standard library wouldn=E2=80=99t have. Instead we could have simpl=
y added pip to the standard library, and then upgrading pip would have dr=
opped it into site-packages and that could have taken precedence, allowin=
g people to upgrade easily.

> =20
> I think the reason the world hasn't burned down because you can shadow
> the stdlib with eggs, is because people *don't*. Or to rephrase,
> because people don't actually need to shadow the stdlib. But I have no
> hard data to support that contention, so if you do, I'll concede the
> point (but I don't think there are many packages that *have* to be
> shipped as eggs these days, if they did they wouldn't work with pip,
> so doesn't that mean that nobody is using the ability of eggs to
> shadow the stdlib=3F)
> =20

People route around technical limitations. In pdb++ case they used to bre=
ak the ability of pip to prevent installation of egg by munging sys.argv =
to remove the =E2=80=94single-version-externally-managed. Now they instal=
l a .pth hack to force it to the front of sys.path.=C2=A0Other examples o=
f ways people route around this is by using monkey patching (either expli=
citly invoked or implicitly via shenagins), an example of this is setupto=
ols monkeypatching distutils on import (which pip uses to override what d=
istutils is). Another example is a more explicit shadowing which is done =
by having some slightly differently named thing available and expecting a=
ll users to do something like try: import unittest2 as unittest except Im=
portError: import unittest.

So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad=
ow the standard library, they just have various ways to do it that have s=
everal gotchas and require people to generally hack around the limitation=

Donald Stufft
PGP: 0x6E3CBCE93372DC=46A // 7C6B 7C5D 5E2B 6356 A926 =4604=46 6E3C BCE9 =
3372 DC=46A

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Message signed with OpenPGP using AMPGpg
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151029/8620cedd/attachment.sig>

More information about the Python-Dev mailing list