[Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen

[Steven D'Aprano]

On Sun, Nov 01, 2015 at 08:53:17AM +0100, Laura Creighton wrote:
> In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes:

> >Scripts which deliberately or unintentionally shadow installed packages 
> >will break from this change. I don't have a problem with this. You can't 
> >fix harmful behaviour without breaking code that depends on that harmful 
> >behaviour. 
> 
> This is a bad idea, if you mean 'shadows anything in site-packages'.
> I write a perfectly good working program, which then silently breaks
> because somebody happens to install a site package with a name
> conflict with my code.

I'm willing to consider that "" should appear in the middle of sys.path:

[standard-library-directories, "", site-package-directories]

but that still allows accidental shadowing of third-party packages. I 
don't think that the problem of accidently shadowing numpy is less 
deserving of a solution than accidently shadowing random. They're both 
problems.

The downside of using a search path, instead of explicit full pathnames, 
is that you can have conflicts where two or more modules have the same 
name, and only the first can be imported. That's unavoidable. We can 
only try to minimize the risk of accidental shadowing, not prevent it 
altogether.


> Can you imagine being in the middle of
> writing and debugging such a thing and have everything start failing 
> because suddenly your program isn't the one being found? How long is 
> it going to take you to stop looking at your own code, and your own 
> setup for the problem and begin looking at what packages got installed 
> by anybody else sharing this machine, and what they are named?

Now just a minute, I'm not proposing that this change occur at random, 
in the middle of a debugging session. Not even in a point (micro) 
release. So the code will only break (if it breaks) when you upgrade 
from version 3.x to 3.6 or higher. Moving to a new minor version number 
is a big step, and people expect that things *could* break.

(They don't necessarily expect that they *will* break, but it's 
certainly a possibility that they might.)

As an experienced developer, what do you do when code that works in one 
version stops working in the next? Everyone I know immediately assumes 
that something has changed in the new Python version, and proceeds from 
there. In this case, they'll be right. Hopefully they will read the 
release notes. Or the documentation. Or ask on StackOverflow.

The same applies for when you install a new package: "Everything worked 
fine yesterday." "What changed?" "Well, I installed a new package with 
the same name as one of my modules." "Well there you go."

Of course we can come up with scenarios that are more confusing. Suppose 
you have a program that you don't use very often, say, once a year, that 
relies on "" being at the front of the path, and then somebody who isn't 
you installs a site-wide third-party package that now shadows one of 
your modules, and you don't find out about for six months.

But this sort of thing can already happen: any time you add a 
third-party module, it may shadow *other* third party modules that are 
found later in the path. So I'm not creating a brand new failure mode, 
I'm just changing how it applies a little.

There are all sorts of debugging strategies available to an experienced 
developer, starting from "print out module.__file__" to "use the 
debugger", which a beginner who has just written "turtle.py" would never 
think of. If I'm going to break anyone's code, I'd rather it be yours 
than theirs, since I have every confidence you can debug any problems 
fairly quickly.

I don't suggest breaking working code lightly, but the issue of 
shadowing the stdlib is an ever-present millstone around every Python 
developer's neck. The stdlib alone, never mind site packages, is too big 
to expect everyone to memorise what names they should not use. We can't 
realistically expect people to avoid shadowing. But we can shift it from 
"easy to shadow, and mostly affect those who are least able to cope" to 
"harder to shadow, and mostly affect those who can cope".



-- 
Steve