[Python-Dev] More optimisation ideas

[Andrew Barnert]

On Feb 1, 2016, at 09:59, mike.romberg at comcast.net wrote:
> 
>  If the stdlib were to use implicit namespace packages
> ( https://www.python.org/dev/peps/pep-0420/ ) and the various
> loaders/importers as well, then python could do what I've done with an
> embedded python application for years.  Freeze the stdlib (or put it
> in a zipfile or whatever is fast).  Then arrange PYTHONPATH to first
> look on the filesystem and then look in the frozen/ziped storage.

This is a great solution for experienced developers, but I think it would be pretty bad for novices or transplants from other languages (maybe even including Python 2).

There are already multiple duplicate questions every month on StackOverflow from people asking "how do I find the source to stdlib module X". The canonical answer starts off by explaining how to import the module and use its __file__, which everyone is able to handle. If we have to instead explain how to work out the .py name from the qualified module name, how to work out the stdlib path from sys.path, and then how to find the source from those two things, with the caveat that it may not be installed at all on some platforms, and how to make sure what they're asking about really is a stdlib module, and how to make sure they aren't shadowing it with a module elsewhere on sys.path, that's a lot more complicated. Especially when you consider that some people on Windows and Mac are writing Python scripts without ever learning how to use the terminal or find their Python packages via Explorer/Finder. 

And meanwhile, other people would be asking why their app runs slower on one machine than another, because they didn't expect that installing python-dev on top of python would slow down startup.

Finally, on Linux and Mac, the stdlib will usually be somewhere that's not user-writable--and we shouldn't expect users to have to mess with stuff in /usr/lib or /System/Library even if they do have sudo access. Of course we could put a "stdlib shadow" location on the sys.path and configure it for /usr/local/lib and /Library and/or for somewhere in -, but that just makes the lookup proceed even more complicated--not to mention that we've just added three stat calls to remove one open, at which point the optimization has probably become a pessimization.