[Python-Dev] More optimisation ideas

[Terry Reedy]

On 2/4/2016 12:18 PM, Sven R. Kunze wrote:
> On 04.02.2016 14:09, Nick Coghlan wrote:
>> On 2 February 2016 at 06:39, Andrew Barnert via Python-Dev
>> <python-dev at python.org>  wrote:
>>> On Feb 1, 2016, at 09:59,mike.romberg at comcast.net  wrote:
>>>>   If the stdlib were to use implicit namespace packages
>>>> (https://www.python.org/dev/peps/pep-0420/  ) and the various
>>>> loaders/importers as well, then python could do what I've done with an
>>>> embedded python application for years.  Freeze the stdlib (or put it
>>>> in a zipfile or whatever is fast).  Then arrange PYTHONPATH to first
>>>> look on the filesystem and then look in the frozen/ziped storage.
>>> This is a great solution for experienced developers, but I think it would be pretty bad for novices or transplants from other languages (maybe even including Python 2).
>>>
>>> There are already multiple duplicate questions every month on StackOverflow from people asking "how do I find the source to stdlib module X". The canonical answer starts off by explaining how to import the module and use its __file__, which everyone is able to handle. If we have to instead explain how to work out the .py name from the qualified module name, how to work out the stdlib path from sys.path, and then how to find the source from those two things, with the caveat that it may not be installed at all on some platforms, and how to make sure what they're asking about really is a stdlib module, and how to make sure they aren't shadowing it with a module elsewhere on sys.path, that's a lot more complicated. Especially when you consider that some people on Windows and Mac are writing Py
>>>   thon scripts without ever learning how to use the terminal or find their Python packages via Explorer/Finder.
>> For folks that *do* know how to use the terminal:
>>
>> $ python3 -m inspect --details inspect
>> Target: inspect
>> Origin: /usr/lib64/python3.4/inspect.py
>> Cached: /usr/lib64/python3.4/__pycache__/inspect.cpython-34.pyc
>> Loader: <_frozen_importlib.SourceFileLoader object at 0x7f0d8d23d9b0>
>>
>> (And if they just want to *read* the source code, then leaving out
>> "--details" prints the full module source, and would work even if the
>> standard library were in a zip archive)

This is completely inadequate as a replacement for loading source into 
an editor, even if just for reading.

First, on Windows, the console defaults to 300 lines.  Print more and 
only the last 300 lines remain.  The max is buffer size is 9999.  But 
setting the buffer to that is obnoxious because the buffer is then 
padded with blank lines to make 9999 lines.  The little rectangle that 
one grabs in the scrollbar is then scaled down to almost nothing, 
becoming hard to grab.

Second is navigation.  No Find, Find-next, or Find-all.  Because of 
padding, moving to the unpadded 'bottom of file' is difficult.

Third, for a repository version, I would have to type, without error, 
instead of 'python3', some version of, for instance, some suffix of 
'F:/python/dev/35/PcBuild/<I forget>/python_d.exe'.  "<I forget>" 
depends, I believe, on the build options.

> I want to see and debug also core Python in PyCharm and this is not
> acceptable.
>
> If you want to make it opt-in, fine. But opt-out is a no-go. I have a
> side-by-side comparison as we use Java and Python in production. It's
> the *ease of access* that makes Python great compared to Java.
>
> @Andrew
> Even for experienced developers it just sucks and there are more
> important things to do.

I agree that removing stdlib python source files by default is an poor 
idea. The disk space saved is trivial.  So, for me, would be nearly all 
of the time saving.

Over recent versions, more and more source files have been linked to in 
the docs.  Guido recently approved of linking the rest.  Removing source 
contradicts this trend.

Easily loading modules, including stdlib modules, into an IDLE Editor 
Window is a documented feature that goes back to the original commit in 
Aug 2000.  We not not usually break stdlib features without 
acknowledgement, some decussion, and a positive decision to do so.

Someone has already mentioned the degredation of tracebacks.

So why not just leave the source files alone in /Lib.  As far as I can 
see, they would not hurt anything   At least on Windows, zip files are 
treated as directories and python35.zip comes before /Lib on sys.path.

The Windows installer currently has an option, selected by default I 
believe, to run compileall.  Add to compileall an option to compile all 
to python35.zip rather than __pycache and  and use that in that 
installer.  Even if the zip is including in the installer, 
compileall-zip + source files would let adventurous people patch their 
stdlib files.

Editing a stdlib file, to see if a confirmed bug disappeared (it did), 
was how I made my first code contribution. If I had had to download and 
setup svn and maybe visual c to try a one line change, I would not have 
done it.

-- 
Terry Jan Reedy