[Python-ideas] Move more parts of interpreter core to stdlib

[Brett Cannon]

On Mon, Aug 26, 2013 at 2:54 PM, Ned Batchelder <ned at nedbatchelder.com>wrote:

> On 8/26/13 1:55 PM, Antoine Pitrou wrote:
>
>> On Mon, 26 Aug 2013 16:36:53 +0200
>> Draic Kin <drekin at gmail.com> wrote:
>>
>>> Hello, it would be nice if reference pure Python implementation existed
>>> for
>>> more parts of interpreter core and the core actually used them. This was
>>> done for import machinery in Python 3.3 making importlib library.
>>>
>>> One potential target for this would be moving the logic of what
>>> python.exe
>>> does – parsing its arguments, finding the script to run, compiling its
>>> code
>>> and running as __main__ module, running REPL if it's in interactive mode
>>> afterwards. There could be a stdlib module exposing this logic, using
>>> argparse to parse the arguments, overhauled version of runpy to run the
>>> __main__ module and overhauled version of code to run the REPL.
>>> Python.exe
>>> would be just thin wrapper which bootstraps the interpreter and runs this
>>> runner module.
>>>
>> The interpreter needs a lot of information to be bootstrapped; you are
>> proposing that the code which extracts that information be run *after*
>> the interpreter is bootstrapped, which creates a nasty temporal problem.
>>
>> In the end, it may make maintenance *more* difficult, rather than less,
>> to rewrite that code in Python.
>>
> It seems to me that this argument could have been made against the import
> rewrite in Python.


Not quite. Antoine's point is that the flags used to start Python are
needed to set up certain things that may influence how using e.g. argparse
works. Importlib is in a unique position because I wrote it from the
beginning to be bootstrapped, so I designed it to deal with bootstrapping
issues. It also led to the code being a little odd and having to work
around things like lacking imports, etc. Argparse (and any of its
dependencies) have not been designed in such a fashion, especially if they
are directly involved in setting key settings in the interpreter.


>  I don't know enough about the various factors to know what the
> differences are between the two scenarios (import and startup) to know
> whether it's a valid argument here or not. Can someone elaborate?
>

The exact location where importlib is bootstrapped is at
http://hg.python.org/cpython/file/8fb3a6f9b0a4/Python/pythonrun.c#l387 . I
think it happens as soon as humanly possible, but it also is in a very
restricted environment that is atypical (e.g. no imports, only uses
built-in modules, can't have any alternative encoding, etc.).


>
> I know it would be great to have the startup logic more accessible.


Sure, but there is also a performance consideration to take in.

I wrote a blog post once on this topic:
http://sayspy.blogspot.ca/2012/12/how-much-of-python-can-be-written-in.html .
Basically you might be able to pull off exceptions in Python because they
are typically such simple chunks of code, but otherwise everything else is
too performance-sensitive. To really dive in you would need to look at the
C code and see what happens at what point to know if unmodified Python code
could be used instead of the C code (or be willing to write it all from
scratch to allow bootstrapping).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130826/88bdfe68/attachment.html>