[Python-Dev] Criticism of execfile() removal in Python3

[Markus Unterwaditzer]

On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:
> Hello,
> 
> I was pleasantly surprised with the response to recent post about
> MicroPython implementation details 
> (https://mail.python.org/pipermail/python-dev/2014-June/134718.html). I
> hope that discussion means that posts about alternative implementations
> are not unwelcome here, so I would like to bring up another (of many)
> issues we faced while implementing MicroPython.
> 
> execfile() builtin function was removed in 3.0. This brings few
> problems:
> 
> 1. It hampers interactive mode - instead of short and easy to type
> execfile("file.py") one needs to use exec(open("file.py").read()). I'm
> sure that's not going to bother a lot of people - after all, the
> easiest way to execute a Python file is to drop back to shell and
> restart python with file name, using all wonders of tab completion. But
> now imagine that Python interpreter runs on bare hardware, and its REPL
> is the only shell. That's exactly what we have with MicroPython's
> Cortex-M port. But it's not really MicroPython-specific, there's
> CPython port to baremetal either - http://www.pycorn.org/ .

As far as i can see, minimizing the amount of characters to type was never a
design goal of the Python language. And because that goal never mattered as
much for the designers as it seems to do for you, the reason for it to get
removed -- reducing the amount of builtins without reducing functionality --
was the only one left.

> 2. Ok, assuming that exec(open().read()) idiom is still a way to go,
> there's a problem - it requires to load entire file to memory. But
> there can be not enough memory. Consider 1Mb file with 900Kb comments
> (autogenerated, for example). execfile() could easily parse it, using
> small buffer. But exec() requires to slurp entire file into memory, and
> 1Mb is much more than heap sizes that we target.

That is a valid concern, but i believe violating the language specification and
adding your own execfile implementation (either as a builtin or in a new stdlib
module) here is justified, even if it means you will have to modify your
existing Python 3 code to use it -- i don't think the majority of software
written in Python will be able to run under such memory constraints without
major modifications anyway.

> Comments, suggestions? Just to set a productive direction, please
> kindly don't consider the problems above as MicroPython's.

A new (not MicroPython-specific) stdlib module containing functions such as
execfile could be considered. Not really for Python-2-compatibility, but for
performance-critical situations.

I am not sure if this is a good solution. Not at all. Even though it's
separated from the builtins, i think it would still sacrifice the purity of the
the language (by which i mean having a minimal composable API), because people
are going to use it anyway. It reminds me of the situation in Python 2 where
developers are trying to use cStringIO with a fallback to StringIO as a matter
of principle, not because they actually need that kind of performance.

Another, IMO better idea which shifts the problem to the MicroPython devs is to
"just" detect code using

    exec(open(...).read())

and transparently rewrite it to something more memory-efficient. This is the
idea i actually think is a good one.


> I very much liked how last discussion went: I was pointed that
> https://docs.python.org/3/reference/index.html is not really a CPython
> reference, it's a *Python* reference, and there were even motion to
> clarify in it some points which came out from MicroPython discussion.
> So, what about https://docs.python.org/3/library/index.html - is it
> CPython, or Python standard library specification? Assuming the latter,
> what we have is that, by removal of previously available feature,
> *Python* became less friendly for interactive usage and less scalable.

"Less friendly for interactive usage" is a strong and vague statement. If
you're going after the amount of characters required to type, yes, absolutely,
but by that terms one could declare Bash and Perl to be superior languages.
Look at it from a different perspective: There are fewer builtins to remember.

> 
> 
> Thanks,
>  Paul                          mailto:pmiscml at gmail.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net