Mailman 3 Criticism of execfile() removal in Python3 - Python-Dev

Criticism of execfile() removal in Python3

older
Issue 21671: CVE-2014-0224 OpenSSL...

Paul Sokolovsky

10 Jun 2014 10 Jun '14

2:23 a.m.

Hello, I was pleasantly surprised with the response to recent post about MicroPython implementation details (https://mail.python.org/pipermail/python-dev/2014-June/134718.html). I hope that discussion means that posts about alternative implementations are not unwelcome here, so I would like to bring up another (of many) issues we faced while implementing MicroPython. execfile() builtin function was removed in 3.0. This brings few problems: 1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()). I'm sure that's not going to bother a lot of people - after all, the easiest way to execute a Python file is to drop back to shell and restart python with file name, using all wonders of tab completion. But now imagine that Python interpreter runs on bare hardware, and its REPL is the only shell. That's exactly what we have with MicroPython's Cortex-M port. But it's not really MicroPython-specific, there's CPython port to baremetal either - http://www.pycorn.org/ . 2. Ok, assuming that exec(open().read()) idiom is still a way to go, there's a problem - it requires to load entire file to memory. But there can be not enough memory. Consider 1Mb file with 900Kb comments (autogenerated, for example). execfile() could easily parse it, using small buffer. But exec() requires to slurp entire file into memory, and 1Mb is much more than heap sizes that we target. Comments, suggestions? Just to set a productive direction, please kindly don't consider the problems above as MicroPython's. I very much liked how last discussion went: I was pointed that https://docs.python.org/3/reference/index.html is not really a CPython reference, it's a *Python* reference, and there were even motion to clarify in it some points which came out from MicroPython discussion. So, what about https://docs.python.org/3/library/index.html - is it CPython, or Python standard library specification? Assuming the latter, what we have is that, by removal of previously available feature, *Python* became less friendly for interactive usage and less scalable. Thanks, Paul mailto:pmiscml@gmail.com

Show replies by date

Steven D'Aprano

10 Jun 10 Jun

3:03 a.m.

On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:

...

execfile() builtin function was removed in 3.0. This brings few problems:

1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()).

If the amount of typing is the problem, that's easy to solve: # do this once def execfile(name): exec(open("file.py").read()) Another possibility is: os.system("python file.py")

...

2. Ok, assuming that exec(open().read()) idiom is still a way to go, there's a problem - it requires to load entire file to memory. But there can be not enough memory. Consider 1Mb file with 900Kb comments (autogenerated, for example). execfile() could easily parse it, using small buffer. But exec() requires to slurp entire file into memory, and 1Mb is much more than heap sizes that we target.

There's nothing stopping alternative implementations having their own implementation-specific standard library modules. steve@orac:/home/s$ jython Jython 2.5.1+ (Release_2_5_1, Aug 4 2010, 07:18:19) [OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_27 Type "help", "copyright", "credits" or "license" for more information.

...

...
...
import java

So you could do this: from upy import execfile execfile("file.py") So long as you make it clear that this is a platform specific module, and don't advertise it as a language feature, I see no reason why you cannot do that. -- Steven

Terry Reedy

3:56 a.m.

On 6/9/2014 11:03 PM, Steven D'Aprano wrote:

...

On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:

...
execfile() builtin function was removed in 3.0.

Because it was hardly ever used. For short bits of code, it is usually inferior to exec with a string in the file. For substantial bits of code, it is generally inferior to 'from file import *' and does not have the option of other forms of import. For startup code that you want every session, it is inferior to PYTHONSTARTUP or custom site module.

...

...
This brings few problems: 1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read())

...

If the amount of typing is the problem, that's easy to solve:

# do this once def execfile(name): exec(open("file.py").read())

Another possibility is:

os.system("python file.py")

...
2. Ok, assuming that exec(open().read()) idiom is still a way to go, there's a problem - it requires to load entire file to memory. But there can be not enough memory. Consider 1Mb file with 900Kb comments (autogenerated, for example). execfile() could easily parse it, using small buffer. But exec() requires to slurp entire file into memory, and 1Mb is much more than heap sizes that we target.

Execfile could slurp the whole file into memory too. Next parse the entire file. Then execute the entire bytecode. Finally toss the bytecode so that the file has to be reparsed next time it is used.

...

There's nothing stopping alternative implementations having their own implementation-specific standard library modules. ... So you could do this:

from upy import execfile execfile("file.py")

So long as you make it clear that this is a platform specific module, and don't advertise it as a language feature, I see no reason why you cannot do that.

If you want execfile as a substitute for 'python -i file' on the unavailable command console, you should have the option to restore globals to initial condition. Something like (untested) # startup entries in globals in CPython 3.4.1 startnames={'__spec__', '__name__', '__builtins__', '__doc__', '__loader__', '__package__'} def execfile(file, encoding='utf-8', restart=): glodict = globals() code = open(file, 'r', encoding=encoding) # don't restart if the file does not open if restart: for name in list(glodict): if name not in startnames: del glodict(name) for statement in statements(code): # statements is statement iterator exec(statement,...globals=glodict, locals=glodict) -- Terry Jan Reedy

Jim Baker

4:41 a.m.

On Mon, Jun 9, 2014 at 9:03 PM, Steven D'Aprano wrote:

...

... There's nothing stopping alternative implementations having their own implementation-specific standard library modules.

steve@orac:/home/s$ jython Jython 2.5.1+ (Release_2_5_1, Aug 4 2010, 07:18:19) [OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_27 Type "help", "copyright", "credits" or "license" for more information.

...
...
...
import java

Small nit: Jython does implement a number of implementation-specific modules in its version of the standard library; jarray comes to mind, which is mostly but not completely superseded by the standard array module. However, the java package namespace is not part of the standard library, it's part of the standard Java ecosystem and it's due to a builtin import hook: Jython 2.7b3+ (default:6cee6fef06f0, Jun 9 2014, 22:29:14) [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_60 Type "help", "copyright", "credits" or "license" for more information.

...

...
...
import sys sys.path ['', '/home/jbaker/jythondev/jython27/dist/Lib', '__classpath__', '__pyclasspath__/', '/home/jbaker/.local/lib/jython2.7/site-packages', '/home/jbaker/jythondev/jython27/dist/Lib/site-packages']

The entry __classpath__ means search CLASSPATH for Java packages; this includes the Java runtime, rt.jar, from which you get package namespaces as java.*, javax.*, sun.*, etc. Another behavior that you get for free in Jython is being able to also import the org.python.* namespace, which is Jython's own runtime. Some of the implementations of standard library modules, such as threading, take advantage of this support. - Jim

Paul Sokolovsky

14 Jun 14 Jun

8:11 p.m.

Hello, On Tue, 10 Jun 2014 13:03:03 +1000 Steven D'Aprano wrote:

...

On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:

...
execfile() builtin function was removed in 3.0. This brings few problems:

1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()).

If the amount of typing is the problem, that's easy to solve:

# do this once def execfile(name): exec(open("file.py").read())

So, you here propose to workaround removal of core language feature either a) on end user side, or b) on "system integrator" side. But such solution is based on big number of assumptions, like: user wants to workaround that at all (hint: they don't, they just want to use it); you say "do this once", but actually it's "do it in each interactive session again and again", and user may not have knowledge to "do it once" instead; that if system integrator does that, the the function is called "execfile": if system integrator didn't have enough Python experience, and read only Python3 spec, they might call it something else, and yet users with a bit of Python experience will expect it be called exactly "execfile" and not anything else.

...

Another possibility is:

os.system("python file.py")

...
2. Ok, assuming that exec(open().read()) idiom is still a way to go, there's a problem - it requires to load entire file to memory. But there can be not enough memory. Consider 1Mb file with 900Kb comments (autogenerated, for example). execfile() could easily parse it, using small buffer. But exec() requires to slurp entire file into memory, and 1Mb is much more than heap sizes that we target.

There's nothing stopping alternative implementations having their own implementation-specific standard library modules.

And here you propose to workaround it on particular implementation's level. But in my original mail, in excerpt that you removed, I kindly asked to skip obvious suggestions (like that particular implementation can do anything it wants). I don't see how working around the issue on user, particular distribution, or particular implementation level help *Python* language in general, and *Python community* in general. So, any bright ideas how to workaround the issue of execfile() removal on *language level*? []

...

So you could do this:

from upy import execfile execfile("file.py")

So long as you make it clear that this is a platform specific module, and don't advertise it as a language feature, I see no reason why you cannot do that.

The case we discuss is clearly different. It's not about "platform specific module", it's about functionality which was in Python all the time, and was suddenly removed in Python3, for not fully clear, or alternatively, not severe enough, reasons. If some implementation is to re-add it, the description like above seems the most truthful way to represent that function. -- Best regards, Paul mailto:pmiscml@gmail.com

Skip Montanaro

15 Jun 15 Jun

12:01 a.m.

...

you say "do this once", but actually it's "do it in each interactive session again and again", ...

That's what your Python startup file is for. I have been running with several tweaked builtin functions for years. Never have to consciously load them. If I wanted execfile badly enough, I'd define it there. I don't think I've used execfile more than a handful of times in the 20-odd years I've been using Python. Perhaps our personal approaches to executing code at the interpreter prompt are radically different, but I think if the lack of execfile is such a big deal for you, you might want to check around to see how other people use interactive mode. Skip

Chris Barker

16 Jun 16 Jun

5:40 p.m.

On Sat, Jun 14, 2014 at 1:11 PM, Paul Sokolovsky wrote:

...

...
...
1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()).

...

...
If the amount of typing is the problem, that's easy to solve:

# do this once def execfile(name): exec(open("file.py").read())

FWIW, when I started using python (15?) years ago -- the first thing I looked for was a way to "just run a file", at the interactive prompt, like I had in MATLAB. I found and used execfile(). However, it wasn't long before I discovered that excecfile() was really kind of a pain, you've got namespaces, and all sorts of stuff that made it often not work like I wanted, and was a pain to type. I stopped using it all together More recently, I discovered iPython and its "run" function -- very nice, it does the obvious stuff for you the way you'd expect. My conclusions: 1) runfile() is not really very usefull, it's fine to hve removed it. 2) the built-in interactive python interpreter is really pretty lame. If you want a good interactive experience, you need something more anyway (iPython, for instance) -- putting execfile() back is only one tiny improvement that's not worth it. So if this is about micropython -- I think it would serve the project very well to have a micropython-specific interactive mode. iPython is fabulous, but though I imagine too heavy weight. But perhaps you could borrow some things from it -- like "run" , for example. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Ethan Furman

5:52 p.m.

On 06/16/2014 10:40 AM, Chris Barker wrote:

...

My conclusions:

1) runfile() is not really very usefull, it's fine to hve removed it.

s/runfile/execfile -- ~Ethan~

Nick Coghlan

10:39 p.m.

On 17 Jun 2014 03:42, "Chris Barker" wrote:

...

On Sat, Jun 14, 2014 at 1:11 PM, Paul Sokolovsky

wrote:

...

...
...
...
1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()).

...
If the amount of typing is the problem, that's easy to solve:

# do this once def execfile(name): exec(open("file.py").read())

FWIW, when I started using python (15?) years ago -- the first thing I

looked for was a way to "just run a file", at the interactive prompt, like I had in MATLAB. I found and used execfile(). Yes, if people are looking for a MATLAB replacement, they want IPython rather than the default REPL. The default one is deliberately minimal, IPython is designed to be a comprehensive numeric and scientific workspace. Cheers, Nick.

Chris Barker

17 Jun 17 Jun

3:59 p.m.

On Mon, Jun 16, 2014 at 3:39 PM, Nick Coghlan wrote:

...

...
FWIW, when I started using python (15?) years ago -- the first thing I looked for was a way to "just run a file", at the interactive prompt, like I had in MATLAB. I found and used execfile().

Yes, if people are looking for a MATLAB replacement, they want IPython rather than the default REPL.

I didn't meant o distract the conversation here -- what I meant was that even before iPython existed, I still dropped using execfile("") it was hardly ever the right thing. And for the micropython example, I'm proposing that a micropython interactive environment would be a really nice thing to build -- and worth doing, even if execfile() was still there. By the way: iPython, while coming from, and heavily used by, the scientific/numeric computing community, is a great tool for all sorts of other python development as well. But probably too heavyweight for micropython. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Nick Coghlan

10 p.m.

On 18 Jun 2014 01:59, "Chris Barker" wrote:

...

By the way: iPython, while coming from, and heavily used by, the

scientific/numeric computing community, is a great tool for all sorts of other python development as well. But probably too heavyweight for micropython. (we're drifting off topic, so this will be my last addition to this subthread) Yes, as great as IPython is, when it's considered out of scope for the standard installers, it's unlikely to be a good fit for a version of Python aimed at running *on* a microcontroller. Running on a Raspberry Pi or remote PC and *talking* to an associated microcontroller is a different story, though. Cheers, Nick.

...

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Nick Coghlan

10 Jun 10 Jun

7:36 a.m.

On 10 June 2014 12:23, Paul Sokolovsky wrote:

...

1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()). I'm sure that's not going to bother a lot of people - after all, the easiest way to execute a Python file is to drop back to shell and restart python with file name, using all wonders of tab completion. But now imagine that Python interpreter runs on bare hardware, and its REPL is the only shell. That's exactly what we have with MicroPython's Cortex-M port. But it's not really MicroPython-specific, there's CPython port to baremetal either - http://www.pycorn.org/ .

https://docs.python.org/3/library/runpy.html#runpy.run_path import runpy file_globals = runpy.run_path("file.py") The standard implementation of run_path reads the whole file into memory, but MicroPython would be free to optimise that and do statement by statement execution instead (while that will pose some challenges in terms of handling encoding cookies, future imports, etc correctly, it's certainly feasible). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Paul Moore

8:41 a.m.

On 10 June 2014 08:36, Nick Coghlan wrote:

...

The standard implementation of run_path reads the whole file into memory, but MicroPython would be free to optimise that and do statement by statement execution instead (while that will pose some challenges in terms of handling encoding cookies, future imports, etc correctly, it's certainly feasible).

... and if they did optimise that way, I would imagine that the patch would be a useful contribution back to the core Python stdlib, rather than remaining a MicroPython-specific optimisation. Paul

Nick Coghlan

9:07 a.m.

On 10 Jun 2014 18:41, "Paul Moore" wrote:

...

On 10 June 2014 08:36, Nick Coghlan wrote:

...
The standard implementation of run_path reads the whole file into memory, but MicroPython would be free to optimise that and do statement by statement execution instead (while that will pose some challenges in terms of handling encoding cookies, future imports, etc correctly, it's certainly feasible).

... and if they did optimise that way, I would imagine that the patch would be a useful contribution back to the core Python stdlib, rather than remaining a MicroPython-specific optimisation.

I believe it's a space/speed trade-off, so I'd be surprised if it made sense for CPython in general. There are also some behavioural differences when it comes to handling syntax errors. Now that I think about the idea a bit more, if the MicroPython folks can get a low memory usage incremental file execution model working, the semantic differences mean it would likely make the most sense as a separate API in runpy, rather than as an implicit change to run_path. Cheers, Nick.

...

Paul

R. David Murray

1:05 p.m.

On Tue, 10 Jun 2014 19:07:40 +1000, Nick Coghlan wrote:

...

On 10 Jun 2014 18:41, "Paul Moore" wrote:

...
On 10 June 2014 08:36, Nick Coghlan wrote:

...
The standard implementation of run_path reads the whole file into memory, but MicroPython would be free to optimise that and do statement by statement execution instead (while that will pose some challenges in terms of handling encoding cookies, future imports, etc correctly, it's certainly feasible).

... and if they did optimise that way, I would imagine that the patch would be a useful contribution back to the core Python stdlib, rather than remaining a MicroPython-specific optimisation.

I believe it's a space/speed trade-off, so I'd be surprised if it made sense for CPython in general. There are also some behavioural differences when it comes to handling syntax errors.

Now that I think about the idea a bit more, if the MicroPython folks can get a low memory usage incremental file execution model working, the semantic differences mean it would likely make the most sense as a separate API in runpy, rather than as an implicit change to run_path.

If it is a separate API, it seems like there's no reason it couldn't be contributed back to CPython. There might be other contexts in which low memory would be the right tradeoff. Although, if key bits end up working at the C level, "contributing back" might require writing separate C for CPython, so that might not happen. --David

Nick Coghlan

1:11 p.m.

On 10 June 2014 23:05, R. David Murray wrote:

...

On Tue, 10 Jun 2014 19:07:40 +1000, Nick Coghlan wrote:

...
I believe it's a space/speed trade-off, so I'd be surprised if it made sense for CPython in general. There are also some behavioural differences when it comes to handling syntax errors.

Now that I think about the idea a bit more, if the MicroPython folks can get a low memory usage incremental file execution model working, the semantic differences mean it would likely make the most sense as a separate API in runpy, rather than as an implicit change to run_path.

If it is a separate API, it seems like there's no reason it couldn't be contributed back to CPython. There might be other contexts in which low memory would be the right tradeoff. Although, if key bits end up working at the C level, "contributing back" might require writing separate C for CPython, so that might not happen.

Yeah, as a separate API it could make sense in CPython - I just didn't go back and revise the first paragraph after writing the second one :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Paul Sokolovsky

14 Jun 14 Jun

8:52 p.m.

Hello, On Tue, 10 Jun 2014 17:36:02 +1000 Nick Coghlan wrote:

...

On 10 June 2014 12:23, Paul Sokolovsky wrote:

...
1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()). I'm sure that's not going to bother a lot of people - after all, the easiest way to execute a Python file is to drop back to shell and restart python with file name, using all wonders of tab completion. But now imagine that Python interpreter runs on bare hardware, and its REPL is the only shell. That's exactly what we have with MicroPython's Cortex-M port. But it's not really MicroPython-specific, there's CPython port to baremetal either - http://www.pycorn.org/ .

https://docs.python.org/3/library/runpy.html#runpy.run_path

import runpy file_globals = runpy.run_path("file.py")

Thanks, it's the most productive response surely. So, at least there's alternative to removed execfile(). Unfortunately, I don't think it's good alternative to execfile() in all respects. It clearly provides API for that functionality, but is that solution of least surprise and is it actually known by users at all (to be useful for them)? Googling for "execfile python 3", top 3 hits I see are stackoverflow questions, *none* of which mentions runpy. So, people either don't consider it viable alternative to execfile, or don't know about it at all (my guess it's the latter). Like with previous discussion, its meaning goes beyond just Python realm - there's competition all around. And internets bring funny examples, like for example http://www.red-lang.org/p/contributions.html (scroll down to diagram, or here's direct link: http://3.bp.blogspot.com/-xhOP35Dm99w/UuXFKgY2dlI/AAAAAAAAAGA/YQu98_pPDjw/s1...) So, didn't you know that Ruby can be used for OS-level development, and Python can't? Or that JavaScript DSL capabilities are better than Python's (that's taking into account that JavaScript DSL capabilities are represented by JSON, whose creators were so arrogant as to disallow even usage of comments in it). So, now suppose there's a discussion of how good different languages are for interactive usage (out of the box apparently). It would be a little hard to defend claim that Python is *excellent* interactive language, if its latest series got -1 on that scale, by removing feature which may be indispensable at times. Knowing that, one subconsciously may start to wonder if Ruby or JavaScript are doing it (in wide sense) better than Python. -- Best regards, Paul mailto:pmiscml@gmail.com

Nick Coghlan

15 Jun 15 Jun

1:28 a.m.

On 15 Jun 2014 06:52, "Paul Sokolovsky" wrote:

...

Hello,

On Tue, 10 Jun 2014 17:36:02 +1000 Nick Coghlan wrote:

...
On 10 June 2014 12:23, Paul Sokolovsky wrote:

...
1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()). I'm sure that's not going to bother a lot of people - after all, the easiest way to execute a Python file is to drop back to shell and restart python with file name, using all wonders of tab completion. But now imagine that Python interpreter runs on bare hardware, and its REPL is the only shell. That's exactly what we have with MicroPython's Cortex-M port. But it's not really MicroPython-specific, there's CPython port to baremetal either - http://www.pycorn.org/ .

https://docs.python.org/3/library/runpy.html#runpy.run_path

import runpy file_globals = runpy.run_path("file.py")

Thanks, it's the most productive response surely. So, at least there's alternative to removed execfile(). Unfortunately, I don't think it's good alternative to execfile() in all respects. It clearly provides API for that functionality, but is that solution of least surprise and is it actually known by users at all (to be useful for them)?

We don't want people instinctively reaching for execfile (or run_path for that matter). It's almost always the wrong answer to a problem (because it runs code in a weird, ill-defined environment and has undefined behaviour when used inside a function), meeting the definition of "attractive nuisance". We moved reload() to imp.reload() and reduce() to functools.reduce() for similar reasons - they're too rarely the right answer to justify having them globally available by default.

...

Googling for "execfile python 3", top 3 hits I see are stackoverflow questions, *none* of which mentions runpy. So, people either don't consider it viable alternative to execfile, or don't know about it at all (my guess it's the latter).

Given the relative age of the two APIs, that seems likely. Adding answers pointing users to the runpy APIs could be useful.

...

Like with previous discussion, its meaning goes beyond just Python realm - there's competition all around. And internets bring funny examples, like for example http://www.red-lang.org/p/contributions.html (scroll down to diagram, or here's direct link:

http://3.bp.blogspot.com/-xhOP35Dm99w/UuXFKgY2dlI/AAAAAAAAAGA/YQu98_pPDjw/s1... )

...

So, didn't you know that Ruby can be used for OS-level development, and Python can't? Or that JavaScript DSL capabilities are better than Python's (that's taking into account that JavaScript DSL capabilities are represented by JSON, whose creators were so arrogant as to disallow even usage of comments in it).

There's a lot of misinformation on the internet. While there is certainly room for the PSF to do more in terms of effectively communicating Python's ubiquity and strengths (and we're working on that), "people with no clue post stuff on the internet" doesn't make a compelling *technical* argument (which is what is needed to get new builtins added).

...

So, now suppose there's a discussion of how good different languages are for interactive usage (out of the box apparently). It would be a little hard to defend claim that Python is *excellent* interactive language, if its latest series got -1 on that scale, by removing feature which may be indispensable at times. Knowing that, one subconsciously may start to wonder if Ruby or JavaScript are doing it (in wide sense) better than Python.

Yes, people get upset when we tell them we consider some aspects of their software designs to be ill-advised. Running other code in the *current* namespace is such a thing - it is typically preferable to run it in a *different* namespace and then access the results, rather than implicitly overwriting the contents of the current namespace. That said, a question still worth asking is whether there is scope for additional runpy APIs that are designed to more easily implement Python 2 and IPython style modes of operation where independent units of code manipulate a shared namespace? That's actually a possibility, but any such proposals need to be presented on python-ideas in terms of the *use case* to be addressed, rather than the fact that execfile() happened to be the preferred solution in Python 2. Regards, Nick.

...

-- Best regards, Paul mailto:pmiscml@gmail.com

Markus Unterwaditzer

14 Jun 14 Jun

9 p.m.

On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:

...

Hello,

I was pleasantly surprised with the response to recent post about MicroPython implementation details (https://mail.python.org/pipermail/python-dev/2014-June/134718.html). I hope that discussion means that posts about alternative implementations are not unwelcome here, so I would like to bring up another (of many) issues we faced while implementing MicroPython.

execfile() builtin function was removed in 3.0. This brings few problems:

1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()). I'm sure that's not going to bother a lot of people - after all, the easiest way to execute a Python file is to drop back to shell and restart python with file name, using all wonders of tab completion. But now imagine that Python interpreter runs on bare hardware, and its REPL is the only shell. That's exactly what we have with MicroPython's Cortex-M port. But it's not really MicroPython-specific, there's CPython port to baremetal either - http://www.pycorn.org/ .

As far as i can see, minimizing the amount of characters to type was never a design goal of the Python language. And because that goal never mattered as much for the designers as it seems to do for you, the reason for it to get removed -- reducing the amount of builtins without reducing functionality -- was the only one left.

...

2. Ok, assuming that exec(open().read()) idiom is still a way to go, there's a problem - it requires to load entire file to memory. But there can be not enough memory. Consider 1Mb file with 900Kb comments (autogenerated, for example). execfile() could easily parse it, using small buffer. But exec() requires to slurp entire file into memory, and 1Mb is much more than heap sizes that we target.

That is a valid concern, but i believe violating the language specification and adding your own execfile implementation (either as a builtin or in a new stdlib module) here is justified, even if it means you will have to modify your existing Python 3 code to use it -- i don't think the majority of software written in Python will be able to run under such memory constraints without major modifications anyway.

...

Comments, suggestions? Just to set a productive direction, please kindly don't consider the problems above as MicroPython's.

A new (not MicroPython-specific) stdlib module containing functions such as execfile could be considered. Not really for Python-2-compatibility, but for performance-critical situations. I am not sure if this is a good solution. Not at all. Even though it's separated from the builtins, i think it would still sacrifice the purity of the the language (by which i mean having a minimal composable API), because people are going to use it anyway. It reminds me of the situation in Python 2 where developers are trying to use cStringIO with a fallback to StringIO as a matter of principle, not because they actually need that kind of performance. Another, IMO better idea which shifts the problem to the MicroPython devs is to "just" detect code using exec(open(...).read()) and transparently rewrite it to something more memory-efficient. This is the idea i actually think is a good one.

...

I very much liked how last discussion went: I was pointed that https://docs.python.org/3/reference/index.html is not really a CPython reference, it's a *Python* reference, and there were even motion to clarify in it some points which came out from MicroPython discussion. So, what about https://docs.python.org/3/library/index.html - is it CPython, or Python standard library specification? Assuming the latter, what we have is that, by removal of previously available feature, *Python* became less friendly for interactive usage and less scalable.

"Less friendly for interactive usage" is a strong and vague statement. If you're going after the amount of characters required to type, yes, absolutely, but by that terms one could declare Bash and Perl to be superior languages. Look at it from a different perspective: There are fewer builtins to remember.

...

Thanks, Paul mailto:pmiscml@gmail.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.ne...

Fabio Zadrozny

10:15 p.m.

On Sat, Jun 14, 2014 at 6:00 PM, Markus Unterwaditzer < markus@unterwaditzer.net> wrote:

...

On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:

...
Hello,

I was pleasantly surprised with the response to recent post about MicroPython implementation details (https://mail.python.org/pipermail/python-dev/2014-June/134718.html). I hope that discussion means that posts about alternative implementations are not unwelcome here, so I would like to bring up another (of many) issues we faced while implementing MicroPython.

execfile() builtin function was removed in 3.0. This brings few problems:

1. It hampers interactive mode - instead of short and easy to type execfile("file.py") one needs to use exec(open("file.py").read()). I'm sure that's not going to bother a lot of people - after all, the easiest way to execute a Python file is to drop back to shell and restart python with file name, using all wonders of tab completion. But now imagine that Python interpreter runs on bare hardware, and its REPL is the only shell. That's exactly what we have with MicroPython's Cortex-M port. But it's not really MicroPython-specific, there's CPython port to baremetal either - http://www.pycorn.org/ .

As far as i can see, minimizing the amount of characters to type was never a design goal of the Python language. And because that goal never mattered as much for the designers as it seems to do for you, the reason for it to get removed -- reducing the amount of builtins without reducing functionality -- was the only one left.

...
2. Ok, assuming that exec(open().read()) idiom is still a way to go, there's a problem - it requires to load entire file to memory. But there can be not enough memory. Consider 1Mb file with 900Kb comments (autogenerated, for example). execfile() could easily parse it, using small buffer. But exec() requires to slurp entire file into memory, and 1Mb is much more than heap sizes that we target.

That is a valid concern, but i believe violating the language specification and adding your own execfile implementation (either as a builtin or in a new stdlib module) here is justified, even if it means you will have to modify your existing Python 3 code to use it -- i don't think the majority of software written in Python will be able to run under such memory constraints without major modifications anyway.

...
Comments, suggestions? Just to set a productive direction, please kindly don't consider the problems above as MicroPython's.

A new (not MicroPython-specific) stdlib module containing functions such as execfile could be considered. Not really for Python-2-compatibility, but for performance-critical situations.

I am not sure if this is a good solution. Not at all. Even though it's separated from the builtins, i think it would still sacrifice the purity of the the language (by which i mean having a minimal composable API), because people are going to use it anyway. It reminds me of the situation in Python 2 where developers are trying to use cStringIO with a fallback to StringIO as a matter of principle, not because they actually need that kind of performance.

Another, IMO better idea which shifts the problem to the MicroPython devs is to "just" detect code using

exec(open(...).read())

and transparently rewrite it to something more memory-efficient. This is the idea i actually think is a good one.

...
I very much liked how last discussion went: I was pointed that https://docs.python.org/3/reference/index.html is not really a CPython reference, it's a *Python* reference, and there were even motion to clarify in it some points which came out from MicroPython discussion. So, what about https://docs.python.org/3/library/index.html - is it CPython, or Python standard library specification? Assuming the latter, what we have is that, by removal of previously available feature, *Python* became less friendly for interactive usage and less scalable.

"Less friendly for interactive usage" is a strong and vague statement. If you're going after the amount of characters required to type, yes, absolutely, but by that terms one could declare Bash and Perl to be superior languages. Look at it from a different perspective: There are fewer builtins to remember.

...

Well, I must say that the exec(open().read()) is not really a proper execfile implementation because it may fail because of encoding issues... (i.e.: one has to check the file encoding to do the open with the proper encoding, otherwise it's possible to end up with gibberish). The PyDev debugger has an implementation (see: https://github.com/fabioz/Pydev/blob/development/plugins/org.python.pydev/py...) which considers the encoding so that the result is ok (but it still has a bug related to utf-8 with bom: https://sw-brainwy.rhcloud.com/tracker/PyDev/346 which I plan to fix soon...) Personally, it's one thing that I think should be restored as the proper implementation is actually quite tricky and the default recommended solution does not work properly on some situations (and if micropython can provide an optimized implementation which'd conform to Python, that'd be one more point to add it back)... Best Regards, Fabio

Greg Ewing

11:18 p.m.

Fabio Zadrozny wrote:

...

Well, I must say that the exec(open().read()) is not really a proper execfile implementation because it may fail because of encoding issues...

It's not far off, though -- all it needs is an optional encoding parameter. -- Greg

Steve Dower

11:36 p.m.

I think the point is that the encoding may be embedded in the file as a coding comment and there's no obvious way to deal with that. Top-posted from my Windows Phone ________________________________ From: Greg Ewingmailto:greg.ewing@canterbury.ac.nz Sent: ‎6/‎14/‎2014 16:19 To: python-dev@python.orgmailto:python-dev@python.org Subject: Re: [Python-Dev] Criticism of execfile() removal in Python3 Fabio Zadrozny wrote:

...

Well, I must say that the exec(open().read()) is not really a proper execfile implementation because it may fail because of encoding issues...

It's not far off, though -- all it needs is an optional encoding parameter. -- Greg _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

Nick Coghlan

15 Jun 15 Jun

1:31 a.m.

On 15 Jun 2014 09:37, "Steve Dower" wrote:

...

I think the point is that the encoding may be embedded in the file as a

coding comment and there's no obvious way to deal with that. Opening source files correctly is the intended use case for tokenize.open(). Cheers, Nick.

Steve Dower

3:15 a.m.

So is exec(tokenize.open(file).read()) the actual replacement for execfile()? Not too bad, but still not obvious (or widely promoted - I'd never heard of it). Top-posted from my Windows Phone ________________________________ From: Nick Coghlanmailto:ncoghlan@gmail.com Sent: ‎6/‎14/‎2014 18:31 To: Steve Dowermailto:Steve.Dower@microsoft.com Cc: Greg Ewingmailto:greg.ewing@canterbury.ac.nz; python-dev@python.orgmailto:python-dev@python.org Subject: Re: [Python-Dev] Criticism of execfile() removal in Python3 On 15 Jun 2014 09:37, "Steve Dower" mailto:Steve.Dower@microsoft.com> wrote:

...

I think the point is that the encoding may be embedded in the file as a coding comment and there's no obvious way to deal with that.

Opening source files correctly is the intended use case for tokenize.open(). Cheers, Nick.

Nick Coghlan

4:42 a.m.

On 15 June 2014 13:15, Steve Dower wrote:

...

So is exec(tokenize.open(file).read()) the actual replacement for execfile()? Not too bad, but still not obvious (or widely promoted - I'd never heard of it).

Yes, that's pretty close. It's still a dubious idea due to the implicit modification of the local namespace (and the resulting differences in behaviour at function level due to the fact that writing to locals() doesn't actually update the local namespace). That said, the "implicit changes to the local namespace are a bad idea" concern applies to exec() in general, so it was the "it's just a shorthand for a particular use of exec" aspect that tipped in the balance in the demise of execfile (this is also implied by the phrasing of the relevant bullet point in PEP 3100). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Joseph Martinot-Lagarde

19 Jun 19 Jun

7:39 p.m.

Le 15/06/2014 05:15, Steve Dower a écrit :

...

So is exec(tokenize.open(file).read()) the actual replacement for execfile()? Not too bad, but still not obvious (or widely promoted - I'd never heard of it).

Another way is to open the file in binary, then exec() checks itself if an encoding is defined in the file. This is what is used in spyder: exec(open(file, 'rb').read()) Here is the discussion for reference: https://bitbucket.org/spyder-ide/spyderlib/pull-request/3/execution-on-curre... This behavior is not indicated in the documentation but is somehow confirmed on stackoverflow: http://stackoverflow.com/questions/6357361/alternative-to-execfile-in-python... --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com

Paul Moore

8:46 p.m.

On 19 June 2014 20:39, Joseph Martinot-Lagarde wrote:

...

Another way is to open the file in binary, then exec() checks itself if an encoding is defined in the file. This is what is used in spyder:

exec(open(file, 'rb').read())

Here is the discussion for reference: https://bitbucket.org/spyder-ide/spyderlib/pull-request/3/execution-on-curre...

It would be good to document this. Could you open a docs bug to get this added? Paul

3598

Age (days ago)

3607

Last active (days ago)

List overview

Download

26 comments

15 participants

participants (15)

Chris Barker
Ethan Furman
Fabio Zadrozny
Greg Ewing
Jim Baker
Joseph Martinot-Lagarde
Markus Unterwaditzer
Nick Coghlan
Paul Moore
Paul Sokolovsky
R. David Murray
Skip Montanaro
Steve Dower
Steven D'Aprano
Terry Reedy

Criticism of execfile() removal in Python3

Jim Baker

Joseph Martinot-Lagarde

tags

participants (15)