Mailman 3 November 2015 - Python-Dev

PYC filename and optimization
by Victor Stinner 06 Nov '15

06 Nov '15

Hi, While working on the FAT Python project to try to optimize CPython, I have an issue with the cached .pyc files. I would like to have one .pyc without optimization when the FAT mode is disabled (default) and one .pyc with optimization when the FAT mode is enabled (-F command line option). Currently, I'm using this filenames: - Lib/__pycache__/os.cpython-36.pyc: default mode - Lib/__pycache__/os.cpython-36.fat-0.pyc: FAT mode With -O: - Lib/__pycache__/os.cpython-36.opt-1.pyc: default mode - Lib/__pycache__/os.cpython-36.fat-1.pyc: FAT mode With -OO: - Lib/__pycache__/os.cpython-36.opt-2.pyc: default mode - Lib/__pycache__/os.cpython-36.fat-2.pyc: FAT mode Is it the "correct" way to name .pyc files? I had to modify unit tests which expected an exact filename. An alternative would be to add a new dot: - Lib/__pycache__/os.cpython-36.fat.pyc - Lib/__pycache__/os.cpython-36.fat.opt-1.pyc - Lib/__pycache__/os.cpython-36.fat.opt-2.pyc Or should I add a flag to the cache tag ("cpython-36")? - Lib/__pycache__/os.cpython-36f.pyc - Lib/__pycache__/os.cpython-36f.opt-1.pyc - Lib/__pycache__/os.cpython-36f.opt-2.pyc CPython adds the "d" suffix to the cache tag in debug build, but in my case the flag would be added dynamically at Python startup. The FAT mode is enabled by the -F command line option, not by a compilation flag. Victor

3 2

Second milestone of FAT Python
by Victor Stinner 05 Nov '15

05 Nov '15

Hi, I'm writing a new "FAT Python" project to try to implement optimizations in CPython (inlining, constant folding, move invariants out of loops, etc.) using a "static" optimizer (not a JIT). For the background, see the thread on python-ideas: https://mail.python.org/pipermail/python-ideas/2015-October/036908.html See also the documentation: https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst I implemented the most basic optimization to test my code: replace calls to builtin functions (with constant arguments) with the result. For example, len("abc") is replaced with 3. I reached the second milestone: it's now possible to run the full Python test suite with these optimizations enabled. It confirms that the optimizations don't break the Python semantic. Example: --- >>> def func(): ... return len("abc") ... >>> import dis >>> dis.dis(func) 2 0 LOAD_GLOBAL 0 (len) 3 LOAD_CONST 1 ('abc') 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 9 RETURN_VALUE >>> len(func.get_specialized()) 1 >>> specialized=func.get_specialized()[0] >>> dis.dis(specialized['code']) 2 0 LOAD_CONST 1 (3) 3 RETURN_VALUE >>> len(specialized['guards']) 2 >>> func() 3 >>> len=lambda obj: "mock" >>> func() 'mock' >>> func.get_specialized() [] --- The function func() has specialized bytecode which returns directly 3 instead of calling len("abc"). The specialized bytecode has two guards dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of these keys is modified, the specialized bytecode is simply removed (when the function is called) and the original bytecode is executed. You cannot expect any speedup at this milestone, it's just to validate the implementation. You can only get speedup if you implement *manually* optimizations. See for example posixpath.isabs() which inlines manually the call to the _get_sep() function. More optimizations will be implemented in the third milestone. I don't know yet if I will be able to implement constant folding, function inlining and/or moving invariants out of loops. Download, compile and test FAT Python with: hg clone http://hg.python.org/sandbox/fatpython ./configure && make && ./python -m test test_astoptimizer test_fat Currently, only 24 functions are specialized in the standard library. Calling a builtin function with constant arguments in not common (it was expected, it's only the first step for my optimizer). But 161 functions are specialized in tests. To be honest, I had to modify some tests to make them pass in FAT mode. But most changes are related to the .pyc filename, or to the exact size in bytes of dictionary objects. FAT Python is still experimental. Currently, the main bug is that the AST optimizer can optimize a call to a function which is not the expected builtin function. I already started to implement code to understand namespaces (detect global and local variables), but it's not enough yet to detect when a builtin is overriden. See TODO.rst for known bugs and limitations. Victor

5 8

Python 3.5.1 release schedule
by Larry Hastings 05 Nov '15

05 Nov '15

At the request of the platform experts, 3.5.1 is now scheduled to happen simultaneously with 2.7.11. That means: Saturday November 21, 2015 tag 3.5.1rc1 Sunday November 22, 2015 release 3.5.1rc1 Saturday December 5, 2015 tag 3.5.1 final Sunday December 6, 2015 release 3.5.1 final I'll update the release schedule soon, //arry/

1 0

2.7.11 plans
by Benjamin Peterson 02 Nov '15

02 Nov '15

Happy November, everyone. It’s nearly time for the next semi-annual instalment of the 2.7 series. I’m planning to release a 2.7.11 release candidate on November 21st and 2.7.11 final on December 5. More than half-done releasing 2.7-ly yours, Benjamin

2 1

Re: [Python-Dev] Generated Bytecode ...
by Meador Inge 02 Nov '15

02 Nov '15

On Fri, Oct 23, 2015 at 3:23 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Note: I propose "noopt" because we already have "optimization level 0" > which still uses optimizations, it's the default mode. It's different > than gcc -O0 which really disables all optimizations. I already prefix > the "noopt" suffix for .pyc files than "opt--1" proposed by Brett in > http://bugs.python.org/issue2506 Strictly speaking, this isn't completely true. Most (all?) C/C++ compilers do some level of optimization when they are "disabled". GCC/Clang, for example, both will do some form of constant folding with -O0. The compilers are just *much* less aggressive with the primary goal of making the debugging experience much more enjoyable. The situation here sounds similar -- we want an option to provide a better debugging and tracing experience, which seems like a very reasonable idea to me. -- Meador

1 0

Python 3.5.1 plans
by Chris Angelico 02 Nov '15

02 Nov '15

PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release (and actually, it has 3.5.0 Final listed as a future release). About when is it likely to happen? The one thing I'm hanging out for is an installer patch on Windows that detects XP and immediately aborts with a convenient error; if the number of emails to python-list is indicative, there are a lot of people out there getting confused. ChrisA [1] https://www.python.org/dev/peps/pep-0478/

5 9

Unable to submit a patch to the tracker
by Serhiy Storchaka 01 Nov '15

01 Nov '15

I'm unable to submit any file to any issue, neither via web-form nor via e-mail. Checked with different browsers from different computers. Meta-tracker doesn't work too. http://psf.upfronthosting.co.za/roundup/meta/issue575

2 2

If you shadow a module in the standard library that IDLE depends on, bad things happen
by Laura Creighton 01 Nov '15

01 Nov '15

see the following: lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py lac@smartwheels:~/junk$ idle-python3.5 hello there Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.5/idlelib/run.py", line 10, in <module> from idlelib import CallTips File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in <module> from idlelib.HyperParser import HyperParser File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in <module> _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_") AttributeError: module 'string' has no attribute 'ascii_letters' IDLE then produces a popup that says: IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess por personal firewall software is blocking the connection. <ok> -------- I think that life would be a whole lot easier for people if instead we got a message: Warning: local file /u/lac/junk/string.py shadows module named string in the Standard Library I think that it is python exec that would have to do this -- though of course the popup could also warn about shadowing in general, instead of sending people on wild goose chases over their firewalls. Would this be hard to do? Laura

10 20

Should PEP 498 specify if rf'...' is valid?
by Ryan Gonzalez 01 Nov '15

01 Nov '15

It mentions fr'...' as a formatted raw string but doesn't say anything about rf'...'. Right now, in implementing PEP 498 support in Howl (https://github.com/howl-editor/howl/pull/118 and https://github.com/howl-editor/howl/commit/1e577da89efc1c1de780634b531f6434…), I assumed both were valid. Should the PEP be more specific? BTW, at the rate language-python is going, GitHub will get syntax highlighting for f-strings in 2050. :D -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.

12 24