Mailman 3 August 2016 - Python-Dev

New calling convention to avoid temporarily tuples when calling functions
by Victor Stinner Aug. 24, 2016

Aug. 24, 2016

Hi, tl;dr I found a way to make CPython 3.6 faster and I validated that there is no performance regression. I'm requesting approval of core developers to start pushing changes. In 2014 during a lunch at Pycon, Larry Hasting told me that he would like to get rid of temporary tuples to call functions in Python. In Python, positional arguments are passed as a tuple to C functions: "PyObject *args". Larry wrote Argument Clinic which gives more control on how C functions are called. But I guess that Larry didn't have time to finish his implementation, since he didn't publish a patch. While trying to optimize CPython 3.6, I wrote a proof-of-concept patch and results were promising: https://bugs.python.org/issue26814#msg264003 https://bugs.python.org/issue26814#msg266359 C functions get a C array "PyObject **args, int nargs". Getting the nth argument become "arg = args[n];" at the C level. This format is not new, it's already used internally in Python/ceval.c. A Python function call made from a Python function already avoids a temporary tuple in most cases: we pass the stack of the first function as the list of arguments to the second function. My patch generalizes the idea to C functions. It works in all directions (C=>Python, Python=>C, C=>C, etc.). Many function calls become faster than Python 3.5 with my full patch, but even faster than Python 2.7! For multiple reasons (not interesting here), tested functions are slower in Python 3.4 than Python 2.7. Python 3.5 is better than Python 3.4, but still slower than Python 2.7 in a few cases. Using my "FASTCALL" patch, all tested function calls become faster or as fast as Python 2.7! But when I ran the CPython benchmark suite, I found some major performance regressions. In fact, it took me 3 months to understand that I didn't run benchmarks correctly and that most benchmarks of the CPython benchmark suite are very unstable. I wrote articles explaining how benchmarks should be run (to be stable) and I patched all benchmarks to use my new perf module which runs benchmarks in multiple processes and computes the average (to make benchmarks more stable). At the end, my minimum FASTCALL patch (issue #27128) doesn't show any major performance regression if you run "correctly" benchmarks :-) https://bugs.python.org/issue27128#msg272197 Most benchmarks are not significant, 14 are faster, and only 4 are slower. According to benchmarks on the "full" FASTCALL patch, the slowdown are temporary and should become quickly speedup (with further changes). My question is now: can I push fastcall-2.patch of the issue #27128? This patch only adds the infrastructure to start working on more useful optimizations, more patches will come, I expect more exciting benchmark results. Overview of the initial FASTCALL patch, see my first message on the issue: https://bugs.python.org/issue27128#msg266422 -- Note: My full FASTCALL patch changes the C API: this is out of the scope of my first simple FASTCALL patch. I will open a new discussion to decide if it's worth it and if yes, how it should be done. Victor

4 9

socket.setsockopt() with optval=NULL
by Christian Heimes Aug. 22, 2016

Aug. 22, 2016

Hi, the socket.setsockopt(level, optname, value) method has two calling variants. When it is called with a buffer-like object as value, it calls the C API function setsockopt() with optval=buffer.buf and optlen=buffer.len. When value is an integer, setsockopt() packs it as int32 and sends it with optlen=4. --- # example.py import socket sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, b'\x00\x00\x00\x00') sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) --- $ strace -e setsockopt ./python example.py setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 For AF_ALG (Linux Kernel crypto) I need a way to call the C API function setsockopt() with optval=NULL and optlen as any arbitrary number. I have been playing with multiple ideas. So far I liked the idea of value=(None, int) most. setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, (None, taglen)) What do you think? Christian

4 3

Deprecating invalid escape sequences: review request
by Emanuel Barry Aug. 22, 2016

Aug. 22, 2016

Hello Python-dev, some time ago I went ahead and implemented a patch to deprecate the invalid escape sequences (e.g. \c, \g, \h, etc.) in str and bytes literals. The change itself is pretty straightforward, and shouldn't be hard to review. The change was split in two patches; one which does the actual deprecation and adds relevant tests, and the other fixes all invalid escape sequences in the entire CPython distribution (this resulted in a substantial patch file of over 2000 lines). I'd like to get this reviewed and merged in time, so I'm asking here. Thanks in advance! http://bugs.python.org/issue27364 -Emanuel

1 0

Anyone know Brendan Scott, author of 'Python for Kids'?
by Terry Reedy Aug. 22, 2016

Aug. 22, 2016

Brendan Scott, author of 'Python for Kids for Dummies' is revising his book to produce a Python 3 version. Great so far. Unfortunately, he thinks that Python 3 turned range() into a (generator) function that produces a generator, one that can be indexed and sliced. I left a comment on his blog explaining that range() still produces arithmetic sequence objects. The difference is that in Py 3, the compact representation is not expanded to a verbose (and redundant) Python list until requested. It is *not* like map and filter, which *were* changed to return iterators. For whatever reason, my comment had no effect and today on https://python4kids.brendanscott.com/2016/08/21/python-for-kids-python-3-pr… he still says, near the top >>> # range(10) is no longer a list. Rather, it's a generator >>> # so the [:] operator slices the generator. You can use list() >>> # to see what list the generator corresponds to. I left a second comment, again with interactive code examples, but I fear the result (nothing) will be the same. We cannot stop people from publishing such misleading dis-information, but we can discourage it, and I think we should, especially if we can before paper copies are printed. Beginners have enough problem with iterables versus iterators without a major beginner book confusing the two. I imagine this leading to tutor-list and Stackoverflow questions like "Why doesn't slicing my generator work? It did for range?" So, if you agree with me, please either write Brendan personally if you know him, or just leave your own comment on the blog. -- Terry Jan Reedy

2 1

Re: [Python-Dev] Failures in test_site.py - how to debug?
by Chris Angelico Aug. 19, 2016

Aug. 19, 2016

On Sat, Aug 20, 2016 at 8:07 AM, Sjoerd Job Postmus <sjoerdjob(a)sjec.nl> wrote: > I'd like to re-iterate my suggestion in case it was missed: split the > current test in 2 tests: > > * Running with `-S` which is for checking that by default the collections > are not imported. (Which is what is currently tested) > * Running without `-S` which is for checking that `site` gets imported. > > I hope that is clear enough. Do you see any problems with such an approach? AIUI this wouldn't test that site is sufficiently slim. The test is supposed to mimic normal startup, then assert that unnecessary modules haven't been loaded. Splitting it would prove that a -S run is nice and slim, but would allow bloat in site.py to pass unnoticed. ChrisA

2 2

Re: [Python-Dev] Failures in test_site.py - how to debug?
by Chris Angelico Aug. 19, 2016

Aug. 19, 2016

On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower <steve.dower(a)python.org> wrote: > Check any .pth files you can find. I suspect mpl_toolkits has some magic in > it to make the namespace package work on 2.7. >>> sys.path ['/usr/local/lib/python36.zip', '/home/rosuav/cpython/Lib', '/home/rosuav/cpython/Lib/plat-x86_64-linux-gnu', '/home/rosuav/cpython/build/lib.linux-x86_64-3.6', '/usr/local/lib/python3.6/site-packages'] /usr/local/lib/python36.zip doesn't exist. In /usr/local/lib/python3.6/site-packages are a bunch of things that I've installed using pip, but grepping for '/usr' in the -v output comes up blank. There is, however, one .pth file there: $ cat /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth import sys, types, os;p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('mpl_toolkits',));ie = os.path.exists(os.path.join(p,'__init__.py'));m = not ie and sys.modules.setdefault('mpl_toolkits', types.ModuleType('mpl_toolkits'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p) Is it possible that that's the cause? And if so, should "python -I" remove /usr/local/lib/python3.6/site-packages from sys.path? ChrisA

4 9

Re: [Python-Dev] Failures in test_site.py - how to debug?
by Sjoerd Job Postmus Aug. 19, 2016

Aug. 19, 2016

1 0

Re: [Python-Dev] Failures in test_site.py - how to debug?
by Sjoerd Job Postmus Aug. 19, 2016

Aug. 19, 2016

1 0

Failures in test_site.py - how to debug?
by Chris Angelico Aug. 19, 2016

Aug. 19, 2016

On my main dev system (Debian Stretch), I've had a single long-standing test failure - test_site.py, StartupImportTests.test_startup_imports. It's annoying (partly because it's such a noisy failure), and doesn't appear to be happening on the buildbots, nor presumably on core devs' computers, so it's probably not a Python bug. How can I go about figuring out what's actually going on here? The test effectively runs this (but with -v, which makes it noisy): $ ./python -I -c 'import sys; print(set(sys.modules))' {'_frozen_importlib_external', 'keyword', '_stat', 'weakref', 'sysconfig', 'genericpath', '_sysconfigdata_m', '_io', '_operator', '_bootlocale', 'encodings.utf_8', 'itertools', '_locale', 'codecs', '_weakrefset', 'operator', '_frozen_importlib', 'types', '_sitebuiltins', 'os', 'posixpath', '_functools', '__main__', '_weakref', '_warnings', '_codecs', 'builtins', 'posix', 'functools', 'io', 'encodings.aliases', 'collections.abc', 'encodings', '_thread', 'collections', 'errno', '_collections', '_collections_abc', '_heapq', '_imp', 'site', 'sys', 'reprlib', '_signal', 'stat', 'zipimport', 'abc', 'encodings.latin_1', 'os.path', 'marshal', 'heapq', 'mpl_toolkits'} It then runs into a failure when it asserts that this set has no intersection with the set of collection_mods. Apparently 'import sys' on my system (even with -I for isolation mode) loads up a bunch of modules that it shouldn't be loading. How do I track down what and why, and figure out whether it's a config problem unique to my system or not? ChrisA

3 3

Summary of Python tracker Issues
by Python tracker Aug. 19, 2016

Aug. 19, 2016

ACTIVITY SUMMARY (2016-08-12 - 2016-08-19) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5594 ( -1) closed 33965 (+56) total 39559 (+55) Open issues with patches: 2438 Issues opened (34) ================== #26988: Add AutoNumberedEnum to stdlib http://bugs.python.org/issue26988 reopened by rhettinger #27611: test_tix cannot import _default_root after test_idle http://bugs.python.org/issue27611 reopened by serhiy.storchaka #27748: Simplify test_winsound http://bugs.python.org/issue27748 opened by zach.ware #27749: multprocessing errors on Windows: WriteFile() argument 1 must http://bugs.python.org/issue27749 opened by wevsty #27752: CSV DictReader default dialect name 'excel' is misleading, as http://bugs.python.org/issue27752 opened by lockywolf #27755: Retire DynOptionMenu with a ttk Combobox http://bugs.python.org/issue27755 opened by jfoo #27756: Add pyd icon for 3.6 http://bugs.python.org/issue27756 opened by Decorater #27759: selectors incorrectly retain invalid file descriptors http://bugs.python.org/issue27759 opened by Mark.Williams #27761: Private _nth_root function loses accuracy http://bugs.python.org/issue27761 opened by steven.daprano #27763: Add complex case to test_builtin abs() http://bugs.python.org/issue27763 opened by Evelyn Mitchell #27764: [Patch] Complete bits necessary for making fcntl's file lockin http://bugs.python.org/issue27764 opened by EdSchouten #27766: Add ChaCha20 Poly1305 to SSL ciphers http://bugs.python.org/issue27766 opened by christian.heimes #27768: ssl: add public API for IA-32 processor capabilities vector http://bugs.python.org/issue27768 opened by christian.heimes #27769: IDLE: Replace All up, no wrap replaces one up, all down http://bugs.python.org/issue27769 opened by Qwert225 #27772: Refer to actual format string when creating “zero padding” http://bugs.python.org/issue27772 opened by bignose #27776: PEP 524: Make os.urandom() blocking on Linux http://bugs.python.org/issue27776 opened by haypo #27777: cgi.FieldStorage can't parse simple body with Content-Length a http://bugs.python.org/issue27777 opened by rr- #27778: PEP 524: Add os.getrandom() http://bugs.python.org/issue27778 opened by haypo #27779: Sync-up docstrings in C version of the the decimal module http://bugs.python.org/issue27779 opened by rhettinger #27780: memory leaks in pgen build step abort build with address sanit http://bugs.python.org/issue27780 opened by geeknik #27781: Change sys.getfilesystemencoding() on Windows to UTF-8 http://bugs.python.org/issue27781 opened by steve.dower #27782: Multi-phase extension module initialization, inconsistent exce http://bugs.python.org/issue27782 opened by xiang.zhang #27784: Random failure of test_TCPServer() of test.test_socketserver.S http://bugs.python.org/issue27784 opened by haypo #27787: Avoid test_main() in test_httplib; gc.collect() dangling threa http://bugs.python.org/issue27787 opened by martin.panter #27788: platform module's version number doesn't match its docstring http://bugs.python.org/issue27788 opened by brett.cannon #27790: test_distutils spews linker messages on Windows http://bugs.python.org/issue27790 opened by terry.reedy #27791: test_threading: test_threads_join_2() failed with "Fatal Pytho http://bugs.python.org/issue27791 opened by haypo #27792: bool % int has inconsistent return type. http://bugs.python.org/issue27792 opened by mark.dickinson #27794: setattr a read-only property; the AttributeError should show t http://bugs.python.org/issue27794 opened by ztane #27796: Expose DirEntry constructor http://bugs.python.org/issue27796 opened by moloney #27797: ASCII file with UNIX line conventions and enough lines throws http://bugs.python.org/issue27797 opened by mjpieters #27799: Fix base64-codec and bz2-codec incremental decoders http://bugs.python.org/issue27799 opened by martin.panter #27800: Regular expressions with multiple repeat codes http://bugs.python.org/issue27800 opened by martin.panter #27802: Add __eq__ and __ne__ to collections.abc.Sequence. http://bugs.python.org/issue27802 opened by neil.g Most recent 15 issues with no replies (15) ========================================== #27799: Fix base64-codec and bz2-codec incremental decoders http://bugs.python.org/issue27799 #27796: Expose DirEntry constructor http://bugs.python.org/issue27796 #27772: Refer to actual format string when creating “zero padding” http://bugs.python.org/issue27772 #27764: [Patch] Complete bits necessary for making fcntl's file lockin http://bugs.python.org/issue27764 #27748: Simplify test_winsound http://bugs.python.org/issue27748 #27741: datetime.datetime.strptime functionality description incorrect http://bugs.python.org/issue27741 #27737: email.header.Header.encode() crashes with IndexError on spaces http://bugs.python.org/issue27737 #27733: Test that self.bell is called in IDLE classes when intended. http://bugs.python.org/issue27733 #27713: Spurious "platform dependent libraries" warnings when running http://bugs.python.org/issue27713 #27705: Updating old C:/Windows/System32/ucrtbased.dll http://bugs.python.org/issue27705 #27701: [posixmodule] [Refactoring patch] Simply call into *at() funct http://bugs.python.org/issue27701 #27696: win_add2path.py does not add user site directory http://bugs.python.org/issue27696 #27691: X509 cert with GEN_RID subject alt name causes SytemError http://bugs.python.org/issue27691 #27679: set_bitfields() unused in _ctypes_test http://bugs.python.org/issue27679 #27666: "stack smashing detected" in PyCursesWindow_Box http://bugs.python.org/issue27666 Most recent 15 issues waiting for review (15) ============================================= #27802: Add __eq__ and __ne__ to collections.abc.Sequence. http://bugs.python.org/issue27802 #27799: Fix base64-codec and bz2-codec incremental decoders http://bugs.python.org/issue27799 #27787: Avoid test_main() in test_httplib; gc.collect() dangling threa http://bugs.python.org/issue27787 #27782: Multi-phase extension module initialization, inconsistent exce http://bugs.python.org/issue27782 #27781: Change sys.getfilesystemencoding() on Windows to UTF-8 http://bugs.python.org/issue27781 #27778: PEP 524: Add os.getrandom() http://bugs.python.org/issue27778 #27776: PEP 524: Make os.urandom() blocking on Linux http://bugs.python.org/issue27776 #27766: Add ChaCha20 Poly1305 to SSL ciphers http://bugs.python.org/issue27766 #27759: selectors incorrectly retain invalid file descriptors http://bugs.python.org/issue27759 #27756: Add pyd icon for 3.6 http://bugs.python.org/issue27756 #27755: Retire DynOptionMenu with a ttk Combobox http://bugs.python.org/issue27755 #27748: Simplify test_winsound http://bugs.python.org/issue27748 #27746: ResourceWarnings in test_asyncio http://bugs.python.org/issue27746 #27744: Add AF_ALG (Linux Kernel crypto) to socket module http://bugs.python.org/issue27744 #27740: Fix doc of Py_CompileStringExFlags http://bugs.python.org/issue27740 Top 10 most discussed issues (10) ================================= #26988: Add AutoNumberedEnum to stdlib http://bugs.python.org/issue26988 27 msgs #27761: Private _nth_root function loses accuracy http://bugs.python.org/issue27761 22 msgs #23591: enum: Add Flags and IntFlags http://bugs.python.org/issue23591 19 msgs #27749: multprocessing errors on Windows: WriteFile() argument 1 must http://bugs.python.org/issue27749 11 msgs #27598: Add Collection to collections.abc and typing http://bugs.python.org/issue27598 10 msgs #27766: Add ChaCha20 Poly1305 to SSL ciphers http://bugs.python.org/issue27766 10 msgs #27781: Change sys.getfilesystemencoding() on Windows to UTF-8 http://bugs.python.org/issue27781 9 msgs #1602: windows console doesn't print or input Unicode http://bugs.python.org/issue1602 8 msgs #27181: Add geometric mean to `statistics` module http://bugs.python.org/issue27181 8 msgs #16764: Make zlib accept keyword-arguments http://bugs.python.org/issue16764 7 msgs Issues closed (53) ================== #9998: ctypes find_library should search LD_LIBRARY_PATH on Linux http://bugs.python.org/issue9998 closed by python-dev #10910: pyport.h FreeBSD/Mac OS X "fix" causes errors in C++ compilati http://bugs.python.org/issue10910 closed by ned.deily #12713: argparse: allow abbreviation of sub commands by users http://bugs.python.org/issue12713 closed by python-dev #12946: PyModule_GetDict() claims it can never fail, but it can http://bugs.python.org/issue12946 closed by berker.peksag #22395: test_pathlib error for complex symlinks on Windows http://bugs.python.org/issue22395 closed by jfoo #24637: locals dictionary in PyRun_String http://bugs.python.org/issue24637 closed by berker.peksag #24853: Py_Finalize doesn't clean up PyImport_Inittab http://bugs.python.org/issue24853 closed by haypo #25628: Make namedtuple "verbose" and "rename" parameters into keyword http://bugs.python.org/issue25628 closed by rhettinger #25805: Failure in test_pkgutil run from command-line http://bugs.python.org/issue25805 closed by brett.cannon #25958: Implicit ABCs have no means of "anti-registration" http://bugs.python.org/issue25958 closed by gvanrossum #26200: SETREF adds unnecessary work in some cases http://bugs.python.org/issue26200 closed by serhiy.storchaka #26754: PyUnicode_FSDecoder() accepts arbitrary iterable http://bugs.python.org/issue26754 closed by serhiy.storchaka #26800: Don't accept bytearray as filenames part 2 http://bugs.python.org/issue26800 closed by serhiy.storchaka #26823: Shrink recursive tracebacks http://bugs.python.org/issue26823 closed by ncoghlan #27173: Modern Unix key bindings for IDLE http://bugs.python.org/issue27173 closed by terry.reedy #27558: SystemError with bare `raise` in threading or multiprocessing http://bugs.python.org/issue27558 closed by haypo #27573: code.interact() should print an exit message http://bugs.python.org/issue27573 closed by steven.daprano #27594: Assertion failure when running "test_ast" tests with coverage. http://bugs.python.org/issue27594 closed by ned.deily #27698: socketpair not in socket.__all__ on Windows http://bugs.python.org/issue27698 closed by haypo #27704: bytes(x) is slow when x is bytearray http://bugs.python.org/issue27704 closed by serhiy.storchaka #27712: Tiny typos in import.rst http://bugs.python.org/issue27712 closed by brett.cannon #27719: Misleading note about "args" attribute in "User-defined Except http://bugs.python.org/issue27719 closed by rhettinger #27720: decimal.Context.to_eng_string wrong docstring http://bugs.python.org/issue27720 closed by rhettinger #27723: Document typing.Text and typing.AnyStr http://bugs.python.org/issue27723 closed by gvanrossum #27725: Use Py_SIZE(x) instead of x->ob_size http://bugs.python.org/issue27725 closed by skrah #27726: ctags -t does not exists in Makefile.pre.in http://bugs.python.org/issue27726 closed by haypo #27734: Memory leaks at Python35-32 http://bugs.python.org/issue27734 closed by r.david.murray #27736: repeated Py_Initialize/PyRun_SimpleString/Py_Finalize segfault http://bugs.python.org/issue27736 closed by ned.deily #27742: Random.randint generates different values in Python2 and Pytho http://bugs.python.org/issue27742 closed by rhettinger #27747: Broken example in the queue module documentation http://bugs.python.org/issue27747 closed by Paulo Gabriel Poiati #27750: Idle Shell freezes when printing 100000+ char lines. http://bugs.python.org/issue27750 closed by terry.reedy #27751: Itertools -> Recipes -> pairwise() http://bugs.python.org/issue27751 closed by YoSTEALTH #27753: py_compile.compile() does not specify if it can accept a file http://bugs.python.org/issue27753 closed by r.david.murray #27754: Changing the meaning of bool.__invert__ http://bugs.python.org/issue27754 closed by benjamin.peterson #27757: eval() does not allow import statements to run. http://bugs.python.org/issue27757 closed by steven.daprano #27758: integer overflow in the _csv module's join_append_data functio http://bugs.python.org/issue27758 closed by python-dev #27760: integer overflow in binascii.b2a_qp http://bugs.python.org/issue27760 closed by python-dev #27762: ConfigParser ignores in-line comments for lines with semi-colo http://bugs.python.org/issue27762 closed by lukasz.langa #27765: Accept UTF-8 encoded bytes as input to json.loads() http://bugs.python.org/issue27765 closed by haypo #27767: Receive "A required privilege is not held by the Client" error http://bugs.python.org/issue27767 closed by ebarry #27770: remove run_until_complete http://bugs.python.org/issue27770 closed by gvanrossum #27771: Add a clear screen button or menu choice for IDLE http://bugs.python.org/issue27771 closed by terry.reedy #27773: Excessive Py_XDECREF in the ssl module: http://bugs.python.org/issue27773 closed by python-dev #27774: Py_DECREF on a non-owned object in _sre http://bugs.python.org/issue27774 closed by python-dev #27775: `shlex.split(None)` causes infinite loop http://bugs.python.org/issue27775 closed by r.david.murray #27783: potential use of unitialized memory in operator.methodcaller http://bugs.python.org/issue27783 closed by python-dev #27785: Module platform: Versions of Windows http://bugs.python.org/issue27785 closed by steve.dower #27786: longobject.c: simplify x_sub(), inline _PyLong_Negate() http://bugs.python.org/issue27786 closed by haypo #27789: test_asyncio Resource Warnings http://bugs.python.org/issue27789 closed by berker.peksag #27793: Double underscore variables in module are mangled when used in http://bugs.python.org/issue27793 closed by r.david.murray #27795: Cygwin compile errors http://bugs.python.org/issue27795 closed by r.david.murray #27798: test_sysconfig: test_srcdir() fails on "x86 Gentoo Installed w http://bugs.python.org/issue27798 closed by zach.ware #27801: test_update_lines_cols from test.test_curses.MiscTests fails o http://bugs.python.org/issue27801 closed by berker.peksag

1 0