Mailman 3 September 2017 - Python-Dev

Pygments in PEPs?
by Antoine Pitrou 08 Sep '17

08 Sep '17

Hi, Is it possible to install pygments on the machine which builds and renders PEPs? It would allow syntax highlighting in PEPs, making example code more readable. Regards Antoine.

4 5

make multissltests
by Christian Heimes 08 Sep '17

08 Sep '17

For your information, You can now automatically compile and check the ssl module with multiple versions of OpenSSL and LibreSSL. The multissltest script downloads tar.gz, compiles the source and installs headers + shared lib into a local directory. It takes rather long the first time because OpenSSL does not support parallel builds. Be prepared to wait an hour. After the script has downloaded and installed OpenSSL/LibreSSL, it recompiles the ssl and hashlib module with every version and runs the SSL-related test suite. make multissltests compiles and runs all tests make multisslcompile only compiles See ./python Tools/ssl/multissltests.py --help for additional options. For now the script covers: * OpenSSL 0.9.8zc * OpenSSL 0.9.8zh * OpenSSL 1.0.1u * OpenSSL 1.0.2 * OpenSSL 1.0.2l * OpenSSL 1.1.0f * LibreSSL 2.3.10 * LibreSSL 2.4.5 * LibreSSL 2.5.3 * LibreSSL 2.5.5 Christian

1 0

PEP 549 v2: now titled Instance Descriptors
by Larry Hastings 08 Sep '17

08 Sep '17

I've updated PEP 549 with a new title--"Instance Descriptors" is a better name than "Instance Properties"--and to clarify my rationale for the PEP. I've also updated the prototype with code cleanups and a new type: "collections.abc.InstanceDescriptor", a base class that allows user classes to be instance descriptors. //arry/

3 4

Lazy initialization of module global state
by Neil Schemenauer 08 Sep '17

08 Sep '17

This is an idea that came out of the lazy module loading (via AST analysis), posted to python-ideas. The essential idea is to split the marshal data stored in the .pyc into smaller pieces and only load the parts as they are accessed. E.g. use a __getattr__ hook on the module to unmarshal+exec the code. I have a very early prototype: https://github.com/warsaw/lazyimport/blob/master/lazy_compile.py Would work like a "compile_all.py" tool. It writes standard .pyc files right now. It is not there yet but I should use the AST analysis, like the lazy module load stuff, to determine if things have potential side-effects on module import. Those things will get loaded eagerly, like they do now. Initially I was thinking of class definitions and functions but now I realize any global state could get this treatment. E.g. if you have a large dictionary global, don't unmarshal it until someone accesses the module attribute. This should be pretty safe to do and should give a significant benefit in startup time and memory usage.

2 1

Summary of Python tracker Issues
by Python tracker 08 Sep '17

08 Sep '17

ACTIVITY SUMMARY (2017-09-01 - 2017-09-08) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6149 (-17) closed 36997 (+87) total 43146 (+70) Open issues with patches: 2321 Issues opened (45) ================== #31069: test_multiprocessing_spawn and test_multiprocessing_forkserver https://bugs.python.org/issue31069 reopened by haypo #31327: bug in dateutil\tz\tz.py https://bugs.python.org/issue31327 opened by jerrykramskoy #31329: Add idlelib module entry to doc https://bugs.python.org/issue31329 opened by terry.reedy #31331: IDLE: Move prompts with input. https://bugs.python.org/issue31331 opened by terry.reedy #31332: Building modules by Clang with Microsoft CodeGen https://bugs.python.org/issue31332 opened by tats.u. #31333: Implement ABCMeta in C https://bugs.python.org/issue31333 opened by levkivskyi #31334: select.poll.poll fails on BSDs with arbitrary negative timeout https://bugs.python.org/issue31334 opened by Riccardo Coccioli #31336: Speed up _PyType_Lookup() for class creation https://bugs.python.org/issue31336 opened by scoder #31338: Use abort() for code we never expect to hit https://bugs.python.org/issue31338 opened by barry #31342: test.bisect module causes tests to fail https://bugs.python.org/issue31342 opened by nascheme #31345: Backport docstring improvements to the C version of OrderedDic https://bugs.python.org/issue31345 opened by rhettinger #31346: Prefer PROTOCOL_TLS_CLIENT/SERVER https://bugs.python.org/issue31346 opened by christian.heimes #31348: webbrowser BROWSER with MacOSXOSAScript type https://bugs.python.org/issue31348 opened by michael-lazar #31349: Embedded initialization ignores Py_SetProgramName() https://bugs.python.org/issue31349 opened by chrullrich #31351: ensurepip discards pip's return code which leads to broken ven https://bugs.python.org/issue31351 opened by Igor Filatov #31353: Implement PEP 553 - built-in debug() https://bugs.python.org/issue31353 opened by barry #31354: Fixing a bug related to LTO only build https://bugs.python.org/issue31354 opened by octavian.soldea #31355: Remove Travis CI macOS job: rely on buildbots https://bugs.python.org/issue31355 opened by haypo #31356: Add context manager to temporarily disable GC https://bugs.python.org/issue31356 opened by rhettinger #31357: Expose `worker_target` and `workitem_cls` as arugments to cust https://bugs.python.org/issue31357 opened by justdoit0823 #31359: `configure` script incorrectly detects symbols as available on https://bugs.python.org/issue31359 opened by Maxime Belanger #31361: Update feedparser.py to prevent theano compiling fail in pytho https://bugs.python.org/issue31361 opened by Wei-Shun Lo #31363: __PYVENV_LAUNCHER__ breaks calling another venv's interpreter https://bugs.python.org/issue31363 opened by Ilya.Kulakov #31366: Missing terminator option when using readline with socketserve https://bugs.python.org/issue31366 opened by Thomas Feldmann #31368: RWF_NONBLOCK https://bugs.python.org/issue31368 opened by YoSTEALTH #31369: re.RegexFlag is not included in __all__ https://bugs.python.org/issue31369 opened by PJB3005 #31371: Remove deprecated tkinter.tix module in 3.7 https://bugs.python.org/issue31371 opened by ned.deily #31372: Add SSLSocket.get_verify_result() https://bugs.python.org/issue31372 opened by christian.heimes #31374: expat: warning: "_POSIX_C_SOURCE" redefined https://bugs.python.org/issue31374 opened by christian.heimes #31375: Add the interpreters module to stdlib (PEP 554). https://bugs.python.org/issue31375 opened by eric.snow #31376: test_multiprocessing_spawn randomly hangs AMD64 FreeBSD 10.x S https://bugs.python.org/issue31376 opened by haypo #31377: remove *_INTERNED opcodes from marshal https://bugs.python.org/issue31377 opened by benjamin.peterson #31378: Missing documentation for sqlite3.OperationalError https://bugs.python.org/issue31378 opened by Leonardo Taglialegne #31380: test_undecodable_filename() in Lib/test/test_httpservers.py br https://bugs.python.org/issue31380 opened by howarthjw #31381: Unable to read the project file "pythoncore.vcxproj". https://bugs.python.org/issue31381 opened by denis-osipov #31382: CGI upload error when file ~< 10kb https://bugs.python.org/issue31382 opened by mschaming #31386: Make return types of wrap_bio and wrap_socket customizable https://bugs.python.org/issue31386 opened by christian.heimes #31387: asyncio should make it easy to enable cooperative SIGINT handl https://bugs.python.org/issue31387 opened by ncoghlan #31388: Provide a way to defer SIGINT handling in the current thread https://bugs.python.org/issue31388 opened by ncoghlan #31389: Give pdb.set_trace() an optional `header` keyword argument https://bugs.python.org/issue31389 opened by barry #31390: pydoc.Helper.keywords missing async and await https://bugs.python.org/issue31390 opened by rsw #31391: Forward-port test_xpickle from 2.7 to 3.x https://bugs.python.org/issue31391 opened by zach.ware #31392: Upgrade installers to OpenSSL 1.1.0f https://bugs.python.org/issue31392 opened by steve.dower #31394: Ellipsis_token.type != token.ELLIPSIS https://bugs.python.org/issue31394 opened by Aivar.Annamaa #31395: Docs Downloads are 404s https://bugs.python.org/issue31395 opened by n8henrie Most recent 15 issues with no replies (15) ========================================== #31395: Docs Downloads are 404s https://bugs.python.org/issue31395 #31392: Upgrade installers to OpenSSL 1.1.0f https://bugs.python.org/issue31392 #31391: Forward-port test_xpickle from 2.7 to 3.x https://bugs.python.org/issue31391 #31390: pydoc.Helper.keywords missing async and await https://bugs.python.org/issue31390 #31386: Make return types of wrap_bio and wrap_socket customizable https://bugs.python.org/issue31386 #31382: CGI upload error when file ~< 10kb https://bugs.python.org/issue31382 #31380: test_undecodable_filename() in Lib/test/test_httpservers.py br https://bugs.python.org/issue31380 #31376: test_multiprocessing_spawn randomly hangs AMD64 FreeBSD 10.x S https://bugs.python.org/issue31376 #31375: Add the interpreters module to stdlib (PEP 554). https://bugs.python.org/issue31375 #31374: expat: warning: "_POSIX_C_SOURCE" redefined https://bugs.python.org/issue31374 #31372: Add SSLSocket.get_verify_result() https://bugs.python.org/issue31372 #31368: RWF_NONBLOCK https://bugs.python.org/issue31368 #31366: Missing terminator option when using readline with socketserve https://bugs.python.org/issue31366 #31363: __PYVENV_LAUNCHER__ breaks calling another venv's interpreter https://bugs.python.org/issue31363 #31357: Expose `worker_target` and `workitem_cls` as arugments to cust https://bugs.python.org/issue31357 Most recent 15 issues waiting for review (15) ============================================= #31392: Upgrade installers to OpenSSL 1.1.0f https://bugs.python.org/issue31392 #31389: Give pdb.set_trace() an optional `header` keyword argument https://bugs.python.org/issue31389 #31386: Make return types of wrap_bio and wrap_socket customizable https://bugs.python.org/issue31386 #31381: Unable to read the project file "pythoncore.vcxproj". https://bugs.python.org/issue31381 #31346: Prefer PROTOCOL_TLS_CLIENT/SERVER https://bugs.python.org/issue31346 #31336: Speed up _PyType_Lookup() for class creation https://bugs.python.org/issue31336 #31325: req_rate is a namedtuple type rather than instance https://bugs.python.org/issue31325 #31315: assertion failure in imp.create_dynamic(), when spec.name is n https://bugs.python.org/issue31315 #31310: semaphore tracker isn't protected against crashes https://bugs.python.org/issue31310 #31308: forkserver process isn't re-launched if it died https://bugs.python.org/issue31308 #31294: ZeroMQSocketListener and ZeroMQSocketHandler examples in the L https://bugs.python.org/issue31294 #31184: Fix data descriptor detection in inspect.getattr_static https://bugs.python.org/issue31184 #31179: Speed-up dict.copy() up to 5.5 times. https://bugs.python.org/issue31179 #31175: Exception while extracting file from ZIP with non-matching fil https://bugs.python.org/issue31175 #31151: socketserver.ForkingMixIn.server_close() leaks zombie processe https://bugs.python.org/issue31151 Top 10 most discussed issues (10) ================================= #29988: with statements are not ensuring that __exit__ is called if __ https://bugs.python.org/issue29988 26 msgs #31338: Use abort() for code we never expect to hit https://bugs.python.org/issue31338 18 msgs #14976: queue.Queue() is not reentrant, so signals and GC can cause de https://bugs.python.org/issue14976 14 msgs #31336: Speed up _PyType_Lookup() for class creation https://bugs.python.org/issue31336 14 msgs #17852: Built-in module _io can lose data from buffered files at exit https://bugs.python.org/issue17852 7 msgs #30403: PEP 547: Running extension modules using -m switch https://bugs.python.org/issue30403 7 msgs #31355: Remove Travis CI macOS job: rely on buildbots https://bugs.python.org/issue31355 7 msgs #30860: Consolidate stateful C globals under a single struct. https://bugs.python.org/issue30860 6 msgs #31333: Implement ABCMeta in C https://bugs.python.org/issue31333 6 msgs #31377: remove *_INTERNED opcodes from marshal https://bugs.python.org/issue31377 6 msgs Issues closed (82) ================== #10746: ctypes c_long & c_bool have incorrect PEP-3118 type codes https://bugs.python.org/issue10746 closed by pitrou #11943: Add TLS-SRP (RFC 5054) support to ssl, _ssl, http, and urllib https://bugs.python.org/issue11943 closed by christian.heimes #12197: non-blocking SSL write fails if a partial write was issued https://bugs.python.org/issue12197 closed by christian.heimes #12633: sys.modules doc entry should reflect restrictions https://bugs.python.org/issue12633 closed by eric.snow #14191: argparse doesn't allow optionals within positionals https://bugs.python.org/issue14191 closed by r.david.murray #15464: ssl: add set_msg_callback function https://bugs.python.org/issue15464 closed by christian.heimes #16988: argparse: PARSER option for nargs not documented https://bugs.python.org/issue16988 closed by r.david.murray #19084: No way to use TLS-PSK from python ssl https://bugs.python.org/issue19084 closed by christian.heimes #20994: Disable TLS Compression https://bugs.python.org/issue20994 closed by christian.heimes #21649: Mention "Recommendations for Secure Use of TLS and DTLS" https://bugs.python.org/issue21649 closed by Mariatta #21818: cookielib documentation references Cookie module, not cookieli https://bugs.python.org/issue21818 closed by berker.peksag #22536: subprocess should include filename in FileNotFoundError except https://bugs.python.org/issue22536 closed by gregory.p.smith #22635: subprocess.getstatusoutput changed behavior in 3.4 (maybe 3.3. https://bugs.python.org/issue22635 closed by gregory.p.smith #23863: Fix EINTR Socket Module issues in 2.7 https://bugs.python.org/issue23863 closed by haypo #24516: SSL create_default_socket purpose insufficiently documented https://bugs.python.org/issue24516 closed by christian.heimes #25674: test_ssl (test_algorithms) failures on bolen-ubuntu slaves: sh https://bugs.python.org/issue25674 closed by christian.heimes #26612: test_ssl: use context manager (with) to fix ResourceWarning https://bugs.python.org/issue26612 closed by haypo #26904: Difflib quick_ratio() could use Counter() https://bugs.python.org/issue26904 closed by mscuthbert #27144: concurrent.futures.as_completed() memory inefficiency https://bugs.python.org/issue27144 closed by pitrou #27340: bytes-like objects with socket.sendall(), SSL, and http.client https://bugs.python.org/issue27340 closed by christian.heimes #27448: Race condition in subprocess.Popen which causes a huge memory https://bugs.python.org/issue27448 closed by gregory.p.smith #27584: New addition of vSockets to the python socket module https://bugs.python.org/issue27584 closed by christian.heimes #27629: Cannot create ssl.SSLSocket without existing socket https://bugs.python.org/issue27629 closed by christian.heimes #27768: ssl: get CPU cap flags for AESNI and PCLMULQDQ https://bugs.python.org/issue27768 closed by christian.heimes #28085: SSL: Add client and server protocols for SSLContext https://bugs.python.org/issue28085 closed by christian.heimes #28191: Support RFC4985 SRVName in SAN name https://bugs.python.org/issue28191 closed by christian.heimes #28196: ssl.match_hostname() should check for SRV-ID and URI-ID https://bugs.python.org/issue28196 closed by christian.heimes #28411: Eliminate PyInterpreterState.modules. https://bugs.python.org/issue28411 closed by eric.snow #28588: Memory leak in OpenSSL thread state https://bugs.python.org/issue28588 closed by christian.heimes #28958: Python should return comperhansive error when SSLContext canno https://bugs.python.org/issue28958 closed by christian.heimes #29136: Add OP_NO_TLSv1_3 https://bugs.python.org/issue29136 closed by christian.heimes #29212: Python 3.6 logging thread name regression with concurrent.futu https://bugs.python.org/issue29212 closed by gregory.p.smith #29588: importing ssl can fail with NameError: name 'PROTOCOL_TLS' is https://bugs.python.org/issue29588 closed by christian.heimes #29627: configparser.ConfigParser.read() has undocumented/unexpected b https://bugs.python.org/issue29627 closed by lukasz.langa #29781: SSLObject.version returns incorrect value before handshake. https://bugs.python.org/issue29781 closed by christian.heimes #29824: Hostname validation in SSL match_hostname() https://bugs.python.org/issue29824 closed by christian.heimes #30096: Update examples in abc documentation to use abc.ABC https://bugs.python.org/issue30096 closed by Mariatta #30102: improve performance of libSSL usage on hashing https://bugs.python.org/issue30102 closed by christian.heimes #30502: Fix buffer handling of OBJ_obj2txt https://bugs.python.org/issue30502 closed by christian.heimes #30622: Fix NPN guard for OpenSSL 1.1 https://bugs.python.org/issue30622 closed by christian.heimes #30640: NULL + 1 in _PyFunction_FastCallDict(), PyEval_EvalCodeEx() https://bugs.python.org/issue30640 closed by haypo #30662: fix OrderedDict.__init__ docstring to reflect PEP 468 https://bugs.python.org/issue30662 closed by Mariatta #30737: Update devguide link to the new URL https://bugs.python.org/issue30737 closed by Mariatta #30828: Out of bounds write in _asyncio_Future_remove_done_callback https://bugs.python.org/issue30828 closed by serhiy.storchaka #30867: Add macro `HAVE_OPENSSL_VERIFY_PARAM` to avoid invalid functio https://bugs.python.org/issue30867 closed by christian.heimes #30912: python 3 git master fails to find libffi and build _ctypes on https://bugs.python.org/issue30912 closed by zach.ware #31075: Collections - ChainMap - Documentation example wrong order lin https://bugs.python.org/issue31075 closed by rhettinger #31144: add initializer to concurrent.futures.ProcessPoolExecutor https://bugs.python.org/issue31144 closed by pitrou #31172: Py_Main() is totally broken on Visual Studio 2017 https://bugs.python.org/issue31172 closed by steve.dower #31178: [EASY] subprocess: TypeError: can't concat str to bytes, in _e https://bugs.python.org/issue31178 closed by gregory.p.smith #31185: Miscellaneous errors in asyncio speedup module https://bugs.python.org/issue31185 closed by serhiy.storchaka #31252: Operator.itemgetter documentation should include dictionary ke https://bugs.python.org/issue31252 closed by rhettinger #31270: Simplify documentation of itertools.zip_longest https://bugs.python.org/issue31270 closed by rhettinger #31281: fileinput inplace does not work with pathlib.Path https://bugs.python.org/issue31281 closed by eric.smith #31301: Python 2.7 SIGSEGV https://bugs.python.org/issue31301 closed by serhiy.storchaka #31326: concurrent.futures: ProcessPoolExecutor.shutdown(wait=True) sh https://bugs.python.org/issue31326 closed by haypo #31328: _sha3 is missing from Setup.dist https://bugs.python.org/issue31328 closed by benjamin.peterson #31330: argparse.RawTextHelpFormatter does not maintain lines separate https://bugs.python.org/issue31330 closed by r.david.murray #31335: str.format should support "{:r}" for any type. https://bugs.python.org/issue31335 closed by al45tair #31337: Small opportunity for NULL dereference in compile.c https://bugs.python.org/issue31337 closed by barry #31339: [2.7] time.asctime() crash with year > 9999 using musl C libra https://bugs.python.org/issue31339 closed by haypo #31340: Use VS 2017 compiler for build https://bugs.python.org/issue31340 closed by steve.dower #31341: remove IRIX support code https://bugs.python.org/issue31341 closed by benjamin.peterson #31343: Include major(), minor(), makedev() from sysmacros https://bugs.python.org/issue31343 closed by christian.heimes #31344: f_trace_opcodes frame attribute to switch to per-opcode tracin https://bugs.python.org/issue31344 closed by ncoghlan #31347: possible undefined behavior in _PyObject_FastCall_Prepend https://bugs.python.org/issue31347 closed by benjamin.peterson #31350: asyncio: Optimize get_event_loop and _get_running_loop https://bugs.python.org/issue31350 closed by Mariatta #31352: Travis CI coverage job fails on Python 3.6 https://bugs.python.org/issue31352 closed by haypo #31358: Pull zlib out to cpython-source-deps https://bugs.python.org/issue31358 closed by zach.ware #31360: argparse mutually_exclusive_group under add_argument_group fai https://bugs.python.org/issue31360 closed by martin.panter #31362: "async" and "await" are not keyword https://bugs.python.org/issue31362 closed by Conor Cal #31364: Possible problem with PR #3377 https://bugs.python.org/issue31364 closed by denis-osipov #31365: Multiplication issue with 16.1 https://bugs.python.org/issue31365 closed by martin.panter #31367: [[]]*int gives x of the same list that cannot be appended to o https://bugs.python.org/issue31367 closed by zach.ware #31370: Remove support for threads-less builds https://bugs.python.org/issue31370 closed by haypo #31373: demoting floating float values to unrepresentable types is und https://bugs.python.org/issue31373 closed by benjamin.peterson #31379: run_profile_task in Makefile should add $(RUNSHARED) https://bugs.python.org/issue31379 closed by xiang.zhang #31383: Issue with _posixsubprocess when importing subprocess32 https://bugs.python.org/issue31383 closed by gregory.p.smith #31384: marshal: remove "current_filename" optimization https://bugs.python.org/issue31384 closed by benjamin.peterson #31385: `import as` does not work when module has same same as parent https://bugs.python.org/issue31385 closed by ncoghlan #31393: Fix the use of PyUnicode_READY() https://bugs.python.org/issue31393 closed by serhiy.storchaka #1198569: string.Template not flexible enough to subclass (regexes) https://bugs.python.org/issue1198569 closed by barry

1 0

PEP 553 V2 - builtin breakpoint() (was Re: PEP 553: Built-in debug())
by Barry Warsaw 08 Sep '17

08 Sep '17

Thanks for all the great feedback folks! Here then is PEP 553 version 2. The notable changes are: * Change the name of the built-in from debug() to breakpoint() * Modify the signature to be breakpoint(*args, **kws) https://www.python.org/dev/peps/pep-0553/ Included below for convenience. Cheers, -Barry PEP: 553 Title: Built-in breakpoint() Author: Barry Warsaw <barry(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 Post-History: 2017-09-05, 2017-09-07 Abstract ======== This PEP proposes adding a new built-in function called ``breakpoint()`` which enters a Python debugger at the point of the call. Additionally, two new names are added to the ``sys`` module to make the debugger pluggable. Rationale ========= Python has long had a great debugger in its standard library called ``pdb``. Setting a break point is commonly written like this:: foo() import pdb; pdb.set_trace() bar() Thus after executing ``foo()`` and before executing ``bar()``, Python will enter the debugger. However this idiom has several disadvantages. * It's a lot to type (27 characters). * It's easy to typo. The PEP author often mistypes this line, e.g. omitting the semicolon, or typing a dot instead of an underscore. * It ties debugging directly to the choice of pdb. There might be other debugging options, say if you're using an IDE or some other development environment. * Python linters (e.g. flake8 [1]_) complain about this line because it contains two statements. Breaking the idiom up into two lines further complicates the use of the debugger, These problems can be solved by modeling a solution based on prior art in other languages, and utilizing a convention that already exists in Python. Proposal ======== The JavaScript language provides a ``debugger`` statement [2]_ which enters the debugger at the point where the statement appears. This PEP proposes a new built-in function called ``breakpoint()`` which enters a Python debugger at the call site. Thus the example above would be written like so:: foo() breakpoint() bar() Further, this PEP proposes two new name bindings for the ``sys`` module, called ``sys.breakpointhook()`` and ``sys.__breakpointhook__``. By default, ``sys.breakpointhook()`` implements the actual importing and entry into ``pdb.set_trace()``, and it can be set to a different function to change the debugger that ``breakpoint()`` enters. ``sys.__breakpointhook__`` then stashes the default value of ``sys.breakpointhook()`` to make it easy to reset. This exactly models the existing ``sys.displayhook()`` / ``sys.__displayhook__`` and ``sys.excepthook()`` / ``sys.__excepthook__`` hooks [3]_. The signature of the built-in is ``breakpoint(*args, **kws)``. The positional and keyword arguments are passed straight through to ``sys.breakpointhook()`` and the signatures must match or a ``TypeError`` will be raised. The return from ``sys.breakpointhook()`` is passed back up to, and returned from ``breakpoint()``. Since ``sys.breakpointhook()`` by default calls ``pdb.set_trace()`` by default it accepts no arguments. Open issues =========== Confirmation from other debugger vendors ---------------------------------------- We want to get confirmation from at least one alternative debugger implementation (e.g. PyCharm) that the hooks provided in this PEP will be useful to them. Breakpoint bytecode ------------------- Related, there has been an idea to add a bytecode that calls ``sys.breakpointhook()``. Whether built-in ``breakpoint()`` emits this bytecode (or gets peephole optimized to the bytecode) is an open issue. The bytecode is useful for debuggers that actively modify bytecode streams to trampoline into their own debugger. Having a "breakpoint" bytecode might allow them to avoid bytecode modification in order to invoke this trampoline. *NOTE*: It probably makes sense to split this idea into a separate PEP. Environment variable -------------------- Should we add an environment variable so that ``sys.breakpointhook()`` can be set outside of the Python invocation? E.g.:: $ export PYTHONBREAKPOINTHOOK=my.debugger:Debugger This would provide execution environments such as IDEs which run Python code inside them, to set an internal breakpoint hook before any Python code executes. Call a fancier object by default -------------------------------- Some folks want to be able to use other ``pdb`` interfaces such as ``pdb.pm()``. Although this is a less commonly used API, it could be supported by binding ``sys.breakpointhook`` to an object that implements ``__call__()``. Calling this object would call ``pdb.set_trace()``, but the object could expose other methods, such as ``pdb.pm()``, making invocation of it as handy as ``breakpoint.pm()``. Implementation ============== A pull request exists with the proposed implementation [4]_. Rejected alternatives ===================== A new keyword ------------- Originally, the author considered a new keyword, or an extension to an existing keyword such as ``break here``. This is rejected on several fronts. * A brand new keyword would require a ``__future__`` to enable it since almost any new keyword could conflict with existing code. This negates the ease with which you can enter the debugger. * An extended keyword such as ``break here``, while more readable and not requiring a ``__future__`` would tie the keyword extension to this new feature, preventing more useful extensions such as those proposed in PEP 548. * A new keyword would require a modified grammar and likely a new bytecode. Each of these makes the implementation more complex. A new built-in breaks no existing code (since any existing module global would just shadow the built-in) and is quite easy to implement. sys.breakpoint() ---------------- Why not ``sys.breakpoint()``? Requiring an import to invoke the debugger is explicitly rejected because ``sys`` is not imported in every module. That just requires more typing and would lead to:: import sys; sys.breakpoint() which inherits several of the problems this PEP aims to solve. Version History =============== * 2017-09-07 * ``debug()`` renamed to ``breakpoint()`` * Signature changed to ``breakpoint(*args, **kws)`` which is passed straight through to ``sys.breakpointhook()``. References ========== .. [1] http://flake8.readthedocs.io/en/latest/ .. [2] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statement… .. [3] https://docs.python.org/3/library/sys.html#sys.displayhook .. [4] https://github.com/python/cpython/pull/3355 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

4 7

PEP 548: More Flexible Loop Control
by R. David Murray 08 Sep '17

08 Sep '17

I've written a PEP proposing a small enhancement to the Python loop control statements. Short version: here's what feels to me like a Pythonic way to spell "repeat until": while: <do stuff> break if <done condition> The PEP goes into some detail on why this feels like a readability improvement in the more general case, with examples taken from the standard library: https://www.python.org/dev/peps/pep-0548/ Unlike Larry, I don't have a prototype, and in fact if this idea meets with approval I'll be looking for a volunteer to do the actual implementation. --David PS: this came to me in a dream on Sunday night, and the more I explored the idea the better I liked it. I have no idea what I was dreaming about that resulted in this being the thing left in my mind when I woke up :)

7 9

HTTPS on bugs.python.org
by Victor Stinner 08 Sep '17

08 Sep '17

Hi, When I go to http://bugs.python.org/ Firefox warns me that the form on the left to login (user, password) sends data in clear text (HTTP). Ok, I switch manually to HTTPS: add "s" in "http://" of the URL. I log in. I go to an issue using HTTPS like https://bugs.python.org/issue31250 I modify an issue using the form and click on [Submit Changes] (or just press Enter): I'm back to HTTP. Truncated URL: http://bugs.python.org/issue31250?@ok_message=msg%20301099%20created%... Hum, again I switch manually to HTTPS by modifying the URL: https://bugs.python.org/issue31250?@ok_message=msg%20301099%20created%... I click on the "clear this message" link: oops, I'm back to the HTTP world... http://bugs.python.org/issue31250 So, would it be possible to enforce HTTPS on the bug tracker? The best would be to always generate HTTPS urls and *maybe* redirect HTTP to HTTPS. Sorry, I don't know what are the best practices. For example, should we use HTTPS only cookies? Victor

8 19

PEP 550 v4
by Yury Selivanov 08 Sep '17

08 Sep '17

Hi, This is the 4th iteration of the PEP that Elvis and I have rewritten from scratch. The specification section has been separated from the implementation section, which makes them easier to follow. During the rewrite, we realized that generators and coroutines should work with the EC in exactly the same way (coroutines used to be created with no LC in prior versions of the PEP). We also renamed Context Keys to Context Variables which seems to be a more appropriate name. Hopefully this update will resolve the remaining questions about the specification and the proposed implementation, and will allow us to focus on refining the API. Yury PEP: 550 Title: Execution Context Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io>, Elvis Pranskevichus <elvis(a)magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2017 Python-Version: 3.7 Post-History: 11-Aug-2017, 15-Aug-2017, 18-Aug-2017, 25-Aug-2017 Abstract ======== This PEP adds a new generic mechanism of ensuring consistent access to non-local state in the context of out-of-order execution, such as in Python generators and coroutines. Thread-local storage, such as ``threading.local()``, is inadequate for programs that execute concurrently in the same OS thread. This PEP proposes a solution to this problem. Rationale ========= Prior to the advent of asynchronous programming in Python, programs used OS threads to achieve concurrency. The need for thread-specific state was solved by ``threading.local()`` and its C-API equivalent, ``PyThreadState_GetDict()``. A few examples of where Thread-local storage (TLS) is commonly relied upon: * Context managers like decimal contexts, ``numpy.errstate``, and ``warnings.catch_warnings``. * Request-related data, such as security tokens and request data in web applications, language context for ``gettext`` etc. * Profiling, tracing, and logging in large code bases. Unfortunately, TLS does not work well for programs which execute concurrently in a single thread. A Python generator is the simplest example of a concurrent program. Consider the following:: def fractions(precision, x, y): with decimal.localcontext() as ctx: ctx.prec = precision yield Decimal(x) / Decimal(y) yield Decimal(x) / Decimal(y**2) g1 = fractions(precision=2, x=1, y=3) g2 = fractions(precision=6, x=2, y=3) items = list(zip(g1, g2)) The expected value of ``items`` is:: [(Decimal('0.33'), Decimal('0.666667')), (Decimal('0.11'), Decimal('0.222222'))] Rather surprisingly, the actual result is:: [(Decimal('0.33'), Decimal('0.666667')), (Decimal('0.111111'), Decimal('0.222222'))] This is because Decimal context is stored as a thread-local, so concurrent iteration of the ``fractions()`` generator would corrupt the state. A similar problem exists with coroutines. Applications also often need to associate certain data with a given thread of execution. For example, a web application server commonly needs access to the current HTTP request object. The inadequacy of TLS in asynchronous code has lead to the proliferation of ad-hoc solutions, which are limited in scope and do not support all required use cases. The current status quo is that any library (including the standard library), which relies on TLS, is likely to be broken when used in asynchronous code or with generators (see [3]_ as an example issue.) Some languages, that support coroutines or generators, recommend passing the context manually as an argument to every function, see [1]_ for an example. This approach, however, has limited use for Python, where there is a large ecosystem that was built to work with a TLS-like context. Furthermore, libraries like ``decimal`` or ``numpy`` rely on context implicitly in overloaded operator implementations. The .NET runtime, which has support for async/await, has a generic solution for this problem, called ``ExecutionContext`` (see [2]_). Goals ===== The goal of this PEP is to provide a more reliable ``threading.local()`` alternative, which: * provides the mechanism and the API to fix non-local state issues with coroutines and generators; * has no or negligible performance impact on the existing code or the code that will be using the new mechanism, including libraries like ``decimal`` and ``numpy``. High-Level Specification ======================== The full specification of this PEP is broken down into three parts: * High-Level Specification (this section): the description of the overall solution. We show how it applies to generators and coroutines in user code, without delving into implementation details. * Detailed Specification: the complete description of new concepts, APIs, and related changes to the standard library. * Implementation Details: the description and analysis of data structures and algorithms used to implement this PEP, as well as the necessary changes to CPython. For the purpose of this section, we define *execution context* as an opaque container of non-local state that allows consistent access to its contents in the concurrent execution environment. A *context variable* is an object representing a value in the execution context. A new context variable is created by calling the ``new_context_var()`` function. A context variable object has two methods: * ``lookup()``: returns the value of the variable in the current execution context; * ``set()``: sets the value of the variable in the current execution context. Regular Single-threaded Code ---------------------------- In regular, single-threaded code that doesn't involve generators or coroutines, context variables behave like globals:: var = new_context_var() def sub(): assert var.lookup() == 'main' var.set('sub') def main(): var.set('main') sub() assert var.lookup() == 'sub' Multithreaded Code ------------------ In multithreaded code, context variables behave like thread locals:: var = new_context_var() def sub(): assert var.lookup() is None # The execution context is empty # for each new thread. var.set('sub') def main(): var.set('main') thread = threading.Thread(target=sub) thread.start() thread.join() assert var.lookup() == 'main' Generators ---------- In generators, changes to context variables are local and are not visible to the caller, but are visible to the code called by the generator. Once set in the generator, the context variable is guaranteed not to change between iterations:: var = new_context_var() def gen(): var.set('gen') assert var.lookup() == 'gen' yield 1 assert var.lookup() == 'gen' yield 2 def main(): var.set('main') g = gen() next(g) assert var.lookup() == 'main' var.set('main modified') next(g) assert var.lookup() == 'main modified' Changes to caller's context variables are visible to the generator (unless they were also modified inside the generator):: var = new_context_var() def gen(): assert var.lookup() == 'var' yield 1 assert var.lookup() == 'var modified' yield 2 def main(): g = gen() var.set('var') next(g) var.set('var modified') next(g) Now, let's revisit the decimal precision example from the `Rationale`_ section, and see how the execution context can improve the situation:: import decimal decimal_prec = new_context_var() # create a new context variable # Pre-PEP 550 Decimal relies on TLS for its context. # This subclass switches the decimal context storage # to the execution context for illustration purposes. # class MyDecimal(decimal.Decimal): def __init__(self, value="0"): prec = decimal_prec.lookup() if prec is None: raise ValueError('could not find decimal precision') context = decimal.Context(prec=prec) super().__init__(value, context=context) def fractions(precision, x, y): # Normally, this would be set by a context manager, # but for simplicity we do this directly. decimal_prec.set(precision) yield MyDecimal(x) / MyDecimal(y) yield MyDecimal(x) / MyDecimal(y**2) g1 = fractions(precision=2, x=1, y=3) g2 = fractions(precision=6, x=2, y=3) items = list(zip(g1, g2)) The value of ``items`` is:: [(Decimal('0.33'), Decimal('0.666667')), (Decimal('0.11'), Decimal('0.222222'))] which matches the expected result. Coroutines and Asynchronous Tasks --------------------------------- In coroutines, like in generators, context variable changes are local and are not visible to the caller:: import asyncio var = new_context_var() async def sub(): assert var.lookup() == 'main' var.set('sub') assert var.lookup() == 'sub' async def main(): var.set('main') await sub() assert var.lookup() == 'main' loop = asyncio.get_event_loop() loop.run_until_complete(main()) To establish the full semantics of execution context in couroutines, we must also consider *tasks*. A task is the abstraction used by *asyncio*, and other similar libraries, to manage the concurrent execution of coroutines. In the example above, a task is created implicitly by the ``run_until_complete()`` function. ``asyncio.wait_for()`` is another example of implicit task creation:: async def sub(): await asyncio.sleep(1) assert var.lookup() == 'main' async def main(): var.set('main') # waiting for sub() directly await sub() # waiting for sub() with a timeout await asyncio.wait_for(sub(), timeout=2) var.set('main changed') Intuitively, we expect the assertion in ``sub()`` to hold true in both invocations, even though the ``wait_for()`` implementation actually spawns a task, which runs ``sub()`` concurrently with ``main()``. Thus, tasks **must** capture a snapshot of the current execution context at the moment of their creation and use it to execute the wrapped coroutine whenever that happens. If this is not done, then innocuous looking changes like wrapping a coroutine in a ``wait_for()`` call would cause surprising breakage. This leads to the following:: import asyncio var = new_context_var() async def sub(): # Sleeping will make sub() run after # `var` is modified in main(). await asyncio.sleep(1) assert var.lookup() == 'main' async def main(): var.set('main') loop.create_task(sub()) # schedules asynchronous execution # of sub(). assert var.lookup() == 'main' var.set('main changed') loop = asyncio.get_event_loop() loop.run_until_complete(main()) In the above code we show how ``sub()``, running in a separate task, sees the value of ``var`` as it was when ``loop.create_task(sub())`` was called. Like tasks, the intuitive behaviour of callbacks scheduled with either ``Loop.call_soon()``, ``Loop.call_later()``, or ``Future.add_done_callback()`` is to also capture a snapshot of the current execution context at the point of scheduling, and use it to run the callback:: current_request = new_context_var() def log_error(e): logging.error('error when handling request %r', current_request.lookup()) async def render_response(): ... async def handle_get_request(request): current_request.set(request) try: return await render_response() except Exception as e: get_event_loop().call_soon(log_error, e) return '500 - Internal Server Error' Detailed Specification ====================== Conceptually, an *execution context* (EC) is a stack of logical contexts. There is one EC per Python thread. A *logical context* (LC) is a mapping of context variables to their values in that particular LC. A *context variable* is an object representing a value in the execution context. A new context variable object is created by calling the ``sys.new_context_var(name: str)`` function. The value of the ``name`` argument is not used by the EC machinery, but may be used for debugging and introspection. The context variable object has the following methods and attributes: * ``name``: the value passed to ``new_context_var()``. * ``lookup()``: traverses the execution context top-to-bottom, until the variable value is found. Returns ``None``, if the variable is not present in the execution context; * ``set()``: sets the value of the variable in the topmost logical context. Generators ---------- When created, each generator object has an empty logical context object stored in its ``__logical_context__`` attribute. This logical context is pushed onto the execution context at the beginning of each generator iteration and popped at the end:: var1 = sys.new_context_var('var1') var2 = sys.new_context_var('var2') def gen(): var1.set('var1-gen') var2.set('var2-gen') # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen', var2: 'var2-gen'}) # ] n = nested_gen() # nested_gen_LC is created next(n) # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen', var2: 'var2-gen'}) # ] var1.set('var1-gen-mod') var2.set('var2-gen-mod') # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen-mod', var2: 'var2-gen-mod'}) # ] next(n) def nested_gen(): # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen', var2: 'var2-gen'}), # nested_gen_LC() # ] assert var1.lookup() == 'var1-gen' assert var2.lookup() == 'var2-gen' var1.set('var1-nested-gen') # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen', var2: 'var2-gen'}), # nested_gen_LC({var1: 'var1-nested-gen'}) # ] yield # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen-mod', var2: 'var2-gen-mod'}), # nested_gen_LC({var1: 'var1-nested-gen'}) # ] assert var1.lookup() == 'var1-nested-gen' assert var2.lookup() == 'var2-gen-mod' yield # EC = [outer_LC()] g = gen() # gen_LC is created for the generator object `g` list(g) # EC = [outer_LC()] The snippet above shows the state of the execution context stack throughout the generator lifespan. contextlib.contextmanager ------------------------- Earlier, we've used the following example:: import decimal # create a new context variable decimal_prec = sys.new_context_var('decimal_prec') # ... def fractions(precision, x, y): decimal_prec.set(precision) yield MyDecimal(x) / MyDecimal(y) yield MyDecimal(x) / MyDecimal(y**2) Let's extend it by adding a context manager:: @contextlib.contextmanager def precision_context(prec): old_rec = decimal_prec.lookup() try: decimal_prec.set(prec) yield finally: decimal_prec.set(old_prec) Unfortunately, this would not work straight away, as the modification to the ``decimal_prec`` variable is contained to the ``precision_context()`` generator, and therefore will not be visible inside the ``with`` block:: def fractions(precision, x, y): # EC = [{}, {}] with precision_context(precision): # EC becomes [{}, {}, {decimal_prec: precision}] in the # *precision_context()* generator, # but here the EC is still [{}, {}] # raises ValueError('could not find decimal precision')! yield MyDecimal(x) / MyDecimal(y) yield MyDecimal(x) / MyDecimal(y**2) The way to fix this is to set the generator's ``__logical_context__`` attribute to ``None``. This will cause the generator to avoid modifying the execution context stack. We modify the ``contextlib.contextmanager()`` decorator to set ``genobj.__logical_context__`` to ``None`` to produce well-behaved context managers:: def fractions(precision, x, y): # EC = [{}, {}] with precision_context(precision): # EC = [{}, {decimal_prec: precision}] yield MyDecimal(x) / MyDecimal(y) yield MyDecimal(x) / MyDecimal(y**2) # EC becomes [{}, {decimal_prec: None}] asyncio ------- ``asyncio`` uses ``Loop.call_soon``, ``Loop.call_later``, and ``Loop.call_at`` to schedule the asynchronous execution of a function. ``asyncio.Task`` uses ``call_soon()`` to further the execution of the wrapped coroutine. We modify ``Loop.call_{at,later,soon}`` to accept the new optional *execution_context* keyword argument, which defaults to the copy of the current execution context:: def call_soon(self, callback, *args, execution_context=None): if execution_context is None: execution_context = sys.get_execution_context() # ... some time later sys.run_with_execution_context( execution_context, callback, args) The ``sys.get_execution_context()`` function returns a shallow copy of the current execution context. By shallow copy here we mean such a new execution context that: * lookups in the copy provide the same results as in the original execution context, and * any changes in the original execution context do not affect the copy, and * any changes to the copy do not affect the original execution context. Either of the following satisfy the copy requirements: * a new stack with shallow copies of logical contexts; * a new stack with one squashed logical context. The ``sys.run_with_execution_context(ec, func, *args, **kwargs)`` function runs ``func(*args, **kwargs)`` with *ec* as the execution context. The function performs the following steps: 1. Set *ec* as the current execution context stack in the current thread. 2. Push an empty logical context onto the stack. 3. Run ``func(*args, **kwargs)``. 4. Pop the logical context from the stack. 5. Restore the original execution context stack. 6. Return or raise the ``func()`` result. These steps ensure that *ec* cannot be modified by *func*, which makes ``run_with_execution_context()`` idempotent. ``asyncio.Task`` is modified as follows:: class Task: def __init__(self, coro): ... # Get the current execution context snapshot. self._exec_context = sys.get_execution_context() self._loop.call_soon( self._step, execution_context=self._exec_context) def _step(self, exc=None): ... self._loop.call_soon( self._step, execution_context=self._exec_context) ... Generators Transformed into Iterators ------------------------------------- Any Python generator can be represented as an equivalent iterator. Compilers like Cython rely on this axiom. With respect to the execution context, such iterator should behave the same way as the generator it represents. This means that there needs to be a Python API to create new logical contexts and run code with a given logical context. The ``sys.new_logical_context()`` function creates a new empty logical context. The ``sys.run_with_logical_context(lc, func, *args, **kwargs)`` function can be used to run functions in the specified logical context. The *lc* can be modified as a result of the call. The ``sys.run_with_logical_context()`` function performs the following steps: 1. Push *lc* onto the current execution context stack. 2. Run ``func(*args, **kwargs)``. 3. Pop *lc* from the execution context stack. 4. Return or raise the ``func()`` result. By using ``new_logical_context()`` and ``run_with_logical_context()``, we can replicate the generator behaviour like this:: class Generator: def __init__(self): self.logical_context = sys.new_logical_context() def __iter__(self): return self def __next__(self): return sys.run_with_logical_context( self.logical_context, self._next_impl) def _next_impl(self): # Actual __next__ implementation. ... Let's see how this pattern can be applied to a real generator:: # create a new context variable decimal_prec = sys.new_context_var('decimal_precision') def gen_series(n, precision): decimal_prec.set(precision) for i in range(1, n): yield MyDecimal(i) / MyDecimal(3) # gen_series is equivalent to the following iterator: class Series: def __init__(self, n, precision): # Create a new empty logical context on creation, # like the generators do. self.logical_context = sys.new_logical_context() # run_with_logical_context() will pushes # self.logical_context onto the execution context stack, # runs self._next_impl, and pops self.logical_context # from the stack. return sys.run_with_logical_context( self.logical_context, self._init, n, precision) def _init(self, n, precision): self.i = 1 self.n = n decimal_prec.set(precision) def __iter__(self): return self def __next__(self): return sys.run_with_logical_context( self.logical_context, self._next_impl) def _next_impl(self): decimal_prec.set(self.precision) result = MyDecimal(self.i) / MyDecimal(3) self.i += 1 return result For regular iterators such approach to logical context management is normally not necessary, and it is recommended to set and restore context variables directly in ``__next__``:: class Series: def __next__(self): old_prec = decimal_prec.lookup() try: decimal_prec.set(self.precision) ... finally: decimal_prec.set(old_prec) Asynchronous Generators ----------------------- The execution context semantics in asynchronous generators does not differ from that of regular generators and coroutines. Implementation ============== Execution context is implemented as an immutable linked list of logical contexts, where each logical context is an immutable weak key mapping. A pointer to the currently active execution context is stored in the OS thread state:: +-----------------+ | | ec | PyThreadState +-------------+ | | | +-----------------+ | | ec_node ec_node ec_node v +------+------+ +------+------+ +------+------+ | NULL | lc |<----| prev | lc |<----| prev | lc | +------+--+---+ +------+--+---+ +------+--+---+ | | | LC v LC v LC v +-------------+ +-------------+ +-------------+ | var1: obj1 | | EMPTY | | var1: obj4 | | var2: obj2 | +-------------+ +-------------+ | var3: obj3 | +-------------+ The choice of the immutable list of immutable mappings as a fundamental data structure is motivated by the need to efficiently implement ``sys.get_execution_context()``, which is to be frequently used by asynchronous tasks and callbacks. When the EC is immutable, ``get_execution_context()`` can simply copy the current execution context *by reference*:: def get_execution_context(self): return PyThreadState_Get().ec Let's review all possible context modification scenarios: * The ``ContextVariable.set()`` method is called:: def ContextVar_set(self, val): # See a more complete set() definition # in the `Context Variables` section. tstate = PyThreadState_Get() top_ec_node = tstate.ec top_lc = top_ec_node.lc new_top_lc = top_lc.set(self, val) tstate.ec = ec_node( prev=top_ec_node.prev, lc=new_top_lc) * The ``sys.run_with_logical_context()`` is called, in which case the passed logical context object is appended to the execution context:: def run_with_logical_context(lc, func, *args, **kwargs): tstate = PyThreadState_Get() old_top_ec_node = tstate.ec new_top_ec_node = ec_node(prev=old_top_ec_node, lc=lc) try: tstate.ec = new_top_ec_node return func(*args, **kwargs) finally: tstate.ec = old_top_ec_node * The ``sys.run_with_execution_context()`` is called, in which case the current execution context is set to the passed execution context with a new empty logical context appended to it:: def run_with_execution_context(ec, func, *args, **kwargs): tstate = PyThreadState_Get() old_top_ec_node = tstate.ec new_lc = sys.new_logical_context() new_top_ec_node = ec_node(prev=ec, lc=new_lc) try: tstate.ec = new_top_ec_node return func(*args, **kwargs) finally: tstate.ec = old_top_ec_node * Either ``genobj.send()``, ``genobj.throw()``, ``genobj.close()`` are called on a ``genobj`` generator, in which case the logical context recorded in ``genobj`` is pushed onto the stack:: PyGen_New(PyGenObject *gen): gen.__logical_context__ = sys.new_logical_context() gen_send(PyGenObject *gen, ...): tstate = PyThreadState_Get() if gen.__logical_context__ is not None: old_top_ec_node = tstate.ec new_top_ec_node = ec_node( prev=old_top_ec_node, lc=gen.__logical_context__) try: tstate.ec = new_top_ec_node return _gen_send_impl(gen, ...) finally: gen.__logical_context__ = tstate.ec.lc tstate.ec = old_top_ec_node else: return _gen_send_impl(gen, ...) * Coroutines and asynchronous generators share the implementation with generators, and the above changes apply to them as well. In certain scenarios the EC may need to be squashed to limit the size of the chain. For example, consider the following corner case:: async def repeat(coro, delay): await coro() await asyncio.sleep(delay) loop.create_task(repeat(coro, delay)) async def ping(): print('ping') loop = asyncio.get_event_loop() loop.create_task(repeat(ping, 1)) loop.run_forever() In the above code, the EC chain will grow as long as ``repeat()`` is called. Each new task will call ``sys.run_in_execution_context()``, which will append a new logical context to the chain. To prevent unbounded growth, ``sys.get_execution_context()`` checks if the chain is longer than a predetermined maximum, and if it is, squashes the chain into a single LC:: def get_execution_context(): tstate = PyThreadState_Get() if tstate.ec_len > EC_LEN_MAX: squashed_lc = sys.new_logical_context() ec_node = tstate.ec while ec_node: # The LC.merge() method does not replace existing keys. squashed_lc = squashed_lc.merge(ec_node.lc) ec_node = ec_node.prev return ec_node(prev=NULL, lc=squashed_lc) else: return tstate.ec Logical Context --------------- Logical context is an immutable weak key mapping which has the following properties with respect to garbage collection: * ``ContextVar`` objects are strongly-referenced only from the application code, not from any of the Execution Context machinery or values they point to. This means that there are no reference cycles that could extend their lifespan longer than necessary, or prevent their collection by the GC. * Values put in the Execution Context are guaranteed to be kept alive while there is a ``ContextVar`` key referencing them in the thread. * If a ``ContextVar`` is garbage collected, all of its values will be removed from all contexts, allowing them to be GCed if needed. * If a thread has ended its execution, its thread state will be cleaned up along with its ``ExecutionContext``, cleaning up all values bound to all context variables in the thread. As discussed earluier, we need ``sys.get_execution_context()`` to be consistently fast regardless of the size of the execution context, so logical context is necessarily an immutable mapping. Choosing ``dict`` for the underlying implementation is suboptimal, because ``LC.set()`` will cause ``dict.copy()``, which is an O(N) operation, where *N* is the number of items in the LC. ``get_execution_context()``, when squashing the EC, is a O(M) operation, where *M* is the total number of context variable values in the EC. So, instead of ``dict``, we choose Hash Array Mapped Trie (HAMT) as the underlying implementation of logical contexts. (Scala and Clojure use HAMT to implement high performance immutable collections [5]_, [6]_.) With HAMT ``.set()`` becomes an O(log N) operation, and ``get_execution_context()`` squashing is more efficient on average due to structural sharing in HAMT. See `Appendix: HAMT Performance Analysis`_ for a more elaborate analysis of HAMT performance compared to ``dict``. Context Variables ----------------- The ``ContextVar.lookup()`` and ``ContextVar.set()`` methods are implemented as follows (in pseudo-code):: class ContextVar: def get(self): tstate = PyThreadState_Get() ec_node = tstate.ec while ec_node: if self in ec_node.lc: return ec_node.lc[self] ec_node = ec_node.prev return None def set(self, value): tstate = PyThreadState_Get() top_ec_node = tstate.ec if top_ec_node is not None: top_lc = top_ec_node.lc new_top_lc = top_lc.set(self, value) tstate.ec = ec_node( prev=top_ec_node.prev, lc=new_top_lc) else: top_lc = sys.new_logical_context() new_top_lc = top_lc.set(self, value) tstate.ec = ec_node( prev=NULL, lc=new_top_lc) For efficient access in performance-sensitive code paths, such as in ``numpy`` and ``decimal``, we add a cache to ``ContextVar.get()``, making it an O(1) operation when the cache is hit. The cache key is composed from the following: * The new ``uint64_t PyThreadState->unique_id``, which is a globally unique thread state identifier. It is computed from the new ``uint64_t PyInterpreterState->ts_counter``, which is incremented whenever a new thread state is created. * The ``uint64_t ContextVar->version`` counter, which is incremented whenever the context variable value is changed in any logical context in any thread. The cache is then implemented as follows:: class ContextVar: def set(self, value): ... # implementation self.version += 1 def get(self): tstate = PyThreadState_Get() if (self.last_tstate_id == tstate.unique_id and self.last_version == self.version): return self.last_value value = self._get_uncached() self.last_value = value # borrowed ref self.last_tstate_id = tstate.unique_id self.last_version = self.version return value Note that ``last_value`` is a borrowed reference. The assumption is that if the version checks are fine, the object will be alive. This allows the values of context variables to be properly garbage collected. This generic caching approach is similar to what the current C implementation of ``decimal`` does to cache the the current decimal context, and has similar performance characteristics. Performance Considerations ========================== Tests of the reference implementation based on the prior revisions of this PEP have shown 1-2% slowdown on generator microbenchmarks and no noticeable difference in macrobenchmarks. The performance of non-generator and non-async code is not affected by this PEP. Summary of the New APIs ======================= Python ------ The following new Python APIs are introduced by this PEP: 1. The ``sys.new_context_var(name: str='...')`` function to create ``ContextVar`` objects. 2. The ``ContextVar`` object, which has: * the read-only ``.name`` attribute, * the ``.lookup()`` method which returns the value of the variable in the current execution context; * the ``.set()`` method which sets the value of the variable in the current execution context. 3. The ``sys.get_execution_context()`` function, which returns a copy of the current execution context. 4. The ``sys.new_execution_context()`` function, which returns a new empty execution context. 5. The ``sys.new_logical_context()`` function, which returns a new empty logical context. 6. The ``sys.run_with_execution_context(ec: ExecutionContext, func, *args, **kwargs)`` function, which runs *func* with the provided execution context. 7. The ``sys.run_with_logical_context(lc:LogicalContext, func, *args, **kwargs)`` function, which runs *func* with the provided logical context on top of the current execution context. C API ----- 1. ``PyContextVar * PyContext_NewVar(char *desc)``: create a ``PyContextVar`` object. 2. ``PyObject * PyContext_LookupVar(PyContextVar *)``: return the value of the variable in the current execution context. 3. ``int PyContext_SetVar(PyContextVar *, PyObject *)``: set the value of the variable in the current execution context. 4. ``PyLogicalContext * PyLogicalContext_New()``: create a new empty ``PyLogicalContext``. 5. ``PyLogicalContext * PyExecutionContext_New()``: create a new empty ``PyExecutionContext``. 6. ``PyExecutionContext * PyExecutionContext_Get()``: return the current execution context. 7. ``int PyExecutionContext_Set(PyExecutionContext *)``: set the passed EC object as the current for the active thread state. 8. ``int PyExecutionContext_SetWithLogicalContext(PyExecutionContext *, PyLogicalContext *)``: allows to implement ``sys.run_with_logical_context`` Python API. Design Considerations ===================== Should ``PyThreadState_GetDict()`` use the execution context? ------------------------------------------------------------- No. ``PyThreadState_GetDict`` is based on TLS, and changing its semantics will break backwards compatibility. PEP 521 ------- :pep:`521` proposes an alternative solution to the problem, which extends the context manager protocol with two new methods: ``__suspend__()`` and ``__resume__()``. Similarly, the asynchronous context manager protocol is also extended with ``__asuspend__()`` and ``__aresume__()``. This allows implementing context managers that manage non-local state, which behave correctly in generators and coroutines. For example, consider the following context manager, which uses execution state:: class Context: def __init__(self): self.var = new_context_var('var') def __enter__(self): self.old_x = self.var.lookup() self.var.set('something') def __exit__(self, *err): self.var.set(self.old_x) An equivalent implementation with PEP 521:: local = threading.local() class Context: def __enter__(self): self.old_x = getattr(local, 'x', None) local.x = 'something' def __suspend__(self): local.x = self.old_x def __resume__(self): local.x = 'something' def __exit__(self, *err): local.x = self.old_x The downside of this approach is the addition of significant new complexity to the context manager protocol and the interpreter implementation. This approach is also likely to negatively impact the performance of generators and coroutines. Additionally, the solution in :pep:`521` is limited to context managers, and does not provide any mechanism to propagate state in asynchronous tasks and callbacks. Can Execution Context be implemented outside of CPython? -------------------------------------------------------- No. Proper generator behaviour with respect to the execution context requires changes to the interpreter. Should we update sys.displayhook and other APIs to use EC? ---------------------------------------------------------- APIs like redirecting stdout by overwriting ``sys.stdout``, or specifying new exception display hooks by overwriting the ``sys.displayhook`` function are affecting the whole Python process **by design**. Their users assume that the effect of changing them will be visible across OS threads. Therefore we cannot just make these APIs to use the new Execution Context. That said we think it is possible to design new APIs that will be context aware, but that is outside of the scope of this PEP. Greenlets --------- Greenlet is an alternative implementation of cooperative scheduling for Python. Although greenlet package is not part of CPython, popular frameworks like gevent rely on it, and it is important that greenlet can be modified to support execution contexts. Conceptually, the behaviour of greenlets is very similar to that of generators, which means that similar changes around greenlet entry and exit can be done to add support for execution context. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Appendix: HAMT Performance Analysis =================================== .. figure:: pep-0550-hamt_vs_dict-v2.png :align: center :width: 100% Figure 1. Benchmark code can be found here: [9]_. The above chart demonstrates that: * HAMT displays near O(1) performance for all benchmarked dictionary sizes. * ``dict.copy()`` becomes very slow around 100 items. .. figure:: pep-0550-lookup_hamt.png :align: center :width: 100% Figure 2. Benchmark code can be found here: [10]_. Figure 2 compares the lookup costs of ``dict`` versus a HAMT-based immutable mapping. HAMT lookup time is 30-40% slower than Python dict lookups on average, which is a very good result, considering that the latter is very well optimized. Thre is research [8]_ showing that there are further possible improvements to the performance of HAMT. The reference implementation of HAMT for CPython can be found here: [7]_. Acknowledgments =============== Thanks to Victor Petrovykh for countless discussions around the topic and PEP proofreading and edits. Thanks to Nathaniel Smith for proposing the ``ContextVar`` design [17]_ [18]_, for pushing the PEP towards a more complete design, and coming up with the idea of having a stack of contexts in the thread state. Thanks to Nick Coghlan for numerous suggestions and ideas on the mailing list, and for coming up with a case that cause the complete rewrite of the initial PEP version [19]_. Version History =============== 1. Initial revision, posted on 11-Aug-2017 [20]_. 2. V2 posted on 15-Aug-2017 [21]_. The fundamental limitation that caused a complete redesign of the first version was that it was not possible to implement an iterator that would interact with the EC in the same way as generators (see [19]_.) Version 2 was a complete rewrite, introducing new terminology (Local Context, Execution Context, Context Item) and new APIs. 3. V3 posted on 18-Aug-2017 [22]_. Updates: * Local Context was renamed to Logical Context. The term "local" was ambiguous and conflicted with local name scopes. * Context Item was renamed to Context Key, see the thread with Nick Coghlan, Stefan Krah, and Yury Selivanov [23]_ for details. * Context Item get cache design was adjusted, per Nathaniel Smith's idea in [25]_. * Coroutines are created without a Logical Context; ceval loop no longer needs to special case the ``await`` expression (proposed by Nick Coghlan in [24]_.) 4. V4 posted on 25-Aug-2017: the current version. * The specification section has been completely rewritten. * Context Key renamed to Context Var. * Removed the distinction between generators and coroutines with respect to logical context isolation. References ========== .. [1] https://blog.golang.org/context .. [2] https://msdn.microsoft.com/en-us/library/system.threading.executioncontext.… .. [3] https://github.com/numpy/numpy/issues/9444 .. [4] http://bugs.python.org/issue31179 .. [5] https://en.wikipedia.org/wiki/Hash_array_mapped_trie .. [6] http://blog.higher-order.net/2010/08/16/assoc-and-clojures-persistenthashma… .. [7] https://github.com/1st1/cpython/tree/hamt .. [8] https://michael.steindorfer.name/publications/oopsla15.pdf .. [9] https://gist.github.com/1st1/9004813d5576c96529527d44c5457dcd .. [10] https://gist.github.com/1st1/dbe27f2e14c30cce6f0b5fddfc8c437e .. [11] https://github.com/1st1/cpython/tree/pep550 .. [12] https://www.python.org/dev/peps/pep-0492/#async-await .. [13] https://github.com/MagicStack/uvloop/blob/master/examples/bench/echoserver.… .. [14] https://github.com/MagicStack/pgbench .. [15] https://github.com/python/performance .. [16] https://gist.github.com/1st1/6b7a614643f91ead3edf37c4451a6b4c .. [17] https://mail.python.org/pipermail/python-ideas/2017-August/046752.html .. [18] https://mail.python.org/pipermail/python-ideas/2017-August/046772.html .. [19] https://mail.python.org/pipermail/python-ideas/2017-August/046775.html .. [20] https://github.com/python/peps/blob/e8a06c9a790f39451d9e99e203b13b3ad73a1d0… .. [21] https://github.com/python/peps/blob/e3aa3b2b4e4e9967d28a10827eed1e9e5960c17… .. [22] https://github.com/python/peps/blob/287ed87bb475a7da657f950b353c71c1248f67e… .. [23] https://mail.python.org/pipermail/python-ideas/2017-August/046801.html .. [24] https://mail.python.org/pipermail/python-ideas/2017-August/046790.html .. [25] https://mail.python.org/pipermail/python-ideas/2017-August/046786.html Copyright ========= This document has been placed in the public domain.

19 129

Consolidate stateful runtime globals
by Antoine Pitrou 07 Sep '17

07 Sep '17

Hello, I'm a bit concerned about https://github.com/python/cpython/commit/76d5abc8684bac4f2fc7cccfe2cd940923… My main gripe is that makes writing C code more tedious. Simple C global variables such as "_once_registry" are now spelled "_PyRuntime.warnings.once_registry". The most egregious example seems to be "_PyRuntime.ceval.gil.locked" (used to be simply "gil_locked"). Granted, C is more verbose than Python, but it doesn't have to become that verbose. I don't know about you, but when code becomes annoying to type, I tend to try and take shortcuts. Regards Antoine.

6 9