Mailman 3 April 2019 - Python-Dev

Proposal: dict.with_values(iterable)
by Inada Naoki 24 Apr '19

24 Apr '19

Hi, all. I propose adding new method: dict.with_values(iterable) # Motivation Python is used to handle data. While dict is not efficient way to handle may records, it is still convenient way. When creating many dicts with same keys, dict need to lookup internal hash table while inserting each keys. It is costful operation. If we can reuse existing keys of dict, we can skip this inserting cost. Additionally, we have "Key-Sharing Dictionary (PEP 412)". When all keys are string, many dict can share one key. It reduces memory consumption. This might be usable for: * csv.DictReader * namedtuple._asdict() * DB-API 2.0 implementations: (e.g. DictCursor of mysqlclient-python) # Draft implementation pull request: https://github.com/python/cpython/pull/12802 with_values(self, iterable, /) Create a new dictionary with keys from this dict and values from iterable. When length of iterable is different from len(self), ValueError is raised. This method does not support dict subclass. ## Memory usage (Key-Sharing dict) >>> import sys >>> keys = tuple("abcdefg") >>> keys ('a', 'b', 'c', 'd', 'e', 'f', 'g') >>> d = dict(zip(keys, range(7))) >>> d {'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4, 'f': 5, 'g': 6} >>> sys.getsizeof(d) 360 >>> keys = dict.fromkeys("abcdefg") >>> d = keys.with_values(range(7)) >>> d {'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4, 'f': 5, 'g': 6} >>> sys.getsizeof(d) 144 ## Speed $ ./python -m perf timeit -o zip_dict.json -s 'keys = tuple("abcdefg"); values=[*range(7)]' 'dict(zip(keys, values))' $ ./python -m perf timeit -o with_values.json -s 'keys = dict.fromkeys("abcdefg"); values=[*range(7)]' 'keys.with_values(values)' $ ./python -m perf compare_to zip_dict.json with_values.json Mean +- std dev: [zip_dict] 935 ns +- 9 ns -> [with_values] 109 ns +- 2 ns: 8.59x faster (-88%) How do you think? Any comments are appreciated. Regards, -- Inada Naoki <songofacandy(a)gmail.com>

9 26

Concurrent.futures: no type discovery for PyCharm
by Ilya Kamenshchikov 23 Apr '19

23 Apr '19

I am using concurrent.futures to parallelize independent tasks on multiple cores once in a while. Each time I have a difficulty remembering the specific syntax and have to look it up in old code or google. I would much prefer to be able to find the source through the PyCharm and have autocompletion. It takes adding two lines to the __init__.py of concurrent.futures: (insert on line 19) from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor I would also guess that it would make the __getattr__ redundant? Am I missing something or can this change be done this way and would indeed be an improvement? Best Regards, -- Ilya Kamenshchikov

9 16

Python Documentation Translation in italian language
by Alessandro Cucci 20 Apr '19

20 Apr '19

Hello folks, I want to start a project for translating the Python Documentation in Italian. I'm reading the PEP545, trying to understand how it works. I founded a Python User Group in my city and I can work with them on the translations, plus next month I will be speaker at Pycon Italy, so I can easily sponsor this project during the talk and reclute more people to work on that. Is there anybody who can help me to start? Thanks, have a nice day. *Alessandro Cucci*

2 1

Summary of Python tracker Issues
by Python tracker 19 Apr '19

19 Apr '19

ACTIVITY SUMMARY (2019-04-12 - 2019-04-19) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7058 ( -5) closed 41363 (+56) total 48421 (+51) Open issues with patches: 2807 Issues opened (33) ================== #30485: Element.findall(path, dict) doesn't insert null namespace https://bugs.python.org/issue30485 reopened by scoder #36345: Deprecate Tools/scripts/serve.py in favour of python -m http.s https://bugs.python.org/issue36345 reopened by matrixise #36620: Documentation missing parameter for Itertools.zip_longest https://bugs.python.org/issue36620 opened by CharlesMerriam #36621: shutil.rmtree follows junctions on windows https://bugs.python.org/issue36621 opened by Jordan Hueckstaedt #36624: cleanup the stdlib and tests with regard to sys.platform usage https://bugs.python.org/issue36624 opened by Michael.Felt #36630: failure of test_colors_funcs in test_curses with ncurses 6.1 https://bugs.python.org/issue36630 opened by xdegaye #36631: test_urllib2net: test_ftp_no_timeout() killed after a timeout https://bugs.python.org/issue36631 opened by vstinner #36632: test_multiprocessing_forkserver: test_rapid_restart() leaked a https://bugs.python.org/issue36632 opened by vstinner #36634: venv: activate.bat fails for venv with parentheses in PATH https://bugs.python.org/issue36634 opened by BWenzel #36635: Add _testinternalcapi module https://bugs.python.org/issue36635 opened by vstinner #36636: Inner exception is not being raised using asyncio.gather https://bugs.python.org/issue36636 opened by Drew Budwin #36640: python ibm_db setup.py post install script does not seem to wo https://bugs.python.org/issue36640 opened by sabakauser #36643: Forward reference is not resolved by dataclasses.fields() https://bugs.python.org/issue36643 opened by mdrachuk #36644: Improve documentation of slice.indices() https://bugs.python.org/issue36644 opened by pewscorner #36645: re.sub() library entry does not adequately document surprising https://bugs.python.org/issue36645 opened by mollison #36646: os.listdir() got permission error in Python3.6 but it's fine i https://bugs.python.org/issue36646 opened by Ryan_D(a)163.com #36647: TextTestRunner doesn't honour "buffer" argument https://bugs.python.org/issue36647 opened by José Luis Segura Lucas #36648: MAP_SHARED isn't proper for anonymous mappings for VxWorks https://bugs.python.org/issue36648 opened by lzhao #36650: Cached method implementation no longer works on Python 3.7.3 https://bugs.python.org/issue36650 opened by jaraco #36654: Add example to tokenize.tokenize https://bugs.python.org/issue36654 opened by Windson Yang #36656: Allow os.symlink(src, target, force=True) to prevent race cond https://bugs.python.org/issue36656 opened by Tom Hale #36658: Py_Initialze() throws error 'unable to load the file system en https://bugs.python.org/issue36658 opened by rvq #36659: distutils UnixCCompiler: Remove standard library path from rpa https://bugs.python.org/issue36659 opened by vstinner #36661: Missing dataclass decorator import in dataclasses module docs https://bugs.python.org/issue36661 opened by mfisherlevine #36662: asdict/astuple Dataclass methods https://bugs.python.org/issue36662 opened by gsakkis #36663: pdb: store whole exception information in locals (via user_exc https://bugs.python.org/issue36663 opened by blueyed #36664: argparse: parser aliases in subparsers stores alias in dest va https://bugs.python.org/issue36664 opened by Peter McEldowney #36665: REPL doesn't ensure builtins are available when implicitly rec https://bugs.python.org/issue36665 opened by ncoghlan #36666: threading.Thread should have way to catch an exception thrown https://bugs.python.org/issue36666 opened by Joel Croteau #36667: pdb: restore SIGINT handler in sigint_handler already https://bugs.python.org/issue36667 opened by blueyed #36668: semaphore_tracker is not reused by child processes https://bugs.python.org/issue36668 opened by tomMoral #36669: weakref proxy doesn't support the matrix multiplication operat https://bugs.python.org/issue36669 opened by bup #36670: test suite broken due to cpu usage feature on win 10/ german https://bugs.python.org/issue36670 opened by LorenzMende Most recent 15 issues with no replies (15) ========================================== #36669: weakref proxy doesn't support the matrix multiplication operat https://bugs.python.org/issue36669 #36668: semaphore_tracker is not reused by child processes https://bugs.python.org/issue36668 #36667: pdb: restore SIGINT handler in sigint_handler already https://bugs.python.org/issue36667 #36663: pdb: store whole exception information in locals (via user_exc https://bugs.python.org/issue36663 #36654: Add example to tokenize.tokenize https://bugs.python.org/issue36654 #36647: TextTestRunner doesn't honour "buffer" argument https://bugs.python.org/issue36647 #36644: Improve documentation of slice.indices() https://bugs.python.org/issue36644 #36643: Forward reference is not resolved by dataclasses.fields() https://bugs.python.org/issue36643 #36621: shutil.rmtree follows junctions on windows https://bugs.python.org/issue36621 #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 #36606: calling super() causes __class__ to be not defined when sys.se https://bugs.python.org/issue36606 #36603: should pty.openpty() set pty/tty inheritable? https://bugs.python.org/issue36603 #36590: Add Bluetooth RFCOMM Support for Windows https://bugs.python.org/issue36590 #36589: Incorrect error handling in curses.update_lines_cols() https://bugs.python.org/issue36589 #36583: Do not swallow exceptions in the _ssl module https://bugs.python.org/issue36583 Most recent 15 issues waiting for review (15) ============================================= #36668: semaphore_tracker is not reused by child processes https://bugs.python.org/issue36668 #36667: pdb: restore SIGINT handler in sigint_handler already https://bugs.python.org/issue36667 #36659: distutils UnixCCompiler: Remove standard library path from rpa https://bugs.python.org/issue36659 #36648: MAP_SHARED isn't proper for anonymous mappings for VxWorks https://bugs.python.org/issue36648 #36645: re.sub() library entry does not adequately document surprising https://bugs.python.org/issue36645 #36635: Add _testinternalcapi module https://bugs.python.org/issue36635 #36634: venv: activate.bat fails for venv with parentheses in PATH https://bugs.python.org/issue36634 #36624: cleanup the stdlib and tests with regard to sys.platform usage https://bugs.python.org/issue36624 #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 #36612: Unittest document is not clear on SetUpClass calls https://bugs.python.org/issue36612 #36610: os.sendfile can return EINVAL on Solaris https://bugs.python.org/issue36610 #36608: Replace bundled pip and setuptools with a downloader in the en https://bugs.python.org/issue36608 #36602: Recursive directory list with pathlib.Path.iterdir https://bugs.python.org/issue36602 #36601: signals can be caught by any thread https://bugs.python.org/issue36601 Top 10 most discussed issues (10) ================================= #35866: concurrent.futures deadlock https://bugs.python.org/issue35866 17 msgs #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 14 msgs #30485: Element.findall(path, dict) doesn't insert null namespace https://bugs.python.org/issue30485 10 msgs #36646: os.listdir() got permission error in Python3.6 but it's fine i https://bugs.python.org/issue36646 10 msgs #27987: obmalloc's 8-byte alignment causes undefined behavior https://bugs.python.org/issue27987 8 msgs #36624: cleanup the stdlib and tests with regard to sys.platform usage https://bugs.python.org/issue36624 8 msgs #16079: list duplicate test names with patchcheck https://bugs.python.org/issue16079 7 msgs #31904: Python should support VxWorks RTOS https://bugs.python.org/issue31904 6 msgs #36634: venv: activate.bat fails for venv with parentheses in PATH https://bugs.python.org/issue36634 6 msgs #32782: memoryview & ctypes: incorrect itemsize for empty array https://bugs.python.org/issue32782 5 msgs Issues closed (53) ================== #2007: cookielib lacks FileCookieJar class for Internet Explorer https://bugs.python.org/issue2007 closed by inada.naoki #15917: hg hook to detect unmerged changesets https://bugs.python.org/issue15917 closed by inada.naoki #16254: Make PyUnicode_AsWideCharString() increase temporary https://bugs.python.org/issue16254 closed by inada.naoki #18610: wsgiref.validate expects wsgi.input read to give exactly one a https://bugs.python.org/issue18610 closed by cheryl.sabella #22991: test_gdb leaves the terminal in raw mode with gdb 7.8.1 https://bugs.python.org/issue22991 closed by xdegaye #23768: assert on getting the representation of a thread in atexit fun https://bugs.python.org/issue23768 closed by xdegaye #27326: SIGSEV in test_window_funcs of test_curses https://bugs.python.org/issue27326 closed by xdegaye #28055: pyhash's siphash24 assumes alignment of the data pointer https://bugs.python.org/issue28055 closed by vstinner #28809: mention asyncio.gather non-deterministic task starting order https://bugs.python.org/issue28809 closed by cheryl.sabella #31658: xml.sax.parse won't accept path objects https://bugs.python.org/issue31658 closed by scoder #32849: Fatal Python error: Py_Initialize: can't initialize sys standa https://bugs.python.org/issue32849 closed by vstinner #32913: Improve regular expression HOWTO https://bugs.python.org/issue32913 closed by brett.cannon #33783: Use proper class markup for random.Random docs https://bugs.python.org/issue33783 closed by vstinner #34814: makesetup: must link C extensions to libpython when compiled i https://bugs.python.org/issue34814 closed by vstinner #35581: Document @typing.type_check_only https://bugs.python.org/issue35581 closed by gvanrossum #35697: _decimal: Implement the previously rejected changes from #7442 https://bugs.python.org/issue35697 closed by vstinner #35755: On Unix, shutil.which() and subprocess no longer look for the https://bugs.python.org/issue35755 closed by vstinner #36071: Add support for Windows ARM32 in ctypes/libffi https://bugs.python.org/issue36071 closed by steve.dower #36227: Add default_namespace argument to xml.etree.ElementTree.tostri https://bugs.python.org/issue36227 closed by scoder #36263: test_hashlib.test_scrypt() fails on Fedora 29 https://bugs.python.org/issue36263 closed by vstinner #36348: test_imaplib.RemoteIMAP_STARTTLSTest.test_logout() fails rando https://bugs.python.org/issue36348 closed by vstinner #36427: Document that PyEval_RestoreThread and PyGILState_Ensure can t https://bugs.python.org/issue36427 closed by pablogsal #36466: Adding a way to strip annotations from compiled bytecode https://bugs.python.org/issue36466 closed by cary #36508: python-config --ldflags must not contain LINKFORSHARED ("-Xlin https://bugs.python.org/issue36508 closed by vstinner #36558: Change time.mktime() return type from float to int? https://bugs.python.org/issue36558 closed by vstinner #36572: python-snappy install issue during Crossbar install with Pytho https://bugs.python.org/issue36572 closed by SilentGhost #36585: test_posix.py fails due to unsupported RWF_HIPRI https://bugs.python.org/issue36585 closed by pablogsal #36593: isinstance check fails for Mock objects with spec executed und https://bugs.python.org/issue36593 closed by pablogsal #36600: re-enable test in nntplib https://bugs.python.org/issue36600 closed by Marcin Niemira #36605: make tags should also parse Modules/_io/*.c and Modules/_io/*. https://bugs.python.org/issue36605 closed by vstinner #36611: Debug memory allocators: remove useless "serialno" field to re https://bugs.python.org/issue36611 closed by vstinner #36616: Optimize thread state handling in function call code https://bugs.python.org/issue36616 closed by vstinner #36622: Inconsistent exponent notation formatting https://bugs.python.org/issue36622 closed by mark.dickinson #36623: Clean unused parser headers https://bugs.python.org/issue36623 closed by pablogsal #36625: Obsolete comments in docstrings in fractions module https://bugs.python.org/issue36625 closed by mark.dickinson #36626: asyncio run_forever blocks indefinitely https://bugs.python.org/issue36626 closed by asvetlov #36627: composing generator expression doesn't work as expected https://bugs.python.org/issue36627 closed by SilentGhost #36628: Enhancement: i-Strings https://bugs.python.org/issue36628 closed by SilentGhost #36629: imaplib test fails with errno 101 https://bugs.python.org/issue36629 closed by vstinner #36633: py_compile.compile: AttributeError on importlib.utils https://bugs.python.org/issue36633 closed by xtreak #36637: Restrict syntax for tuple literals with one element https://bugs.python.org/issue36637 closed by brett.cannon #36638: typeperf.exe is not in all skus of Windows SKUs https://bugs.python.org/issue36638 closed by steve.dower #36639: Provide list.rindex() https://bugs.python.org/issue36639 closed by rhettinger #36641: make docstring in C const https://bugs.python.org/issue36641 closed by inada.naoki #36642: make unicodedata "const" https://bugs.python.org/issue36642 closed by inada.naoki #36649: Windows Store app install registry keys have incorrect paths https://bugs.python.org/issue36649 closed by steve.dower #36651: Asyncio Event Loop documentation inconsistency (call_later and https://bugs.python.org/issue36651 closed by asvetlov #36652: Non-embedded zip distribution https://bugs.python.org/issue36652 closed by steve.dower #36653: Dictionary Key is without ' ' quotes https://bugs.python.org/issue36653 closed by mrabarnett #36655: Division Precision Problem https://bugs.python.org/issue36655 closed by christian.heimes #36657: AttributeError https://bugs.python.org/issue36657 closed by xtreak #36660: TypeError https://bugs.python.org/issue36660 closed by vstinner #1402289: Allow mappings as globals (was: Fix dictionary subclass ...) https://bugs.python.org/issue1402289 closed by rhettinger

1 0

PEP 591 discussion (final qualifier) happening at typing-sig@
by Michael Sullivan 18 Apr '19

18 Apr '19

I've submitted PEP 591 (Adding a final qualifier to typing) for discussion to typing-sig [1]. Here's the abstract: This PEP proposes a "final" qualifier to be added to the ``typing`` module---in the form of a ``final`` decorator and a ``Final`` type annotation---to serve three related purposes: * Declaring that a method should not be overridden * Declaring that a class should not be subclassed * Declaring that a variable or attribute should not be reassigned Full text at https://www.python.org/dev/peps/pep-0591/ -sully [1] https://mail.python.org/mailman3/lists/typing-sig.python.org/

3 3

Adding shlex.join?
by Paul Ganssle 17 Apr '19

17 Apr '19

Hey all, I've been reviewing old "awaiting review" PRs recently, and about a week ago I found PR #7605 <https://github.com/python/cpython/pull/7605>, adding shlex.join(), with a corresponding bug at bpo-22454 <https://bugs.python.org/issue22454>. The PR's implementation is simple and seems reasonable and decently well-tested, but it has been unreviewed for ~10 months. The reason I'm bringing it up here is that I believe the major blocker here is getting agreement to actually add the function. There doesn't seem to be much /opposition/ in the BPO issue, but given how infrequently the shlex module is changed I'm worried that there may be no one around who feels confident to judge how the interface should evolve. Does anyone feel strongly about this issue? Is there anyone who wants to make a yes/no decision on this feature? Best, Paul P.S. The PR's submitter seems responsive. I made a comment on the documentation and it was addressed in something like 5 minutes.

4 3

PEP 7: Adding anonymous union / struct
by Inada Naoki 17 Apr '19

17 Apr '19

Hi, all. PEP 7 includes some C99 features. I propose to add include anonymous union and struct to the list. https://www.geeksforgeeks.org/g-fact-38-anonymous-union-and-structure/ Anonymous union and struct are C11 feature, not C99. But gcc and MSVC supported it as language extension from before C11. Anonymous union is useful when all union members have different names. Especially, when we need to add dummy member only for padding / alignment, union name looks too verbose: ... # in some struct union { struct { int member1; int member2; } s; long double _dummy; // for largest alignment. } u; ... x.u.s.member1 = 42; vs ... union { struct { int member1; int member2; }; long double _dummy; // for largest alignment. }; ... x.member1 = 42; Does anyone know compiler which can be use to compile Python but doesn't support anonymous union / struct? Regards, -- Inada Naoki <songofacandy(a)gmail.com>

2 2

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
by Michael Sullivan 17 Apr '19

17 Apr '19

> The main question is if anyone ever used Py_TRACE_REFS? Does someone > use sys.getobjects() or PYTHONDUMPREFS environment variable? I used sys.getobjects() today to track down a memory leak in the mypyc-compiled version of mypy. We were leaking memory badly but no sign of the leak was showing up in mypy's gc.get_objects() based profiler. Using a debug build and switching to sys.getobjects() showed that we were badly leaking int objects. A quick inspection of the values in question (large and random looking) suggested we were leaking hash values, and that quickly pointed me to https://github.com/mypyc/mypyc/pull/562. I don't have any strong feelings about whether to keep it in the "default" debug build, though. I was using a debug build that I built myself with every debug feature that seemed potentially useful. -sully

3 6

PEP 578: Python Runtime Audit Hooks
by Steve Dower 16 Apr '19

16 Apr '19

Hi all Time is short, but I'm hoping to get PEP 578 (formerly PEP 551) into Python 3.8. Here's the current text for review and comment before I submit to the Steering Council. The formatted text is at https://www.python.org/dev/peps/pep-0578/ (update just pushed, so give it an hour or so, but it's fundamentally the same as what's there) No Discourse post, because we don't have a python-dev equivalent there yet, so please reply here for this one. Implementation is at https://github.com/zooba/cpython/tree/pep-578/ and my backport to 3.7 (https://github.com/zooba/cpython/tree/pep-578-3.7/) is already getting some real use (though this will not be added to 3.7, unless people *really* want it, so the backport is just for reference). Cheers, Steve ===== PEP: 578 Title: Python Runtime Audit Hooks Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 16-Jun-2018 Python-Version: 3.8 Post-History: Abstract ======== This PEP describes additions to the Python API and specific behaviors for the CPython implementation that make actions taken by the Python runtime visible to auditing tools. Visibility into these actions provides opportunities for test frameworks, logging frameworks, and security tools to monitor and optionally limit actions taken by the runtime. This PEP proposes adding two APIs to provide insights into a running Python application: one for arbitrary events, and another specific to the module import system. The APIs are intended to be available in all Python implementations, though the specific messages and values used are unspecified here to allow implementations the freedom to determine how best to provide information to their users. Some examples likely to be used in CPython are provided for explanatory purposes. See PEP 551 for discussion and recommendations on enhancing the security of a Python runtime making use of these auditing APIs. Background ========== Python provides access to a wide range of low-level functionality on many common operating systems. While this is incredibly useful for "write-once, run-anywhere" scripting, it also makes monitoring of software written in Python difficult. Because Python uses native system APIs directly, existing monitoring tools either suffer from limited context or auditing bypass. Limited context occurs when system monitoring can report that an action occurred, but cannot explain the sequence of events leading to it. For example, network monitoring at the OS level may be able to report "listening started on port 5678", but may not be able to provide the process ID, command line, parent process, or the local state in the program at the point that triggered the action. Firewall controls to prevent such an action are similarly limited, typically to process names or some global state such as the current user, and in any case rarely provide a useful log file correlated with other application messages. Auditing bypass can occur when the typical system tool used for an action would ordinarily report its use, but accessing the APIs via Python do not trigger this. For example, invoking "curl" to make HTTP requests may be specifically monitored in an audited system, but Python's "urlretrieve" function is not. Within a long-running Python application, particularly one that processes user-provided information such as a web app, there is a risk of unexpected behavior. This may be due to bugs in the code, or deliberately induced by a malicious user. In both cases, normal application logging may be bypassed resulting in no indication that anything out of the ordinary has occurred. Additionally, and somewhat unique to Python, it is very easy to affect the code that is run in an application by manipulating either the import system's search path or placing files earlier on the path than intended. This is often seen when developers create a script with the same name as the module they intend to use - for example, a ``random.py`` file that attempts to import the standard library ``random`` module. This is not sandboxing, as this proposal does not attempt to prevent malicious behavior (though it enables some new options to do so). See the `Why Not A Sandbox`_ section below for further discussion. Overview of Changes =================== The aim of these changes is to enable both application developers and system administrators to integrate Python into their existing monitoring systems without dictating how those systems look or behave. We propose two API changes to enable this: an Audit Hook and Verified Open Hook. Both are available from Python and native code, allowing applications and frameworks written in pure Python code to take advantage of the extra messages, while also allowing embedders or system administrators to deploy builds of Python where auditing is always enabled. Only CPython is bound to provide the native APIs as described here. Other implementations should provide the pure Python APIs, and may provide native versions as appropriate for their underlying runtimes. Auditing events are likewise considered implementation specific, but are bound by normal feature compatibility guarantees. Audit Hook ---------- In order to observe actions taken by the runtime (on behalf of the caller), an API is required to raise messages from within certain operations. These operations are typically deep within the Python runtime or standard library, such as dynamic code compilation, module imports, DNS resolution, or use of certain modules such as ``ctypes``. The following new C APIs allow embedders and CPython implementors to send and receive audit hook messages:: # Add an auditing hook typedef int (*hook_func)(const char *event, PyObject *args, void *userData); int PySys_AddAuditHook(hook_func hook, void *userData); # Raise an event with all auditing hooks int PySys_Audit(const char *event, PyObject *args); # Internal API used during Py_Finalize() - not publicly accessible void _Py_ClearAuditHooks(void); The new Python APIs for receiving and raising audit hooks are:: # Add an auditing hook sys.addaudithook(hook: Callable[[str, tuple]]) # Raise an event with all auditing hooks sys.audit(str, *args) Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time, including before ``Py_Initialize()``, or by calling ``sys.addaudithook()`` from Python code. Hooks cannot be removed or replaced. When events of interest are occurring, code can either call ``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The string argument is the name of the event, and the tuple contains arguments. A given event name should have a fixed schema for arguments, which should be considered a public API (for each x.y version release), and thus should only change between feature releases with updated documentation. For maximum compatibility, events using the same name as an event in the reference interpreter CPython should make every attempt to use compatible arguments. Including the name or an abbreviation of the implementation in implementation-specific event names will also help prevent collisions. For example, a ``pypy.jit_invoked`` event is clearly distinguised from an ``ipy.jit_invoked`` event. When an event is audited, each hook is called in the order it was added with the event name and tuple. If any hook returns with an exception set, later hooks are ignored and *in general* the Python runtime should terminate. This is intentional to allow hook implementations to decide how to respond to any particular event. The typical responses will be to log the event, abort the operation with an exception, or to immediately terminate the process with an operating system exit call. When an event is audited but no hooks have been set, the ``audit()`` function should impose minimal overhead. Ideally, each argument is a reference to existing data rather than a value calculated just for the auditing call. As hooks may be Python objects, they need to be freed during ``Py_Finalize()``. To do this, we add an internal API ``_Py_ClearAuditHooks()`` that releases any Python hooks and any memory held. This is an internal function with no public export, and we recommend it raise its own audit event for all current hooks to ensure that unexpected calls are observed. Below in `Suggested Audit Hook Locations`_, we recommend some important operations that should raise audit events. Python implementations should document which operations will raise audit events, along with the event schema. It is intentional that ``sys.addaudithook(print)`` be a trivial way to display all messages. Verified Open Hook ------------------ Most operating systems have a mechanism to distinguish between files that can be executed and those that can not. For example, this may be an execute bit in the permissions field, a verified hash of the file contents to detect potential code tampering, or file system path restrictions. These are an important security mechanism for preventing execution of data or code that is not approved for a given environment. Currently, Python has no way to integrate with these when launching scripts or importing modules. The new public C API for the verified open hook is:: # Set the handler typedef PyObject *(*hook_func)(PyObject *path, void *userData) int PyImport_SetOpenForImportHook(hook_func handler, void *userData) # Open a file using the handler PyObject *PyImport_OpenForImport(const char *path) The new public Python API for the verified open hook is:: # Open a file using the handler importlib.util.open_for_import(path : str) -> io.IOBase The ``importlib.util.open_for_import()`` function is a drop-in replacement for ``open(str(pathlike), 'rb')``. Its default behaviour is to open a file for raw, binary access. To change the behaviour a new handler should be set. Handler functions only accept ``str`` arguments. The C API ``PyImport_OpenForImport`` function assumes UTF-8 encoding. A custom handler may be set by calling ``PyImport_SetOpenForImportHook()`` from C at any time, including before ``Py_Initialize()``. However, if a hook has already been set then the call will fail. When ``open_for_import()`` is called with a hook set, the hook will be passed the path and its return value will be returned directly. The returned object should be an open file-like object that supports reading raw bytes. This is explicitly intended to allow a ``BytesIO`` instance if the open handler has already read the entire file into memory. Note that these hooks can import and call the ``_io.open()`` function on CPython without triggering themselves. They can also use ``_io.BytesIO`` to return a compatible result using an in-memory buffer. If the hook determines that the file should not be loaded, it should raise an exception of its choice, as well as performing any other logging. All import and execution functionality involving code from a file will be changed to use ``open_for_import()`` unconditionally. It is important to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go through this function - an audit hook that includes the code from these calls is the best opportunity to validate code that is read from the file. Given the current decoupling between import and execution in Python, most imported code will go through both ``open_for_import()`` and the log hook for ``compile``, and so care should be taken to avoid repeating verification steps. There is no Python API provided for changing the open hook. To modify import behavior from Python code, use the existing functionality provided by ``importlib``. API Availability ---------------- While all the functions added here are considered public and stable API, the behavior of the functions is implementation specific. Most descriptions here refer to the CPython implementation, and while other implementations should provide the functions, there is no requirement that they behave the same. For example, ``sys.addaudithook()`` and ``sys.audit()`` should exist but may do nothing. This allows code to make calls to ``sys.audit()`` without having to test for existence, but it should not assume that its call will have any effect. (Including existence tests in security-critical code allows another vector to bypass auditing, so it is preferable that the function always exist.) ``importlib.util.open_for_import(path)`` should at a minimum always return ``_io.open(path, 'rb')``. Code using the function should make no further assumptions about what may occur, and implementations other than CPython are not required to let developers override the behavior of this function with a hook. Suggested Audit Hook Locations ============================== The locations and parameters in calls to ``sys.audit()`` or ``PySys_Audit()`` are to be determined by individual Python implementations. This is to allow maximum freedom for implementations to expose the operations that are most relevant to their platform, and to avoid or ignore potentially expensive or noisy events. Table 1 acts as both suggestions of operations that should trigger audit events on all implementations, and examples of event schemas. Table 2 provides further examples that are not required, but are likely to be available in CPython. Refer to the documentation associated with your version of Python to see which operations provide audit events. .. csv-table:: Table 1: Suggested Audit Hooks :header: "API Function", "Event Name", "Arguments", "Rationale" :widths: 2, 2, 3, 6 ``PySys_AddAuditHook``, ``sys.addaudithook``, "", "Detect when new audit hooks are being added. " ``PyImport_SetOpenForImportHook``, ``setopenforimporthook``, "", " Detects any attempt to set the ``open_for_import`` hook. " "``compile``, ``exec``, ``eval``, ``PyAst_CompileString``, ``PyAST_obj2mod``", ``compile``, "``(code, filename_or_none)``", " Detect dynamic code compilation, where ``code`` could be a string or AST. Note that this will be called for regular imports of source code, including those that were opened with ``open_for_import``. " "``exec``, ``eval``, ``run_mod``", ``exec``, "``(code_object,)``", " Detect dynamic execution of code objects. This only occurs for explicit calls, and is not raised for normal function invocation. " ``import``, ``import``, "``(module, filename, sys.path, sys.meta_path, sys.path_hooks)``", "Detect when modules are imported. This is raised before the module name is resolved to a file. All arguments other than the module name may be ``None`` if they are not used or available. " "``open``", ``open``, "``(path, mode, flags)``", "Detect when a file is about to be opened. *path* and *mode* are the usual parameters to ``open`` if available, while *flags* is provided instead of *mode* in some cases. " ``PyEval_SetProfile``, ``sys.setprofile``, "", "Detect when code is injecting trace functions. Because of the implementation, exceptions raised from the hook will abort the operation, but will not be raised in Python code. Note that ``threading.setprofile`` eventually calls this function, so the event will be audited for each thread. " ``PyEval_SetTrace``, ``sys.settrace``, "", "Detect when code is injecting trace functions. Because of the implementation, exceptions raised from the hook will abort the operation, but will not be raised in Python code. Note that ``threading.settrace`` eventually calls this function, so the event will be audited for each thread. " "``_PyObject_GenericSetAttr``, ``check_set_special_type_attr``, ``object_set_class``, ``func_set_code``, ``func_set_[kw]defaults``"," ``object.__setattr__``","``(object, attr, value)``","Detect monkey patching of types and objects. This event is raised for the ``__class__`` attribute and any attribute on ``type`` objects. " "``_PyObject_GenericSetAttr``",``object.__delattr__``,"``(object, attr)``","Detect deletion of object attributes. This event is raised for any attribute on ``type`` objects. " "``Unpickler.find_class``",``pickle.find_class``,"``(module_name, global_name)``","Detect imports and global name lookup when unpickling. " .. csv-table:: Table 2: Potential CPython Audit Hooks :header: "API Function", "Event Name", "Arguments", "Rationale" :widths: 2, 2, 3, 6 ``_PySys_ClearAuditHooks``, ``sys._clearaudithooks``, "", "Notifies hooks they are being cleaned up, mainly in case the event is triggered unexpectedly. This event cannot be aborted. " ``code_new``, ``code.__new__``, "``(bytecode, filename, name)``", " Detect dynamic creation of code objects. This only occurs for direct instantiation, and is not raised for normal compilation. " ``func_new_impl``, ``function.__new__``, "``(code,)``", "Detect dynamic creation of function objects. This only occurs for direct instantiation, and is not raised for normal compilation. " "``_ctypes.dlopen``, ``_ctypes.LoadLibrary``", ``ctypes.dlopen``, " ``(module_or_path,)``", "Detect when native modules are used. " ``_ctypes._FuncPtr``, ``ctypes.dlsym``, "``(lib_object, name)``", " Collect information about specific symbols retrieved from native modules. " ``_ctypes._CData``, ``ctypes.cdata``, "``(ptr_as_int,)``", "Detect when code is accessing arbitrary memory using ``ctypes``. " "``new_mmap_object``",``mmap.__new__``,"``(fileno, map_size, access, offset)``", "Detects creation of mmap objects. On POSIX, access may have been calculated from the ``prot`` and ``flags`` arguments. " ``sys._getframe``, ``sys._getframe``, "``(frame_object,)``", "Detect when code is accessing frames directly. " ``sys._current_frames``, ``sys._current_frames``, "", "Detect when code is accessing frames directly. " "``socket.bind``, ``socket.connect``, ``socket.connect_ex``, ``socket.getaddrinfo``, ``socket.getnameinfo``, ``socket.sendmsg``, ``socket.sendto``", ``socket.address``, "``(address,)``", "Detect access to network resources. The address is unmodified from the original call. " "``member_get``, ``func_get_code``, ``func_get_[kw]defaults`` ",``object.__getattr__``,"``(object, attr)``","Detect access to restricted attributes. This event is raised for any built-in members that are marked as restricted, and members that may allow bypassing imports. " "``urllib.urlopen``",``urllib.Request``,"``(url, data, headers, method)``", "Detects URL requests. " Performance Impact ================== The important performance impact is the case where events are being raised but there are no hooks attached. This is the unavoidable case - once a developer has added audit hooks they have explicitly chosen to trade performance for functionality. Performance impact with hooks added are not of interest here, since this is opt-in functionality. Analysis using the Python Performance Benchmark Suite [1]_ shows no significant impact, with the vast majority of benchmarks showing between 1.05x faster to 1.05x slower. In our opinion, the performance impact of the set of auditing points described in this PEP is negligible. Rejected Ideas ============== Separate module for audit hooks ------------------------------- The proposal is to add a new module for audit hooks, hypothetically ``audit``. This would separate the API and implementation from the ``sys`` module, and allow naming the C functions ``PyAudit_AddHook`` and ``PyAudit_Audit`` rather than the current variations. Any such module would need to be a built-in module that is guaranteed to always be present. The nature of these hooks is that they must be callable without condition, as any conditional imports or calls provide opportunities to intercept and suppress or modify events. Given it is one of the most core modules, the ``sys`` module is somewhat protected against module shadowing attacks. Replacing ``sys`` with a sufficiently functional module that the application can still run is a much more complicated task than replacing a module with only one function of interest. An attacker that has the ability to shadow the ``sys`` module is already capable of running arbitrary code from files, whereas an ``audit`` module could be replaced with a single line in a ``.pth`` file anywhere on the search path:: import sys; sys.modules['audit'] = type('audit', (object,), {'audit': lambda *a: None, 'addhook': lambda *a: None}) Multiple layers of protection already exist for monkey patching attacks against either ``sys`` or ``audit``, but assignments or insertions to ``sys.modules`` are not audited. This idea is rejected because it makes it trivial to suppress all calls to ``audit``. Flag in sys.flags to indicate "audited" mode -------------------------------------------- The proposal is to add a value in ``sys.flags`` to indicate when Python is running in a "secure" or "audited" mode. This would allow applications to detect when some features are enabled or when hooks have been added and modify their behaviour appropriately. Currently, we are not aware of any legitimate reasons for a program to behave differently in the presence of audit hooks. Both application-level APIs ``sys.audit`` and ``importlib.util.open_for_import`` are always present and functional, regardless of whether the regular ``python`` entry point or some alternative entry point is used. Callers cannot determine whether any hooks have been added (except by performing side-channel analysis), nor do they need to. The calls should be fast enough that callers do not need to avoid them, and the program is responsible for ensuring that any added hooks are fast enough to not affect application performance. The argument that this is "security by obscurity" is valid, but irrelevant. Security by obscurity is only an issue when there are no other protective mechanisms; obscurity as the first step in avoiding attack is strongly recommended (see `this article <https://danielmiessler.com/study/security-by-obscurity/>`_ for discussion). This idea is rejected because there are no appropriate reasons for an application to change its behaviour based on whether these APIs are in use. Why Not A Sandbox ================= Sandboxing CPython has been attempted many times in the past, and each past attempt has failed. Fundamentally, the problem is that certain functionality has to be restricted when executing the sandboxed code, but otherwise needs to be available for normal operation of Python. For example, completely removing the ability to compile strings into bytecode also breaks the ability to import modules from source code, and if it is not completely removed then there are too many ways to get access to that functionality indirectly. There is not yet any feasible way to generically determine whether a given operation is "safe" or not. Further information and references available at [2]_. This proposal does not attempt to restrict functionality, but simply exposes the fact that the functionality is being used. Particularly for intrusion scenarios, detection is significantly more important than early prevention (as early prevention will generally drive attackers to use an alternate, less-detectable, approach). The availability of audit hooks alone does not change the attack surface of Python in any way, but they enable defenders to integrate Python into their environment in ways that are currently not possible. Since audit hooks have the ability to safely prevent an operation occuring, this feature does enable the ability to provide some level of sandboxing. In most cases, however, the intention is to enable logging rather than creating a sandbox. Relationship to PEP 551 ======================= This API was originally presented as part of `PEP 551 <https://www.python.org/dev/peps/pep-0551/>`_ Security Transparency in the Python Runtime. For simpler review purposes, and due to the broader applicability of these APIs beyond security, the API design is now presented separately. PEP 551 is an informational PEP discussing how to integrate Python into a secure or audited environment. References ========== .. [1] Python Performance Benchmark Suite `<https://github.com/python/performance>`_ .. [2] Python Security model - Sandbox `<https://python-security.readthedocs.io/security.html#sandbox>`_ Copyright ========= Copyright (c) 2019 by Microsoft Corporation. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0 or later (the latest version is presently available at http://www.opencontent.org/openpub/).

7 25

bpo-36558: Change time.mktime() return type from float to int?
by Victor Stinner 16 Apr '19

16 Apr '19

Hi, time.mktime() looks "inconsistent" to me and I would like to change it, but I'm not sure how it impacts backward compatibility. https://bugs.python.org/issue36558 time.mktime() returns a floating point number: >>> type(time.mktime(time.localtime())) <class 'float'> The documentation says: "It returns a floating point number, for compatibility with :func:`.time`." time.time() returns a float because it has sub-second resolution, but the C function mktime() returns an integer number of seconds. Would it make sense to change mktime() return type from float to int? I would like to change mktime() return type to make the function more consistent: all inputs are integers, it sounds wrong to me to return float. The result should be integer as well. How much code would it break? I guess that the main impact are unit tests relying on repr(time.mktime(t)) exact value. But it's easy to fix the tests: use int(time.mktime(t)) or "%.0f" % time.mktime(t) to never get ".0", or use float(time.mktime(t))) to explicitly cast for a float (that which be a bad but quick fix). Note: I wrote and implemented the PEP 564 to avoid any precision loss. mktime() will not start loosing precision before year 285,422,891 (which is quite far in the future ;-)). Victor -- Night gathers, and now my watch begins. It shall not end until my death.

5 6