Python-Dev March 2019

python-dev@python.org

83 participants
79 discussions

Strange umask(?)/st_mode issue
by Steve Dower 01 Apr '19

01 Apr '19

Hi I'm trying to track down the cause of the failed Linux tests on this build (it's also been happening on random PR builds for the last day or two, but this is the first I've seen it happen on already merged code): https://dev.azure.com/Python/cpython/_build/results?buildId=40189 One of the failed tests is this one from test_tarfile.py (and I think this is representative of the others): def test_file_mode(self): # Test for issue #8464: Create files with correct # permissions. if os.path.exists(tmpname): support.unlink(tmpname) original_umask = os.umask(0o022) try: tar = tarfile.open(tmpname, self.mode) tar.close() mode = os.stat(tmpname).st_mode & 0o777 self.assertEqual(mode, 0o644, "wrong file permissions") finally: os.umask(original_umask) It's failing because the mode of the new TAR file is 0o666 rather than 0o644. I'd guess that this means the umask() call is not having effect? But honestly this is beyond me - I don't have any idea what would cause Python to ignore that, and I don't see anything in the commit history that could be at fault. I'm hoping someone can tell me that it's definitely a platform configuration issue or it's definitely interference between two of our tests (which run in random order, which would explain the intermittent nature of this, but I haven't spotted any tests that seem likely either). To see the other failed PR builds, the full list is at https://dev.azure.com/Python/cpython/_build?definitionId=9 and most of the ones from today have failed because of whatever is causing it. Any help? Thanks, Steve

6 14

Removing PendingDeprecationWarning
by Inada Naoki 01 Apr '19

01 Apr '19

Hi, all. I'm thinking about removing PendingDeprecationWarning. (previous discussion: https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038) It was added "not be printed by default" version of DeprecationWarning. But DeprecationWarning is not printed by default now. We use PendingDeprecationWarning for N-2 release, and change it to DeprecationWarning for N-1 release. But this workflow seems not worth enough for now. I want to stop using PendingDeprecationWarning for new deprecation. More aggressively, I want to remove PendingDeprecationWarning class, and `PendingDeprecationWarning = DeprecationWarning` for backward compatibility. How do you think? May I do it in Python 3.8? -- Inada Naoki <songofacandy(a)gmail.com>

10 34

Tests for internal functionality
by Ivan Pozdeev 30 Mar '19

30 Mar '19

In https://github.com/python/cpython/pull/6541 , I was requested to add tests for an internal C function. As I wrote in https://github.com/python/cpython/pull/6541#issuecomment-445514807 , it's not clear from the codebase 1) where tests for internal (as opposed to public) functionality should be located * I only ran across very few tests for private functionality and they were located alongside the public tests. See e.g. https://github.com/python/cpython/pull/12140 -- site._getuserbase is a private function. 2) what spec one should test against since there's no official docs for internal functionality That time, they let it slide, but of course this is not something to make a habit of. It still bothers me that I left unfinished business there. Any thoughts? -- Regards, Ivan

3 2

Summary of Python tracker Issues
by Python tracker 29 Mar '19

29 Mar '19

ACTIVITY SUMMARY (2019-03-22 - 2019-03-29) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7048 ( -3) closed 41176 (+76) total 48224 (+73) Open issues with patches: 2809 Issues opened (50) ================== #22189: collections.UserString missing some str methods https://bugs.python.org/issue22189 reopened by rhettinger #28190: Cross-build _curses failed if host ncurses headers and target https://bugs.python.org/issue28190 reopened by doko #33356: Windows 10 buildbot: test__xxsubinterpreters.test_already_runn https://bugs.python.org/issue33356 reopened by vstinner #35155: Clarify Protocol Handlers in urllib.request Docs https://bugs.python.org/issue35155 reopened by orsenthil #36150: Possible assertion failures due to _ctypes.c's PyCData_reduce( https://bugs.python.org/issue36150 reopened by serhiy.storchaka #36344: install_certificates.command too complicated https://bugs.python.org/issue36344 reopened by ned.deily #36402: test_threading: test_threads_join_2() failed with "Fatal Pytho https://bugs.python.org/issue36402 opened by vstinner #36403: AsyncIterator on 3.7: __aiter__ no longer honors finally block https://bugs.python.org/issue36403 opened by asksol #36404: Document PendingDeprecationWarning as deprecated https://bugs.python.org/issue36404 opened by inada.naoki #36406: doctest.testmod(empty_package) raises TypeError in 3.7 (and no https://bugs.python.org/issue36406 opened by Dutcho #36408: Tkinter multi-processing performance, Linux 10-25 times faster https://bugs.python.org/issue36408 opened by james.mccormack #36409: plistlib old API should be removed https://bugs.python.org/issue36409 opened by bigfootjon #36411: Python 3 f.tell() gets out of sync with file pointer in binary https://bugs.python.org/issue36411 opened by pewscorner #36414: Multiple test failures in GCC and Clang optional builds on Tra https://bugs.python.org/issue36414 opened by xtreak #36416: bytes.rpartition bug in online documentation https://bugs.python.org/issue36416 opened by pewscorner #36417: unicode.isdecimal bug in online Python 2 documentation https://bugs.python.org/issue36417 opened by pewscorner #36418: urllib.parse.*Result: support _replace for additional computed https://bugs.python.org/issue36418 opened by cheryl.sabella #36419: IDLE autocomplete: refactor and polish code and tests https://bugs.python.org/issue36419 opened by terry.reedy #36422: tempfile.TemporaryDirectory() removes entire directory tree ev https://bugs.python.org/issue36422 opened by riccardomurri #36424: Pickle fails on frozen dataclass that has slots https://bugs.python.org/issue36424 opened by drhagen #36426: exec() issue when used inside function https://bugs.python.org/issue36426 opened by schperplata #36427: Document that PyEval_RestoreThread and PyGILState_Ensure can t https://bugs.python.org/issue36427 opened by pablogsal #36429: Fix starting IDLE with pyshell https://bugs.python.org/issue36429 opened by terry.reedy #36432: Running python test suite fails on macOS 10.14.4 with resource https://bugs.python.org/issue36432 opened by ned.deily #36434: Zipfile breaks if signalled during write() https://bugs.python.org/issue36434 opened by and800 #36437: method_descriptor surprising error message when self is passed https://bugs.python.org/issue36437 opened by vstinner #36438: Python 3.5.7 import error on Cross compile https://bugs.python.org/issue36438 opened by Brian Spratke #36439: Inconsistencies with datetime.fromtimestamp(t) when t < 0 https://bugs.python.org/issue36439 opened by Tiger-222 #36440: more helpful diagnostics for parser module https://bugs.python.org/issue36440 opened by A. Skrobov #36442: Different ValueError for the same operation in List and Tuple https://bugs.python.org/issue36442 opened by HardikPatel #36445: bus error in test_gil test on armhf running with 64bit kernel https://bugs.python.org/issue36445 opened by doko #36448: Message "You will need to rebuild pythoncore to see the change https://bugs.python.org/issue36448 opened by jdemeyer #36449: __aexit__ is not called when a context manager is used in an a https://bugs.python.org/issue36449 opened by Andrey Lemets #36453: pkgutil.get_importer only return the first valid path_hook(imp https://bugs.python.org/issue36453 opened by Windson Yang #36454: test_time: test_monotonic() failed on AMD64 FreeBSD 10-STABLE https://bugs.python.org/issue36454 opened by vstinner #36456: task.cancel unbound recursion https://bugs.python.org/issue36456 opened by Dima.Tisnek #36457: functools.singledispatchmethod interacts poorly with subclasse https://bugs.python.org/issue36457 opened by Tim Mitchell2 #36460: Add AMP MIME type support https://bugs.python.org/issue36460 opened by Daniel Black #36461: timeit: Additional changes for autorange https://bugs.python.org/issue36461 opened by cheryl.sabella #36463: python37.dll crashing 0xc000041d https://bugs.python.org/issue36463 opened by Savagery #36464: Python 2.7 build install fails intermittently with -j on MacOS https://bugs.python.org/issue36464 opened by madscientist #36465: No longer enable Py_TRACE_REFS by default in debug build https://bugs.python.org/issue36465 opened by vstinner #36466: Adding a way to strip annotations from compiled bytecode https://bugs.python.org/issue36466 opened by cary #36467: IDLE to help with invisible characters https://bugs.python.org/issue36467 opened by rhettinger #36468: Treeview: wrong color change https://bugs.python.org/issue36468 opened by guillaumeb #36469: Stuck during interpreter exit, attempting to take the GIL https://bugs.python.org/issue36469 opened by mocramis #36470: dataclasses.replace raises an exception if InitVar with defaul https://bugs.python.org/issue36470 opened by Greg Kuhn #36471: PEP 432, PEP 587: Add _Py_RunMain() https://bugs.python.org/issue36471 opened by vstinner #36472: Some old PR with CLA not signed https://bugs.python.org/issue36472 opened by eamanu #36473: Detect all dictionary changes during iteration https://bugs.python.org/issue36473 opened by thomas.perl Most recent 15 issues with no replies (15) ========================================== #36468: Treeview: wrong color change https://bugs.python.org/issue36468 #36467: IDLE to help with invisible characters https://bugs.python.org/issue36467 #36466: Adding a way to strip annotations from compiled bytecode https://bugs.python.org/issue36466 #36457: functools.singledispatchmethod interacts poorly with subclasse https://bugs.python.org/issue36457 #36456: task.cancel unbound recursion https://bugs.python.org/issue36456 #36454: test_time: test_monotonic() failed on AMD64 FreeBSD 10-STABLE https://bugs.python.org/issue36454 #36445: bus error in test_gil test on armhf running with 64bit kernel https://bugs.python.org/issue36445 #36438: Python 3.5.7 import error on Cross compile https://bugs.python.org/issue36438 #36437: method_descriptor surprising error message when self is passed https://bugs.python.org/issue36437 #36434: Zipfile breaks if signalled during write() https://bugs.python.org/issue36434 #36424: Pickle fails on frozen dataclass that has slots https://bugs.python.org/issue36424 #36419: IDLE autocomplete: refactor and polish code and tests https://bugs.python.org/issue36419 #36418: urllib.parse.*Result: support _replace for additional computed https://bugs.python.org/issue36418 #36417: unicode.isdecimal bug in online Python 2 documentation https://bugs.python.org/issue36417 #36404: Document PendingDeprecationWarning as deprecated https://bugs.python.org/issue36404 Most recent 15 issues waiting for review (15) ============================================= #36473: Detect all dictionary changes during iteration https://bugs.python.org/issue36473 #36471: PEP 432, PEP 587: Add _Py_RunMain() https://bugs.python.org/issue36471 #36466: Adding a way to strip annotations from compiled bytecode https://bugs.python.org/issue36466 #36465: No longer enable Py_TRACE_REFS by default in debug build https://bugs.python.org/issue36465 #36448: Message "You will need to rebuild pythoncore to see the change https://bugs.python.org/issue36448 #36440: more helpful diagnostics for parser module https://bugs.python.org/issue36440 #36434: Zipfile breaks if signalled during write() https://bugs.python.org/issue36434 #36429: Fix starting IDLE with pyshell https://bugs.python.org/issue36429 #36427: Document that PyEval_RestoreThread and PyGILState_Ensure can t https://bugs.python.org/issue36427 #36416: bytes.rpartition bug in online documentation https://bugs.python.org/issue36416 #36409: plistlib old API should be removed https://bugs.python.org/issue36409 #36406: doctest.testmod(empty_package) raises TypeError in 3.7 (and no https://bugs.python.org/issue36406 #36404: Document PendingDeprecationWarning as deprecated https://bugs.python.org/issue36404 #36398: A possible crash in structseq.c's structseq_repr() https://bugs.python.org/issue36398 #36394: test_multiprocessing_spawn fails on Windows7 3.x buildbot https://bugs.python.org/issue36394 Top 10 most discussed issues (10) ================================= #36345: Deprecate Tools/scripts/serve.py in favour of python -m http.s https://bugs.python.org/issue36345 12 msgs #36305: Inconsistent behavior of pathlib.WindowsPath with drive paths https://bugs.python.org/issue36305 10 msgs #36347: Renaming the constants for the .flags of PyMemberDef https://bugs.python.org/issue36347 10 msgs #35983: tp_dealloc trashcan shouldn't be called for subclasses https://bugs.python.org/issue35983 8 msgs #36460: Add AMP MIME type support https://bugs.python.org/issue36460 6 msgs #36414: Multiple test failures in GCC and Clang optional builds on Tra https://bugs.python.org/issue36414 5 msgs #36427: Document that PyEval_RestoreThread and PyGILState_Ensure can t https://bugs.python.org/issue36427 5 msgs #36469: Stuck during interpreter exit, attempting to take the GIL https://bugs.python.org/issue36469 5 msgs #10716: Modernize pydoc to use better HTML and separate CSS https://bugs.python.org/issue10716 4 msgs #31327: bug in dateutil\tz\tz.py https://bugs.python.org/issue31327 4 msgs Issues closed (78) ================== #5340: Change in cgi behavior breaks existing software https://bugs.python.org/issue5340 closed by inada.naoki #6422: timeit called from within Python should allow autoranging https://bugs.python.org/issue6422 closed by cheryl.sabella #12445: dict view values objects are missing tp_richcmp and tp_as_numb https://bugs.python.org/issue12445 closed by inada.naoki #19879: imageop: bug in error handler https://bugs.python.org/issue19879 closed by inada.naoki #21019: PyMethodDef ml_name is char* instead of const char* https://bugs.python.org/issue21019 closed by inada.naoki #21022: PyMemberDef doc is char* instead of const char* https://bugs.python.org/issue21022 closed by inada.naoki #22327: test_gdb failures on Ubuntu 14.10 https://bugs.python.org/issue22327 closed by cheryl.sabella #22503: Signal stack overflow in faulthandler_user https://bugs.python.org/issue22503 closed by vstinner #23126: Add Python hook function to replace NameError https://bugs.python.org/issue23126 closed by inada.naoki #23205: Unit test needed for IDLE's GrepDialog.py's findfiles() https://bugs.python.org/issue23205 closed by cheryl.sabella #25614: Lib/code.py: InteractiveConsole.raw_input writes prompt to std https://bugs.python.org/issue25614 closed by cheryl.sabella #27464: Document that SplitResult & friends are namedtuples https://bugs.python.org/issue27464 closed by martin.panter #27797: ASCII file with UNIX line conventions and enough lines throws https://bugs.python.org/issue27797 closed by inada.naoki #30348: IDLE: Add test_autocomplete unittests https://bugs.python.org/issue30348 closed by terry.reedy #31292: `python setup.py check --restructuredtext` fails when a includ https://bugs.python.org/issue31292 closed by cheryl.sabella #31822: Document that urllib.parse.{Defrag,Split,Parse}Result are name https://bugs.python.org/issue31822 closed by cheryl.sabella #32217: freeze.py fails to work. https://bugs.python.org/issue32217 closed by cheryl.sabella #32419: Add unittest support for pyc projects https://bugs.python.org/issue32419 closed by inada.naoki #32823: Regression in test -j behavior and time in 3.7.0b1 https://bugs.python.org/issue32823 closed by terry.reedy #32950: profiling python gc https://bugs.python.org/issue32950 closed by inada.naoki #33043: Add a 'Contributing to Docs' link at the bottom of docs.python https://bugs.python.org/issue33043 closed by willingc #33319: `subprocess.run` documentation doesn't tell is using `stdout=P https://bugs.python.org/issue33319 closed by gregory.p.smith #33832: doc Add "magic method" entry to Glossary https://bugs.python.org/issue33832 closed by eric.araujo #34085: doc Improve wording on classmethod/staticmethod https://bugs.python.org/issue34085 closed by cheryl.sabella #34745: asyncio ssl memory leak https://bugs.python.org/issue34745 closed by thehesiod #34826: io.BufferedReader crashes in 2.7 on memoryview attribute acces https://bugs.python.org/issue34826 closed by inada.naoki #34995: functools.cached_property does not maintain the wrapped method https://bugs.python.org/issue34995 closed by inada.naoki #35165: Possible wrong method name in attribute references doc https://bugs.python.org/issue35165 closed by cheryl.sabella #35194: A typo in a constant in cp932 codec https://bugs.python.org/issue35194 closed by inada.naoki #35528: [DOC] [LaTeX] Sphinx 2.0 uses GNU FreeFont as default for xela https://bugs.python.org/issue35528 closed by mdk #36064: docs: urllib.request.Request not accepting iterables data type https://bugs.python.org/issue36064 closed by mdk #36143: Auto-generate Lib/keyword.py https://bugs.python.org/issue36143 closed by pablogsal #36202: Calling Py_DecodeLocale() before _PyPreConfig_Write() can prod https://bugs.python.org/issue36202 closed by vstinner #36218: .sort() segfaults consistently on crafted input https://bugs.python.org/issue36218 closed by rhettinger #36236: Python crash on macOS when CWD is invalid https://bugs.python.org/issue36236 closed by ned.deily #36245: PCBuild/build.bat errors, probably from space characters in pa https://bugs.python.org/issue36245 closed by steve.dower #36260: Cpython/Lib vulnerability found and request a patch submission https://bugs.python.org/issue36260 closed by krnick #36286: Random failure in test_idle https://bugs.python.org/issue36286 closed by serhiy.storchaka #36298: Lib/pyclbr.py crashes when the package spec cannot be determin https://bugs.python.org/issue36298 closed by brett.cannon #36301: Add _Py_PreInitialize() function https://bugs.python.org/issue36301 closed by vstinner #36318: Adding support for setting the "disabled" attribute of loggers https://bugs.python.org/issue36318 closed by maggyero #36326: Teach inpsect.getdoc() to read __slots__ with an optional data https://bugs.python.org/issue36326 closed by rhettinger #36340: 3.7.3rc1 Install Certificates fails on macOS https://bugs.python.org/issue36340 closed by ned.deily #36354: Use CreateProcessW for Python 2.7 on Windows. https://bugs.python.org/issue36354 closed by gregory.p.smith #36366: Patcher stop method should be idempotent https://bugs.python.org/issue36366 closed by brett.cannon #36370: Check for PyErr_Occurred() after PyImport_GetModule(). https://bugs.python.org/issue36370 closed by skrah #36381: Deprecate "#" argument format without PY_SSIZE_T_CLEAN https://bugs.python.org/issue36381 closed by inada.naoki #36387: Refactor getenvironment() in _winapi.c https://bugs.python.org/issue36387 closed by serhiy.storchaka #36396: Remove fgBg param of idlelib.config.GetHighlight() https://bugs.python.org/issue36396 closed by terry.reedy #36401: Readonly properties should be marked as such in help() https://bugs.python.org/issue36401 closed by rhettinger #36405: Use dict unpacking in idlelib https://bugs.python.org/issue36405 closed by terry.reedy #36407: xml.dom.minidom wrong indentation writing for CDATA section https://bugs.python.org/issue36407 closed by serhiy.storchaka #36410: Proposal to make strip/lstrip/rstrip more explicit https://bugs.python.org/issue36410 closed by rhettinger #36412: A possible crash in dictobject.c's new_dict() https://bugs.python.org/issue36412 closed by inada.naoki #36413: Brief Tour of the Standard Library — Part II example problem https://bugs.python.org/issue36413 closed by rhettinger #36415: [math] Implement pow2 function https://bugs.python.org/issue36415 closed by serhiy.storchaka #36420: f_trace_opcodes setting and accessing opcodes https://bugs.python.org/issue36420 closed by anthony shaw #36421: A possible double decref in _ctypes.c's PyCArrayType_new() https://bugs.python.org/issue36421 closed by serhiy.storchaka #36423: www.pythontest.net is down https://bugs.python.org/issue36423 closed by vstinner #36425: Add Simplified Chinese to the language switcher https://bugs.python.org/issue36425 closed by mdk #36428: Support AutoAwait? https://bugs.python.org/issue36428 closed by gvanrossum #36430: A possible reference leak in itertools.count() https://bugs.python.org/issue36430 closed by serhiy.storchaka #36431: Use dict unpacking for merging two dicts https://bugs.python.org/issue36431 closed by serhiy.storchaka #36433: Unhelpful error message in classmethoddescr_call() https://bugs.python.org/issue36433 closed by inada.naoki #36435: multiprocessing crashes in Python 3.7.3 on Windows https://bugs.python.org/issue36435 closed by paul.moore #36436: _testcapi.pymem_buffer_overflow() doesn't handle memory alloca https://bugs.python.org/issue36436 closed by vstinner #36441: Cannot create venv with debug binaries installed https://bugs.python.org/issue36441 closed by steve.dower #36443: Disable coerce_c_locale and utf8_mode by default in _PyPreConf https://bugs.python.org/issue36443 closed by vstinner #36444: Python initialization: remove _PyMainInterpreterConfig https://bugs.python.org/issue36444 closed by vstinner #36446: Need Windows x86-64 MSI installer for Python >= 3.x.x https://bugs.python.org/issue36446 closed by zach.ware #36447: test__xxsubinterpreters leaked references and memory blocks https://bugs.python.org/issue36447 closed by vstinner #36450: 3D Array Assignment to all 2D sub-arrays https://bugs.python.org/issue36450 closed by eric.snow #36451: Docs zh-cn roots error in nav bar https://bugs.python.org/issue36451 closed by mdk #36452: Detect dict iteration "overflow" when changing keys https://bugs.python.org/issue36452 closed by inada.naoki #36455: collections.abc.Sequence doesn't support reflection/introspect https://bugs.python.org/issue36455 closed by josh.r #36458: Compile errors --without-threads https://bugs.python.org/issue36458 closed by ned.deily #36459: A possible double PyMem_FREE() due to tokenizer.c's tok_nextc( https://bugs.python.org/issue36459 closed by serhiy.storchaka #36462: CVE-2019-9674 : zip bomb vulnerability in Lib/zipfile.py https://bugs.python.org/issue36462 closed by brett.cannon

1 0

PEP 570
by Pablo Galindo Salgado 29 Mar '19

29 Mar '19

Hi everyone, We have submitted PEP 570 for consideration to the steering council: https://github.com/python/steering-council/issues/4 The discussion is happening on discourse: https://discuss.python.org/t/pep-570-python-positional-only-parameters/1078 To eliminate splitting the discussion into two forums (discourse and python-dev), we kindly ask you to go to discourse if you want to participate in the debate :) Here is the full document of the PEP: https://www.python.org/dev/peps/pep-0570/ Pablo Galindo Salgado

5 4

PEP 556 threaded garbage collection & linear recursion in gc
by Gregory P. Smith 28 Mar '19

28 Mar '19

https://www.python.org/dev/peps/pep-0556/ This PEP is currently Deferred as nobody is actively working on a test implementation. A situation came up the other day where I *believe* this could've helped. Scenario (admittedly not one most environments run into): A Python process with a C++ extension module implementing a threaded server (threads spawned by C++) that could call back into CPython within server request handlers. (ie: how all threaded servers tend to work regardless of core loop implementation language) Python code in the application had done something (unknown to me, I didn't dive into their code) that built up large enough presumably nested or recursive data structures that the garbage collector, when triggered, would wind up in very deep recursion. This caused a stack overflow as the C++ spawned threads were only being given a 256k stack (to conserve virtual address space - there can potentially be a _ton_ of threads in this code). That had a C++ stack trace 1000+ levels deep repeating the pattern of ... @ 0x564d59bd21de 32 func_dealloc @ 0x564d59bce0c1 32 cell_dealloc @ 0x564d5839db41 48 tupledealloc @ 0x564d59bd21de 32 func_dealloc @ 0x564d59bce0c1 32 cell_dealloc @ 0x564d5839db41 48 tupledealloc ... If our gc were done on a thread of its own spawned by Python, with a typical normal larger default stack size (8mb) this would've been less likely to trigger a crash (though obviously still possible if the recursion is linear). I know, there are obvious workarounds to this situation of all sorts. My point is more that synchronously triggering gc _within_ a thread that happens to invoke the periodic "hey, lets do another gc run" logic the eval loop is undesirable. I'm not making any plans to work on an implementation for this PEP, deferred seems accurate. Just dropping a note that I'd still be interested in seeing something other than synchronous gc in arbitrary threads happen when a process has multiple threads. *Another* take away from this is that it *appears* possible to cause our gc to go into linear recursion. Yuck! I'd file an issue on that, but doing so requires making some example code to construct such a scenario first... Food for thought in case these kinds of things are something anyone else has encountered. -Greg

5 6

VxWorks and Python
by Kuhl, Brian 28 Mar '19

28 Mar '19

Hi Python developers, Victor suggested I post a little background to help the maintainers understand VxWorks a little better. It can be found here https://github.com/Wind-River/cpython/wiki/Python-on-VxWorks If you'd like more detail in a particular area, please feel free to reply on or off list. I'd also like to mention again, that Wind River is interested in setting up a short term consulting arrangement, with any core dev that's available. To help accelerate Peixing's PRs acceptance https://github.com/python/cpython/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+… and generally improve Cpython. Wind River operates globally so don't let you location dissuade you. Many thanks for all your help, Brian

1 0

Mistake in 3.7.3 Changelog
by Dima Tisnek 28 Mar '19

28 Mar '19

Hi list, Sorry, I'm not sure where to post this. The Changelog reads: bpo-12477: Fix bug in parsermodule when parsing a state in a DFA that has two or more arcs with labels of the same type. Patch by Pablo Galindo. The `12477` is actually a master github pull request number, not a Python bug number. Correct BPO is https://bugs.python.org/issue36256 Cheers, d

2 2

PEP 587: Python Initialization Configuration
by Victor Stinner 27 Mar '19

27 Mar '19

Hi, Since Steve Dower asked me to write a formal PEP for my proposal of a new C API to initialize Python, here you have! This new PEP will be shortly rendered as HTML at: https://www.python.org/dev/peps/pep-0587/ The Rationale might be a little bit weak at this point and there are still 3 small open questions, but the whole PEP should give you a better overview of my proposal than my previous email :-) I chose to start with a new PEP rather than updating PEP 432. I didn't feel able to just "update" the PEP 432, my proposal is different. The main difference between my PEP 587 and Nick Coghlan's PEP 432 is that my PEP 587 only allows to configure Python *before* its initialization, whereas Nick wanted to give control between its "Initializing" and "Initialized" initialization phases. Moreover, my PEP only uses C types rather Nick wanted to use Python types. By setting the private PyConfig._init_main field to 1, my PEP 587 allows to stop Python initialization early to get a bare minimum working Python only with builtin types and no importlib. If I understood correctly, with my PEP, it remains possible to *extend* Python C API later to implement what Nick wants. Nick: please correct me if I'm wrong :-) Victor PEP: 587 Title: Python Initialization Configuration Author: Victor Stinner <vstinner(a)redhat.com> Discussions-To: python-dev(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Mar-2018 Python-Version: 3.8 Abstract ======== Add a new C API to configure the Python Initialization providing finer control on the whole configuration and better error reporting. Rationale ========= Python is highly configurable but its configuration evolved organically: configuration parameters is scattered all around the code using different ways to set them (mostly global configuration variables and functions). A straightforward and reliable way to configure Python is needed. Some configuration parameters are not accessible from the C API, or not easily. The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as input whereas the Python filesystem encoding is set during the initialization. Python Initialization C API =========================== This PEP proposes to add the following new structures, functions and macros. New structures (4): * ``PyConfig`` * ``PyInitError`` * ``PyPreConfig`` * ``PyWideCharList`` New functions (8): * ``Py_PreInitialize(config)`` * ``Py_PreInitializeFromArgs(config, argc, argv)`` * ``Py_PreInitializeFromWideArgs(config, argc, argv)`` * ``Py_InitializeFromConfig(config)`` * ``Py_InitializeFromArgs(config, argc, argv)`` * ``Py_InitializeFromWideArgs(config, argc, argv)`` * ``Py_UnixMain(argc, argv)`` * ``Py_ExitInitError(err)`` New macros (6): * ``Py_INIT_OK()`` * ``Py_INIT_ERR(MSG)`` * ``Py_INIT_USER_ERR(MSG)`` * ``Py_INIT_NO_MEMORY()`` * ``Py_INIT_EXIT(EXITCODE)`` * ``Py_INIT_FAILED(err)`` PyWideCharList -------------- ``PyWideCharList`` is a list of ``wchar_t*`` strings. Example to initialize a string from C static array:: static wchar_t* argv[2] = { L"-c", L"pass", }; PyWideCharList config_argv = PyWideCharList_INIT; config_argv.length = Py_ARRAY_LENGTH(argv); config_argv.items = argv; ``PyWideCharList`` structure fields: * ``length`` (``Py_ssize_t``) * ``items`` (``wchar_t**``) If *length* is non-zero, *items* must be non-NULL and all strings must be non-NULL. .. note:: The "WideChar" name comes from the existing Python C API. Example: ``PyUnicode_FromWideChar(const wchar_t *str, Py_ssize_t size)``. PyInitError ----------- ``PyInitError`` is a structure to store an error message or an exit code for the Python Initialization. Example:: PyInitError alloc(void **ptr, size_t size) { *ptr = PyMem_RawMalloc(size); if (*ptr == NULL) { return Py_INIT_NO_MEMORY(); } return Py_INIT_OK(); } int main(int argc, char **argv) { void *ptr; PyInitError err = alloc(&ptr, 16); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } PyMem_Free(ptr); return 0; } ``PyInitError`` fields: * ``exitcode`` (``int``): if greater or equal to zero, argument passed to ``exit()`` * ``msg`` (``const char*``): error message * ``prefix`` (``const char*``): string displayed before the message, usually rendered as ``prefix: msg``. * ``user_err`` (int): if non-zero, the error is caused by the user configuration, otherwise it's an internal Python error. Macro to create an error: * ``Py_INIT_OK()``: success * ``Py_INIT_NO_MEMORY()``: memory allocation failure (out of memory) * ``Py_INIT_ERR(MSG)``: Python internal error * ``Py_INIT_USER_ERR(MSG)``: error caused by user configuration * ``Py_INIT_EXIT(STATUS)``: exit Python with the specified status Other macros and functions: * ``Py_INIT_FAILED(err)``: Is the result an error or an exit? * ``Py_ExitInitError(err)``: call ``exit(status)`` for an error created by ``Py_INIT_EXIT(status)``, call ``Py_FatalError(msg)`` for other errors. Must not be called for an error created by ``Py_INIT_OK()``. Pre-Initialization with PyPreConfig ----------------------------------- ``PyPreConfig`` structure is used to pre-initialize Python: * Set the memory allocator * Configure the LC_CTYPE locale * Set the UTF-8 mode Example using the pre-initialization to enable the UTF-8 Mode:: PyPreConfig preconfig = PyPreConfig_INIT; preconfig.utf8_mode = 1; PyInitError err = Py_PreInitialize(&preconfig); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* at this point, Python will speak UTF-8 */ Py_Initialize(); /* ... use Python API here ... */ Py_Finalize(); Functions to pre-initialize Python: * ``PyInitError Py_PreInitialize(const PyPreConfig *config)`` * ``PyInitError Py_PreInitializeFromArgs( const PyPreConfig *config, int argc, char **argv)`` * ``PyInitError Py_PreInitializeFromWideArgs( const PyPreConfig *config, int argc, wchar_t **argv)`` These functions can be called with *config* set to ``NULL``. The caller is responsible to handler error using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. ``PyPreConfig`` fields: * ``allocator``: name of the memory allocator (ex: ``"malloc"``) * ``coerce_c_locale_warn``: if non-zero, emit a warning if the C locale is coerced. * ``coerce_c_locale``: if equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE to decide if it should be coerced. * ``dev_mode``: see ``PyConfig.dev_mode`` * ``isolated``: see ``PyConfig.isolated`` * ``legacy_windows_fs_encoding`` (Windows only): if non-zero, set the Python filesystem encoding to ``"mbcs"``. * ``use_environment``: see ``PyConfig.use_environment`` * ``utf8_mode``: if non-zero, enable the UTF-8 mode The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``, ``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python enable them depending on the user configuration. Initialization with PyConfig ---------------------------- The ``PyConfig`` structure contains all parameters to configure Python. Example of Python initialization enabling the isolated mode:: PyConfig config = PyConfig_INIT; config.isolated = 1; PyInitError err = Py_InitializeFromConfig(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* ... use Python API here ... */ Py_Finalize(); Functions to initialize Python: * ``PyInitError Py_InitializeFromConfig(const PyConfig *config)`` * ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int argc, char **argv)`` * ``PyInitError Py_InitializeFromWideArgs(const PyConfig *config, int argc, wchar_t **argv)`` These functions can be called with *config* set to ``NULL``. The caller is responsible to handler error using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. PyConfig fields: * ``argv``: ``sys.argv`` * ``base_exec_prefix``: ``sys.base_exec_prefix`` * ``base_prefix``: ``sys.base_prefix`` * ``buffered_stdio``: if equals to 0, enable unbuffered mode, make stdout and stderr streams to be unbuffered. * ``bytes_warning``: if equals to 1, issue a warning when comparing ``bytes`` or ``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If equal or greater to 2, raise a ``BytesWarning`` exception. * ``dll_path`` (Windows only): Windows DLL path * ``dump_refs``: if non-zero, display all objects still alive at exit * ``exec_prefix``: ``sys.exec_prefix`` * ``executable``: ``sys.executable`` * ``faulthandler``: if non-zero, call ``faulthandler.enable()`` * ``filesystem_encoding``: Filesystem encoding, ``sys.getfilesystemencoding()`` * ``filesystem_errors``: Filesystem encoding errors, ``sys.getfilesystemencodeerrors()`` * ``hash_seed``, ``use_hash_seed``: randomized hash function seed * ``home``: Python home * ``import_time``: if non-zero, profile import time * ``inspect``: enter interactive mode after executing a script or a command * ``install_signal_handlers``: install signal handlers? * ``interactive``: interactive mode * ``legacy_windows_stdio`` (Windows only): if non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. * ``malloc_stats``: if non-zero, dump memory allocation statistics at exit * ``module_search_path_env``: ``PYTHONPATH`` environment variale value * ``module_search_paths``, ``use_module_search_paths``: ``sys.path`` * ``optimization_level``: compilation optimization level * ``parser_debug``: if non-zero, turn on parser debugging output (for expert only, depending on compilation options). * ``prefix``: ``sys.prefix`` * ``program_name``: Program name * ``program``: ``argv[0]`` or an empty string * ``pycache_prefix``: ``.pyc`` cache prefix * ``quiet``: quiet mode (ex: don't display the copyright and version messages even in interactive mode) * ``run_command``: ``-c COMMAND`` argument * ``run_filename``: ``python3 SCRIPT`` argument * ``run_module``: ``python3 -m MODULE`` argument * ``show_alloc_count``: show allocation counts at exit * ``show_ref_count``: show total reference count at exit * ``site_import``: import the ``site`` module at startup? * ``skip_source_first_line``: skip the first line of the source * ``stdio_encoding``, ``stdio_errors``: encoding and encoding errors of ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` * ``tracemalloc``: if non-zero, call ``tracemalloc.start(value)`` * ``user_site_directory``: if non-zero, add user site directory to ``sys.path`` * ``verbose``: if non-zero, enable verbose mode * ``warnoptions``: options of the ``warnings`` module to build filters * ``write_bytecode``: if non-zero, write ``.pyc`` files * ``xoptions``: ``sys._xoptions`` There are also private fields which are for internal-usage only: * ``_check_hash_pycs_mode`` * ``_frozen`` * ``_init_main`` * ``_install_importlib`` New Py_UnixMain() function -------------------------- Python 3.7 provides a high-level ``Py_Main()`` function which requires to pass command line arguments as ``wchar_t*`` strings. It is non-trivial to use the correct encoding to decode bytes. Python has its own set of issues with C locale coercion and UTF-8 Mode. This PEP adds a new ``Py_UnixMain()`` function which takes command line arguments as bytes:: int Py_UnixMain(int argc, char **argv) Memory allocations and Py_DecodeLocale() ---------------------------------------- New pre-initialization and initialization APIs use constant ``PyPreConfig`` or ``PyConfig`` structures. If memory is allocated dynamically, the caller is responsible to release it. Using static strings is just fine. Python memory allocation functions like ``PyMem_RawMalloc()`` must not be used before Python pre-initialization. Using ``malloc()`` and ``free()`` is always safe. ``Py_DecodeLocale()`` must only be used after the pre-initialization. XXX Open Questions ================== This PEP is still a draft with open questions which should be answered: * Do we need to add an API for import ``inittab``? * What about the stable ABI? Should we add a version into ``PyPreConfig`` and ``PyConfig`` structures somehow? The Windows API is known for its ABI stability and it stores the structure size into the structure directly. Do the same? * The PEP 432 stores ``PYTHONCASEOK`` into the config. Do we need to add something for that into ``PyConfig``? How would it be exposed at the Python level for ``importlib``? Passed as an argument to ``importlib._bootstrap._setup()`` maybe? Backwards Compatibility ======================= This PEP only adds a new API: it leaves the existing API unchanged and has no impact on the backwards compatibility. Alternative: PEP 432 ==================== This PEP is inspired by Nick Coghlan's PEP 432 with a main difference: it only allows to configure Python before its initialization. The PEP 432 uses three initialization phases: Pre-Initialization, Initializing, Initialized. It is possible to configure Python between Initializing and Initialized phases using Python objects. This PEP only uses C types like ``int`` and ``wchar_t*`` (and ``PyWideCharList`` structure). All parameters must be configured at once before the Python initialization using the ``PyConfig`` structure. Annex: Python Configuration =========================== Priority and Rules ------------------ Priority of configuration parameters, highest to lowest: * ``PyConfig`` * ``PyPreConfig`` * Configuration files * Command line options * Environment variables * Global configuration variables Priority of warning options, highest to lowest: * ``PyConfig.warnoptions`` * ``PyConfig.dev_mode`` (add ``"default"``) * ``PYTHONWARNINGS`` environment variables * ``-W WARNOPTION`` command line argument * ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater than 1, or add ``"default::BytesWarning``) Rules on ``PyConfig`` and ``PyPreConfig`` parameters: * If ``isolated`` is non-zero, ``use_environment`` and ``user_site_directory`` are set to 0 * If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to 0 * If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``, ``faulthandler`` is set to 1, and ``"default"`` filter is added to ``warnoptions``. But ``PYTHONMALLOC`` has the priority over ``dev_mode`` to set the memory allocator. Configuration Files ------------------- Python configuration files: * ``pyvenv.cfg`` * ``python._pth`` (Windows only) * ``pybuilddir.txt`` (Unix only) Global Configuration Variables ------------------------------ Global configuration variables mapped to ``PyPreConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` ``Py_UTF8Mode`` ``utf8_mode`` ``Py_UTF8Mode`` ``utf8_mode`` ======================================== ================================ Global configuration variables mapped to ``PyConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_BytesWarningFlag`` ``bytes_warning`` ``Py_DebugFlag`` ``parser_debug`` ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_FrozenFlag`` ``_frozen`` ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` ``Py_IgnoreEnvironmentFlag`` ``use_environment`` ``Py_InspectFlag`` ``inspect`` ``Py_InteractiveFlag`` ``interactive`` ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` ``Py_NoSiteFlag`` ``site_import`` ``Py_NoUserSiteDirectory`` ``user_site_directory`` ``Py_OptimizeFlag`` ``optimization_level`` ``Py_QuietFlag`` ``quiet`` ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` ``Py_VerboseFlag`` ``verbose`` ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_BytesWarningFlag`` ``bytes_warning`` ``Py_DebugFlag`` ``parser_debug`` ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_FrozenFlag`` ``_frozen`` ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` ``Py_IgnoreEnvironmentFlag`` ``use_environment`` ``Py_InspectFlag`` ``inspect`` ``Py_InteractiveFlag`` ``interactive`` ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` ``Py_NoSiteFlag`` ``site_import`` ``Py_NoUserSiteDirectory`` ``user_site_directory`` ``Py_OptimizeFlag`` ``optimization_level`` ``Py_QuietFlag`` ``quiet`` ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` ``Py_VerboseFlag`` ``verbose`` ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` ======================================== ================================ ``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are only available on Windows. Command Line Arguments ---------------------- Usage:: python3 [options] python3 [options] -c COMMAND python3 [options] -m MODULE python3 [options] SCRIPT Command line options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-b`` ``bytes_warning++`` ``-B`` ``write_bytecode = 0`` ``-c COMMAND`` ``run_module = COMMAND`` ``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE`` ``-d`` ``parser_debug++`` ``-E`` ``use_environment = 0`` ``-i`` ``inspect++`` and ``interactive++`` ``-I`` ``isolated = 1`` ``-m MODULE`` ``run_module = MODULE`` ``-O`` ``optimization_level++`` ``-q`` ``quiet++`` ``-R`` ``use_hash_seed = 0`` ``-s`` ``user_site_directory = 0`` ``-S`` ``site_import`` ``-t`` ignored (kept for backwards compatibility) ``-u`` ``buffered_stdio = 0`` ``-v`` ``verbose++`` ``-W WARNING`` add ``WARNING`` to ``warnoptions`` ``-x`` ``skip_source_first_line = 1`` ``-X XOPTION`` add ``XOPTION`` to ``xoptions`` ================================ ================================ ``-h``, ``-?`` and ``-V`` options are handled outside ``PyConfig``. Environment Variables --------------------- Environment variables mapped to ``PyPreConfig`` fields: ================================= ============================================= Variable ``PyPreConfig`` field ================================= ============================================= ``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding`` ``PYTHONMALLOC`` ``allocator`` ``PYTHONUTF8`` ``utf8_mode`` ================================= ============================================= Environment variables mapped to ``PyConfig`` fields: ================================= ==================================== Variable ``PyConfig`` field ================================= ==================================== ``PYTHONDEBUG`` ``parser_debug`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONDONTWRITEBYTECODE`` ``write_bytecode`` ``PYTHONDUMPREFS`` ``dump_refs`` ``PYTHONEXECUTABLE`` ``program_name`` ``PYTHONFAULTHANDLER`` ``faulthandler`` ``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed`` ``PYTHONHOME`` ``home`` ``PYTHONINSPECT`` ``inspect`` ``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors`` ``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio`` ``PYTHONMALLOCSTATS`` ``malloc_stats`` ``PYTHONNOUSERSITE`` ``user_site_directory`` ``PYTHONOPTIMIZE`` ``optimization_level`` ``PYTHONPATH`` ``module_search_path_env`` ``PYTHONPROFILEIMPORTTIME`` ``import_time`` ``PYTHONPYCACHEPREFIX,`` ``pycache_prefix`` ``PYTHONTRACEMALLOC`` ``tracemalloc`` ``PYTHONUNBUFFERED`` ``buffered_stdio`` ``PYTHONVERBOSE`` ``verbose`` ``PYTHONWARNINGS`` ``warnoptions`` ================================= ==================================== ``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are specific to Windows. ``PYTHONDEVMODE`` is mapped to ``PyPreConfig.dev_mode`` and ``PyConfig.dev_mode``. Annex: Python 3.7 API ===================== Python 3.7 has 4 functions in its C API to initialize and finalize Python: * ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python * ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python Python can be configured using scattered global configuration variables (like ``Py_IgnoreEnvironmentFlag``) and using the following functions: * ``PyImport_AppendInittab()`` * ``PyImport_ExtendInittab()`` * ``PyMem_SetAllocator()`` * ``PyMem_SetupDebugHooks()`` * ``PyObject_SetArenaAllocator()`` * ``Py_SetPath()`` * ``Py_SetProgramName()`` * ``Py_SetPythonHome()`` * ``Py_SetStandardStreamEncoding()`` * ``PySys_AddWarnOption()`` * ``PySys_AddXOption()`` * ``PySys_ResetWarnOptions()`` There is also a high-level ``Py_Main()`` function. Copyright ========= This document has been placed in the public domain.

1 0

Rejecting PEP 502
by Brett Cannon 27 Mar '19

27 Mar '19

https://www.python.org/dev/peps/pep-0502/ Thanks to Mike G. Miller for the informational PEP, but since the PEP was written with an opinionated voice and the details of the PEP aren't critical as they should be captured by PEP 498 anyway, we didn't think it was worth trying to touch the PEP up to change its voice.

1 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev March 2019