Mailman 3 April 2019 - Python-Dev

Summary of Python tracker Issues
by Python tracker 12 Apr '19

12 Apr '19

ACTIVITY SUMMARY (2019-04-05 - 2019-04-12) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7063 ( +7) closed 41307 (+76) total 48370 (+83) Open issues with patches: 2815 Issues opened (60) ================== #25160: Stop using deprecated imp module; imp should now emit a real D https://bugs.python.org/issue25160 reopened by ncoghlan #35934: Add socket.create_server() utility function https://bugs.python.org/issue35934 reopened by vstinner #36235: distutils.sysconfig.customize_compiler() overrides CFLAGS var https://bugs.python.org/issue36235 reopened by vstinner #36537: except statement block incorrectly assumes end of scope(?). https://bugs.python.org/issue36537 opened by Saim Raza #36538: _thread.interrupt_main() no longer interrupts Lock.wait https://bugs.python.org/issue36538 opened by gregory.p.smith #36540: PEP 570: Python Positional-Only Parameters https://bugs.python.org/issue36540 opened by pablogsal #36541: Make lib2to3 grammar more closely match Python https://bugs.python.org/issue36541 opened by thatch #36542: Allow to overwrite the signature for Python functions https://bugs.python.org/issue36542 opened by serhiy.storchaka #36543: Remove old-deprecated ElementTree features (part 2) https://bugs.python.org/issue36543 opened by serhiy.storchaka #36545: Python 3.5 OOM during test_socket on make https://bugs.python.org/issue36545 opened by dekken #36546: Add quantiles() to the statistics module https://bugs.python.org/issue36546 opened by rhettinger #36548: Make the repr of re flags more readable https://bugs.python.org/issue36548 opened by serhiy.storchaka #36550: Avoid creating AttributeError exceptions in the debugger https://bugs.python.org/issue36550 opened by blueyed #36551: Optimize list comprehensions with preallocate size and protect https://bugs.python.org/issue36551 opened by anthony shaw #36552: Replace OverflowError with ValueError when calculating length https://bugs.python.org/issue36552 opened by anthony shaw #36553: inspect.is_decorator_call(frame) https://bugs.python.org/issue36553 opened by smarie #36556: Trashcan causing duplicated __del__ calls https://bugs.python.org/issue36556 opened by jdemeyer #36557: Python (Launcher)3.7.3 CMDLine install/uninstall https://bugs.python.org/issue36557 opened by mattcher_h #36558: Change time.mktime() return type from float to int? https://bugs.python.org/issue36558 opened by vstinner #36560: test_functools leaks randomly 1 memory block https://bugs.python.org/issue36560 opened by vstinner #36563: pdbrc is read twice if current directory is the home directory https://bugs.python.org/issue36563 opened by blueyed #36564: Infinite loop with short maximum line lengths in EmailPolicy https://bugs.python.org/issue36564 opened by p-ganssle #36567: DOC: manpage directive doesn't create hyperlink https://bugs.python.org/issue36567 opened by cheryl.sabella #36568: Typo in socket.CAN_RAW_FD_FRAMES library documentation https://bugs.python.org/issue36568 opened by Carl Cerecke #36569: @staticmethod seems to work with setUpClass, but docs say it s https://bugs.python.org/issue36569 opened by Peter de Blanc #36572: python-snappy install issue during Crossbar install with Pytho https://bugs.python.org/issue36572 opened by telatoa #36573: zipfile zipfile.BadZipFile: Bad CRC-32 for file '11_02_2019.pd https://bugs.python.org/issue36573 opened by Jozef Cernak #36576: Some test_ssl and test_asyncio tests fail with OpenSSL 1.1.1 o https://bugs.python.org/issue36576 opened by vstinner #36580: unittest.mock does not understand dataclasses https://bugs.python.org/issue36580 opened by John Parejko2 #36581: __dir__ on unittest.mock not safe for all spec types https://bugs.python.org/issue36581 opened by Dylan Semler #36582: collections.UserString encode method returns a string https://bugs.python.org/issue36582 opened by trey #36583: Do not swallow exceptions in the _ssl module https://bugs.python.org/issue36583 opened by serhiy.storchaka #36585: test_posix.py fails due to unsupported RWF_HIPRI https://bugs.python.org/issue36585 opened by jdemeyer #36586: multiprocessing.Queue.close doesn't behave as documented https://bugs.python.org/issue36586 opened by graingert #36587: race in logging code when fork() https://bugs.python.org/issue36587 opened by cagney #36589: Incorrect error handling in curses.update_lines_cols() https://bugs.python.org/issue36589 opened by ZackerySpytz #36590: Add Bluetooth RFCOMM Support for Windows https://bugs.python.org/issue36590 opened by topnotcher #36593: Trace function interferes with MagicMock isinstance? https://bugs.python.org/issue36593 opened by nedbat #36594: Undefined behavior due to incorrect usage of %p in format stri https://bugs.python.org/issue36594 opened by ZackerySpytz #36595: IDLE: Add search to Squeezed Output text viewer. https://bugs.python.org/issue36595 opened by Shane Smith #36596: tarfile module considers anything starting with 512 bytes of z https://bugs.python.org/issue36596 opened by cks #36600: re-enable test in nntplib https://bugs.python.org/issue36600 opened by Marcin Niemira #36601: signals can be caught by any thread https://bugs.python.org/issue36601 opened by jdemeyer #36602: Recursive directory list with pathlib.Path.iterdir https://bugs.python.org/issue36602 opened by Epic_Wink #36603: should pty.openpty() set pty/tty inheritable? https://bugs.python.org/issue36603 opened by cagney #36605: make tags should also parse Modules/_io/*.c and Modules/_io/*. https://bugs.python.org/issue36605 opened by vstinner #36606: calling super() causes __class__ to be not defined when sys.se https://bugs.python.org/issue36606 opened by xtreak #36607: asyncio.all_tasks() crashes if asyncio is used in multiple thr https://bugs.python.org/issue36607 opened by Nick Davies #36608: Replace bundled pip and setuptools with a downloader in the en https://bugs.python.org/issue36608 opened by webknjaz #36609: activate.ps1 in venv for Windows should encoded with BOM https://bugs.python.org/issue36609 opened by 정한솔 #36610: os.sendfile can return EINVAL on Solaris https://bugs.python.org/issue36610 opened by kulikjak #36611: Debug memory allocators: remove useless "serialno" field to re https://bugs.python.org/issue36611 opened by vstinner #36612: Unittest document is not clear on SetUpClass calls https://bugs.python.org/issue36612 opened by vrpolakatcisco #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 opened by gescheit #36614: Popen output on windows server 2019 https://bugs.python.org/issue36614 opened by weispinc #36615: why call _Py_set_inheritable(0) from os.open() when O_CLOEXEC? https://bugs.python.org/issue36615 opened by cagney #36616: Optimize thread state handling in function call code https://bugs.python.org/issue36616 opened by jdemeyer #36617: The rich comparison operators are second class citizens https://bugs.python.org/issue36617 opened by bup #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 opened by vstinner #36619: when is os.posix_spawn(setsid=True) safe? https://bugs.python.org/issue36619 opened by cagney Most recent 15 issues with no replies (15) ========================================== #36619: when is os.posix_spawn(setsid=True) safe? https://bugs.python.org/issue36619 #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 #36616: Optimize thread state handling in function call code https://bugs.python.org/issue36616 #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 #36607: asyncio.all_tasks() crashes if asyncio is used in multiple thr https://bugs.python.org/issue36607 #36606: calling super() causes __class__ to be not defined when sys.se https://bugs.python.org/issue36606 #36603: should pty.openpty() set pty/tty inheritable? https://bugs.python.org/issue36603 #36596: tarfile module considers anything starting with 512 bytes of z https://bugs.python.org/issue36596 #36594: Undefined behavior due to incorrect usage of %p in format stri https://bugs.python.org/issue36594 #36590: Add Bluetooth RFCOMM Support for Windows https://bugs.python.org/issue36590 #36589: Incorrect error handling in curses.update_lines_cols() https://bugs.python.org/issue36589 #36585: test_posix.py fails due to unsupported RWF_HIPRI https://bugs.python.org/issue36585 #36583: Do not swallow exceptions in the _ssl module https://bugs.python.org/issue36583 #36572: python-snappy install issue during Crossbar install with Pytho https://bugs.python.org/issue36572 #36567: DOC: manpage directive doesn't create hyperlink https://bugs.python.org/issue36567 Most recent 15 issues waiting for review (15) ============================================= #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 #36612: Unittest document is not clear on SetUpClass calls https://bugs.python.org/issue36612 #36611: Debug memory allocators: remove useless "serialno" field to re https://bugs.python.org/issue36611 #36610: os.sendfile can return EINVAL on Solaris https://bugs.python.org/issue36610 #36608: Replace bundled pip and setuptools with a downloader in the en https://bugs.python.org/issue36608 #36605: make tags should also parse Modules/_io/*.c and Modules/_io/*. https://bugs.python.org/issue36605 #36602: Recursive directory list with pathlib.Path.iterdir https://bugs.python.org/issue36602 #36601: signals can be caught by any thread https://bugs.python.org/issue36601 #36600: re-enable test in nntplib https://bugs.python.org/issue36600 #36594: Undefined behavior due to incorrect usage of %p in format stri https://bugs.python.org/issue36594 #36593: Trace function interferes with MagicMock isinstance? https://bugs.python.org/issue36593 #36590: Add Bluetooth RFCOMM Support for Windows https://bugs.python.org/issue36590 #36589: Incorrect error handling in curses.update_lines_cols() https://bugs.python.org/issue36589 #36585: test_posix.py fails due to unsupported RWF_HIPRI https://bugs.python.org/issue36585 Top 10 most discussed issues (10) ================================= #36551: Optimize list comprehensions with preallocate size and protect https://bugs.python.org/issue36551 24 msgs #30458: [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection https://bugs.python.org/issue30458 13 msgs #36560: test_functools leaks randomly 1 memory block https://bugs.python.org/issue36560 13 msgs #36537: except statement block incorrectly assumes end of scope(?). https://bugs.python.org/issue36537 9 msgs #33608: Add a cross-interpreter-safe mechanism to indicate that an obj https://bugs.python.org/issue33608 8 msgs #36533: logging regression with threading + fork are mixed in 3.7.1rc2 https://bugs.python.org/issue36533 8 msgs #36389: Add gc.enable_object_debugger(): detect corrupted Python objec https://bugs.python.org/issue36389 7 msgs #36611: Debug memory allocators: remove useless "serialno" field to re https://bugs.python.org/issue36611 7 msgs #18748: io.IOBase destructor silence I/O error on close() by default https://bugs.python.org/issue18748 6 msgs #36573: zipfile zipfile.BadZipFile: Bad CRC-32 for file '11_02_2019.pd https://bugs.python.org/issue36573 6 msgs Issues closed (73) ================== #2281: Enhanced cPython profiler with high-resolution timer https://bugs.python.org/issue2281 closed by inada.naoki #12910: urllib.quote quotes too many chars, e.g., '()' https://bugs.python.org/issue12910 closed by orsenthil #14017: Make it easy to create a new TextIOWrapper based on an existin https://bugs.python.org/issue14017 closed by ncoghlan #16712: collections.abc.Sequence should not provide __reversed__ https://bugs.python.org/issue16712 closed by inada.naoki #17396: modulefinder fails if module contains syntax error https://bugs.python.org/issue17396 closed by ncoghlan #17561: Add socket.bind_socket() convenience function https://bugs.python.org/issue17561 closed by giampaolo.rodola #19417: Bdb: add a unittest file (test.test_bdb) https://bugs.python.org/issue19417 closed by xdegaye #19476: Add a dedicated specification for module "reloading" to the la https://bugs.python.org/issue19476 closed by eric.snow #21318: sdist fails with symbolic links do non-existing files https://bugs.python.org/issue21318 closed by cheryl.sabella #25922: canceling a repair install breaks the ability to uninstall, re https://bugs.python.org/issue25922 closed by cheryl.sabella #27181: Add geometric mean to `statistics` module https://bugs.python.org/issue27181 closed by rhettinger #28351: statistics.geometric_mean can enter infinite loop for Decimal https://bugs.python.org/issue28351 closed by cheryl.sabella #28626: Tutorial: rearrange discussion of output formatting to encoura https://bugs.python.org/issue28626 closed by cheryl.sabella #29209: Remove old-deprecated ElementTree features https://bugs.python.org/issue29209 closed by serhiy.storchaka #29707: os.path.ismount() always returns false for mount --bind on sam https://bugs.python.org/issue29707 closed by christian.heimes #30134: BytesWarning is missing from the documents https://bugs.python.org/issue30134 closed by inada.naoki #30661: Support tarfile.PAX_FORMAT in shutil.make_archive https://bugs.python.org/issue30661 closed by ncoghlan #31155: Encode set, frozenset, bytearray, and iterators as json arrays https://bugs.python.org/issue31155 closed by inada.naoki #31512: Add non-elevated symlink support for dev mode Windows 10 https://bugs.python.org/issue31512 closed by steve.dower #32534: Speed-up list.insert: use memmove() https://bugs.python.org/issue32534 closed by inada.naoki #33228: Use Random.choices in tempfile https://bugs.python.org/issue33228 closed by inada.naoki #33456: site.py: by default, a virtual environment is *not* isolated f https://bugs.python.org/issue33456 closed by vinay.sajip #33461: json.loads(encoding=<not None>) does not emit deprecation warn https://bugs.python.org/issue33461 closed by inada.naoki #33722: Document builtins in mock_open https://bugs.python.org/issue33722 closed by jcrotts #34060: regrtest: log "CPU usage" on Windows https://bugs.python.org/issue34060 closed by vstinner #34139: Remove stale unix datagram socket before binding https://bugs.python.org/issue34139 closed by asvetlov #34144: venv activate.bat reset codepage fails on windows 10 https://bugs.python.org/issue34144 closed by cheryl.sabella #34160: ElementTree not preserving attribute order https://bugs.python.org/issue34160 closed by scoder #34805: Explicitly specify `MyClass.__subclasses__()` returns classes https://bugs.python.org/issue34805 closed by cheryl.sabella #35376: modulefinder skips nested modules with same name as top-level https://bugs.python.org/issue35376 closed by ncoghlan #35416: Fix potential resource warnings in distutils https://bugs.python.org/issue35416 closed by inada.naoki #35488: pathlib Path.match does not behave as described https://bugs.python.org/issue35488 closed by anthony shaw #35848: readinto is not a method on io.TextIOBase https://bugs.python.org/issue35848 closed by benjamin.peterson #35906: [CVE-2019-9947] Header Injection in urllib https://bugs.python.org/issue35906 closed by gregory.p.smith #35936: Give modulefinder some much-needed updates. https://bugs.python.org/issue35936 closed by ncoghlan #36050: Why does http.client.HTTPResponse._safe_read use MAXAMOUNT https://bugs.python.org/issue36050 closed by inada.naoki #36378: Add support to load from paths to json.load https://bugs.python.org/issue36378 closed by inada.naoki #36416: bytes.rpartition bug in online documentation https://bugs.python.org/issue36416 closed by inada.naoki #36495: Out-of-bounds array reads in Python/ast.c https://bugs.python.org/issue36495 closed by levkivskyi #36501: Remove POSIX.1e ACLs in tests that rely on default permissions https://bugs.python.org/issue36501 closed by Ivan.Pozdeev #36503: remove references to aix3 and aix4 in \*.py https://bugs.python.org/issue36503 closed by inada.naoki #36504: Signed integer overflow in _ctypes.c's PyCArrayType_new() https://bugs.python.org/issue36504 closed by serhiy.storchaka #36506: [security] CVE-2019-10268: An arbitrary execution vulnerabilit https://bugs.python.org/issue36506 closed by serhiy.storchaka #36513: Add support for building arm32 nuget package https://bugs.python.org/issue36513 closed by steve.dower #36527: unused parameter warnings in Include/object.h (affecting build https://bugs.python.org/issue36527 closed by inada.naoki #36532: Example of logging.formatter with new str.format style https://bugs.python.org/issue36532 closed by spaceman_spiff #36535: Windows build failure when use the code from the GitHub master https://bugs.python.org/issue36535 closed by Manjusaka #36539: Distutils VC 6.0 Errors When Using mingw-w64 GCC https://bugs.python.org/issue36539 closed by danyeaw #36544: cannot import hashlib when openssl is missing https://bugs.python.org/issue36544 closed by xdegaye #36547: bedevere is not working https://bugs.python.org/issue36547 closed by xtreak #36549: str.capitalize should titlecase the first character not upperc https://bugs.python.org/issue36549 closed by steve.dower #36554: unittest.TestCase: "subTest" cannot be used together with "deb https://bugs.python.org/issue36554 closed by dmaurer #36555: PEP484 @overload vs. str/bytes https://bugs.python.org/issue36555 closed by levkivskyi #36559: "import random" should import hashlib on demand (nor load Open https://bugs.python.org/issue36559 closed by rhettinger #36561: Python argparse doesn't work in the presence of my custom modu https://bugs.python.org/issue36561 closed by xtreak #36562: Can't call a method from a module built in Python C API https://bugs.python.org/issue36562 closed by jjppof #36565: Reference hunting (python3 -m test -R 3:3) doesn't work if the https://bugs.python.org/issue36565 closed by vstinner #36566: Support password masking in getpass.getpass() https://bugs.python.org/issue36566 closed by cheryl.sabella #36570: ftplib timeouts for misconfigured server https://bugs.python.org/issue36570 closed by giampaolo.rodola #36571: Lib/smtplib.py have some pep8 issues https://bugs.python.org/issue36571 closed by Marcin Niemira #36574: Error with self in python https://bugs.python.org/issue36574 closed by steven.daprano #36575: Use _PyTime_GetPerfCounter() in lsprof https://bugs.python.org/issue36575 closed by inada.naoki #36577: setup doesn't report missing _ssl and _hashlib https://bugs.python.org/issue36577 closed by christian.heimes #36578: multiprocessing pool + subprocess ValueError: empty range for https://bugs.python.org/issue36578 closed by SilentGhost #36579: test_venv: test_with_pip() hangs on PPC64 AIX 3.x https://bugs.python.org/issue36579 closed by vstinner #36584: cython nametuple TypeError https://bugs.python.org/issue36584 closed by serhiy.storchaka #36588: change sys.platform() to just "aix" for AIX https://bugs.python.org/issue36588 closed by vstinner #36591: Should be a typing.UserNamedTuple https://bugs.python.org/issue36591 closed by levkivskyi #36592: is behave different for integers in 3.6 and 3.7 https://bugs.python.org/issue36592 closed by eric.smith #36597: Travis CI: doctest failure https://bugs.python.org/issue36597 closed by inada.naoki #36598: mock side_effect should be checked for iterable not callable https://bugs.python.org/issue36598 closed by xtreak #36599: doctest document says dict order is unstable https://bugs.python.org/issue36599 closed by inada.naoki #36604: Add recipe to itertools https://bugs.python.org/issue36604 closed by rhettinger

1 0

PEP-582 and multiple Python installations
by Calvin Spealman 12 Apr '19

12 Apr '19

(I originally posted this to python-ideas, where I was told none of this PEP's authors subscribe so probably no one will see it there, so I'm posting it here to raise the issue where it can get seen and hopefully discussed) While the PEP does show the version number as part of the path to the actual packages, implying support for multiple versions, this doesn't seem to be spelled out in the actual text. Presumably __pypackages__/3.8/ might sit beside __pypackages__/3.9/, etc. to keep future versions capable of installing packages for each version, the way virtualenv today is bound to one version of Python. I'd like to raise a potential edge case that might be a problem, and likely an increasingly common one: users with multiple installations of the *same* version of Python. This is actually a common setup for Windows users who use WSL, Microsoft's Linux-on-Windows solution, as you could have both the Windows and Linux builds of a given Python version installed on the same machine. The currently implied support for multiple versions would not be able to separate these and could create problems if users pip install a Windows binary package through Powershell and then try to run a script in Bash from the same directory, causing the Linux version of Python to try to use Windows python packages. I'm not actually sure what the solution here is. Mostly I wanted to raise the concern, because I'm very keen on WSL being a great entry path for new developers and I want to make that a better experience, not a more confusing one. Maybe that version number could include some other unique identify, maybe based on Python's own executable. A hash maybe? I don't know if anything like that already exists to uniquely identify a Python build or installation. -- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma(a)redhat.com M: +1.336.210.5107 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

6 5

(no subject)
by Robert Okadar 10 Apr '19

10 Apr '19

Hi community, I have developed a tkinter GUI component, Python v3.7. It runs very well in Linux but seeing a huge performance impact in Windows 10. While in Linux an almost real-time performance is achieved, in Windows it is slow to an unusable level. The code is somewhat stripped down from the original, but the performance difference is the same anyway. The columns can be resized by clicking on the column border and dragging it. Resizing works only for the top row (but it resizes the entire column). In this demo, all bindings are avoided to exclude influence on the component performance and thus not included. If you resize the window (i.e., if you maximize it), you must call the function table.fit() from IDLE shell. Does anyone know where is this huge difference in performance coming from? Can anything be done about it? All the best, -- Robert Okadar IT Consultant Schedule an *online meeting <https://calendly.com/aranea-network/60min>* with me! Visit *aranea-mreze.hr* <http://aranea-mreze.hr> or call * +385 91 300 8887*

4 4

Need help to fix HTTP Header Injection vulnerability
by Victor Stinner 10 Apr '19

10 Apr '19

Hi, In May 2017, user "Orange" found a vulnerability in the urllib fix for CVE-2016-5699 (HTTP Header Injection vulnerability): https://bugs.python.org/issue30458 It allows to inject arbitrary HTTP headers. Copy of their message: """ Hi, the patch in CVE-2016-5699 can be broke by an addition space. http://www.cvedetails.com/cve/CVE-2016-5699/ https://hg.python.org/cpython/rev/bf3e1c9b80e9 https://hg.python.org/cpython/rev/1c45047c5102 import urllib, urllib2 urllib.urlopen('http://127.0.0.1\r\n\x20hihi\r\n :11211') urllib2.urlopen('http://127.0.0.1\r\n\x20hihi\r\n :11211') """ Last month, the same bug has been rediscovered by user "ragdoll.guo": https://bugs.python.org/issue36276 Almost one year after the bug has been reported, no one came with a solution. I'm not comfortable with having known security issues impacting HTTP. Can someone please have a look at the issue and try to write a change to fix the issue? According to Karthikeyan Singaravelan, the Go language fixed a similar issue in Go 1.12: throw an error if the URL contains any control character. If we decide that the issue is not a security issue, we should document the behavior properly and close the issue. See also this related issue: "urlopen URL with unescaped space" https://bugs.python.org/issue14826 Victor -- Night gathers, and now my watch begins. It shall not end until my death.

5 8

New Python Initialization API
by Victor Stinner 09 Apr '19

09 Apr '19

Hi, I would like to add a new C API to initialize Python. I would like your opinion on the whole API before making it public. The code is already implemented. Doc of the new API: https://pythondev.readthedocs.io/init_config.html To make the API public, _PyWstrList, _PyInitError, _PyPreConfig, _PyCoreConfig and related functions should be made public. By the way, I would suggest to rename "_PyCoreConfig" to just "PyConfig" :-) I don't think that "core init" vs "main init" is really relevant: more about that below. Let's start with two examples using the new API. Example of simple initialization to enable isolated mode: _PyCoreConfig config = _PyCoreConfig_INIT; config.isolated = 1; _PyInitError err = _Py_InitializeFromConfig(&config); if (_Py_INIT_FAILED(err)) { _Py_ExitInitError(err); } /* ... use Python API here ... */ Py_Finalize(); Example using the pre-initialization to enable the UTF-8 Mode (and use the "legacy" Py_Initialize() function): _PyPreConfig preconfig = _PyPreConfig_INIT; preconfig.utf8_mode = 1; _PyInitError err = _Py_PreInitialize(&preconfig); if (_Py_INIT_FAILED(err)) { _Py_ExitInitError(err); } /* at this point, Python will only speak UTF-8 */ Py_Initialize(); /* ... use Python API here ... */ Py_Finalize(); Since November 2017, I'm refactoring the Python Initialization code to cleanup the code and prepare a new ("better") API to configure Python Initialization. I just fixed the last issues that Nick Coghlan asked me to fix (add a pre-initialization step: done, fix mojibake: done). My work is inspired by Nick Coghlan's PEP 432, but it is not implementing it directly. I had other motivations than Nick even if we are somehow going towards the same direction. Nick wants to get a half-initialized Python ("core init"), configure Python using the Python API and Python objects, and then finish the implementation ("main init"). I chose a different approach: put *everything* into a single C structure (_PyCoreConfig) using C types. Using the structure, you should be able to do what Nick wanted to do, but with C rather than Python. Nick: please tell me if I'm wrong :-) This work is also connected to Eric Snow's work on sub-interpreters (PEP 554) and moving global variables into structures. For example, I'm using his _PyRuntime structure to store a new "preconfig" state (pre-initialization configuration, more about that below). In November 2017, when I started to work on the Python Initialization (bpo-32030), I identified the following problems: * Many parts of the code were interdependent * Code executed early in Py_Main() used the Python API before the Python API was fully initialized. Like code parsing -W command line option which used PyUnicode_FromWideChar() and PyList_Append(). * Error handling used Py_FatalError() which didn't let the caller to decide how to handle the error. Moreover, exit() was used to exit Python, whereas libpython shouldn't do that: a library should not exit the whole process! (imagine when Python is embedded inside an application) One year and a half later, I implemented the following solutions: * Py_Main() and Py_Initialize() code has been reorganized to respect priorities between global configuration variables (ex: Py_IgnoreEnvironmentFlag), environment variables (ex: PYTHONPATH), command line arguments (ex: -X utf8), configuration files (ex: pyenv.cfg), and the new _PyPreConfig and _PyCoreConfig structures which store the whole configuration. * Python Initialization no longer uses the Python API but only C types like wchar_t* strings, a new _PyWstrList structure and PyMem_RawMalloc() memory allocator (PyMem_Malloc() is no longer used during init). * The code has been modified to use a new _PyInitError structure. The caller of the top function gets control to cleanup everything before handling the error (display a fatal error message or simply exit Python). The new _PyCoreConfig structure has the top-priority and provides a single structure for all configuration parameters. It becomes possible to override the code computing the "path configuration" like sys.path to fully control where Python looks to import modules. It becomes possible to use an empty list of paths to only allow builtin modules. A new "pre-initialization" steps is responsible to configure the bare minimum before the Python initialization: memory allocators and encodings (LC_CTYPE locale and the UTF-8 mode). The LC_CTYPE is no longer coerced and the UTF-8 Mode is no longer enabled automatically depending on the user configuration to prevent mojibake. Previously, calling Py_DecodeLocale() to get a Unicode wchar_t* string from a bytes wchar* string created mojibake when called before Py_Initialize() if the LC_CTYPE locale was coerced and/or if the UTF-8 Mode was enabled. The pre-initialization step ensures that the encodings and memory allocators are well defined *before* Py_Initialize() is called. Since the new API is currently private, I didn't document it in Python. Moreover, the code changed a lot last year :-) But it should now be way more stable. I started to document it in a separated webpage: https://pythondev.readthedocs.io/init_config.html The plan is to put it in the Python documentation once it becomes public. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

8 27

Deprecating "instance method" class
by Jeroen Demeyer 08 Apr '19

08 Apr '19

During my investigations related to low-level function/method classes, I came across the "instance method" class. There is a C API for it: https://docs.python.org/3.7/c-api/method.html However, it's not used/exposed anywhere in CPython, except as _testcapi.instancemethod (for testing its functionality) This class was discussed at https://mail.python.org/pipermail/python-3000/2007-December/011456.html and implemented in https://bugs.python.org/issue1587 Reading that old thread, there are use cases presented related to classic classes, wrapping Kogut (http://kokogut.sourceforge.net/kogut.html) objects and Pyrex. But classic classes no longer exist and the latter two use cases aren't actually needed if you read the thread to the end. So there are no surviving use cases from that thread. Does anybody know actual use cases or any code in the wild using it? To me, the fact that it's only exposed in the C API is a good sign that it's not really useful. So, should we deprecate the instance method class? Jeroen.

7 21

bedevere pipelines hang on github
by Xavier de Gaye 08 Apr '19

08 Apr '19

bedevere/issue-number and bedevere/news are not triggered for some reason at https://github.com/python/cpython/pull/12708 and hang forever with "Expected — Waiting for status to be reported ". Xavier

3 3

Summary of Python tracker Issues
by Python tracker 05 Apr '19

05 Apr '19

ACTIVITY SUMMARY (2019-03-29 - 2019-04-05) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7056 ( +8) closed 41231 (+55) total 48287 (+63) Open issues with patches: 2816 Issues opened (44) ================== #36260: Cpython/Lib vulnerability found and request a patch submission https://bugs.python.org/issue36260 reopened by krnick #36474: RecursionError resets trace function set via sys.settrace https://bugs.python.org/issue36474 opened by blueyed #36475: PyEval_AcquireLock() and PyEval_AcquireThread() do not handle https://bugs.python.org/issue36475 opened by eric.snow #36476: Runtime finalization assumes all other threads have exited. https://bugs.python.org/issue36476 opened by eric.snow #36478: backport of pickle fixes to Python 3.5.7 uses C99 for loops https://bugs.python.org/issue36478 opened by Anthony Sottile #36479: Exit threads when interpreter is finalizing rather than runtim https://bugs.python.org/issue36479 opened by eric.snow #36481: telnetlib process_rawq() callback https://bugs.python.org/issue36481 opened by Quanthir #36484: Can't reorder TLS 1.3 ciphersuites https://bugs.python.org/issue36484 opened by Eman Alashwali #36485: Establish a uniform way to clear all caches in a given module https://bugs.python.org/issue36485 opened by serhiy.storchaka #36486: Bugs and inconsistencies in unicodedata https://bugs.python.org/issue36486 opened by dscorbett #36487: Make C-API docs clear about what the "main" interpreter is https://bugs.python.org/issue36487 opened by nanjekyejoannah #36488: os.sendfile() on BSD and macOS does not return bytes sent on E https://bugs.python.org/issue36488 opened by giampaolo.rodola #36489: add filename_extension_map and/or content-types_map dict(s) to https://bugs.python.org/issue36489 opened by Daniel Black #36490: Modernize function signature format in Archiving section of sh https://bugs.python.org/issue36490 opened by CAM-Gerlach #36494: bdb.Bdb.set_trace: should set f_trace_lines = True https://bugs.python.org/issue36494 opened by blueyed #36495: Out-of-bounds array reads in Python/ast.c https://bugs.python.org/issue36495 opened by blarsen #36497: Undocumented behavior in csv.Sniffer (preferred delimiters) https://bugs.python.org/issue36497 opened by thomas #36500: Add "regen-*" equivalent projects for Windows builds https://bugs.python.org/issue36500 opened by anthony shaw #36501: Remove POSIX.1e ACLs in tests that rely on default permissions https://bugs.python.org/issue36501 opened by Ivan.Pozdeev #36502: The behavior of str.isspace() for U+00A0 and U+202F is differe https://bugs.python.org/issue36502 opened by Jun #36503: remove references to aix3 and aix4 in \*.py https://bugs.python.org/issue36503 opened by Michael.Felt #36504: Signed integer overflow in _ctypes.c's PyCArrayType_new() https://bugs.python.org/issue36504 opened by ZackerySpytz #36506: [security] CVE-2019-10268: An arbitrary execution vulnerabilit https://bugs.python.org/issue36506 opened by bigbigliang #36508: python-config --ldflags must not contain LINKFORSHARED ("-Xlin https://bugs.python.org/issue36508 opened by vstinner #36509: Add iot layout for windows iot containers https://bugs.python.org/issue36509 opened by Paul Monson #36511: Add Windows ARM32 buildbot https://bugs.python.org/issue36511 opened by Paul Monson #36512: future_factory argument for Thread/ProcessPoolExecutor https://bugs.python.org/issue36512 opened by stefanhoelzl #36513: Add support for building arm32 nuget package https://bugs.python.org/issue36513 opened by Paul Monson #36515: unaligned memory access in the _sha3 extension https://bugs.python.org/issue36515 opened by doko #36516: Python Launcher can not recognize pyw file as Python GUI Scrip https://bugs.python.org/issue36516 opened by gjj2828 #36517: typing.NamedTuple does not support mixins https://bugs.python.org/issue36517 opened by rectalogic #36518: Avoid conflicts when pass arbitrary keyword arguments to Pytho https://bugs.python.org/issue36518 opened by serhiy.storchaka #36519: Blake2b/s implementations have minor GIL issues https://bugs.python.org/issue36519 opened by gwk #36520: Email header folded incorrectly https://bugs.python.org/issue36520 opened by Jonathan Horn #36521: Consider removing docstrings from co_consts in code objects https://bugs.python.org/issue36521 opened by rhettinger #36523: missing docs for IOBase writelines https://bugs.python.org/issue36523 opened by Marcin Niemira #36527: unused parameter warnings in Include/object.h (affecting build https://bugs.python.org/issue36527 opened by AMDmi3 #36528: Remove duplicate tests in Lib/tests/re_tests.py https://bugs.python.org/issue36528 opened by xtreak #36529: Python from WindowsStore: can't install package using "-m pip" https://bugs.python.org/issue36529 opened by Ilya Kazakevich #36531: PyType_FromSpec wrong behavior with multiple Py_tp_members https://bugs.python.org/issue36531 opened by eelizondo #36532: Example of logging.formatter with new str.format style https://bugs.python.org/issue36532 opened by spaceman_spiff #36533: logging regression with threading + fork are mixed in 3.7.1rc2 https://bugs.python.org/issue36533 opened by gregory.p.smith #36534: tarfile: handling Windows (path) illegal characters in archive https://bugs.python.org/issue36534 opened by CristiFati #36535: Windows build failure when use the code from the GitHub master https://bugs.python.org/issue36535 opened by Manjusaka Most recent 15 issues with no replies (15) ========================================== #36535: Windows build failure when use the code from the GitHub master https://bugs.python.org/issue36535 #36531: PyType_FromSpec wrong behavior with multiple Py_tp_members https://bugs.python.org/issue36531 #36529: Python from WindowsStore: can't install package using "-m pip" https://bugs.python.org/issue36529 #36528: Remove duplicate tests in Lib/tests/re_tests.py https://bugs.python.org/issue36528 #36527: unused parameter warnings in Include/object.h (affecting build https://bugs.python.org/issue36527 #36523: missing docs for IOBase writelines https://bugs.python.org/issue36523 #36520: Email header folded incorrectly https://bugs.python.org/issue36520 #36517: typing.NamedTuple does not support mixins https://bugs.python.org/issue36517 #36516: Python Launcher can not recognize pyw file as Python GUI Scrip https://bugs.python.org/issue36516 #36515: unaligned memory access in the _sha3 extension https://bugs.python.org/issue36515 #36513: Add support for building arm32 nuget package https://bugs.python.org/issue36513 #36512: future_factory argument for Thread/ProcessPoolExecutor https://bugs.python.org/issue36512 #36511: Add Windows ARM32 buildbot https://bugs.python.org/issue36511 #36509: Add iot layout for windows iot containers https://bugs.python.org/issue36509 #36503: remove references to aix3 and aix4 in \*.py https://bugs.python.org/issue36503 Most recent 15 issues waiting for review (15) ============================================= #36532: Example of logging.formatter with new str.format style https://bugs.python.org/issue36532 #36531: PyType_FromSpec wrong behavior with multiple Py_tp_members https://bugs.python.org/issue36531 #36528: Remove duplicate tests in Lib/tests/re_tests.py https://bugs.python.org/issue36528 #36527: unused parameter warnings in Include/object.h (affecting build https://bugs.python.org/issue36527 #36523: missing docs for IOBase writelines https://bugs.python.org/issue36523 #36518: Avoid conflicts when pass arbitrary keyword arguments to Pytho https://bugs.python.org/issue36518 #36516: Python Launcher can not recognize pyw file as Python GUI Scrip https://bugs.python.org/issue36516 #36515: unaligned memory access in the _sha3 extension https://bugs.python.org/issue36515 #36513: Add support for building arm32 nuget package https://bugs.python.org/issue36513 #36512: future_factory argument for Thread/ProcessPoolExecutor https://bugs.python.org/issue36512 #36509: Add iot layout for windows iot containers https://bugs.python.org/issue36509 #36508: python-config --ldflags must not contain LINKFORSHARED ("-Xlin https://bugs.python.org/issue36508 #36504: Signed integer overflow in _ctypes.c's PyCArrayType_new() https://bugs.python.org/issue36504 #36503: remove references to aix3 and aix4 in \*.py https://bugs.python.org/issue36503 #36501: Remove POSIX.1e ACLs in tests that rely on default permissions https://bugs.python.org/issue36501 Top 10 most discussed issues (10) ================================= #36485: Establish a uniform way to clear all caches in a given module https://bugs.python.org/issue36485 13 msgs #36466: Adding a way to strip annotations from compiled bytecode https://bugs.python.org/issue36466 12 msgs #36469: Stuck during interpreter exit, attempting to take the GIL https://bugs.python.org/issue36469 10 msgs #36506: [security] CVE-2019-10268: An arbitrary execution vulnerabilit https://bugs.python.org/issue36506 8 msgs #6721: Locks in the standard library should be sanitized on fork https://bugs.python.org/issue6721 7 msgs #36533: logging regression with threading + fork are mixed in 3.7.1rc2 https://bugs.python.org/issue36533 6 msgs #35866: concurrent.futures deadlock https://bugs.python.org/issue35866 5 msgs #36384: ipaddress Should not reject IPv4 addresses with leading zeroes https://bugs.python.org/issue36384 5 msgs #30661: Support tarfile.PAX_FORMAT in shutil.make_archive https://bugs.python.org/issue30661 4 msgs #35224: PEP 572: Assignment Expressions https://bugs.python.org/issue35224 4 msgs Issues closed (53) ================== #17110: sys.argv docs should explaining how to handle encoding issues https://bugs.python.org/issue17110 closed by inada.naoki #20844: SyntaxError: encoding problem: iso-8859-1 on Windows https://bugs.python.org/issue20844 closed by inada.naoki #21269: Provide args and kwargs attributes on mock call objects https://bugs.python.org/issue21269 closed by xtreak #22831: Use "with" to avoid possible fd leaks https://bugs.python.org/issue22831 closed by serhiy.storchaka #24214: UTF-8 incremental decoder doesn't support surrogatepass correc https://bugs.python.org/issue24214 closed by serhiy.storchaka #25451: tkinter: PhotoImage transparency methods https://bugs.python.org/issue25451 closed by serhiy.storchaka #29202: Improve dict iteration https://bugs.python.org/issue29202 closed by inada.naoki #31182: Suggested Enhancements to zipfile & tarfile command line inter https://bugs.python.org/issue31182 closed by brett.cannon #32413: Document that locals() may return globals() https://bugs.python.org/issue32413 closed by brett.cannon #32531: gdb.execute can not put string value. https://bugs.python.org/issue32531 closed by berker.peksag #32538: Multiprocessing Manager on 3D list - no change of the list pos https://bugs.python.org/issue32538 closed by berker.peksag #33261: inspect.isgeneratorfunction fails on hand-created methods https://bugs.python.org/issue33261 closed by petr.viktorin #34430: Symmetrical chaining futures in asyncio.future.wrap_future https://bugs.python.org/issue34430 closed by huji #35272: sqlite3 get the connected database url https://bugs.python.org/issue35272 closed by berker.peksag #35403: support application/wasm in mimetypes and http.server https://bugs.python.org/issue35403 closed by martin.panter #35838: ConfigParser: document optionxform must be idempotent https://bugs.python.org/issue35838 closed by inada.naoki #36010: Please provide a .zip Windows release of Python that is not cr https://bugs.python.org/issue36010 closed by steve.dower #36026: Different error message when sys.settrace is used https://bugs.python.org/issue36026 closed by inada.naoki #36085: Enable better DLL resolution https://bugs.python.org/issue36085 closed by steve.dower #36157: Document PyInterpreterState_Main(). https://bugs.python.org/issue36157 closed by eric.snow #36293: Nonblocking read sys.stdin raises error https://bugs.python.org/issue36293 closed by martin.panter #36322: Argument typo in dbm.ndbm.open https://bugs.python.org/issue36322 closed by brett.cannon #36377: Python 'datastructures.html' docs page needs improvement becau https://bugs.python.org/issue36377 closed by rhettinger #36404: Document PendingDeprecationWarning is not so useful. https://bugs.python.org/issue36404 closed by inada.naoki #36426: exec() issue when used inside function https://bugs.python.org/issue36426 closed by ncoghlan #36434: Zipfile breaks if signalled during write() https://bugs.python.org/issue36434 closed by serhiy.storchaka #36440: more helpful diagnostics for parser module https://bugs.python.org/issue36440 closed by pablogsal #36442: Different ValueError for the same operation in List and Tuple https://bugs.python.org/issue36442 closed by serhiy.storchaka #36445: bus error in test_gil test on armhf running with 64bit kernel https://bugs.python.org/issue36445 closed by doko #36448: Message "You will need to rebuild pythoncore to see the change https://bugs.python.org/issue36448 closed by steve.dower #36468: Treeview: wrong color change https://bugs.python.org/issue36468 closed by ned.deily #36472: Some old PR with CLA not signed https://bugs.python.org/issue36472 closed by brett.cannon #36473: dictkeysobject: Add maximum iteration check for .values() and https://bugs.python.org/issue36473 closed by inada.naoki #36477: Subinterpreters are not finalized during runtime finalization. https://bugs.python.org/issue36477 closed by eric.snow #36480: .strip() unexpected output on Windows https://bugs.python.org/issue36480 closed by eric.smith #36482: let struct's internal cache use FIFO policy https://bugs.python.org/issue36482 closed by rhettinger #36483: Missing line in documentation example https://bugs.python.org/issue36483 closed by martin.panter #36491: sum function's start optional parameter documented in help but https://bugs.python.org/issue36491 closed by rhettinger #36492: Deprecate passing some conflicting arguments by keyword https://bugs.python.org/issue36492 closed by serhiy.storchaka #36493: Add math.midpoint(a,b) function https://bugs.python.org/issue36493 closed by scoder #36496: Local variables can be used uninitialized in _PyPreConfig_Read https://bugs.python.org/issue36496 closed by vstinner #36498: combining dict comprehensing and lists lead to IndexError https://bugs.python.org/issue36498 closed by SilentGhost #36499: unpickling of a datetime object in 3.5 fails when pickled with https://bugs.python.org/issue36499 closed by josh.r #36505: PYTHON-CAN with vector https://bugs.python.org/issue36505 closed by SilentGhost #36507: frozenset type breaks ZFC https://bugs.python.org/issue36507 closed by rhettinger #36510: Regular Expression Dot-Star patter matching - re- text skippi https://bugs.python.org/issue36510 closed by SilentGhost #36514: -m switch revisited https://bugs.python.org/issue36514 closed by ronaldoussoren #36522: http/client.py does not print duplicate header values in debug https://bugs.python.org/issue36522 closed by serhiy.storchaka #36524: identity operator https://bugs.python.org/issue36524 closed by SilentGhost #36525: Deprecate instancemethod https://bugs.python.org/issue36525 closed by christian.heimes #36526: python crash when loading some .pyc file https://bugs.python.org/issue36526 closed by serhiy.storchaka #36530: Document codecs decode_encode() and encode_decode() APIs https://bugs.python.org/issue36530 closed by gregory.p.smith #36536: is there a python implementation of the cpython commandline in https://bugs.python.org/issue36536 closed by larry

1 0

how to rerun the job “Azure Pipelines PR”?
by Xin, Peixing 02 Apr '19

02 Apr '19

Hi, Experts: Anyone can tell how to rerun the job “Azure Pipelines PR” for my PR? Sometimes my PR failed but this is caused by externals. The next day this external issue was fixed then I might want to rerun this specific job on my PR to get the new result. How can I reach this? [cid:image001.png@01D4E97B.F67B1E20] Thanks, Peixing

3 2

Strange umask(?)/st_mode issue
by Steve Dower 02 Apr '19

02 Apr '19

Hi I'm trying to track down the cause of the failed Linux tests on this build (it's also been happening on random PR builds for the last day or two, but this is the first I've seen it happen on already merged code): https://dev.azure.com/Python/cpython/_build/results?buildId=40189 One of the failed tests is this one from test_tarfile.py (and I think this is representative of the others): def test_file_mode(self): # Test for issue #8464: Create files with correct # permissions. if os.path.exists(tmpname): support.unlink(tmpname) original_umask = os.umask(0o022) try: tar = tarfile.open(tmpname, self.mode) tar.close() mode = os.stat(tmpname).st_mode & 0o777 self.assertEqual(mode, 0o644, "wrong file permissions") finally: os.umask(original_umask) It's failing because the mode of the new TAR file is 0o666 rather than 0o644. I'd guess that this means the umask() call is not having effect? But honestly this is beyond me - I don't have any idea what would cause Python to ignore that, and I don't see anything in the commit history that could be at fault. I'm hoping someone can tell me that it's definitely a platform configuration issue or it's definitely interference between two of our tests (which run in random order, which would explain the intermittent nature of this, but I haven't spotted any tests that seem likely either). To see the other failed PR builds, the full list is at https://dev.azure.com/Python/cpython/_build?definitionId=9 and most of the ones from today have failed because of whatever is causing it. Any help? Thanks, Steve

6 14