Mailman 3 December 2019 - Python-Dev

Adding a toml module to the standard lib?
by Bastian Venthur Dec. 16, 2019

Dec. 16, 2019

Hi, I'd like to discuss the idea to add a module to parse TOML [toml-lang] to Python's standard library. PEP-0518 -- Specifying Minimum Build System Requirements for Python Projects [pep] suggests to store build system dependencies in `pyproject.toml`, yet Python itself does not support this format. Various packaging related projects like pip and pipenv already support PEP-0518 and vendored one of the existing TOML libraries in order to read `pyproject.toml` files. Besides that, TOML seems a better alternative to .cfg/.ini, .json -- both of which are already supported by Python's standard lib and parsing/dumping TOML properly is tricky enough to solve it properly [requirements]. There are a couple of TOML implementations out there [toml, pytoml, tomlkit] and one would have to find out which one to prefer and migrate into the stdlib. If the result of this discussion is leaning towards adding TOML, I'd volunteer to do it. This includes: coordinating with the maintainer of the chosen library, writing the PEP (hopefully with some help) and maintain the module for at least two years. Cheers, Bastian [toml-lang]: https://github.com/toml-lang/toml [pep]: https://www.python.org/dev/peps/pep-0518 [pip]: https://github.com/pypa/pip [pipenv]: https://github.com/pypa/pipenv [toml]: https://github.com/uiri/toml [pytoml]: https://github.com/avakar/pytoml [tomlkit]: https://github.com/sdispater/tomlkit [requirements]: https://devguide.python.org/stdlibchanges/#requirements -- Dr. Bastian Venthur http://venthur.de Debian Developer venthur at debian org

8 9

Handling cross-distro variations when running autoreconf?
by Nick Coghlan Dec. 16, 2019

Dec. 16, 2019

Hi folks, While reviewing https://github.com/python/cpython/pull/17303/files, I noticed that the configure script update removed the options for `--runstatedir`. Those options appear to come from a Debian patch that other distros don't yet have: https://sources.debian.org/patches/autoconf/2.69-11/add-runstatedir.patch/ Since I use Fedora, running autoreconf locally on the review branch didn't add those options back, but did add in various macros related to Fedora's modular build system (we don't use those macros explicitly ourselves, but presumably some of the m4 macros we do use include them in their expansions when run on Fedora systems, so aclocal picked them up). Does anyone have any recommendations for dealing with this? My current plan is to revert back to the configure script from master, run autoreconf, and then use `git add -p` to only add in the desired changes, leaving everything else as it was on master. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 4

Emit a SyntaxWarning for unhashables literals in hashable dependant literals.
by mental na Dec. 15, 2019

Dec. 15, 2019

I'm not sure if this is the right place to bring this up, python-ideas seemed like language issues and python-dev seemed like CPython issues. There are several unhashable builtin types present in CPython, as of today the ones I've noticed are: lists, dicts, sets, and bytearrays. Two of these are containers that require (at least partially) their members to be hashable: dicts and sets By this logic you cannot have a set of lists or a dict of sets to ints. CPython however does not stop or warn you if you attempt to do such a thing until it hits `BUILD_MAP` or `BUILD_SET` during runtime. This is reasonable behavior when it's not possible to infer the member type of the container i.e. ``{f(x) for x in iterable}`` or ``{f(x): y for zip(xs, ys)}`` However, given the situation where literals are nested i.e. ``{[*gen] for gen in gens}`` or ``{{green: eggs}, {and_: ham}}`` this presents an unavoidable exception at runtime. I suggest emitting a SyntaxWarning when encountering these cases of literals that produce unhashable types that are used in literals that produce types where the members must be hashable. I don't think it should be a SyntaxError because it's not a language issue, its an implementation issue. I don't think it should be a linters responsibility because for the most part linters should consider language issues/idioms not side-effects from the running implementation. I do understand that such cases this issue addresses may be uncommon and once you do get that TypeError raised its a relatively quick and easy fix, but consider this being present in code paths that don't get taken as frequently, large codebases where it becomes difficult to keep track of small one liner literals like this or even for the newer programmers toying with Python through CPython and naively using unhashables in places they shouldn't be. Either way I'm interested in hearing what the core team thinks of this suggestion, thanks in advance! :D

4 5

Documenting sorted/min/max prerequisites
by Steven D'Aprano Dec. 15, 2019

Dec. 15, 2019

The list.sort method is documented to only use less than: https://docs.python.org/3/library/stdtypes.html#list.sort but I don't think that is correct, it seems to use greater than if it exists and less than doesn't. My understanding is that items need to define one of l.t. or g.t. to make it sortable, not the full complement of six rich comparison methods. The same seems to be the case for sorted, min and max, but none of them have documented that fact: https://docs.python.org/3/library/functions.html#max https://docs.python.org/3/library/functions.html#min https://docs.python.org/3/library/functions.html#sorted Should we document that all four functions will use l.t. if it exists, otherwise g.t. if it exists, but don't need both? -- Steven

3 6

Please be more precise when commenting on PEP 611.
by Mark Shannon Dec. 13, 2019

Dec. 13, 2019

Hi everyone, Thanks again for all your comments on PEP 611. I would like to ask a favour; please be more specific in your comments. Ideally state which part of the PEP you are disagreeing with and why you disagree with the relevant part of the rationale/motivation. Also, when asking for limits to be raised or removed entirely, could you state what you perceive to be the costs and benefits of larger limits. What do you believe is an acceptable cost in memory or runtime for larger limits? For example, you might say that the limit of one million lines of code per module is too small, and that it is worth a small, say 1%, impact on speed to allow a larger of limit of 100 million. If you believe a limit would have no cost, then please give a explanation of why that is so. Merely saying that you would like a larger limit is pointless. If there were no cost to arbitrarily large limits, then I wouldn't have proposed the PEP in the first place. Bear in mind that the costs of higher limits are paid by everyone, but the benefits are gained by few. Cheers, Mark.

13 22

PEP 611: The one million limit.
by Mark Shannon Dec. 13, 2019

Dec. 13, 2019

Hi Everyone, Thanks for all your feedback on my proposed PEP. I've editing the PEP in light of all your comments and it is now hopefully more precise and with better justification. https://github.com/python/peps/pull/1249 Cheers, Mark.

10 12

Summary of Python tracker Issues
by Python tracker Dec. 13, 2019

Dec. 13, 2019

ACTIVITY SUMMARY (2019-12-06 - 2019-12-13) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7174 ( -6) closed 43617 (+58) total 50791 (+52) Open issues with patches: 2818 Issues opened (30) ================== #37404: asyncio sock_recv blocks on ssl sockets. https://bugs.python.org/issue37404 reopened by christian.heimes #38989: pip install selects 32 bit wheels for 64 bit python if vcvarsa https://bugs.python.org/issue38989 opened by Ryan Thornton #38993: cProfile behaviour issue with decorator and math.factorial() l https://bugs.python.org/issue38993 opened by AVicennA #38999: Python launcher on Windows does not detect active venv https://bugs.python.org/issue38999 opened by Alexandros Karypidis #39005: test_faulthandler: test_dump_traceback_later_file() fails rand https://bugs.python.org/issue39005 opened by vstinner #39010: ProactorEventLoop raises unhandled ConnectionResetError https://bugs.python.org/issue39010 opened by Jonathan Slenders #39011: ElementTree attributes replace "\r" with "\n" https://bugs.python.org/issue39011 opened by mefistotelis #39014: test_concurrent_futures: test_crash() timed out on AMD64 Fedor https://bugs.python.org/issue39014 opened by vstinner #39015: DeprecationWarnings of implicitly truncations by __int__ appea https://bugs.python.org/issue39015 opened by lukasz.langa #39017: Infinite loop in the tarfile module https://bugs.python.org/issue39017 opened by jvoisin #39019: Missing class getitems in standard library classes https://bugs.python.org/issue39019 opened by BTaskaya #39020: [AIX] module _curses fails to build since ESCDELAY has been ad https://bugs.python.org/issue39020 opened by Michael.Felt #39021: multiprocessing is_alive() between children processes https://bugs.python.org/issue39021 opened by matttimms #39025: Windows Python Launcher does not update PATH to Scripts direct https://bugs.python.org/issue39025 opened by bluebird #39026: pystate.h contains non-relative of initconfig.h include causin https://bugs.python.org/issue39026 opened by gaige #39027: run_coroutine_threadsafe uses wrong TimeoutError https://bugs.python.org/issue39027 opened by janust #39028: ENH: Fix performance issue in keyword extraction https://bugs.python.org/issue39028 opened by seberg #39029: TestMaildir.test_clean fails randomly under parallel tests https://bugs.python.org/issue39029 opened by xtreak #39030: Ctypes unions with bitfield members that do not share memory https://bugs.python.org/issue39030 opened by dankreso #39031: Inconsistency with lineno and col_offset info when parsing eli https://bugs.python.org/issue39031 opened by lys.nikolaou #39032: wait_for and Condition.wait still not playing nicely https://bugs.python.org/issue39032 opened by criches #39033: zipimport raises NameError: name '_boostrap_external' is not d https://bugs.python.org/issue39033 opened by misho88 #39034: Documentation: Coroutines https://bugs.python.org/issue39034 opened by agarus #39035: Travis CI fail on backports: pyvenv not installed https://bugs.python.org/issue39035 opened by vstinner #39036: Add center_char attribute to str type https://bugs.python.org/issue39036 opened by lovi #39037: Wrong trial order of __exit__ and __enter__ in the with statem https://bugs.python.org/issue39037 opened by maggyero #39038: OverflowError in tarfile.open https://bugs.python.org/issue39038 opened by jvoisin #39039: zlib.error with tarfile.open https://bugs.python.org/issue39039 opened by jvoisin #39040: Wrong attachement filename when mail mime header was too long https://bugs.python.org/issue39040 opened by mkaiser #1021318: PyThreadState_Next not thread safe https://bugs.python.org/issue1021318 reopened by vstinner Most recent 15 issues with no replies (15) ========================================== #39035: Travis CI fail on backports: pyvenv not installed https://bugs.python.org/issue39035 #39032: wait_for and Condition.wait still not playing nicely https://bugs.python.org/issue39032 #39030: Ctypes unions with bitfield members that do not share memory https://bugs.python.org/issue39030 #39021: multiprocessing is_alive() between children processes https://bugs.python.org/issue39021 #39019: Missing class getitems in standard library classes https://bugs.python.org/issue39019 #39017: Infinite loop in the tarfile module https://bugs.python.org/issue39017 #39014: test_concurrent_futures: test_crash() timed out on AMD64 Fedor https://bugs.python.org/issue39014 #39005: test_faulthandler: test_dump_traceback_later_file() fails rand https://bugs.python.org/issue39005 #38976: Add support for HTTP Only flag in MozillaCookieJar https://bugs.python.org/issue38976 #38963: multiprocessing processes seem to "bleed" user information (GI https://bugs.python.org/issue38963 #38961: Flaky detection of compiler vendor https://bugs.python.org/issue38961 #38960: DTrace FreeBSD build fix https://bugs.python.org/issue38960 #38958: asyncio REPL swallows KeyboardInterrupt while editing https://bugs.python.org/issue38958 #38953: Untokenize and retokenize does not round-trip https://bugs.python.org/issue38953 #38947: dataclass defaults behave inconsistently for init=True/init=Fa https://bugs.python.org/issue38947 Most recent 15 issues waiting for review (15) ============================================= #39040: Wrong attachement filename when mail mime header was too long https://bugs.python.org/issue39040 #39033: zipimport raises NameError: name '_boostrap_external' is not d https://bugs.python.org/issue39033 #39031: Inconsistency with lineno and col_offset info when parsing eli https://bugs.python.org/issue39031 #39028: ENH: Fix performance issue in keyword extraction https://bugs.python.org/issue39028 #39019: Missing class getitems in standard library classes https://bugs.python.org/issue39019 #38981: better name for re.error Exception class. https://bugs.python.org/issue38981 #38976: Add support for HTTP Only flag in MozillaCookieJar https://bugs.python.org/issue38976 #38975: Add direct anchors to regex syntax documentation https://bugs.python.org/issue38975 #38960: DTrace FreeBSD build fix https://bugs.python.org/issue38960 #38956: argparse.BooleanOptionalAction should not add the default valu https://bugs.python.org/issue38956 #38952: asyncio cannot handle Python3 IPv4Address https://bugs.python.org/issue38952 #38945: Remove newline characters from uu encoding methods https://bugs.python.org/issue38945 #38942: Possible assertion failures in csv.Dialect() https://bugs.python.org/issue38942 #38938: Possible performance improvement for heaqq.merge() https://bugs.python.org/issue38938 #38937: NameError in list comprehension within .pth file https://bugs.python.org/issue38937 Top 10 most discussed issues (10) ================================= #37228: UDP sockets created by create_datagram_endpoint() allow by def https://bugs.python.org/issue37228 20 msgs #38999: Python launcher on Windows does not detect active venv https://bugs.python.org/issue38999 9 msgs #1021318: PyThreadState_Next not thread safe https://bugs.python.org/issue1021318 9 msgs #39033: zipimport raises NameError: name '_boostrap_external' is not d https://bugs.python.org/issue39033 7 msgs #38699: socket: change listen() default backlog from 128 to 4096? https://bugs.python.org/issue38699 5 msgs #38981: better name for re.error Exception class. https://bugs.python.org/issue38981 5 msgs #39010: ProactorEventLoop raises unhandled ConnectionResetError https://bugs.python.org/issue39010 5 msgs #39031: Inconsistency with lineno and col_offset info when parsing eli https://bugs.python.org/issue39031 5 msgs #20443: __code__. co_filename should always be an absolute path https://bugs.python.org/issue20443 4 msgs #25160: Stop using deprecated imp module; imp should now emit a real D https://bugs.python.org/issue25160 4 msgs Issues closed (59) ================== #9529: Make re match object iterable https://bugs.python.org/issue9529 closed by serhiy.storchaka #27822: Fail to create _SelectorTransport with unbound socket https://bugs.python.org/issue27822 closed by vstinner #30386: Add a build infrastructure for Android https://bugs.python.org/issue30386 closed by xdegaye #30458: [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection https://bugs.python.org/issue30458 closed by gregory.p.smith #32461: the first build after a change to Makefile.pre.in uses the old https://bugs.python.org/issue32461 closed by xdegaye #32907: pathlib: test_resolve_common fails on Windows w/ longusername https://bugs.python.org/issue32907 closed by cheryl.sabella #32949: Simplify "with"-related opcodes https://bugs.python.org/issue32949 closed by serhiy.storchaka #33125: Windows 10 ARM64 platform support https://bugs.python.org/issue33125 closed by steve.dower #33762: Make tempfiles subclass IOBase https://bugs.python.org/issue33762 closed by terry.reedy #33880: namedtuple should use NFKD to find duplicate members https://bugs.python.org/issue33880 closed by rhettinger #34776: Postponed annotations break inspection of dataclasses https://bugs.python.org/issue34776 closed by lukasz.langa #35805: email package folds msg-id identifiers using RFC2047 encoded w https://bugs.python.org/issue35805 closed by maxking #36214: AC_RUN_IFELSE macros not used as arguments of AC_CACHE_VAL et https://bugs.python.org/issue36214 closed by xdegaye #37226: Asyncio Fatal Error on SSL Transport - IndexError Deque Index https://bugs.python.org/issue37226 closed by ben.brown #37535: Client SSLSocket with select.select() always returns readable https://bugs.python.org/issue37535 closed by christian.heimes #38449: regression - mimetypes guess_type is confused by ; in the file https://bugs.python.org/issue38449 closed by corona10 #38529: Python 3.8 improperly warns about closing properly closed stre https://bugs.python.org/issue38529 closed by asvetlov #38547: test_pty fails when using setsid() https://bugs.python.org/issue38547 closed by vstinner #38594: importlib.metadata documentation deficiencies https://bugs.python.org/issue38594 closed by jaraco #38614: Add timeout constants to test.support https://bugs.python.org/issue38614 closed by vstinner #38652: Remove/update provisional note for asyncio.BufferedProtocol https://bugs.python.org/issue38652 closed by asvetlov #38669: patch.object should raise another error when first argument is https://bugs.python.org/issue38669 closed by cjw296 #38673: REPL shows continuation prompt (...) when comment or space ent https://bugs.python.org/issue38673 closed by gvanrossum #38697: test.support.find_unused_port() race condition: test_socket fa https://bugs.python.org/issue38697 closed by vstinner #38698: While parsing email message id: UnboundLocalError https://bugs.python.org/issue38698 closed by maxking #38708: parse_message_id in email module is very buggy / crashy https://bugs.python.org/issue38708 closed by maxking #38849: test_timestamp_naive fails on android https://bugs.python.org/issue38849 closed by xdegaye #38916: Remove array.fromstring() and array.tostring() aliases, deprec https://bugs.python.org/issue38916 closed by vstinner #38943: Idle autocomplete window doesn't show up https://bugs.python.org/issue38943 closed by terry.reedy #38944: Idle completion windows should close on Escape key https://bugs.python.org/issue38944 closed by terry.reedy #38949: incorrect prefix, exec_prefix in distutils.command.install https://bugs.python.org/issue38949 closed by xdegaye #38978: Implement __class_getitem__ for Future, Task, Queue https://bugs.python.org/issue38978 closed by asvetlov #38979: ContextVar[str] should return ContextVar class, not None https://bugs.python.org/issue38979 closed by asvetlov #38986: Suppport TaskWakeupMethWrapper.__self__ to conform asyncio _fo https://bugs.python.org/issue38986 closed by asvetlov #38987: 3.8.0 on GNU/Linux fails to find shared library https://bugs.python.org/issue38987 closed by christian.heimes #38990: Import genericpath fails with python -S https://bugs.python.org/issue38990 closed by serhiy.storchaka #38991: Remove test.support.strip_python_stderr() https://bugs.python.org/issue38991 closed by vstinner #38992: testFsum failure caused by constant folding of a float express https://bugs.python.org/issue38992 closed by mark.dickinson #38994: Implement __class_getitem__ for PathLike https://bugs.python.org/issue38994 closed by asvetlov #38995: reverse search (ctrl-r) doest not work https://bugs.python.org/issue38995 closed by ned.deily #38996: introduction of default values for collection.namedtuple https://bugs.python.org/issue38996 closed by pablogsal #38997: test__xxsubinterpreters test_atexit test_capi test_threading a https://bugs.python.org/issue38997 closed by vstinner #38998: dict.setdefault (setdefault of dictionary) https://bugs.python.org/issue38998 closed by eric.smith #39000: Range causing unstable output(Windows64) https://bugs.python.org/issue39000 closed by mark.dickinson #39001: possible problem with 64-bit mingw DECREF https://bugs.python.org/issue39001 closed by Dave Lawrence #39002: Small typo in Lib/test/test_statistics.py: tranlation -> trans https://bugs.python.org/issue39002 closed by rhettinger #39003: test_unparse leaked [35, 5, 6] references: fail randomly, pass https://bugs.python.org/issue39003 closed by pablogsal #39004: test_largefile: TestSocketSendfile.test_it() failed on AMD64 F https://bugs.python.org/issue39004 closed by vstinner #39006: test_asyncio: sendfile tests fail on AMD64 Debian root 3.7 whe https://bugs.python.org/issue39006 closed by vstinner #39007: Add audit hooks to winreg module https://bugs.python.org/issue39007 closed by steve.dower #39008: PySys_Audit should require PY_SSIZE_T_CLEAN https://bugs.python.org/issue39008 closed by steve.dower #39009: Small typo in Lib/test/test__locale.py: thousauds -> thousands https://bugs.python.org/issue39009 closed by pablogsal #39012: nuget package published as 3.8.1-c1 instead of rc1 https://bugs.python.org/issue39012 closed by steve.dower #39013: SyntaxError: 'break' outside loop for legal Expression https://bugs.python.org/issue39013 closed by serhiy.storchaka #39016: Negative Refcount in Python 3.8 https://bugs.python.org/issue39016 closed by Christian.Tismer #39018: IndexError exception on corrupted zip file https://bugs.python.org/issue39018 closed by serhiy.storchaka #39022: Synchronize importlib.metadata with importlib_metadata 1.3 https://bugs.python.org/issue39022 closed by jaraco #39023: random.seed with string and version 1 not deterministic in 3.5 https://bugs.python.org/issue39023 closed by rhettinger #39024: Compiling relative paths test fails (install via pyenv) https://bugs.python.org/issue39024 closed by ned.deily

1 0

PEP 611 -- why limit coroutines and classes?
by Guido van Rossum Dec. 12, 2019

Dec. 12, 2019

I want to question two specific limits. (a) Limiting the number of classes, in order to potentially save space in object headers, sounds like a big C API change, and I think it's better to lift this idea out of PEP 611 and debate the and cons separately. (b) Why limit coroutines? It's just another Python object and has no operating resources associated with it. Perhaps your definition of coroutine is different, and you are thinking of OS threads? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

10 16

Python 3.8 problem with PySide
by Christian Tismer Dec. 12, 2019

Dec. 12, 2019

Hi guys, during the last few weeks I have been struggling quite much in order to make PySide run with Python 3.8 at all. The expected problems were refcounting leaks due to changed handling of heaptypes. But in fact, the runtime behavior was much worse, because I always got negative refcounts! After exhaustive searching through the different 3.8 commits, I could isolate the three problems with logarithmic search. The hard problem was this: Whenever PySide creates a new type, it crashes in PyType_Ready. The reason is the existence of the Py_TPFLAGS_METHOD_DESCRIPTOR flag. During the PyType_Ready call, the function mro() is called. This mro() call results in a negative refcount, because something behaves differently since this flag is set by default in mro(). When I patched this flag away during the type_new call, everything worked ok. I don't understand why this problem affects PySide at all. Here is the code that would normally be only the newType line: // PYSIDE-939: This is a temporary patch that circumvents the problem // with Py_TPFLAGS_METHOD_DESCRIPTOR until this is finally solved. PyObject *ob_PyType_Type = reinterpret_cast<PyObject *>(&PyType_Type); PyObject *mro = PyObject_GetAttr(ob_PyType_Type, Shiboken::PyName::mro()); auto hold = Py_TYPE(mro)->tp_flags; Py_TYPE(mro)->tp_flags &= ~Py_TPFLAGS_METHOD_DESCRIPTOR; auto *newType = reinterpret_cast<SbkObjectType *>(type_new(metatype, args, kwds)); Py_TYPE(mro)->tp_flags = hold; I would really like to understand the reason for this unexpected effect. Does this ring a bell? I have no clue what is wrong with PySide, if it is wrong at all. Thanks -- Chris -- Christian Tismer :^) tismer(a)stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023

5 12

Feedback on PEP 611 (so far) from the Steering Council
by Barry Warsaw Dec. 12, 2019

Dec. 12, 2019

The Python Steering Council discussed PEP 611 at today’s meeting. Here is our feedback so far: * The Steering Council reserves the right to be the BDFL-Delegate for this PEP * The PEP should clearly delineate two aspects: - A language generic part (i.e. applies to all implementations of Python) which exposes implementation limits in a way that Python code can read at runtime. Think sys.getrecursionlimit(), sys.maxsize, and sys.implementation - An implementation-specific part where the PEP would propose specific limits for the CPython implementation. Other implementations of the Python language would be free to adjust such limits up or down as they deem appropriate. * We encourage the PEP authors and proponents to gather actual performance data that can be used to help us evaluate whether this PEP is a good idea or not. By all means, continue to discuss and refine the PEP. When the PEP author is ready for a pronouncement, you can email the Steering Council. Cheers, -Barry

3 2