Python-Dev August 2021

python-dev@python.org

75 participants
39 discussions

How about using modern C++ in development of CPython ?

by redradist＠gmail.com

Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...

1 day, 20 hours

PEP 646 (Variadic Generics): final call for comments

by Matthew Rahtz

Hi everyone, We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council. If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.) Thanks! Matthew and Pradeep

6 days, 2 hours

PEP 467 feedback from the Steering Council

by Barry Warsaw

Hello Nick, Ethan, The Python Steering Council reviewed PEP 467 -- Minor API improvements for binary sequences at our 2021-07-26 meeting. Thank you for work on this PEP. We’re generally very favorable for adding to Python 3.11 the features and APIs described in the PEP. We have some requests for changes that we’d like you to consider. * The Python-Version in the PEP needs to target Python 3.11 of course. * We think it would be better if bytes.fromsize()’s second argument was a keyword-enabled or keyword-only argument. We understand the rationale given in the PEP for not doing so, but ultimately we think the readability of (at least allowing) a keyword argument to be more compelling. Some possible options include `fill`, `value`, or `byte`. * We all really dislike the word “ord” as in `bytes.fromord()`. We understand the symmetry of this choice, but we also feel like we have an opportunity to make it more understandable, so we recommend `bytes.fromint()` and `bytearray.fromint()`. * We think the `bchr()` built-in is not necessary. Given the `.fromint()` methods, it’s better not to duplicate the functionality, and everything has a cost. A built-in that exists only for the symmetry described in the PEP is just a little extra complexity for little value. Let us know what you think about making these changes. We aren’t making acceptance contingent on these changes, but we do think they make the PEP and the new APIs better. -Barry (on behalf of the Python Steering Council)

4 months, 2 weeks

Discrepancy between what aiter() and `async for` requires on purpose?

by Brett Cannon

If you look at https://github.com/python/cpython/blob/b11a951f16f0603d98de24fee5c023df83ea… you will see that `async for` requires that the iterator returned from `__aiter__` define `__anext__`. But if you look at aiter() which uses PyObject_GetAiter() from https://github.com/python/cpython/blob/f0a6fde8827d5d4f7a1c741ab1a8b206b66f… and PyAiter_Check() from https://github.com/python/cpython/blob/f0a6fde8827d5d4f7a1c741ab1a8b206b66f… you will notice that aiter() requires `__anext__` *and* `__aiter__` on the async iterator that gets returned from __aiter__. Now the docs for aiter() at https://docs.python.org/3.10/library/functions.html#aiter points out that the async iterator is expected to define both methods as does the glossary definition for "asynchronous iterator" ( https://docs.python.org/3.8/glossary.html#term-asynchronous-iterator). So my question is whether the discrepancy between what `async for` expects and what `aiter()` expects on purpose? https://bugs.python.org/issue31861 was the issue for creating aiter() and I didn't notice a discussion of this difference. The key reason I'm asking is this does cause a deviation compared to the relationship between `for` and `iter()` (which does not require `__iter__` to be defined on the iterator, although collections.abc.Iterator does). It also makes the glossary definition being linked from https://docs.python.org/3.10/reference/compound_stmts.html#the-async-for-st… incorrect.

4 months, 2 weeks

Making code object APIs unstable

by Guido van Rossum

In 3.11 we're changing a lot of details about code objects. Part of this is the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine Grained Error Locations in Tracebacks). As a result, the set of fields of the code object is changing. This is fine, the structure is part of the internal API anyway. But there's a problem with two public API functions, PyCode_New() and PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their signatures are incompatible with previous versions, and they have to be since the set of values needed to create a code object is different. (The types.CodeType constructor signature is also changed, and so is its replace() method, but these aren't part of any stable API). Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the PEP 387 stable ABI. What should we do? A. We could deprecate them, keep (restore) their old signatures, and create crippled code objects (no exception table, no endline/column tables, qualname defaults to name). B. We could deprecate them, restore the old signatures, and always raise an error when they are called. C. We could just delete them. D. We could keep them, with modified signatures, and to heck with ABI compatibility for these two. E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add the posonlyargcount (which is the only difference between the two), and d*mn the torpedoes. F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for PyCode_New() (and deprecate it). If these weren't part of the stable ABI, I'd choose (E). But because they are, I think only (A) or (B) are our options. The problem with (C) is that if there's code that links to them but doesn't call them (except in some corner case that the user can avoid), the code won't link even though it would work fine. The problem with (D) is that if it *is* called by code expecting the old signature it will segfault. I'm not keen on (A) since it can cause broken code objects when used to copy a code object with some modified metadata (e.g. a different filename), since there's no way to pass the exception table (and several other fields, but the exception table is an integral part of the code now). Code wanting to make slight modifications to code objects such as changing co_name or co_filename should switch to the .replace() API, which is much better at this (though calling it from C is a pain, it's possible). Code wanting to synthesize code should be updated for each release; we should probably require such code to be built with the internal API and use _PyCode_New(), which takes a struct argument containing all the necessary fields. Thoughts? I'm especially interested in Petr's opinion given that this is a case where we'd like to deprecate something in the stable ABI. See also discussion in https://bugs.python.org/issue40222 (esp. near the end). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

4 months, 2 weeks

API's to interface python with Eclipse to execute and debug C/C++ Source files.

by Chandrakant Shrimantrao

Hi, We would like to know if there is any option available to execute and debug( setting break point and delete break point) C/C++ Source files Python API's inside Eclipse IDE after interfacing Pydev with Eclipse. Please let us know. Thanks and regards. Chandrakant shrimantrao. L&T Technology Services Ltd www.LTTS.com L&T Technology Services Limited (LTTS) is committed to safeguard your data privacy. For more information to view our commitment towards data privacy under GDPR, please visit the privacy policy on our website www.Ltts.com. This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.

4 months, 3 weeks

Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries

by Ammar Askar

Hey everyone, As part of PEP 588, migrating bugs.python.org issues to Github, there are two current mailing list related items that need a replacement or need to be turned down. 1. Weekly summary emails with bug counts and issues from the week, example: https://mail.python.org/archives/list/python-dev@python.org/thread/JRFJ4QH7… 2. Emails sent to the new-bugs-announce and python-bugs-list for new issues and comments to existing issues. I wanted to quickly gauge if anyone is relying on these mailing lists/emails or if it's a convenient part of your workflow. Feel free to chime in here or on the migration issues directly with your use-case: 1. https://github.com/psf/gh-migration/issues/6 2. https://github.com/psf/gh-migration/issues/7 Thanks, Ammar

4 months, 3 weeks

PEP-535 (Rich comparison chaining) Discussion?

by Angus Hollands

Hi all, PEP 535 was deferred until Python 3.8. I would like to discuss the PEP given that some time has passed, and I _personally_ would benefit from its acceptance. ## A little about me: As a reasonably long-time Python user (well, 16 years), I have gradually moved from using only the core language to working within the DSLs (NumPy et al.) of the Scientific Python ecosystem. Most recently, I've been using Awkward Arrray (https://github.com/scikit-hep/awkward-1.0), which generalises the regular NumPy array programming paradigm into "jagged" data structures, and was initially aimed at the High Energy Physics (HEP) domain. ## TL;DR of the PEP's Motivations: Various aspects of NumPy's API have become a "lingua-franca" in the Scientific Python domain, such as strongly-overloaded index operations. One such overload allows "mask arrays" — arrays with boolean entries corresponding to a per-element mask — to select a subset of the array. These mask arrays are conveniently built from the rich comparison operators that Python supports, e.g. ```python x = np.array(...) y = x[(2 < x) & (x < 8)] ``` As discussed in PEP 535, the limitation on comparison chaining (namely that it is not possible for rich comparisons) makes the above syntax more clunky than it might otherwise be. The main repercussion of this limitation is lower readability: - operator precedence of bitwise and/or requires a reasonable number of parentheses (https://docs.python.org/3/reference/expressions.html#operator-precedence) - repetition of the rich-object (array) With PEP 535 (and the necessary library changes), this would look more like ```python x = np.array(...) y = x[2 < x < 8] ``` This is already well described in the PEP, so I'll leave the example there! ## Community Survey The Oct 2020 Python Developers Survey (https://www.jetbrains.com/lp/python-developers-survey-2020/) suggests that 62% of respondents use NumPy. Additionally, 37% of respondents list "Simple syntax, syntactic sugar, easy to learn" and 30% list "Easy to write & read code, high-level language" amongst their three most liked features in the Python language (to clarify, these answers may overlap by respondent!) From these results, I would make a tentative case that this PEP would have a reasonable impact on the Python community, given that libraries benefitting from this proposal (e.g. NumPy) are used by a "majority" of those surveyed. ## Conclusion This is a bit of a long opener to this list (apologies!). Is there any interest in pushing this PEP along? Thanks, Angus Hollands

4 months, 3 weeks

[RELEASE] Python 3.9.7 and 3.8.12 are now available

by Łukasz Langa

Python 3.9.7 Get it here: https://www.python.org/downloads/release/python-397/ <https://www.python.org/downloads/release/python-397/> Python 3.9.7 is the newest major stable release of the Python programming language, and it contains many new features and optimizations. There’s been 187 commits since 3.9.6 which is a similar amount compared to 3.8 at the same stage of the release cycle. See the change log <https://docs.python.org/release/3.9.7/whatsnew/changelog.html> for details. On macOS, we encourage you to use the universal2 binary installer variant whenever possible. The legacy 10.9+ Intel-only variant will not be provided for Python 3.10 and the universal2 variant will become the default download for 3.9.8. You may need to upgrade third-party components, like pip, to later versions. You may experience differences in behavior in IDLE and other Tk-based applications due to using the newer version of Tk. As always, if you encounter problems when using this installer variant, please check https://bugs.python.org <https://bugs.python.org/> for existing reports and for opening new issues. The next Python 3.9 maintenance release will be 3.9.8, currently scheduled for 2021-11-01. <https://discuss.python.org/t/python-3-9-7-and-3-8-12-now-available/10401#th…>The Second Security-Only Release of Python 3.8 Get it here: https://www.python.org/downloads/release/python-3812/ <https://www.python.org/downloads/release/python-3812/> Security content in this release contains four fixes. There are also four additional fixes for bugs that might have lead to denial-of-service attacks. Finally, while we’re not providing binary installers anymore, for those users who produce installers, we upgraded the OpenSSL version used to 1.1.1l. Take a look at the change log <https://docs.python.org/release/3.8.12/whatsnew/changelog.html> for details. According to the release calendar specified in PEP 569 <https://www.python.org/dev/peps/pep-0569/>, Python 3.8 is now in “security fixes only” stage of its life cycle: 3.8 branch only accepts security fixes and releases of those are made irregularly in source-only form until October 2024. Python 3.8 isn’t receiving regular bug fixes anymore, and binary installers are no longer provided for it. Python 3.8.10 was the last full bugfix release of Python 3.8 with binary installers. <https://discuss.python.org/t/python-3-9-7-and-3-8-12-now-available/10401#se…>Security releases of 3.7.12 and 3.6.15 Those aren’t ready just yet but are soon to follow. Similarly to 3.8, Python 3.7 and 3.6 are now in “security fixes only” stage of their life cycle. Python 3.7 will be providing source archives until June 2023 while Python 3.6 ends its life in December 2021. <https://discuss.python.org/t/python-3-9-7-and-3-8-12-now-available/10401#we…>We hope you enjoy the new releases Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Łukasz Langa @ambv <https://discuss.python.org/u/ambv>

4 months, 3 weeks

(Adaptive) Shivers sort variant of Tim sort

by Laurent Lyaudet

Hello, I developed a C sorting algorithms library that currently has a few dozens of algorithms and thousands of variants when choosing parameters (like threshold for switching to insertion sort, etc.). (I compile each variant on demand in custom tests to avoid having a bloated source file.) My benchmarks could be improved but however I found that Shivers' sort and adaptive Shivers' sort (aka Jugé's sort) performs better than Tim's sort. I don't know if it will still be true with the benchmarks in Python. If you're open to the idea that maybe Tim's sort can be improved and are willing to benchmark it, it can be done easily by switching the main natural merge strategy like I did here : https://github.com/LLyaudet/TSODLULS/commit/2968c4b4ca58ae794157dc9636ed870… The relevant code is not very long : /** * This strategy is from ShiversSort: * see the research report by Olin Shivers, Georgia Institute of Technology, 2002. * It uses bitwise tricks to check condition on floor of logarithms in base 2 of runs lengths. * When I benchmark it, it is slightly faster than Tim's sort strategy. */ #define TSODLULS_natural_merge_main_strategy__Shivers_sort \ while(merge_state.i_run_instances_count > 1){\ size_t i_merge_at = merge_state.i_run_instances_count - 2;\ size_t i_order_of_magnitude = merge_state.arr_run_instances[i_merge_at + 1].i_length;\ if(i_order_of_magnitude < ((~i_order_of_magnitude) & merge_state.arr_run_instances[i_merge_at].i_length) ){\ break;\ }\ i_compare_result = TSODLULS_MERGE_TWO_RUNS(&merge_state, i_merge_at);\ if(i_compare_result != 0){\ goto clean_and_return_error;\ }\ }\ /** * This strategy is from AdaptiveShiversSort: * see the articles by Vincent Jugé, for example 1024 Bulletin de la SIF, avril 2020, in French, * or the preprint on arXiv or SODA 2020 proceedings. * It uses bitwise tricks to check condition on floor of logarithms in base 2 of runs lengths. * When I benchmark it, it is slightly faster than Tim's sort strategy. * Despite its ressemblance with Shivers's sort, * the distinct properties of this strategy make that JugéSort would probably be a better name than AdaptiveShiversSort, * or an even better name for the whole algorithm should be TimShiversJugéSort and I must have missed many names ;) * With AdaptiveShiversSort we avoid 'é' and diacritics in function names ;P */ #define TSODLULS_natural_merge_main_strategy__adaptive_Shivers_sort \ while(merge_state.i_run_instances_count > 2){\ size_t i_merge_at = merge_state.i_run_instances_count - 3;\ size_t i_order_of_magnitude = merge_state.arr_run_instances[i_merge_at + 1].i_length | merge_state.arr_run_instances[i_merge_at + 2].i_length;\ if(i_order_of_magnitude < ((~i_order_of_magnitude) & merge_state.arr_run_instances[i_merge_at].i_length) ){\ break;\ }\ i_compare_result = TSODLULS_MERGE_TWO_RUNS(&merge_state, i_merge_at);\ if(i_compare_result != 0){\ goto clean_and_return_error;\ }\ }\ In my library the merge strategy is a parameter and I switch its macro when compiling competitors algorithms on demand. I hope it may help. Thanks for your time. If you want to look further at my library, it works with gcc and glibc. I have not tested it on other platforms and whilst I have no compilation warning on my personal dev laptop, I do have one when I run it on the laptop I'm currently writing this email. The tests passes nevertheless. Best regards, Laurent Lyaudet

4 months, 4 weeks

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev August 2021