Mailman 3 April 2020 - Python-Dev

Mixed Python/C debugging
by Skip Montanaro April 26, 2023

April 26, 2023

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (… [View More]flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido) [View Less]

7 17

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: … [View More]$Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: [View Less]

8 14

Boundaries between numbers and identifiers
by Serhiy Storchaka April 15, 2021

April 15, 2021

In Python 2.5 `0or[]` was accepted by the Python parser. It became an error in 2.6 because "0o" became recognizing as an incomplete octal number. `1or[]` still is accepted. On other hand, `1if 2else 3` is accepted despites the fact that "2e" can be recognized as an incomplete floating point number. In this case the tokenizer pushes "e" back and returns "2". Shouldn't it do the same with "0o"? It is possible to make `0or[]` be parseable again. Python implementation is able to tokenize … [View More]

8 10

pathlib.Path: inconsistent symlink_to() and link_to()
by Barney Gale Jan. 22, 2021

Jan. 22, 2021

Hi, Pathlib's symlink_to() and link_to() methods have different argument orders, so: a.symlink_to(b) # Creates a symlink from A to B a.link_to(b) # Creates a hard link from B to A I don't think link_to() was intended to be implemented this way, as the docs say "Create a hard link pointing to a path named target.". It's also inconsistent with everything else in pathlib, most obviously symlink_to(). Bug report here: https://bugs.python.org/issue39291 This /really/ irks me. … [View More]

4 5

test_ssl.py hangs with SSL 1.1 built with no threads
by mig28suarez＠hotmail.com May 17, 2020

May 17, 2020

Hello! This is my first time posting to the group. I've been running builds of Python 3.7.x on CentOS Linux release 7.7 (64bit/Intel Core 2 Duo) and I ran into hangs with test_ssl.py when using latest SSL 1.1.1d sources. I've done a full compilation from source for Python 3.7.7 and SSL 1.1 in my workspaces. From what I can tell the problem is when SSL 1.1 is built with no threading there is no locking enabled by python. This one line change will make the hangs in test_ssl.py go away: … [View More]

2 1

Re: Improvement to SimpleNamespace
by Raymond Hettinger May 7, 2020

May 7, 2020

[GvR] > We should not try to import JavaScript's object model into Python. Yes, I get that. Just want to point-out that working with heavily nested dictionaries (typical for JSON) is no fun with square brackets and quotation marks. Raymond

8 10

PEP 617: New PEG parser for CPython
by Guido van Rossum May 7, 2020

May 7, 2020

Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros Nikolaou and myself have been working on a new parser for CPython. We are now far enough along that we present a PEP we've written: https://www.python.org/dev/peps/pep-0617/ Hopefully the PEP speaks for itself. We are hoping for a speedy resolution so we can land the code we've written before 3.9 beta 1. If people insist I can post a copy of the entire PEP here on the list, but since a lot of it is just background … [View More]

22 58

Moving tzdata into the python organization
by Paul Ganssle May 3, 2020

May 3, 2020

Hi all, PEP 615 specifies that we will add a first party tzdata package <https://www.python.org/dev/peps/pep-0615/#the-tzdata-python-package>. I created this package at pganssle/tzdata <https://github.com/pganssle/tzdata>, but I believe the final home for it should be python/tzdata. Are there any objections to me moving it into the python org as is? Thanks! Paul

2 2

Adding a "call_once" decorator to functools
by tom＠tomforb.es May 1, 2020

May 1, 2020

Hello, After a great discussion in python-ideas[1][2] it was suggested that I cross-post this proposal to python-dev to gather more comments from those who don't follow python-ideas. The proposal is to add a "call_once" decorator to the functools module that, as the name suggests, calls a wrapped function once, caching the result and returning it with subsequent invocations. The rationale behind this proposal is that: 1. Developers are using "lru_cache" to achieve this right now, which is less … [View More]

15 23

Virtual machine bleeds into generator implementation?
by Skip Montanaro April 30, 2020

April 30, 2020

This is more an observation and question than anything else, but perhaps it will stimulate some ideas from the experts. Consider this trivial generator function: def gen(a): yield a When the YIELD_VALUE instruction is executed, it executes (in the non-async case): retval = POP(); f->f_stacktop = stack_pointer; goto exiting; This is fine as far as it goes. However, execution eventually leads to Objects/genobject.c where we hit this code (I think after falling off … [View More]

3 4

PEP 554 comments
by Antoine Pitrou April 30, 2020

April 30, 2020

Hello, First, I would like to say that I have no fondamental problem with this PEP. While I agree with Nathaniel that the rationale given about the CSP concurrency model seems a bit weak, the author is obviously expressing his opinion there and I won't object to that. However, I think the PEP is desirable for other reasons. Mostly, I hope that by making the subinterpreters functionality available to pure Python programmers (while it was formally an advanced and arcane part of the C API), we … [View More]will spur of bunch of interesting third-party experimentations, including possibilities that we on python-dev have not thought about. The appeal of the PEP for experimentations is multiple: 1) ability to concurrently run independent execution environments without spawning child processes (which on some platforms and in some situations may not be very desirable: for example on Windows where the cost of spawning is rather high; also, child processes may crash, and sometimes it is not easy for the parent to recover, especially if a synchronization primitive is left in an unexpected state) 2) the potential for parallelizing CPU-bound pure Python code in a single process, if a per-interpreter GIL is finally implemented 3) easier support for sharing large data between separate execution environments, without the hassle of setting up shared memory or the fragility of relying on fork() semantics (and as I said, I hope people find other applications) As for the argument that we already have asyncio and several other packages, I actually think that combining these different concurrency mechanisms would be interesting complex applications (such as distributed systems). For that, however, I think the PEP as currently written is a bit lacking, see below. Now for the detailed comments. * I think the module should indeed be provisional. Experimentation may discover warts that call for a change in the API or semantics. Let's not prevent ourselves from fixing those issues. * The "association" timing seems quirky and potentially annoying: an interpreter only becomes associated with a channel the first time it calls recv() or send(). How about, instead, associating an interpreter with a channel as soon as that channel is given to it through `Interpreter.run(..., channels=...)` (or received through `recv()`)? * How hard would it be, in the current implementation, to add buffering to channels? It doesn't have to be infinite: you can choose a fixed buffer size (or make it configurable in the create() function, which allows passing 0 for unbuffered). Like Nathaniel, I think unbuffered channels will quickly be annoying to work with (yes, you can create a helper thread... now you have one additional thread per channel, which isn't pretty -- especially with the GIL). * In the same vein, I think channels should allow adding readiness callbacks (that are called whenever a channel becomes ready for sending or receiving, respectively). This would make it easy to plug them into an event loop or other concurrency systems (such as Future-based concurrency). Note that each interpreter "associated" with a channel should be able to set its own readiness callback: so one callback per Python object representing the channel, but potentially multiple callbacks for the underlying channel primitive. (how would the callback be scheduled for execution in the right interpreter? perhaps using `_PyEval_AddPendingCall()` or a similar mechanism?) * I think either `interpreters.get_main()` or `interpreters.is_main()` is desirable. Inevitable, the slight differences between main and non-main interpreters will surface in non-trivial applications (finalization issues in distributed systems can really be hairy). It seems this should be mostly costless to provide, so let's do it. * I do think a minimal synchronization primitive would be nice. Either a Lock (in the Python sense) or a Semaphore: both should be relatively easy to provide, by wrapping an OS-level synchronization primitive. Then you can recreate all high-level synchronization primitives, like the threading and multiprocessing modules do (using a Lock or a Semaphore, respectively). (note you should be able to emulate a semaphore using blocking send() and recv() calls, but that's probably not very efficient, and efficiency is important) Of course, I hope these are all actionable before beta1 :-) If not, here is my preferential priority list: * High priority: fix association timing * High priority: either buffering /or/ readiness callbacks * Middle priority: get_main() /or/ is_main() * Middle / low priority: a simple synchronization primitive But I would stress the more of these we provide, the more we encourage people to experiment without pulling too much of their hair. (also, of course, I hope other people read the PEP and emit feedback) Best regards Antoine. [View Less]

7 29

[RELEASE] Python 3.8.3rc1 is now ready for testing
by Łukasz Langa April 29, 2020

April 29, 2020

Python 3.8.3rc1 is the release candidate of the third maintenance release of Python 3.8. Go get it here: https://www.python.org/downloads/release/python-383rc1/ <https://www.python.org/downloads/release/python-383rc1/> Assuming no critical problems are found prior to 2020-05-11, the scheduled release date for 3.8.3, no code changes are planned between this release candidate and the final release. That being said, please keep in mind that this is a pre-release and as such its main … [View More]

1 0

PEP 554 for 3.9 or 3.10?
by Eric Snow April 29, 2020

April 29, 2020

Hi all, It was nice to speak with many of you at the online language summit this week. It was the next best thing to seeing you all in person! :) With the 3.9 feature freeze coming up I'm considering options for PEP 554. I'm hopeful to have a pronouncement from Antoine in the near future. If that comes in time for 3.9, we will have the implementation ready to go. It is already nearly complete and the remaining work should be done in the next week or so. I'm looking for feedback on a … [View More]nagging worry I have with merging PEP 554 into 3.9 (if it is accepted) without having the per-interpreter GIL. See below. I'll update the PEP later with relevant info from this discussion. wishing-you-were-all-here-ly yours, -eric +++++++++++++++++++++++++++++++++++++++++++++++++ (++++++++++ Dilemma ============ Many folks have conflated PEP 554 with having a per-interpreter GIL. In fact, I was careful to avoid any mention of parallelism or the GIL in the PEP. Nonetheless some are expecting that when PEP 554 lands we will reach multi-core nirvana. While PEP 554 might be accepted and the implementation ready in time for 3.9, the separate effort toward a per-interpreter GIL is unlikely to be sufficiently done in time. That will likely happen in the next couple months (for 3.10). So...would it be sufficiently problematic for users if we land PEP 554 in 3.9 without per-interpreter GIL? Options ============ Here are the options as I see them (if the PEP is accepted in time for 3.9): 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 (they get parallelism for free in 3.10) 2. like 1, but mark the module as provisional until per-interpreter GIL lands 3. do not merge PEP 554 until per-interpreter GIL is merged 4. like 3, but publish a 3.9-only module to PyPI in the meantime Context ============ Like I said, PEP 554 purposefully does not talk about parallelism or the GIL. Instead it focuses strictly on the following (with an emphasis on minimalism): * exposing the existing C-API * supporting a new concurrency model (CSP) Regarding that first one, the value is in exposing the existing functionality to a broader group of people. keep in mind that subinterpreters currently have various other limitations (aside from sharing the GIL): * nothing about them has been optimized (e.g. interpreter startup, data sharing, data passing) * extension modules with process-global state may break * some bugs still remain in the corners Currently, there is a chicken-and-egg problem. It is unlikely that there will be sufficient motivation to address those limitations until the community starts really using subinterpreters. Hence PEP 554. My Position ============ At this point I think we should go with option #1 (or possibly #2). IMHO, we would be fine merging PEP 554 without per-interpreter GIL. Here's why: * the two objectives of the PEP have value for the community (as explained in the PEP) * the sooner we can get the functionality of subinterpreters out to the broader world, the better * I don't think the lack of parallelism in 3.9 will trip anyone up My only concern is that folks try them out in 3.9, get frustrated by the limitations, and then there is a mental barrier to trying them again in the future when the limitations have been reduced or eliminated. However, I don't think that will be a problem for people that would benefit from serious use of subinterpreters. Furthermore, if we're clear that "for now" subinterpreters still share the GIL in 3.9 then I'm not so worried about folks getting confused. In fact, as long as we're clear about all the current limitations then there isn't much downside to option #1. Marking the module as "provisional" may help communicate that there are limitations that impact use. [View Less]

16 60

Comments on PEP 554 (Multiple Interpreters in the Stdlib)
by Mark Shannon April 29, 2020

April 29, 2020

Hi, I'm generally in favour of PEP 554, but I don't think it is ready to be accepted in its current form. My main objection is that without per-subinterpeter GILs (SILs?) PEP 554 provides no value over threading or multi-processing. Multi-processing provides true parallelism and threads provide shared memory concurrency. If per-subinterpeter GILs are possible then, and only then, sub-interpreters will provide true parallelism and (limited) shared memory concurrency. The problem is that … [View More]we don't know whether we can implement per-subinterpeter GILs without too large a negative performance impact. I think we can, but we can't say so for certain. So, IMO, we should not accept PEP 554 until we know for sure that per-subinterpeter GILs can be implemented efficiently. Detailed critique ----------------- I don't see how `list_all()` can be both safe and accurate. The Java equivalent makes no guarantees of accuracy. Attempting to lock the list is likely to lead to deadlock and not locking it will lead to races; potentially dangerous ones. I think it would be best to drop this. `list_all_channels()`. See `list_all()` above. `.destroy()` is either misleading or unsafe. What does this do? >>> is.destroy() >>> is.run() If `run()` raises an exception then the interpreter must exist. Rename to `close()` perhaps? `Channel.interpreters` see `list_all()` and `list_all_channels()` above. How does `is_shareable()` work? Are you proposing some mechanism to transfer an object from one sub-interpreter to another? How would that work? If objects are not shared but serialized, why not use marshal or pickle instead of inventing a third serialization protocol? It would be clearer if channels only dealt with simple, contiguous binary data. As it stands the PEP doesn't state what form the received object will take. Once channels supporting the transfer of bytes objects work, then it is simple to pass more complex objects using pickle or marshal. Channels need a more detailed description of their lifespan. Ideally a state machine. For example: How does an interpreter detach from the receiving end of a channel that is never empty? What happens if an interpreter deletes the last reference to a non-empty channel? On the receiving end, or on the sending end? Isn't the lack of buffering in channels a recipe for deadlocks? What is the mechanism for reliably copying exceptions from one sub-interpreter to another in the `run()` method? If `run()` can raise an exception, why not let it return values? Cheers, Mark. [View Less]

5 8

[RELEASE] Python 3.9.0a6 is now available for testing
by Łukasz Langa April 29, 2020

April 29, 2020

On behalf of the entire Python development community, and the currently serving Python release team in particular, I’m pleased to announce the release of Python 3.9.0a6. Get it here: https://www.python.org/downloads/release/python-390a6/ <https://www.python.org/downloads/release/python-390a6/> This is an early developer preview of Python 3.9 Python 3.9 is still in development. This release, 3.9.0a6, is the last out of six planned alpha releases. Alpha releases are intended to make it … [View More]easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2020-05-18) and, if necessary, may be modified or deleted up until the release candidate phase (2020-08-10). Please keep in mind that this is a preview release and its use is not recommended for production environments. Major new features of the 3.9 series, compared to 3.8 Many new features for Python 3.9 are still being planned and written. Among the new major new features and changes so far: PEP 584 <https://www.python.org/dev/peps/pep-0584/>, Union Operators in dict PEP 585 <https://www.python.org/dev/peps/pep-0585/>, Type Hinting Generics In Standard Collections PEP 593 <https://www.python.org/dev/peps/pep-0593/>, Flexible function and variable annotations PEP 602 <https://www.python.org/dev/peps/pep-0602/>, Python adopts a stable annual release cadence PEP 616 <https://www.python.org/dev/peps/pep-0616/>, String methods to remove prefixes and suffixes PEP 617 <https://www.python.org/dev/peps/pep-0617/>, New PEG parser for CPython BPO 38379 <https://bugs.python.org/issue38379>, garbage collection does not block on resurrected objects; BPO 38692 <https://bugs.python.org/issue38692>, os.pidfd_open added that allows process management without races and signals; BPO 39926 <https://bugs.python.org/issue39926>, Unicode support updated to version 13.0.0 BPO 1635741 <https://bugs.python.org/issue1635741>, when Python is initialized multiple times in the same process, it does not leak memory anymore A number of Python builtins (range, tuple, set, frozenset, list) are now sped up using PEP 590 <https://www.python.org/dev/peps/pep-0590> vectorcall A number of standard library modules (audioop, ast, grp, _hashlib, pwd, _posixsubprocess, random, select, struct, termios, zlib) are now using the stable ABI defined by PEP 384 <https://www.python.org/dev/peps/pep-0384/>. (Hey, fellow core developer, if a feature you find important is missing from this list, let Łukasz know <mailto:lukasz@python.org>.) The next pre-release, the first beta release of Python 3.9, will be 3.9.0b1. It is currently scheduled for 2020-05-18. Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Łukasz Langa @ambv <https://discuss.python.org/u/ambv> [View Less]

5 5

killing static types (for sub-interpreters?)
by Jim J. Jewett April 29, 2020

April 29, 2020

Why do sub-interpreters require (separate and) heap-allocated types? It seems types that are statically allocated are a pretty good use for immortal objects, where you never change the refcount ... and then I don't see why you need more than one copy.

4 5

Announcement: pip 20.1b1 beta release
by Sumana Harihareswara April 28, 2020

April 28, 2020

On behalf of the PyPA, I am pleased to announce that a beta release of pip, pip 20.1b1, has been released. I hope you will help us by testing this and checking for new bugs in how pip deals with complicated projects and dependencies. The highlights for this release are: * Significant speedups when building local directories, by changing behavior to perform in-place builds, instead of copying to temporary directories. * Significant speedups in `pip list --outdated`, by parallelizing network … [View More]

3 6

Improvement to SimpleNamespace
by Raymond Hettinger April 28, 2020

April 28, 2020

SimpleNamespace() is really good at giving attribute style-access. I would like to make that functionality available to the JSON module (or just about anything else that accepts a custom dict) by adding the magic methods for mappings so that this works: catalog = json.load(f, object_hook=SimpleNamespace) print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity']) # currently possible with dict() print(catalog.clothing.mens.shoes.extra_wide.quantity]) … [View More]

14 19

Request For Review: Add support for CAN_J1939 sockets (bpo-40291)
by Karl Ding April 28, 2020

April 28, 2020

Hi all, Could someone take a look at the following PR to add support for CAN_J1939 to the socket module? I'd like to try landing this for 3.9. This enhancement would be useful for anyone working in automotive and/or dealing with the SAE J1939 CAN protocol. This feature is available on Linux 5.4+ (Ubuntu 20.04 LTS ships with a compatible kernel). PR Link: https://github.com/python/cpython/pull/19538 BPO Link: https://bugs.python.org/issue40291 You may find the following links useful if you … [View More]

2 1

Collecting questions for a Steering Council Q&A
by Victor Stinner April 27, 2020

April 27, 2020

Copy of Brett Cannon's message: https://discuss.python.org/t/collecting-questions-for-a-steering-council-q-… --- As part of PyCon US 2020 (*), we are planning to record a Q&A with the Steering Council. We are using this topic to solicit questions from the community. We plan to leave this open for about a week. Thanks for all of those who leave a question (which adhere to the PSF Code of Conduct )! P.S. Do make sure to check out the various talks that have been posted for the conference … [View More]

1 0

Summary of Python tracker Issues
by Python tracker April 24, 2020

April 24, 2020

ACTIVITY SUMMARY (2020-04-17 - 2020-04-24) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7452 ( +9) closed 44675 (+57) total 52127 (+66) Open issues with patches: 2942 Issues opened (50) ================== #34037: asyncio: BaseEventLoop.close() shutdowns the executor without https://bugs.python.org/issue34037 reopened by vstinner #40286: Add … [View More]

1 0

Accepted PEP 615: Support for the IANA Time Zone Database in the Standard Library
by Barry Warsaw April 21, 2020

April 21, 2020

The Python Steering Council accepts PEP 615 - Support for the IANA Time Zone Database in the Standard Library: https://www.python.org/dev/peps/pep-0615/ Congratulations Paul Ganssle! This is a fantastic, well written PEP, and we appreciate Paul’s engagement with the SC to clear up our last questions and concerns. We look forward to its availability in Python 3.9. Thank you Paul, and everyone who helped contribute to this important new module. Cheers, -Barry (on behalf of the Python Steering Council)

4 3

[RELEASE] Python 2.7.18, the end of an era
by Benjamin Peterson April 21, 2020

April 21, 2020

I'm eudaemonic to announce the immediate availability of Python 2.7.18. Python 2.7.18 is a special release. I refer, of course, to the fact that "2.7.18" is the closest any Python version number will ever approximate e, Euler's number. Simply exquisite! A less transcendent property of Python 2.7.18 is that it is the last Python 2.7 release and therefore the last Python 2 release. It's time for the CPython community to say a fond but firm farewell to Python 2. Users still on Python 2 can use e … [View More]to compute the instantaneously compounding interest on their technical debt. Download this unique, commemorative Python release on python.org: https://www.python.org/downloads/release/python-2718/ Python 2.7 has been under active development since the release of Python 2.6, more than 11 years ago. Over all those years, CPython's core developers and contributors sedulously applied bug fixes to the 2.7 branch, no small task as the Python 2 and 3 branches diverged. There were large changes midway through Python 2.7's life such as PEP 466's feature backports to the ssl module and hash randomization. Traditionally, these features would never have been added to a branch in maintenance mode, but exceptions were made to keep Python 2 users secure. Thank you to CPython's community for such dedication. Python 2.7 was lucky to have the services of two generations of binary builders and operating system experts, Martin von Löwis and Steve Dower for Windows, and Ronald Oussoren and Ned Deily for macOS. The reason we provided binary Python 2.7 releases for macOS 10.9, an operating system obsoleted by Apple 4 years ago, or why the "Microsoft Visual C++ Compiler for Python 2.7" exists is the dedication of these individuals. I thank the past and present Python release managers, Barry Warsaw, Ned Deily, Georg Brandl, Larry Hastings, and Łukasz Langa for their advice and support over the years. I've learned a lot from them—like don't be the sucker who volunteers to manage the release right before a big compatibility break! Python 3 would be nowhere without the critical work of the wider community. Library maintainers followed CPython by maintaining Python 2 support for many years but also threw their weight behind the Python 3 statement (https://python3statement.org). Linux distributors chased Python 2 out of their archives. Users migrated hundreds of millions of lines of code, developed porting guides, and kept Python 2 in their brain while Python 3 gained 10 years of improvements. Finally, thank you to GvR for creating Python 0.9, 1, 2, and 3. Long live Python 3+! Signing off, Benjamin 2.7 release manager [View Less]

4 3

Re: Accepting PEP 617: New PEG parser for CPython
by Raymond Hettinger April 20, 2020

April 20, 2020

This will be a nice improvement. Raymond

1 0

Re: PEP 616 "String methods to remove prefixes and suffixes" accepted
by Raymond Hettinger April 20, 2020

April 20, 2020

Please consider adding underscores to the names: remove_prefix() and remove_suffix(). The latter method causes a mental hiccup when first read as removes-uffix, forcing mental backtracking to get to remove-suffix. We had a similar problem with addinfourl initially being read as add-in-four-l before mentally backtracking to add-info-url. The PEP says this alternative was considered, but I disagree with the rationale given in the PEP. The reason that "startswith" and "endswith" don't have … [View More]

1 0

PEP 616 "String methods to remove prefixes and suffixes" accepted
by Victor Stinner April 20, 2020

April 20, 2020

Hi, The Python Steering Council accepts the PEP 616 "String methods to remove prefixes and suffixes": https://www.python.org/dev/peps/pep-0616/ Congrats Dennis Sweeney! We just have one last request: we expect the documentation to explain well the difference between removeprefix()/removesuffix() and lstrip()/strip()/rstrip(), since it is the rationale of the PEP ;-) You can find the WIP implementation at: * https://github.com/python/cpython/pull/18939 * https://bugs.python.org/issue39939 … [View More]

3 2

Accepting PEP 617: New PEG parser for CPython
by Brett Cannon April 20, 2020

April 20, 2020

The steering council is happy to announce that we have accepted PEP 617! Thanks to the PEP authors for all their hard work (which includes sending a PR to update the acceptance of the PEP 😉).

2 1

Further PEP 615 Discussion: Equality and hash of ZoneInfo
by Paul Ganssle April 20, 2020

April 20, 2020

A few weeks ago, I submitted PEP 615 to the steering council <https://github.com/python/steering-council/issues/22> for a decision. There's been a decent amount of discussion there with some very good questions. I think they are mostly resolved (though I'm happy to have other people look over the logic of some of my responses there), but Victor brought up the question of the equality and hash behavior of ZoneInfo, and while I'm leaning one way, I could easily be convinced - particularly … [View More]

3 3

Summary of Python tracker Issues
by Python tracker April 17, 2020

April 17, 2020

ACTIVITY SUMMARY (2020-04-10 - 2020-04-17) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7443 (+10) closed 44618 (+55) total 52061 (+65) Open issues with patches: 2932 Issues opened (40) ================== #25680: Selector.select() hangs when there is nothing to select https://bugs.python.org/issue25680 reopened by gvanrossum #40247: Logged out … [View More]

1 0

The Python 2 death march
by Benjamin Peterson April 16, 2020

April 16, 2020

Hi all, It's finally time to schedule the last releases in Python 2's life. There will be two more releases of Python 2.7: Python 2.7.17 and Python 2.7.18. Python 2.7.17 release candidate 1 will happen on October 5th followed by the final release on October 19th. I'm going to time Python 2.7.18 to coincide with PyCon 2020 in April, so attendees can enjoy some collective catharsis. We'll still say January 1st is the official EOL date. Thanks to Sumana Harihareswara, there's now a FAQ about … [View More]

24 38

Re: Improvement to SimpleNamespace
by Raymond Hettinger April 15, 2020

April 15, 2020

[Serhiy] > As a workaround you can use > > object_hook=lambda x: SimpleNamespace(**x) That doesn't suffice because some valid JSON keys are not valid identifiers. You still need a way to get past those when they arise: catalog.books.fiction['Paradise Lost'].isbn Also, it still leaves you with using setattr(ns, attrname, attrvalue) or tricks with vars() when doing updates. The AttrDict recipe is popular for a reason. Raymond

3 2

PEP: Modify the C API to hide implementation details
by Victor Stinner April 15, 2020

April 15, 2020

Hi, Here is a first draft a PEP which summarize the research work I'm doing on CPython C API since 2017 and the changes that me and others already made since Python 3.7 towards an "opaque" C API. The PEP is also a collaboration with developers of PyPy, HPy, Rust-CPython and many others! Thanks to everyone who helped me to write it down! Maybe this big document should be reorganized as multiple smaller better defined goals: as multiple PEPs. The PEP is quite long and talks about things which … [View More]

16 43

Providing fix for modulefinder.py regression on Windows
by Barry Scott April 15, 2020

April 15, 2020

modulefinder.py does not open source files in "rb" which prevents compile() from applying the encoding rules. This first showed up for me on Windows with Python 3.8. I'd like to provide patches for 3.8 (if any more releases) and 3.9. What is the process I should follow? I have signed the Contributor Agreement. Here is my test case and the results on Windows with 3.7. import modulefinder with open('import_functools.py', 'w') as f: f.write('import functools\n') mf = modulefinder.… [View More]

5 6

Do we need port some extension modules to the modules directory?
by Hai Shi April 13, 2020

April 13, 2020

Hi, I notice some modules not in modules directory(for example: _warnings、marshal in python directory). Do we need port those modules to modules directory? Sorry for my noisy info. Bests Wishs, Hai Shi

3 4

Python2 as 𝑣 → 𝑒
by Mike Miller April 12, 2020

April 12, 2020

Unless I've read something wrong, it looks like the final Python 2 release (2.7.18) should approximate the math constant e: >>> import math >>> math.e 2.718281828459045 Aka: Python as 𝑣 → 𝑒 Python ≈ 𝑒 Would be fun to note that in the announcement/docs somehow. -Mike

4 3

Fixing Copy on Writes with Immortal Instances
by Eddie Elizondo April 11, 2020

April 11, 2020

Copy on writes are a big problem in large Python application that rely on multiple processes sharing the same memory. I just sent out a patch introduces Immortal Instances which allows the user to bypass reference counting on specific objects to avoid CoW on forked processes. This has helped in reducing the memory constraints in some of our applications. The PR has a more in-depth explanation about the change. Looking forward to get your thoughts. Github PR: https://github.com/python/cpython/… [View More]

1 0

Summary of Python tracker Issues
by Python tracker April 10, 2020

April 10, 2020

ACTIVITY SUMMARY (2020-04-03 - 2020-04-10) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7433 (+32) closed 44563 (+37) total 51996 (+69) Open issues with patches: 2934 Issues opened (50) ================== #39984: Move pending calls from _PyRuntime to PyInterpreterState https://bugs.python.org/issue39984 reopened by vstinner #40178: Convert the … [View More]

1 0

Need help with test_ctypes failing on Windows (test_load_dll_with_flags)
by Guido van Rossum April 7, 2020

April 7, 2020

I have a large PR (https://github.com/python/cpython/pull/18239, for PEP 585) that's failing in the Azures pipeline on Win32 and Win64 only. My trusty assistant who has a Windows laptop couldn't reproduce the failure. Can I buy a hint from someone? Steve? The relevant failure output is: ====================================================================== ERROR: test_load_dll_with_flags (ctypes.test.test_loading.LoaderTest) [WinDLL('_sqlite3.dll', winmode=0)] ---------------------------------… [View More]

4 5

Problem with instantiating a C extension class from another class
by Musbur April 6, 2020

April 6, 2020

Hello, (I've asked this question before on python-list but only got scarce and ultimately unhelpful answers. Although this isn't about Python development itself, I'm assuming that there is more CPython knowledge on this list than on the other, so please bear with me.) I've written a C extension module, each defining a class (called Series and CLHist, respectively). I can import both modules in Python and use the classes. Everything works fine, also according to valgrind. Now I want to return … [View More]a Series instance directly from a CLHist method, and that immediately crashes Python with a segfault. Here's a bit of code and just the _new, _init, and _finalize methods which I've sprinkled with debugging code. When I instantiate a Series object from Python, I see output from all three methods. When I call pyseries_new() from C, none of the three functions are called and it crashes. At some point I thought maybe I should call PyType_Ready() from the C side before instantiating Series. In a "standard" (system) Python installation, it printed the desired debugging output and crashed later. Under a debugging Python installation I built it outputs nothing except an interesting post-mortem "Fatal Python error: UNREF invalid object" (see below). Here's my C code with the boilerplate stuff left out. typedef struct { PyObject_HEAD struct clh_series *series; } Series; static PyObject *Series_new(PyTypeObject *type, PyObject *args, PyObject *kw) { Series *self; self = (Series *) type->tp_alloc(type, 0); fprintf(stderr, "Series_new(%p)\n", self); self->series = NULL; return (PyObject*)self; } static int Series_init(Series *self, PyObject *args, PyObject *kw) { fprintf(stderr, "Series_init(%p)\n", self); self->series = NULL; return 0; } static void Series_finalize(PyObject *self) { fprintf(stderr, "Series_finalize(%p)\n", self); clh_series_free(((Series*)self)->series); } /* To create a new Series object directly from C */ PyObject *pyseries_new(struct clh_series *series) { Series *pyseries; pyseries = PyObject_New(Series, &series_type); PyObject_Init((PyObject *)pyseries, &series_type); pyseries->series = series; return (PyObject *) pyseries; } Here's the debugging output: * ob object : <refcnt 0 at 0x7f1a4e2f97a8> type : tuple refcount: 0 address : 0x7f1a4e2f97a8 * op->_ob_prev->_ob_next object : <_Series object at 0x7f1a4e2ee9d0> type : _Series refcount: 1 address : 0x7f1a4e2ee9d0 * op->_ob_next->_ob_prev object : <refcnt 0 at 0x7f1a4e2f97a8> type : tuple refcount: 0 address : 0x7f1a4e2f97a8 Fatal Python error: UNREF invalid object [View Less]

3 3

Question about bytecode stability
by Jonathan Goble April 6, 2020

April 6, 2020

I'm getting ready to get back into an old personal project with some new ideas. Before I do, I need to ask a question regarding the stability of CPython bytecode. Before you say it, I am well aware and fully understand that the opcodes may change in value and semantics between 3.x and 3.(x+1) without notice, and that's acceptable to me. My question is, are the opcodes guaranteed stable across the lifetime of a single 3.x release? In other words, are they guaranteed to not change values or … [View More]

5 6

[RELEASE] Python 2.7.18 release candidate 1
by Benjamin Peterson April 6, 2020

April 6, 2020

Greetings, 2.7.18 release candidate 1, a testing release for the last release of the Python 2.7 series, is now available for download. The CPython core developers stopped applying routine bugfixes to the 2.7 branch on January 1. 2.7.18 will includes fixes that were made between the release of 2.7.17 and the end of 2019. A final—very final!—release is expected in 2 weeks. Downloads are at: https://www.python.org/downloads/release/python-2718rc1/ The full changelog is at https://raw.… [View More]

1 0

How to enable tracemalloc for the test suite?
by Skip Montanaro April 6, 2020

April 6, 2020

I've got a memory issue in my modified Python interpreter I'm trying to debug. Output at the end of the problematic unit test looks like this: ... == Tests result: FAILURE then SUCCESS == 1 test OK. 1 re-run test: test_rattlesnake Total duration: 2.9 sec Tests result: FAILURE then SUCCESS Debug memory block at address p=0x55a227969080: API '' 0 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd): at p-7: 0x00 *** OUCH at p-6: 0x00 **… [View More]

3 8

Any thoughts about a control flow optimizer for CPython?
by joannah nanjekye April 6, 2020

April 6, 2020

Hey all, From my CS theory, a control flow graph models a program flow and one of its main characteristics is it has one entry and exit point. IIRC, CPython’s compilation process involves generation of a control flow graph. Contrary to peephole optimizations, optimizations on the control flow graph are more global allowing us to have complex and global optimizations like branch and checkpoint eliminations etc. I have seen several implementations of control flow optimizations. The one I am … [View More]

4 7

Set PyEval_SetTrace to method from class
by Leandro Müller April 5, 2020

April 5, 2020

Hello. I need to set a PyEval_SetTrace with a new class instance. All examples use simple function. PyEval_SetTrace(trace_trampoline, obj); Following my problem: I start three threads, but I need to check trace C just one thread. If I run a while on C trace on thread, it stops all threads. So, I've been thinking to run Trace with a new instance class C. Any suggestions? Att. Leandro Müller

2 1

Summary of Python tracker Issues
by Python tracker April 3, 2020

April 3, 2020

ACTIVITY SUMMARY (2020-03-27 - 2020-04-03) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7401 (+44) closed 44526 (+40) total 51927 (+84) Open issues with patches: 2909 Issues opened (65) ================== #23574: datetime: support leap seconds https://bugs.python.org/issue23574 reopened by vstinner #35455: Solaris thread_time doesn't work with … [View More]

1 0

PEP 616 -- String methods to remove prefixes and suffixes
by Dennis Sweeney April 2, 2020

April 2, 2020

Browser Link: https://www.python.org/dev/peps/pep-0616/ PEP: 616 Title: String methods to remove prefixes and suffixes Author: Dennis Sweeney <sweeney.dennis650(a)gmail.com> Sponsor: Eric V. Smith <eric(a)trueblade.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 19-Mar-2020 Python-Version: 3.9 Post-History: 30-Aug-2002 Abstract ======== This is a proposal to add two new methods, ``cutprefix`` and ``cutsuffix``, to the APIs of Python's various string … [View More]objects. In particular, the methods would be added to Unicode ``str`` objects, binary ``bytes`` and ``bytearray`` objects, and ``collections.UserString``. If ``s`` is one these objects, and ``s`` has ``pre`` as a prefix, then ``s.cutprefix(pre)`` returns a copy of ``s`` in which that prefix has been removed. If ``s`` does not have ``pre`` as a prefix, an unchanged copy of ``s`` is returned. In summary, ``s.cutprefix(pre)`` is roughly equivalent to ``s[len(pre):] if s.startswith(pre) else s``. The behavior of ``cutsuffix`` is analogous: ``s.cutsuffix(suf)`` is roughly equivalent to ``s[:-len(suf)] if suf and s.endswith(suf) else s``. Rationale ========= There have been repeated issues [#confusion]_ on the Bug Tracker and StackOverflow related to user confusion about the existing ``str.lstrip`` and ``str.rstrip`` methods. These users are typically expecting the behavior of ``cutprefix`` and ``cutsuffix``, but they are surprised that the parameter for ``lstrip`` is interpreted as a set of characters, not a substring. This repeated issue is evidence that these methods are useful, and the new methods allow a cleaner redirection of users to the desired behavior. As another testimonial for the usefulness of these methods, several users on Python-Ideas [#pyid]_ reported frequently including similar functions in their own code for productivity. The implementation often contained subtle mistakes regarding the handling of the empty string (see `Specification`_). Specification ============= The builtin ``str`` class will gain two new methods with roughly the following behavior:: def cutprefix(self: str, pre: str, /) -> str: if self.startswith(pre): return self[len(pre):] return self[:] def cutsuffix(self: str, suf: str, /) -> str: if suf and self.endswith(suf): return self[:-len(suf)] return self[:] The only difference between the real implementation and the above is that, as with other string methods like ``replace``, the methods will raise a ``TypeError`` if any of ``self``, ``pre`` or ``suf`` is not an instace of ``str``, and will cast subclasses of ``str`` to builtin ``str`` objects. Note that without the check for the truthyness of ``suf``, ``s.cutsuffix('')`` would be mishandled and always return the empty string due to the unintended evaluation of ``self[:-0]``. Methods with the corresponding semantics will be added to the builtin ``bytes`` and ``bytearray`` objects. If ``b`` is either a ``bytes`` or ``bytearray`` object, then ``b.cutsuffix()`` and ``b.cutprefix()`` will accept any bytes-like object as an argument. Note that the ``bytearray`` methods return a copy of ``self``; they do not operate in place. The following behavior is considered a CPython implementation detail, but is not guaranteed by this specification:: >>> x = 'foobar' * 10**6 >>> x.cutprefix('baz') is x is x.cutsuffix('baz') True >>> x.cutprefix('') is x is x.cutsuffix('') True That is, for CPython's immutable ``str`` and ``bytes`` objects, the methods return the original object when the affix is not found or if the affix is empty. Because these types test for equality using shortcuts for identity and length, the following equivalent expressions are evaluated at approximately the same speed, for any ``str`` objects (or ``bytes`` objects) ``x`` and ``y``:: >>> (True, x[len(y):]) if x.startswith(y) else (False, x) >>> (True, z) if x != (z := x.cutprefix(y)) else (False, x) The two methods will also be added to ``collections.UserString``, where they rely on the implementation of the new ``str`` methods. Motivating examples from the Python standard library ==================================================== The examples below demonstrate how the proposed methods can make code one or more of the following: Less fragile: The code will not depend on the user to count the length of a literal. More performant: The code does not require a call to the Python built-in ``len`` function. More descriptive: The methods give a higher-level API for code readability, as opposed to the traditional method of string slicing. refactor.py ----------- - Current:: if fix_name.startswith(self.FILE_PREFIX): fix_name = fix_name[len(self.FILE_PREFIX):] - Improved:: fix_name = fix_name.cutprefix(self.FILE_PREFIX) c_annotations.py: ----------------- - Current:: if name.startswith("c."): name = name[2:] - Improved:: name = name.cutprefix("c.") find_recursionlimit.py ---------------------- - Current:: if test_func_name.startswith("test_"): print(test_func_name[5:]) else: print(test_func_name) - Improved:: print(test_finc_name.cutprefix("test_")) deccheck.py ----------- This is an interesting case because the author chose to use the ``str.replace`` method in a situation where only a prefix was intended to be removed. - Current:: if funcname.startswith("context."): self.funcname = funcname.replace("context.", "") self.contextfunc = True else: self.funcname = funcname self.contextfunc = False - Improved:: if funcname.startswith("context."): self.funcname = funcname.cutprefix("context.") self.contextfunc = True else: self.funcname = funcname self.contextfunc = False - Arguably further improved:: self.contextfunc = funcname.startswith("context.") self.funcname = funcname.cutprefix("context.") test_i18n.py ------------ - Current:: if test_func_name.startswith("test_"): print(test_func_name[5:]) else: print(test_func_name) - Improved:: print(test_finc_name.cutprefix("test_")) - Current:: if creationDate.endswith('\\n'): creationDate = creationDate[:-len('\\n')] - Improved:: creationDate = creationDate.cutsuffix('\\n') shared_memory.py ---------------- - Current:: reported_name = self._name if _USE_POSIX and self._prepend_leading_slash: if self._name.startswith("/"): reported_name = self._name[1:] return reported_name - Improved:: if _USE_POSIX and self._prepend_leading_slash: return self._name.cutprefix("/") return self._name build-installer.py ------------------ - Current:: if archiveName.endswith('.tar.gz'): retval = os.path.basename(archiveName[:-7]) if ((retval.startswith('tcl') or retval.startswith('tk')) and retval.endswith('-src')): retval = retval[:-4] - Improved:: if archiveName.endswith('.tar.gz'): retval = os.path.basename(archiveName[:-7]) if retval.startswith(('tcl', 'tk')): retval = retval.cutsuffix('-src') Depending on personal style, ``archiveName[:-7]`` could also be changed to ``archiveName.cutsuffix('.tar.gz')``. test_core.py ------------ - Current:: if output.endswith("\n"): output = output[:-1] - Improved:: output = output.cutsuffix("\n") cookiejar.py ------------ - Current:: def strip_quotes(text): if text.startswith('"'): text = text[1:] if text.endswith('"'): text = text[:-1] return text - Improved:: def strip_quotes(text): return text.cutprefix('"').cutsuffix('"') - Current:: if line.endswith("\n"): line = line[:-1] - Improved:: line = line.cutsuffix("\n") fixdiv.py --------- - Current:: def chop(line): if line.endswith("\n"): return line[:-1] else: return line - Improved:: def chop(line): return line.cutsuffix("\n") test_concurrent_futures.py -------------------------- In the following example, the meaning of the code changes slightly, but in context, it behaves the same. - Current:: if name.endswith(('Mixin', 'Tests')): return name[:-5] elif name.endswith('Test'): return name[:-4] else: return name - Improved:: return name.cutsuffix('Mixin').cutsuffix('Tests').cutsuffix('Test') msvc9compiler.py ---------------- - Current:: if value.endswith(os.pathsep): value = value[:-1] - Improved:: value = value.cutsuffix(os.pathsep) test_pathlib.py --------------- - Current:: self.assertTrue(r.startswith(clsname + '('), r) self.assertTrue(r.endswith(')'), r) inner = r[len(clsname) + 1 : -1] - Improved:: self.assertTrue(r.startswith(clsname + '('), r) self.assertTrue(r.endswith(')'), r) inner = r.cutprefix(clsname + '(').cutsuffix(')') Rejected Ideas ============== Expand the lstrip and rstrip APIs --------------------------------- Because ``lstrip`` takes a string as its argument, it could be viewed as taking an iterable of length-1 strings. The API could therefore be generalized to accept any iterable of strings, which would be successively removed as prefixes. While this behavior would be consistent, it would not be obvious for users to have to call ``'foobar'.cutprefix(('foo,))`` for the common use case of a single prefix. Allow multiple prefixes ----------------------- Some users discussed the desire to be able to remove multiple prefixes, calling, for example, ``s.cutprefix('From: ', 'CC: ')``. However, this adds ambiguity about the order in which the prefixes are removed, especially in cases like ``s.cutprefix('Foo', 'FooBar')``. After this proposal, this can be spelled explicitly as ``s.cutprefix('Foo').cutprefix('FooBar')``. Remove multiple copies of a prefix ---------------------------------- This is the behavior that would be consistent with the aforementioned expansion of the ``lstrip/rstrip`` API -- repeatedly applying the function until the argument is unchanged. This behavior is attainable from the proposed behavior via the following:: >>> s = 'foo' * 100 + 'bar' >>> while s != (s := s.cutprefix("foo")): pass >>> s 'bar' The above can be modififed by chaining multiple ``cutprefix`` calls together to achieve the full behavior of the ``lstrip``/``rstrip`` generalization, while being explicit in the order of removal. While the proposed API could later be extended to include some of these use cases, to do so before any observation of how these methods are used in practice would be premature and may lead to choosing the wrong behavior. Raising an exception when not found ----------------------------------- There was a suggestion that ``s.cutprefix(pre)`` should raise an exception if ``not s.startswith(pre)``. However, this does not match with the behavior and feel of other string methods. There could be ``required=False`` keyword added, but this violates the KISS principle. Alternative Method Names ------------------------ Several alternatives method names have been proposed. Some are listed below, along with commentary for why they should be rejected in favor of ``cutprefix`` (the same arguments hold for ``cutsuffix``) ``ltrim`` "Trim" does in other languages (e.g. JavaScript, Java, Go, PHP) what ``strip`` methods do in Python. ``lstrip(string=...)`` This would avoid adding a new method, but for different behavior, it's better to have two different methods than one method with a keyword argument that select the behavior. ``cut_prefix`` All of the other methods of the string API, e.g. ``str.startswith()``, use ``lowercase`` rather than ``lower_case_with_underscores``. ``cutleft``, ``leftcut``, or ``lcut`` The explicitness of "prefix" is preferred. ``removeprefix``, ``deleteprefix``, ``withoutprefix``, etc. All of these might have been acceptable, but they have more characters than ``cut``. Some suggested that the verb "cut" implies mutability, but the string API already contains verbs like "replace", "strip", "split", and "swapcase". ``stripprefix`` Users may benefit from the mnemonic that "strip" means working with sets of characters, while other methods work with substrings, so re-using "strip" here should be avoided. Reference Implementation ======================== See the pull request on GitHub [#pr]_. References ========== .. [#pr] GitHub pull request with implementation (https://github.com/python/cpython/pull/18939) .. [#pyid] Discussion on Python-Ideas (https://mail.python.org/archives/list/python-ideas@python.org/thread/RJARZS…) .. [#confusion] Comment listing Bug Tracker and StackOverflow issues (https://mail.python.org/archives/list/python-ideas@python.org/message/GRGAF…) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

33 137

Debug C++
by Leandro Müller April 2, 2020

April 2, 2020

Hi Are there any away to debug C module during python debug? Att. Leandro Müller

4 3

Moving threadstate to thread-local storage.
by Mark Shannon April 2, 2020

April 2, 2020

Hi, As an experiment, I thought I would try moving the thread state (what you get from _PyThreadState_GET() ) to TLS. https://github.com/python/cpython/compare/master...markshannon:threadstate_… It works, passing all the tests, and seems sound. It is a small patch (< 50 lines) and doesn't increase the overall code size. My branch is GCC/Clang only, so will need a bit of extra code for Windows. It should only need a few more lines; I haven't done it as I don't have a Windows machine … [View More]

3 4

Re: Python-Dev Digest, Vol 201, Issue 1
by Ilya Kamenshchikov April 2, 2020

April 2, 2020

One potentially serious question: what should `enumerate.__reversed__` > do when given a starting value? > reversed(enumerate('abc', 1)) > Should that yield...? > # treat the start value as a start value > (1, 'c'), (0, 'b'), (-1, 'a') > # treat the start value as an end value > (3, 'c'), (2, 'b'), (1, 'a') > Something else? > My preference would be to treat the starting value as an ending value. My idea is largely that as long as any iterable … [View More]has defined length and order, it should be reversible, and this should yield the same as if I did this reverse countdown with some pointer. It just so happens that `enumerate` depends fully in this regard on the iterable it enumerates, if it has defined length, then enumerate will also have defined length. Maybe enumerate should also start defining `len` whenever underlying iterable supports it. Regarding the starting value, I think it is important to keep symmetry: enumerate('abc', 1) is [ (1, 'a'), (2, 'b'), (3, 'c') ], therefore reversed(enumerate('abc', 1)) should be [(3, 'c'), (2, 'b'), (1, 'a')] Best Regards, -- Ilya Kamen On Thu, Apr 2, 2020 at 5:55 AM <python-dev-request(a)python.org> wrote: > Send Python-Dev mailing list submissions to > python-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman3/lists/python-dev.python.org/ > or, via email, send a message with subject or body 'help' to > python-dev-request(a)python.org > > You can reach the person managing the list at > python-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-Dev digest..." > > Today's Topics: > > 1. Re: reversed enumerate (Steven D'Aprano) > 2. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert) > 3. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert) > > > ---------------------------------------------------------------------- > > Date: Thu, 2 Apr 2020 13:20:02 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-Dev] Re: reversed enumerate > To: python-dev(a)python.org > Cc: python-ideas(a)python.org > Message-ID: <20200402022000.GC16830(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Ilya, > > I'm not sure that this mailing list (Python-Dev) is the right place for > this discussion, I think that Python-Ideas (CCed) is the correct place. > > For the benefit of Python-Ideas, I have left your entire post below, to > establish context. > > [Ilya] > > I needed reversed(enumerate(x: list)) in my code, and have discovered > > that it wound't work. This is disappointing because operation is well > > defined. > > It isn't really well-defined, since enumerate can operate on infinite > iterators, and you cannot reverse an infinite stream. Consider: > > def values(): > while True: > yield random.random() > > a, b = reversed(enumerate(values()) > > What should the first pair of (a, b) be? > > However, having said that, I think that your idea is not unreasonable. > `enumerate(it)` in the most general case isn't reversable, but if `it` > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > be too. > > My personal opinion is that this is a fairly obvious and straightforward > enhancement, one which (hopefully!) shouldn't require much, if any, > debate. I don't think we need a new class for this, I think enhancing > enumerate to be reversable if its underlying iterator is reversable > makes good sense. > > But if you can show some concrete use-cases, especially one or two from > the standard library, that would help your case. Or some other languages > which offer this functionality as standard. > > On the other hand, I think that there is a fairly lightweight work > around. Define a helper function: > > def countdown(n): > while True: > yield n > n -= 1 > > then call it like this: > > # reversed(enumerate(seq)) > zip(countdown(len(seq)-1), reversed(seq))) > > So it isn't terribly hard to work around this. But I agree that it would > be nice if enumerate encapsulated this for the caller. > > One potentially serious question: what should `enumerate.__reversed__` > do when given a starting value? > > reversed(enumerate('abc', 1)) > > Should that yield...? > > # treat the start value as a start value > (1, 'c'), (0, 'b'), (-1, 'a') > > # treat the start value as an end value > (3, 'c'), (2, 'b'), (1, 'a') > > Something else? > > My preference would be to treat the starting value as an ending value. > > > Steven > > > On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote: > > Hi, > > > > I needed reversed(enumerate(x: list)) in my code, and have discovered > that > > it wound't work. This is disappointing because operation is well defined. > > It is also well defined for str type, range, and - in principle, but not > > yet in practice - on dictionary iterators - keys(), values(), items() as > > dictionaries are ordered now. > > It would also be well defined on any user type implementing __iter__, > > __len__, __reversed__ - think numpy arrays, some pandas dataframes, > tensors. > > > > That's plenty of usecases, therefore I guess it would be quite useful to > > avoid hacky / inefficient solutions like described here: > > https://code.activestate.com/lists/python-list/706205/. > > > > If deemed useful, I would be interested in implementing this, maybe > > together with __reversed__ on dict keys, values, items. > > > > Best Regards, > > -- > > Ilya Kamen > > > > ----------- > > p.s. > > > > *Sketch* of what I am proposing: > > > > class reversible_enumerate: > > > > def __init__(self, iterable): > > self.iterable = iterable > > self.ctr = 0 > > > > def __iter__(self): > > for e in self.iterable: > > yield self.ctr, e > > self.ctr += 1 > > > > def __reversed__(self): > > try: > > ri = reversed(self.iterable) > > except Exception as e: > > raise Exception( > > "enumerate can only be reversed if iterable to > > enumerate can be reversed and has defined length." > > ) from e > > > > try: > > l = len(self.iterable) > > except Exception as e: > > raise Exception( > > "enumerate can only be reversed if iterable to > > enumerate can be reversed and has defined length." > > ) from e > > > > indexes = range(l-1, -1, -1) > > for i, e in zip(indexes, ri): > > yield i, e > > > > for i, c in reversed(reversible_enumerate("Hello World")): > > print(i, c) > > > > for i, c in reversed(reversible_enumerate([11, 22, 33])): > > > > print(i, c) > > > _______________________________________________ > > Python-Dev mailing list -- python-dev(a)python.org > > To unsubscribe send an email to python-dev-leave(a)python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/NDDKDUD… > > Code of Conduct: http://python.org/psf/codeofconduct/ > > ------------------------------ > > Date: Wed, 1 Apr 2020 20:42:12 -0700 > From: Andrew Barnert <abarnert(a)yahoo.com> > Subject: [Python-Dev] Re: [Python-ideas] Re: reversed enumerate > To: Steven D'Aprano <steve(a)pearwood.info> > Cc: python-dev(a)python.org, python-ideas(a)python.org > Message-ID: <647BE271-D7ED-4E50-A807-3D748B00FF1D(a)yahoo.com> > Content-Type: text/plain; charset=utf-8 > > Before jumping in: > > In many cases, when you want to reverse an enumerate, it’s small and > fixed-sized, so there’s a trivial way to do this: Just store the enumerate > iterator in a tuple, and tuples are reversible. > > for idx, value in reversed(tuple(enumerate(stuff))): > > But of course there are some cases where this isn’t appropriate, like > enumerating a fixed-size but huge input. > > > On Apr 1, 2020, at 19:23, Steven D'Aprano <steve(a)pearwood.info> wrote: > > > > [Ilya] > >> I needed reversed(enumerate(x: list)) in my code, and have discovered > >> that it wound't work. This is disappointing because operation is well > >> defined. > > > > It isn't really well-defined, since enumerate can operate on infinite > > iterators, and you cannot reverse an infinite stream. > > ... > > > However, having said that, I think that your idea is not unreasonable. > > `enumerate(it)` in the most general case isn't reversable, but if `it` > > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > > be too. > > Agreed—but this is just a small piece of a much wider issue. Today, > enumerate is always an Iterator. It’s never reversible. But it’s also not > sized, or subscriptable, or in-testable, even if you give it inputs that > are. And it’s not just enumerate—the same is true for map, filter, zip, > itertools.islice, itertools.dropwhile, etc. > > There’s no reason these things couldn’t all be views, just like the > existing dict views (and other things like memoryview and third-party > things like numpy array slices). In fact, they already are in Swift, and > will be in C++20. > > > My personal opinion is that this is a fairly obvious and straightforward > > enhancement, one which (hopefully!) shouldn't require much, if any, > > debate. I don't think we need a new class for this, I think enhancing > > enumerate to be reversable if its underlying iterator is reversable > > makes good sense. > > Actually, that doesn’t work—it has to be Sized as well. > > More generally, it’s rarely _quite_ as simple as just “views support the > same operations as the things they view”. An enumerate can be a Sequence if > its input is, but a filter can’t. A map with multiple inputs isn’t > Reversible unless they’re all not just Reversible but Sized, although a map > with only one input doesn’t need it to be Sized. And so on. But none of > these things are hard, it’s just a bunch of work to go through all the > input types for all the view types and write up the rules. (Or steal them > from another language or library that already did that work…) > > > But if you can show some concrete use-cases, especially one or two from > > the standard library, that would help your case. Or some other languages > > which offer this functionality as standard. > > Agreed. I don’t think we need to wait until someone designs and writes a > complete viewtools library and submits it for stdlib inclusion before we > can consider adding just one extension to one iterator. But I do think we > want to add the one(s) that are most useful if any, not just whichever ones > people think of first. I’ve personally wanted to reverse a map or a filter > more often than an enumerate, but examples would easily convince me that > that’s just me, and reversing enumerate is more needed. > > > One potentially serious question: what should `enumerate.__reversed__` > > do when given a starting value? > > > > reversed(enumerate('abc', 1)) > > I don’t think this is a problem. When you reversed(tuple(enumerate('abc', > 1))) today, what do you get? You presumably don’t even need to look that up > or try it out. It would be pretty confusing if it were different without > the tuple. > > > ------------------------------ > > Date: Wed, 1 Apr 2020 20:42:12 -0700 > From: Andrew Barnert <abarnert(a)yahoo.com> > Subject: [Python-Dev] Re: [Python-ideas] Re: reversed enumerate > To: Steven D'Aprano <steve(a)pearwood.info> > Cc: python-dev(a)python.org, python-ideas(a)python.org > Message-ID: <647BE271-D7ED-4E50-A807-3D748B00FF1D(a)yahoo.com> > Content-Type: text/plain; charset=utf-8 > > Before jumping in: > > In many cases, when you want to reverse an enumerate, it’s small and > fixed-sized, so there’s a trivial way to do this: Just store the enumerate > iterator in a tuple, and tuples are reversible. > > for idx, value in reversed(tuple(enumerate(stuff))): > > But of course there are some cases where this isn’t appropriate, like > enumerating a fixed-size but huge input. > > > On Apr 1, 2020, at 19:23, Steven D'Aprano <steve(a)pearwood.info> wrote: > > > > [Ilya] > >> I needed reversed(enumerate(x: list)) in my code, and have discovered > >> that it wound't work. This is disappointing because operation is well > >> defined. > > > > It isn't really well-defined, since enumerate can operate on infinite > > iterators, and you cannot reverse an infinite stream. > > ... > > > However, having said that, I think that your idea is not unreasonable. > > `enumerate(it)` in the most general case isn't reversable, but if `it` > > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > > be too. > > Agreed—but this is just a small piece of a much wider issue. Today, > enumerate is always an Iterator. It’s never reversible. But it’s also not > sized, or subscriptable, or in-testable, even if you give it inputs that > are. And it’s not just enumerate—the same is true for map, filter, zip, > itertools.islice, itertools.dropwhile, etc. > > There’s no reason these things couldn’t all be views, just like the > existing dict views (and other things like memoryview and third-party > things like numpy array slices). In fact, they already are in Swift, and > will be in C++20. > > > My personal opinion is that this is a fairly obvious and straightforward > > enhancement, one which (hopefully!) shouldn't require much, if any, > > debate. I don't think we need a new class for this, I think enhancing > > enumerate to be reversable if its underlying iterator is reversable > > makes good sense. > > Actually, that doesn’t work—it has to be Sized as well. > > More generally, it’s rarely _quite_ as simple as just “views support the > same operations as the things they view”. An enumerate can be a Sequence if > its input is, but a filter can’t. A map with multiple inputs isn’t > Reversible unless they’re all not just Reversible but Sized, although a map > with only one input doesn’t need it to be Sized. And so on. But none of > these things are hard, it’s just a bunch of work to go through all the > input types for all the view types and write up the rules. (Or steal them > from another language or library that already did that work…) > > > But if you can show some concrete use-cases, especially one or two from > > the standard library, that would help your case. Or some other languages > > which offer this functionality as standard. > > Agreed. I don’t think we need to wait until someone designs and writes a > complete viewtools library and submits it for stdlib inclusion before we > can consider adding just one extension to one iterator. But I do think we > want to add the one(s) that are most useful if any, not just whichever ones > people think of first. I’ve personally wanted to reverse a map or a filter > more often than an enumerate, but examples would easily convince me that > that’s just me, and reversing enumerate is more needed. > > > One potentially serious question: what should `enumerate.__reversed__` > > do when given a starting value? > > > > reversed(enumerate('abc', 1)) > > I don’t think this is a problem. When you reversed(tuple(enumerate('abc', > 1))) today, what do you get? You presumably don’t even need to look that up > or try it out. It would be pretty confusing if it were different without > the tuple. > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > > ------------------------------ > > End of Python-Dev Digest, Vol 201, Issue 1 > ****************************************** > [View Less]

1 0

Re: reversed enumerate
by Ilya Kamenshchikov April 2, 2020

April 2, 2020

Re: steve(a)pearwood.info > It isn't really well-defined, since enumerate can operate on infinite > iterators, and you cannot reverse an infinite stream. It is well defined on any iterable that itself is reversible and has defined length. For standard types that's lists, strings, dictionary iterators and some of the collections (deque, dictionary-based counter). Re: storchaka(a)gmail.com > Could you please provide evidence that this feature would be quite > useful? How much … [View More]

1 0

reversed enumerate
by Ilya Kamenshchikov April 2, 2020

April 2, 2020

Hi, I needed reversed(enumerate(x: list)) in my code, and have discovered that it wound't work. This is disappointing because operation is well defined. It is also well defined for str type, range, and - in principle, but not yet in practice - on dictionary iterators - keys(), values(), items() as dictionaries are ordered now. It would also be well defined on any user type implementing __iter__, __len__, __reversed__ - think numpy arrays, some pandas dataframes, tensors. That's plenty of … [View More]

5 4