From bitsink at gmail.com Mon Apr 1 00:16:35 2019 From: bitsink at gmail.com (Nam Nguyen) Date: Sun, 31 Mar 2019 21:16:35 -0700 Subject: [Python-ideas] Built-in parsing library In-Reply-To: References:

Message-ID: On Sun, Mar 31, 2019 at 12:13 PM David Mertz wrote: > I just found this nice summary. It's not complete, but it looks well > written. https://tomassetti.me/parsing-in-python/ > > On Sun, Mar 31, 2019, 3:09 PM David Mertz wrote: > >> There are about a half dozen widely used parsing libraries for Python. >> Each one of them takes a dramatically different approach to the defining a >> grammar. Each one has been debugged for over a decade. >> >> While I can imagine proposing one for inclusion in the standard library, >> you'd have to choose one (or write a new one) and explain why that one is >> better for everyone (or at least a better starting point) than all the >> others are. >> > I'm not at that stage, yet. By the way, it still is not clear to me if you think having one in the stdlib is desirable. > You're also have to explain why it needs to be in the standard library >> rather than installed by 'pip install someparser'. >> > Installing a package out of stdlib does not solve the problem that motivated this thread. The libraries included in the stdlib can't use those parsers. Cheers, Nam > >> On Sat, Mar 30, 2019, 1:58 PM Nam Nguyen wrote: >> >>> Hello list, >>> >>> What do you think of a universal parsing library in the stdlib mainly >>> for use by other libraries in the stdlib? >>> >>> Through out the years we have had many issues with protocol parsing. >>> Some have even introduced security bugs. The main cause of these issues is >>> the use of simple regular expressions. >>> >>> Having a universal parsing library in the stdlib would help cut down >>> these issues. Such a library should be minimal yet encompassing, and whole >>> parse trees should be entirely expressible in code. I am thinking of >>> combinatoric parsing as the main candidate that fits this bill. >>> >>> What do you say? >>> >>> Thanks! >>> Nam >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Mon Apr 1 01:07:34 2019 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Mon, 1 Apr 2019 07:07:34 +0200 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> Message-ID: <639C3854-415A-4153-86FE-BD713F3859DE@killingar.net> > On 1 Apr 2019, at 02:23, David Mertz wrote: > >> On Sun, Mar 31, 2019, 8:11 PM Steven D'Aprano wrote: >> Regarding later proposals to add support for multiple affixes, to >> recursively delete the affix repeatedly, and to take an additional >> argument to limit how many affixes will be removed: YAGNI. > > > That's simply not true, and I think it's clearly illustrated by the example I gave a few times. Not just conceivably, but FREQUENTLY I write code to accomplish the effect of the suggested: > > basename = fname.rstrip(('.jpg', '.gif', '.png')) > > I probably do this MORE OFTEN than removing a single suffix. Doing this with a for loop and without_suffix is fine though. Without without_suffix it's suddenly error prone. With a without_suffix that takes a typle it's unclear what happens without reading the code. I think a single string argument is a great sweet spot: avoid the most error prone part and keep the loop in user code. / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Apr 1 01:06:23 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Apr 2019 18:06:23 +1300 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <20190331235950.GJ31406@ando.pearwood.info> References:

Message-ID: We do have a parser generator in the standard library: https://github.com/python/cpython/tree/master/Lib/lib2to3/pgen2 On Sun, Mar 31, 2019 at 9:17 PM Nam Nguyen wrote: > On Sun, Mar 31, 2019 at 12:13 PM David Mertz wrote: > >> I just found this nice summary. It's not complete, but it looks well >> written. https://tomassetti.me/parsing-in-python/ >> >> On Sun, Mar 31, 2019, 3:09 PM David Mertz wrote: >> >>> There are about a half dozen widely used parsing libraries for Python. >>> Each one of them takes a dramatically different approach to the defining a >>> grammar. Each one has been debugged for over a decade. >>> >>> While I can imagine proposing one for inclusion in the standard library, >>> you'd have to choose one (or write a new one) and explain why that one is >>> better for everyone (or at least a better starting point) than all the >>> others are. >>> >> > I'm not at that stage, yet. By the way, it still is not clear to me if you > think having one in the stdlib is desirable. > > >> You're also have to explain why it needs to be in the standard library >>> rather than installed by 'pip install someparser'. >>> >> > Installing a package out of stdlib does not solve the problem that > motivated this thread. The libraries included in the stdlib can't use those > parsers. > > Cheers, > Nam > > >> >>> On Sat, Mar 30, 2019, 1:58 PM Nam Nguyen wrote: >>> >>>> Hello list, >>>> >>>> What do you think of a universal parsing library in the stdlib mainly >>>> for use by other libraries in the stdlib? >>>> >>>> Through out the years we have had many issues with protocol parsing. >>>> Some have even introduced security bugs. The main cause of these issues is >>>> the use of simple regular expressions. >>>> >>>> Having a universal parsing library in the stdlib would help cut down >>>> these issues. Such a library should be minimal yet encompassing, and whole >>>> parse trees should be entirely expressible in code. I am thinking of >>>> combinatoric parsing as the main candidate that fits this bill. >>>> >>>> What do you say? >>>> >>>> Thanks! >>>> Nam >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Mon Apr 1 01:15:15 2019 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Mon, 1 Apr 2019 07:15:15 +0200 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> Message-ID: > On 1 Apr 2019, at 04:58, David Mertz wrote: > > >> Lacking a good set of semantics for removing multiple affixes at once, we shouldn't rush to guess what people want. You don't even know what behaviour YOU want, let alone what the community as a whole needs. > > > This is both dumb and dishonest. There are basically two choices, both completely clear. I think the more obvious one is to treat several prefixes or suffixes as substring class, much as .[rl]strip() does character class. Please don't say "dumb and dishonest". Especially not when you directly follow up by radically redefining what you want. To get the same semantics as strip() it must follow that "foofoobarfoo".without_suffix(("foo", "bar")) == "" / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamtlu at gmail.com Mon Apr 1 01:43:46 2019 From: jamtlu at gmail.com (James Lu) Date: Mon, 1 Apr 2019 01:43:46 -0400 Subject: [Python-ideas] Built-in parsing library In-Reply-To: References:

Message-ID: Stack-based LL(1) push down automata can be implemented by hand, indeed isn?t that that a textmateLanguage file is? There?s also the option of using Iro to generate a tmLanguage. From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Apr 1 02:00:16 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 1 Apr 2019 15:00:16 +0900 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <07170fd4-0195-f2d4-d2d7-77f7272e263b@potatochowder.com> References: <1E2D83BC-326E-4278-A46C-7FE4F7FE2560@getmailspring.com> <669F88F5-3CB2-4B33-A451-621307BDBD0F@killingar.net>

Message-ID: <23713.44193.803088.183109@turnbull.sk.tsukuba.ac.jp> David Mertz writes: > While I can imagine proposing one for inclusion in the standard > library, you'd have to choose one (or write a new one) and explain > why that one is better for everyone (or at least a better starting > point) than all the others are. In principle, no, one just needs to explain why this battery fits most of the toys encountered in practice. That's good enough, and if during discussion somebody shows another one is better on a lot of fronts, sure, do that instead. We should avoid letting the perfect be the enemy of the good (as people keep insisting about str.cutsuffix). Politically, sure, it's almost 100% certain that somebody will object that there's a whole class of cases handled by the PackMule parser that the ShavedYacc parser doesn't handle, and somebody else will point out the opposite, so neither is acceptable. Ignore them, they're both wrong about "acceptable". ;-) > You're also have to explain why it needs to be in the standard > library rather than installed by 'pip install someparser'. Again, the bar isn't so high as "needs". There's a balance of equities, such as people with Python installations restricted by QA or security vetting, applications where you really don't want to spend most of your hour allocated to teaching the feature downloading requirements, and cases where pretty much everybody performs the task frequently (for some value of frequently), vs. costs of maintenance (we generally require that a core developer vouch for someone who volunteers to take responsibility for it for 3-5 years) and effects on complexity of learning Python (usually not great for such a module, since the excess burden on documentation ends up being one line in the TOC and a half-dozen in the index). Yes, Nam should be prepared for pushback on both grounds. Most pressingly, without a specific package being proposed, discussion will just go in circles indefinitely. But a parser generator package is something that's been lurking, waiting for an enthusiastic proponent for a long time. There's a lot of low-level support for it. Maybe it just needs a specific proposal to take off. And maybe it won't. He won't know unless he tries. Steve P.S. Guido mentioned lib2to3.pgen2, which is in the stdlib. But help(pgen2) isn't very helpful, so there's at least some documentation work to be done there. From p.f.moore at gmail.com Mon Apr 1 03:20:03 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 1 Apr 2019 08:20:03 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <20190331235950.GJ31406@ando.pearwood.info> References:

<23709.35761.793748.866930@turnbull.sk.tsukuba.ac.jp> <20190330013736.GU31406@ando.pearwood.info> <23711.45063.758826.321466@turnbull.sk.tsukuba.ac.jp> <20190331044323.GB6059@ando.pearwood.info> <07170fd4-0195-f2d4-d2d7-77f7272e263b@potatochowder.com> <5CA14701.7090504@canterbury.ac.nz> <20190331235950.GJ31406@ando.pearwood.info> Message-ID: On Mon, 1 Apr 2019 at 01:01, Steven D'Aprano wrote: > > On Mon, Apr 01, 2019 at 12:02:25PM +1300, Greg Ewing wrote: > > Dan Sommers wrote: > > > > >without_prefix > > >without_suffix > > > > > >They're a little longer, but IMO "without" helps > > >reenforce the immutability of the underlying string. > > > > We don't seem to worry about that distinction for other > > string methods, such as lstrip and rstrip. > > Perhaps we ought to. In the spirit of today's date, let me propose > renaming existing string methods to be more explicit, e.g.: > > str.new_string_in_uppercase > str.new_string_with_substrings_replaced > str.new_string_filled_to_the_given_length_with_zeroes_on_the_left > str.new_string_with_character_translations_not_natural_language_translations > > The best thing is that there will no longer be any confusion as to > whether you are looking at a Unicode string or a byte-string: > > a = a.new_string_trimmed_on_the_left() > a = a.new_bytes_trimmed_on_the_left() > > *wink* In order to support duck typing can I suggest that we also add bytes.new_string_trimmed_on_the_left() str.new_bytes_trimmed_on_the_left() These will do the obvious thing, so they do not need documenting. Obviously only needed for Python 3.x, as in 2.x people never use variables that may sometimes be strings and sometimes bytes. *wink* (although the joke is obvious, and so should not need documenting :-P) Paul From cspealma at redhat.com Mon Apr 1 08:26:08 2019 From: cspealma at redhat.com (Calvin Spealman) Date: Mon, 1 Apr 2019 08:26:08 -0400 Subject: [Python-ideas] PEP-582 and multiple Python versions Message-ID: While the PEP does show the version number as part of the path to the actual packages, implying support for multiple versions, this doesn't seem to be spelled out in the actual text. Presumably __pypackages__/3.8/ might sit beside __pypackages__/3.9/, etc. to keep future versions capable of installing packages for each version, the way virtualenv today is bound to one version of Python. I'd like to raise a potential edge case that might be a problem, and likely an increasingly common one: users with multiple installations of the *same* version of Python. This is actually a common setup for Windows users who use WSL, Microsoft's Linux-on-Windows solution, as you could have both the Windows and Linux builds of a given Python version installed on the same machine. The currently implied support for multiple versions would not be able to separate these and could create problems if users pip install a Windows binary package through Powershell and then try to run a script in Bash from the same directory, causing the Linux version of Python to try to use Windows python packages. I'm not actually sure what the solution here is. Mostly I wanted to raise the concern, because I'm very keen on WSL being a great entry path for new developers and I want to make that a better experience, not a more confusing one. Maybe that version number could include some other unique identify, maybe based on Python's own executable. A hash maybe? I don't know if anything like that already exists to uniquely identify a Python build or installation. -- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma at redhat.com M: +1.336.210.5107 TRIED. TESTED. TRUSTED. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Mon Apr 1 09:41:28 2019 From: mertz at gnosis.cx (David Mertz) Date: Mon, 1 Apr 2019 09:41:28 -0400 Subject: [Python-ideas] Built-in parsing library In-Reply-To: <23713.44193.803088.183109@turnbull.sk.tsukuba.ac.jp> References:

<23713.44193.803088.183109@turnbull.sk.tsukuba.ac.jp> Message-ID: On Mon, Apr 1, 2019 at 2:16 AM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > In principle, no, one just needs to explain why this battery fits most > of the toys encountered in practice. That's good enough, and if > during discussion somebody shows another one is better on a lot of > fronts, sure, do that instead. OK, I'll acknowledge my comment might have overstated the bar to overcome. A parser added to the standard library doesn't need to be perfect for everyone. But adding to stdlib *does* provide a kind of endorsement of the right default way to go about things. Among the popular third party libraries, we have several very different attitudes towards designing grammars. On the one hand, there are formal differences in the power of different grammars?some are LL, some LR, some LALR, some Early. Also, some libraries separate parser from lexer while others are scannerless. But most of these can all parse the simple cases fine, so that's good for the 90% coverage. However, cross-cutting that formal power issue, there are two main programming styles used by different libraries. Some libraries use BNF definitions of a grammar as another mini-language inside Python. Exactly where those BNF definitions live varies, but using them is largely similar (i.e. are they in a separate file, in docstrings, contents of variables, etc). And sure, EBNF vs. BNF proper. But other libraries instead use Python functions or classes to define the productions, where each class/function is effectively one term of the grammar. Typically this latter style allows triggering events as soon as some production is encountered?the event could be "accumulate a counter", or "write an output string", or "perform a computation", or other things. There are lots of good arguments for why to use different libraries along the axes I mention, on all sides. What is not possible is to reconcile the very different decisions into a common denominator. Something in the standard library would have to be partisan in selecting one particular approach as the "official" one. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Apr 1 10:24:55 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 1 Apr 2019 23:24:55 +0900 Subject: [Python-ideas] Built-in parsing library In-Reply-To: References:

<23713.44193.803088.183109@turnbull.sk.tsukuba.ac.jp> Message-ID: <23714.7991.700779.260358@turnbull.sk.tsukuba.ac.jp> David Mertz writes: > OK, I'll acknowledge my comment might have overstated the bar to overcome. > A parser added to the standard library doesn't need to be perfect for > everyone. But adding to stdlib *does* provide a kind of endorsement of the > right default way to go about things. Indeed it does, but TOOWTDI is not absolute. > However, cross-cutting that formal power issue, there are two main > programming styles used by different libraries. I concede this tends to raise the bar quite a bit. > Something in the standard library would have to be partisan in > selecting one particular approach as the "official" one. Perhaps. Even there, though, we have an example: XML. We gotcher SAX, we gotcher DOM, we gotcher ElementTree, we gotcher expat. I think XML processing is probably a *lot* more used and in a lot more modes than general parsing. But the analogy is valid, even though I can't say it's powerful *enough*. There definitely is a bar to clear. I don't know if it's worth Nam's effort to try to clear it -- there's no guarantee of success on something like this. I just think we shouldn't be *too* discouraging. And I personally think parsing formal languages is an important enough field to deserve consideration for stdlib inclusion, even if it's not going to be used every day. Steve From antoine.pietri1 at gmail.com Mon Apr 1 10:27:50 2019 From: antoine.pietri1 at gmail.com (Antoine Pietri) Date: Mon, 1 Apr 2019 16:27:50 +0200 Subject: [Python-ideas] Backward-incompatible changes for Python 4 Message-ID: While the switch to Python 3 did an excellent job in removing some of the old inconsistencies we had in the language, pretty much everyone agrees that some other backwards-incompatible changes could be made to remove some old warts and bring even more consistency to Python. Since Python 4 is getting closer and closer, I think it?s time to finally discuss some of the most obvious changes we should do for Python 4. Here is the list I compiled: - The / operator returns floats, which loses information when both of the operands are integer. In Python 4, ?1 / 2? should return a decimal.Decimal. To ease the transition, we propose to add a new ?from __future__ import decimal_division? in Python 3.9 to enable this behavior. - As most of the Python ecosystem is moving towards async, some of the old I/O-blocking APIs should be progressively migrated to an async by default model. The most obvious candidate to start this transition is the print function, which blocks on the I/O of flushes. We propose to make ?print? an async coroutine. In Python 3.9, this feature could be optionally enabled with ?from __future__ import print_coroutine?. - To ease compatibility with the Windows API, the PyUnicode* objects should be internally represented as an array of uint16_t, as it would avoid the conversion overhead from UCS. CPython migration details are left as an exercise for the developer. We think more changes are obviously warranted (e.g adding a new string formatting module, changing the semantic of the import system, using := in with statements...), but these changes will need specific threads of their own. So, can you think of other backward-incompatible changes that should be done in Python 4? Don't hesitate to add your own ideas :-) Thanks, -- Antoine Pietri From 2QdxY4RzWzUUiLuE at potatochowder.com Mon Apr 1 10:41:40 2019 From: 2QdxY4RzWzUUiLuE at potatochowder.com (Dan Sommers) Date: Mon, 1 Apr 2019 10:41:40 -0400 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References: Message-ID: <22ff9916-6971-ad7d-c908-05a114954abf@potatochowder.com> On 4/1/19 10:27 AM, Antoine Pietri wrote: > - The / operator returns floats, which loses information when both of > the operands are integer. In Python 4, ?1 / 2? should return a > decimal.Decimal. To ease the transition, we propose to add a new ?from > __future__ import decimal_division? in Python 3.9 to enable this > behavior. ?1 / 2? should be a syntax error. "1 / 2" should return a string. 1 / 2 should return a fractions.Fraction. From solipsis at pitrou.net Mon Apr 1 10:50:30 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 1 Apr 2019 16:50:30 +0200 Subject: [Python-ideas] Backward-incompatible changes for Python 4 References: <22ff9916-6971-ad7d-c908-05a114954abf@potatochowder.com> Message-ID: <20190401165030.38d94ec3@fsol> On Mon, 1 Apr 2019 10:41:40 -0400 Dan Sommers <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > On 4/1/19 10:27 AM, Antoine Pietri wrote: > > > - The / operator returns floats, which loses information when both of > > the operands are integer. In Python 4, ?1 / 2? should return a > > decimal.Decimal. To ease the transition, we propose to add a new ?from > > __future__ import decimal_division? in Python 3.9 to enable this > > behavior. > > ?1 / 2? should be a syntax error. > > "1 / 2" should return a string. > > 1 / 2 should return a fractions.Fraction. And 01 / 04 / 2019 should return a April 1st datetime. (except in the US, of course) Regards Antoine. From ericfahlgren at gmail.com Mon Apr 1 10:55:04 2019 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Mon, 1 Apr 2019 07:55:04 -0700 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: <20190401165030.38d94ec3@fsol> References: <22ff9916-6971-ad7d-c908-05a114954abf@potatochowder.com> <20190401165030.38d94ec3@fsol> Message-ID: On Mon, Apr 1, 2019 at 7:50 AM Antoine Pitrou wrote: > > And 01 / 04 / 2019 should return a April 1st datetime. > > (except in the US, of course) > Where it would of course be I / IIII / MMXVIIII, unless you have ISO-8601 set in which case it would be MMXVIIII - IIII - I -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Mon Apr 1 10:58:16 2019 From: toddrjen at gmail.com (Todd) Date: Mon, 1 Apr 2019 10:58:16 -0400 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References: Message-ID: On Mon, Apr 1, 2019, 10:28 Antoine Pietri wrote: > While the switch to Python 3 did an excellent job in removing some of > the old inconsistencies we had in the language, pretty much everyone > agrees that some other backwards-incompatible changes could be made to > remove some old warts and bring even more consistency to Python. > > Since Python 4 is getting closer and closer, I think it?s time to > finally discuss some of the most obvious changes we should do for > Python 4. Here is the list I compiled: > > - The / operator returns floats, which loses information when both of > the operands are integer. In Python 4, ?1 / 2? should return a > decimal.Decimal. To ease the transition, we propose to add a new ?from > __future__ import decimal_division? in Python 3.9 to enable this > behavior. > - As most of the Python ecosystem is moving towards async, some of the > old I/O-blocking APIs should be progressively migrated to an async by > default model. The most obvious candidate to start this transition is > the print function, which blocks on the I/O of flushes. We propose to > make ?print? an async coroutine. In Python 3.9, this feature could be > optionally enabled with ?from __future__ import print_coroutine?. > - To ease compatibility with the Windows API, the PyUnicode* objects > should be internally represented as an array of uint16_t, as it would > avoid the conversion overhead from UCS. CPython migration details are > left as an exercise for the developer. > > We think more changes are obviously warranted (e.g adding a new string > formatting module, changing the semantic of the import system, using > := in with statements...), but these changes will need specific > threads of their own. > > So, can you think of other backward-incompatible changes that should > be done in Python 4? Don't hesitate to add your own ideas :-) > > Thanks, > We should probably just get rid of floats entirely and use decimals internally. Floats just have too much unexpected behavior. Anyone wanting real IEEE floats can still use numpy float scalars. Another thing is that a lot of people in numeric computing are used to open-ended indices starting at 1. I would like to see the starting index and whether an index is open-ended or closed-ended configurable on a per-module basis and/or with a context manager. Currently there is no empty set literal. This is a hold-over from when there were no sets. Now would be a good opportunity to add one. I suggest {} become an empty set and {:} be an empty dict. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Apr 1 11:03:01 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 1 Apr 2019 16:03:01 +0100 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References:

Message-ID: On Mon, 1 Apr 2019 at 15:59, Todd wrote: > Currently there is no empty set literal. This is a hold-over from when there were no sets. Now would be a good opportunity to add one. I suggest {} become an empty set and {:} be an empty dict. There should be no need for two styles - now that Python has type inference, it should be possible for users to just type {} and have the interpreter work out which was intended from context. Any ambiguity can easily be resolved by using a type hint. Paul From boxed at killingar.net Mon Apr 1 11:35:31 2019 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Mon, 1 Apr 2019 17:35:31 +0200 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References: Message-ID: Please let's all agree that April 1 is the worst day of the year. > On 1 Apr 2019, at 16:27, Antoine Pietri wrote: > > While the switch to Python 3 did an excellent job in removing some of > the old inconsistencies we had in the language, pretty much everyone > agrees that some other backwards-incompatible changes could be made to > remove some old warts and bring even more consistency to Python. > > Since Python 4 is getting closer and closer, I think it?s time to > finally discuss some of the most obvious changes we should do for > Python 4. Here is the list I compiled: > > - The / operator returns floats, which loses information when both of > the operands are integer. In Python 4, ?1 / 2? should return a > decimal.Decimal. To ease the transition, we propose to add a new ?from > __future__ import decimal_division? in Python 3.9 to enable this > behavior. > - As most of the Python ecosystem is moving towards async, some of the > old I/O-blocking APIs should be progressively migrated to an async by > default model. The most obvious candidate to start this transition is > the print function, which blocks on the I/O of flushes. We propose to > make ?print? an async coroutine. In Python 3.9, this feature could be > optionally enabled with ?from __future__ import print_coroutine?. > - To ease compatibility with the Windows API, the PyUnicode* objects > should be internally represented as an array of uint16_t, as it would > avoid the conversion overhead from UCS. CPython migration details are > left as an exercise for the developer. > > We think more changes are obviously warranted (e.g adding a new string > formatting module, changing the semantic of the import system, using > := in with statements...), but these changes will need specific > threads of their own. > > So, can you think of other backward-incompatible changes that should > be done in Python 4? Don't hesitate to add your own ideas :-) > > Thanks, > > -- > Antoine Pietri > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From p.f.moore at gmail.com Mon Apr 1 14:33:11 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 1 Apr 2019 19:33:11 +0100 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References:

Message-ID: On Mon, 1 Apr 2019 at 16:36, Anders Hovm?ller wrote: > > Please let's all agree that April 1 is the worst day of the year. Maybe in Python 4, datetime.datetime should silently convert 1st April to the 2nd? :-P Paul From brett at python.org Mon Apr 1 14:27:19 2019 From: brett at python.org (Brett Cannon) Date: Mon, 1 Apr 2019 11:27:19 -0700 Subject: [Python-ideas] PEP-582 and multiple Python versions In-Reply-To: References: Message-ID: I just wanted to warn people that I don't know if any of the authors of PEP 582 subscribe to python-ideas and they have not brought it forward for discussion yet, so there's no guarantee of a response. On Mon, Apr 1, 2019 at 5:27 AM Calvin Spealman wrote: > While the PEP does show the version number as part of the path to the > actual packages, implying support for multiple versions, this doesn't seem > to be spelled out in the actual text. Presumably __pypackages__/3.8/ might > sit beside __pypackages__/3.9/, etc. to keep future versions capable of > installing packages for each version, the way virtualenv today is bound to > one version of Python. > > I'd like to raise a potential edge case that might be a problem, and > likely an increasingly common one: users with multiple installations of the > *same* version of Python. This is actually a common setup for Windows users > who use WSL, Microsoft's Linux-on-Windows solution, as you could have both > the Windows and Linux builds of a given Python version installed on the > same machine. The currently implied support for multiple versions would not > be able to separate these and could create problems if users pip install a > Windows binary package through Powershell and then try to run a script in > Bash from the same directory, causing the Linux version of Python to try to > use Windows python packages. > > I'm not actually sure what the solution here is. Mostly I wanted to raise > the concern, because I'm very keen on WSL being a great entry path for new > developers and I want to make that a better experience, not a more > confusing one. Maybe that version number could include some other unique > identify, maybe based on Python's own executable. A hash maybe? I don't > know if anything like that already exists to uniquely identify a Python > build or installation. > > -- > > CALVIN SPEALMAN > > SENIOR QUALITY ENGINEER > > cspealma at redhat.com M: +1.336.210.5107 > > TRIED. TESTED. TRUSTED. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Mon Apr 1 15:02:21 2019 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 1 Apr 2019 20:02:21 +0100 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References:

Message-ID: <1c854d5a-63cb-8987-b2b9-a8deed0ecac9@mrabarnett.plus.com> On 2019-04-01 19:33, Paul Moore wrote: > On Mon, 1 Apr 2019 at 16:36, Anders Hovm?ller wrote: >> >> Please let's all agree that April 1 is the worst day of the year. > > Maybe in Python 4, datetime.datetime should silently convert 1st April > to the 2nd? :-P > Converting silently is not Pythonic. It should raise an exception. :-) From njs at pobox.com Mon Apr 1 15:17:23 2019 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 1 Apr 2019 12:17:23 -0700 Subject: [Python-ideas] Built-in parsing library In-Reply-To: References:

Message-ID: On Sun, Mar 31, 2019 at 9:17 PM Nam Nguyen wrote: > Installing a package out of stdlib does not solve the problem that motivated this thread. The libraries included in the stdlib can't use those parsers. Can you be more specific about exactly which code in the stdlib you think should be rewritten to use a parsing library? -n -- Nathaniel J. Smith -- https://vorpus.org From jelle.zijlstra at gmail.com Mon Apr 1 16:05:30 2019 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Mon, 1 Apr 2019 13:05:30 -0700 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References: Message-ID: El lun., 1 abr. 2019 a las 7:28, Antoine Pietri () escribi?: > While the switch to Python 3 did an excellent job in removing some of > the old inconsistencies we had in the language, pretty much everyone > agrees that some other backwards-incompatible changes could be made to > remove some old warts and bring even more consistency to Python. > > Since Python 4 is getting closer and closer, I think it?s time to > finally discuss some of the most obvious changes we should do for > Python 4. Here is the list I compiled: > > - The / operator returns floats, which loses information when both of > the operands are integer. In Python 4, ?1 / 2? should return a > decimal.Decimal. To ease the transition, we propose to add a new ?from > __future__ import decimal_division? in Python 3.9 to enable this > behavior. > More broadly, one of the best changes in Python 3 was the sanitization of the string/unicode logic: in Python 2 str and unicode were mostly-but-not-always interchangeable, but not always, and that led to a lot of hard to debug errors. Python 3 fixed this by separating the two more cleanly. Python 4 has the opportunity to do something similar to separate out another pair of easily confused types: int and float. Broadly speaking, we should use float for human-understandable numbers, and int for things that map directly to memory offsets in the computer, and we should avoid mixing them. This suggests the following changes: - int + float (and generally any mixed operation between ints and floats) should throw a TypeError - len() should return a float - list.__getitem__ should only accepts ints, not floats - integer overflow should use two's complement wraparound instead of infinite precision > - As most of the Python ecosystem is moving towards async, some of the > old I/O-blocking APIs should be progressively migrated to an async by > default model. The most obvious candidate to start this transition is > the print function, which blocks on the I/O of flushes. We propose to > make ?print? an async coroutine. In Python 3.9, this feature could be > optionally enabled with ?from __future__ import print_coroutine?. > - To ease compatibility with the Windows API, the PyUnicode* objects > should be internally represented as an array of uint16_t, as it would > avoid the conversion overhead from UCS. CPython migration details are > left as an exercise for the developer. > > We think more changes are obviously warranted (e.g adding a new string > formatting module, changing the semantic of the import system, using > := in with statements...), but these changes will need specific > threads of their own. > > So, can you think of other backward-incompatible changes that should > be done in Python 4? Don't hesitate to add your own ideas :-) > > Thanks, > > -- > Antoine Pietri > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bitsink at gmail.com Mon Apr 1 16:08:02 2019 From: bitsink at gmail.com (Nam Nguyen) Date: Mon, 1 Apr 2019 13:08:02 -0700 Subject: [Python-ideas] Built-in parsing library In-Reply-To: References:

Message-ID: Sure! Same examples mentioned in Victor's https://vstinner.github.io/tag/security.html could have been fixed by having a more proper parser. This one that I helped author was also a parsing issue. https://python-security.readthedocs.io/vuln/bpo-30500_urllib_connects_to_a_wrong_host.html Thanks for the pointer to pgen2, Guido. I have only quickly skimmed through it and thought it was really closely tied to the Python language. Maybe I'm wrong, so I'll need some time to try it out on some of those previous security fixes. Cheers, Nam On Mon, Apr 1, 2019 at 12:17 PM Nathaniel Smith wrote: > On Sun, Mar 31, 2019 at 9:17 PM Nam Nguyen wrote: > > Installing a package out of stdlib does not solve the problem that > motivated this thread. The libraries included in the stdlib can't use those > parsers. > > Can you be more specific about exactly which code in the stdlib you > think should be rewritten to use a parsing library? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ricocotam at gmail.com Mon Apr 1 17:12:44 2019 From: ricocotam at gmail.com (Adrien Ricocotam) Date: Mon, 1 Apr 2019 23:12:44 +0200 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References:

Message-ID: Am I being fooled ? I guess yes That?s the worst idea I ever heard. Python is supposed to be easy to use, don?t change it into Rust ! On Mon 1 Apr 2019 at 22:06, Jelle Zijlstra wrote: > > > El lun., 1 abr. 2019 a las 7:28, Antoine Pietri (< > antoine.pietri1 at gmail.com>) escribi?: > >> While the switch to Python 3 did an excellent job in removing some of >> the old inconsistencies we had in the language, pretty much everyone >> agrees that some other backwards-incompatible changes could be made to >> remove some old warts and bring even more consistency to Python. >> >> Since Python 4 is getting closer and closer, I think it?s time to >> finally discuss some of the most obvious changes we should do for >> Python 4. Here is the list I compiled: >> >> - The / operator returns floats, which loses information when both of >> the operands are integer. In Python 4, ?1 / 2? should return a >> decimal.Decimal. To ease the transition, we propose to add a new ?from >> __future__ import decimal_division? in Python 3.9 to enable this >> behavior. >> > More broadly, one of the best changes in Python 3 was the sanitization of > the string/unicode logic: in Python 2 str and unicode were > mostly-but-not-always interchangeable, but not always, and that led to a > lot of hard to debug errors. Python 3 fixed this by separating the two more > cleanly. Python 4 has the opportunity to do something similar to separate > out another pair of easily confused types: int and float. > > Broadly speaking, we should use float for human-understandable numbers, > and int for things that map directly to memory offsets in the computer, and > we should avoid mixing them. This suggests the following changes: > - int + float (and generally any mixed operation between ints and floats) > should throw a TypeError > - len() should return a float > - list.__getitem__ should only accepts ints, not floats > - integer overflow should use two's complement wraparound instead of > infinite precision > > >> - As most of the Python ecosystem is moving towards async, some of the >> old I/O-blocking APIs should be progressively migrated to an async by >> default model. The most obvious candidate to start this transition is >> the print function, which blocks on the I/O of flushes. We propose to >> make ?print? an async coroutine. In Python 3.9, this feature could be >> optionally enabled with ?from __future__ import print_coroutine?. >> - To ease compatibility with the Windows API, the PyUnicode* objects >> should be internally represented as an array of uint16_t, as it would >> avoid the conversion overhead from UCS. CPython migration details are >> left as an exercise for the developer. >> >> We think more changes are obviously warranted (e.g adding a new string >> formatting module, changing the semantic of the import system, using >> := in with statements...), but these changes will need specific >> threads of their own. >> >> So, can you think of other backward-incompatible changes that should >> be done in Python 4? Don't hesitate to add your own ideas :-) >> >> Thanks, >> >> -- >> Antoine Pietri >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 2QdxY4RzWzUUiLuE at potatochowder.com Mon Apr 1 17:37:42 2019 From: 2QdxY4RzWzUUiLuE at potatochowder.com (Dan Sommers) Date: Mon, 1 Apr 2019 17:37:42 -0400 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References:

Message-ID: On 4/1/19 11:35 AM, Anders Hovm?ller wrote: > Please let's all agree that April 1 is the worst day of the year. I can't reproduce your problem. What version of Python are you running, and on what OS? Please show us your program, and tell us what you expected it to do and what it did that failed to meet those expectations. Copy and paste exactly any error messages or tracebacks you received. See also http://www.sscce.org/ for more information. From greg.ewing at canterbury.ac.nz Mon Apr 1 17:43:19 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 02 Apr 2019 10:43:19 +1300 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References:

Message-ID: <5CA285F7.70801@canterbury.ac.nz> Paul Moore wrote: > now that Python has type > inference, it should be possible for users to just type {} and have > the interpreter work out which was intended from context. Or have {} return an ambiguous object that turns into a dict or set depending on what is done to it. We could call it a quict (quantum dict). To support this, we would also have to add a third possible value for type bool: >>> x = {} >>> isinstance(x, dict) Maybe >>> isinstance(x, set) Maybe >>> x['foo'] = 42 >>> isinstance(x, dict) True >>> isinstance(x, set) False -- Greg From tjreedy at udel.edu Mon Apr 1 18:12:00 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 1 Apr 2019 18:12:00 -0400 Subject: [Python-ideas] Built-in parsing library In-Reply-To: References:

Message-ID: On 4/1/2019 1:14 AM, Guido van Rossum wrote: > We do have a parser generator in the standard library: > https://github.com/python/cpython/tree/master/Lib/lib2to3/pgen2 It is effectively undocumented and by inference discouraged from use. The entry for lib2to3 in the 2to3 doc: https://docs.python.org/3/library/2to3.html#module-lib2to3 " lib2to3 - 2to3?s library Source code: Lib/lib2to3/ Note: The lib2to3 API should be considered unstable and may change drastically in the future. help(pgen) is not much more helpful. : Help on package lib2to3.pgen2 in lib2to3: NAME lib2to3.pgen2 - The pgen2 package. PACKAGE CONTENTS conv driver grammar literals parse pgen token tokenize FILE c:\programs\python38\lib\lib2to3\pgen2\__init__.py -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Mon Apr 1 18:05:09 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 02 Apr 2019 11:05:09 +1300 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References:

Message-ID: <5CA28B15.8000105@canterbury.ac.nz> Anders Hovm?ller wrote: > Please let's all agree that April 1 is the worst day of the year. Agreed. In light of that, I propose that the datetime module in Python 4 be changed so that April 1 does not exist: >>> m31 = date(2019, 3, 31) >>> m31 + timedelta(days = 1) datetime.date(2019, 4, 2) This would remove a large amount of confusion from the world, and ensure that Python never receives any more backwards incompatible changes. Obviously, removing a whole day from the year will create problems keeping the calendar in step with the seasons. To compensate, it will be necessary to add approximately 1.25 days worth of leap seconds to each year. This works out to about one leap second every 5 minutes. If a suitable algorithm is devised for distributing these "leap minutes" as evenly as possible over the year, this should cause minimal disruption. -- Greg From cs at cskk.id.au Mon Apr 1 18:29:33 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 2 Apr 2019 09:29:33 +1100 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: <5CA285F7.70801@canterbury.ac.nz> References: <5CA285F7.70801@canterbury.ac.nz> Message-ID: <20190401222933.GA92321@cskk.homeip.net> On 02Apr2019 10:43, Greg Ewing wrote: >Paul Moore wrote: >>now that Python has type >>inference, it should be possible for users to just type {} and have >>the interpreter work out which was intended from context. > >Or have {} return an ambiguous object that turns into a dict or >set depending on what is done to it. > >We could call it a quict (quantum dict). [...] I have concerns that this may lead to excessive memory use. My personal interpretation of QM is the continuous many worlds form, where _all_ values of the field are valid and real, and where the supposedly "collapse" on observation/interaction is just a measurement effect: we in the classical domain measure a particular value of the field and all of our future classical view flows consistent with that measurement, but there is _no_ value collapse of the system; the classical operation with the specific value is just a specific view of the uncollapsed QM state space. This is analogous to Python "bool(some-float-value)": we get a True or False but the underlying value is unchanged. As such, the quict type should be managed by an underlying state with a thread local view; when current thread performs a dict-like or set-like operation on the quict that thread should get a thread-local dict or set flavour view of the quict, with the underlying quict still open. In this way multiple threads get their "collapsed" view of the underlying quict, directly supporting many worlds programme execution. Obviously, any operations which do not induce a dict/set "measurement" leave the quict in the uncollapsed view. And it follows that "print(quict)" of an uncollapsed quict should choose a dict or set view for that thread at random and from then on the quict would have that flavour in that thread. For simple dict/set quicts this presents a fairly capped memory use (two flavours, and per thread view state) but companion types such as the quoat have much more scope for heavy memory consumption. Cheers, Cameron Simpson From jcgoble3 at gmail.com Mon Apr 1 18:33:40 2019 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Mon, 1 Apr 2019 18:33:40 -0400 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: <5CA28B15.8000105@canterbury.ac.nz> References:

<5CA28B15.8000105@canterbury.ac.nz> Message-ID: On Mon, Apr 1, 2019 at 6:12 PM Greg Ewing wrote: > Obviously, removing a whole day from the year will create problems > keeping the calendar in step with the seasons. To compensate, it > will be necessary to add approximately 1.25 days worth of leap > seconds to each year. This works out to about one leap second > every 5 minutes. If a suitable algorithm is devised for distributing > these "leap minutes" as evenly as possible over the year, this > should cause minimal disruption. > Far more disruption than you think, because that would result in daylight at midnight and nighttime at noon for a good chunk of the year. Instead, I suggest permanently extending February to 29 days instead, with a 30th day in leap years. This would limit the disruption to a single month (March), and only by an offset of one day. I never understood what February did wrong to be disrespected with such a short month anyway. Instead, February would be equal in length to April most of the time, and every four years (at least within our lifetimes *cough2100cough*) it would get to gloat over being longer than April. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Apr 1 18:49:15 2019 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 2 Apr 2019 09:49:15 +1100 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: References:

<5CA28B15.8000105@canterbury.ac.nz> Message-ID: On Tue, Apr 2, 2019 at 9:34 AM Jonathan Goble wrote: > > On Mon, Apr 1, 2019 at 6:12 PM Greg Ewing wrote: >> >> Obviously, removing a whole day from the year will create problems >> keeping the calendar in step with the seasons. To compensate, it >> will be necessary to add approximately 1.25 days worth of leap >> seconds to each year. This works out to about one leap second >> every 5 minutes. If a suitable algorithm is devised for distributing >> these "leap minutes" as evenly as possible over the year, this >> should cause minimal disruption. > > > Far more disruption than you think, because that would result in daylight at midnight and nighttime at noon for a good chunk of the year. Instead, I suggest permanently extending February to 29 days instead, with a 30th day in leap years. This would limit the disruption to a single month (March), and only by an offset of one day. I never understood what February did wrong to be disrespected with such a short month anyway. Instead, February would be equal in length to April most of the time, and every four years (at least within our lifetimes *cough2100cough*) it would get to gloat over being longer than April. > You don't know what heinous crimes February committed, because they were overshadowed by March which violated the normal rules by not just having a single id(), but multiple. Beware the IDs of March. ChrisA From steve at pearwood.info Mon Apr 1 20:52:52 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Apr 2019 11:52:52 +1100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> Message-ID: <20190402005251.GU31406@ando.pearwood.info> On Mon, Apr 01, 2019 at 02:29:44PM +1100, Chris Angelico wrote: > The multiple affix case has exactly two forms: > > 1) Tearing multiple affixes off (eg stripping "asdf.jpg.png" down to > just "asdf"), which most people are saying "no, don't do that, it > doesn't make sense and isn't needed" Perhaps I've missed something obvious (its been a long thread, and I'm badly distracted with hardware issues that are causing me some considerable grief), but I haven't seen anyone say "don't do that". But I have seen David Mertz say that this was the best behaviour: [quote] fname = 'silly.jpg.png.gif.png.jpg.gif.jpg' I'm honestly not sure what behavior would be useful most often for this oddball case. For the suffixes, I think "remove them all" is probably the best [end quote] I'd also like to point out that this is not an oddball case. There are two popular platforms where file extensions are advisory not mandatory (Linux and Mac), but even on Windows it is possible to get files with multiple, meaningful, extensions (foo.tar.gz for example) as well as periods used in place of spaces (a.funny.cat.video.mp4). > 2) Removing one of several options, which implies that one option is a > strict subpiece of another (eg stripping off "test" and "st") I take it you're only referring to the problematic cases, because there's the third option, where none of the affixes to be removed clash: spam.cut_suffix(("ed", "ing")) But that's pretty uninteresting and a simple loop or repeated call to the method will work fine: spam.cut_suffix("ed").cut_suffix("ing") just as we do with replace: spam.replace(",", "").replace(" ", "") If you only have a few affixes to work with, this is fine. If you have a lot, you may want a helper function, but that's okay. > If anyone is advocating for #1, I would agree with saying YAGNI. David Mertz did. > But #2 is an extremely unlikely edge case, and whatever semantics are > chosen for it, *normal* usage will not be affected. Not just unlikely, but "extremely" unlikely? Presumably you didn't just pluck that statement out of thin air, but have based it on an objective and statistically representative review of existing code and projections of future uses of these new methods. How could I possibly argue with that? Except to say that I think it is recklessly irresponsible for people engaged in language design to dismiss edge cases which will cause users real bugs and real pain so easily. We're not designing for our personal toolbox, we're designing for hundreds of thousands of other people with widely varying needs. It might be rare for you, but for somebody it will be happening ten times a day. And for somebody else, it will only happen once a year, but when it does, their code won't raise an exception it will just silently do the wrong thing. This is why replace does not take a set of multiple targets to replace. The user, who knows their own use-case and what behaviour they want, can write their own multiple-replace function, and we don't have to guess what they want. The point I am making is not that we must not ever support multiple affixes, but that we shouldn't rush that decision. Let's pick the low-hanging fruit, and get some real-world experience with the function before deciding how to handle the multiple affix case. [...] > Or all the behaviours actually do the same thing anyway. In this thread, I keep hearing this message: "My own personal use-case will never be affected by clashing affixes, so I don't care what behaviour we build into the language, so long as we pick something RIGHT NOW and don't give the people actually affected time to use the method and decide what works best in practice for them." Like for the str.replace method, the final answer might be "there is no best behaviour and we should refuse to choose". Why are we rushing to permanently enshrine one specific behaviour into the builtins before any of the users of the feature have a chance to use it and decide for themselves which suits them best? Now is better than never. Although never is often better than *right* now. Somebody (I won't name names, but they know who they are) wrote to me off-list some time ago and accused me of being arrogant and thinking I know more than everyone else. Well perhaps I am, but I'm not so arrogant as to think that I can choose the right behaviour for clashing affixes for other people when my own use-cases don't have clashing affixes. [...] > Sure, but I've often wanted to do something like "strip off a prefix > of http:// or https://", or something else that doesn't have a > semantic that's known to the stdlib. I presume there's a reason you aren't using urllib.parse and you just need a string without the leading scheme. If you're doing further parsing, the stdlib has the right batteries for that. (Aside: perhaps urllib.parse.ParseResult should get an attribute to return the URL minus the scheme? That seems like it would be useful.) > Also, this is still fairly > verbose, and a lot of people are going to reach for a regex, just > because it can be done in one line of code. Okay, they will use a regex. Is this a problem? We're not planning on banning regexes are we? If they're happy using regexes, and don't care that it will be perhaps 3 times slower, let them. > > I posted links to prior art. Unless I missed something, not one of those > > languages or libraries supports multiple affixes in the one call. > > And they don't support multiple affixes in startswith/endswith either, > but we're very happy to have that in Python. But not until we had a couple of releases of experience with them: https://docs.python.org/2.7/library/stdtypes.html#str.endswith And .replace still only takes a single target to be replaced. [...] > We don't have to worry about edge cases that are > unlikely to come up in real-world code, And you are making that pronouncement on the basis of what? Your gut feeling? Perhaps you're thinking too narrowly. Here's a partial list of English prefixes that somebody doing text processing might want to remove to get at the root word: a an ante anti auto circum co com con contra contro de dis en ex extra hyper il im in ir inter intra intro macro micro mono non omni post pre pro sub sym syn tele un uni up I count fourteen clashes: a: an ante anti an: ante anti co: com con contra contro ex: extra in: inter intra intro un: uni (That's over a third of this admittedly incomplete list of prefixes.) I can think of at least one English suffix pair that clash: -ify, -fy. How about other languages? How comfortable are you to say that nobody doing text processing in German or Hindi will need to deal with clashing affixes? -- Steven From rosuav at gmail.com Mon Apr 1 21:12:40 2019 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 2 Apr 2019 12:12:40 +1100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <20190402005251.GU31406@ando.pearwood.info> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> Message-ID: On Tue, Apr 2, 2019 at 11:53 AM Steven D'Aprano wrote: > The point I am making is not that we must not ever support multiple > affixes, but that we shouldn't rush that decision. Let's pick the > low-hanging fruit, and get some real-world experience with the function > before deciding how to handle the multiple affix case. I still haven't seen anyone actually give a good reason for not going with "first wins", other than a paranoia that we don't have any real-world use-cases. And there are PLENTY of real-world use-cases where any semantics will have the same effect, and only a few where it would be at all important (and in all of those, "first wins" has been the correct semantic). By saying "let's add the method, but not give it all the power yet", you just create more version problems. "Oh, so I can use cutprefix back as far as 3.8, but if I use more than one prefix, now I have to say that this requires 3.9." Why not just give it the full power straight away? Are you actually expecting to find enough use-cases where "longest wins" or some other definition will be better? You can debate whether it's "extremely unlikely" to matter or it's "reasonably common" or whatever, but unless it ever matters AND has to be something other than first-match-wins, there's no reason not to lock in those semantics. ChrisA From mertz at gnosis.cx Mon Apr 1 21:34:21 2019 From: mertz at gnosis.cx (David Mertz) Date: Mon, 1 Apr 2019 21:34:21 -0400 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <20190402005251.GU31406@ando.pearwood.info> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> Message-ID: On Mon, Apr 1, 2019, 8:54 PM Steven D'Aprano wrote: > The point I am making is not that we must not ever support multiple > affixes, but that we shouldn't rush that decision. Let's pick the > low-hanging fruit, and get some real-world experience with the function > before deciding how to handle the multiple affix case. > There are exactly two methods of strings that deal specifically with affixes currently. Startswith and endswith. Both of those allow specifying multiple affixes. That's pretty strong real-world experience, and breaking the symmetry for no reason is merely confusing. Especially since the consistency would be obviously as commonly useful. Now look, the sky won't fall if a single-affix-only method is added. For that matter, it won't if nothing is added. In fact, the single affix version makes it a little bit easier to write a custom function handling multiple affixes. And the sky won't fall if the remove-just-one semantics are used rather than remove-from-class. But adding methods with sneakily helpful capabilities often helps users greatly. A lot of folks in this thread didn't even know about passing a tuple to str.startswith() a few days ago. I'm pretty sure that capability was added by Raymond, who has an amazingly good sense of what little tricks can prove really powerful. Apologies to a different developer if it wasn't him, but congrats and thanks to you if so. Somebody (I won't name names, but they know who they are) wrote to me > off-list some time ago and accused me of being arrogant and thinking I know > more than everyone else. Well perhaps I am, but I'm not so arrogant as to > think that I can choose the right behaviour for clashing affixes for other > people when my own use-cases don't have clashing affixes. > That could be me... Unless it's someone else :-). I think my intent was a bit different than you characterize, but I'm very guilty of presuming too much also. So mea culpa. > Sure, but I've often wanted to do something like "strip off a prefix > > of http:// or https://", or something else that doesn't have a > > semantic that's known to the stdlib. > > I presume there's a reason you aren't using urllib.parse and you just need > a string without the leading scheme. If you're doing further parsing, the > stdlib has the right batteries for that. > I know there are lots of specialized string manipulations in the STDLIB. Yeah, I could use os.path.splitext, and os.path.split, and urllib.parse.something, and lots of other things I rarely use. A lot of us like to manipulate strings in generically stringy ways. But not until we had a couple of releases of experience with them: > > https://docs.python.org/2.7/library/stdtypes.html#l.endswith > Ok. Fair point. I used Python 2.4 without the multiple affix option. Here's a partial list of English prefixes that somebody doing text > processing might want to remove to get at the root word: > > a an ante anti auto circum co com con contra contro de dis > en ex extra hyper il im in ir inter intra intro macro micro > mono non omni post pre pro sub sym syn tele un uni up > > I count fourteen clashes: > > a: an ante anti > an: ante anti > co: com con contra contro > ex: extra > in: inter intra intro > un: uni > This seems like a good argument for remove-all-from-class. :-) stem = word.lstrip(prefix_tup) But the we really need 'word.porter_stemmer()' as a built-in method. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Apr 1 22:03:26 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Apr 2019 13:03:26 +1100 Subject: [Python-ideas] Lessons learned from an API design mistake [was New explicit methods to trim strings] In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> Message-ID: <20190402020325.GW31406@ando.pearwood.info> I think the point Chris made about statistics.mode is important enough to start a new subthread about API design, and the lessons learned. On Mon, Apr 01, 2019 at 02:29:44PM +1100, Chris Angelico wrote: > We're basically debating collision semantics here. It's on par with > asking "how should statistics.mode() cope with multiple modes?". > Should the introduction of statistics.mode() have been delayed pending > a thorough review of use-cases, or is it okay to make it do what most > people want, and then be prepared to revisit its edge-case handling? > > (For those who don't know, mode() was changed in 3.8 to return the > first mode encountered, in contrast to previous behaviour where it > would raise an exception.) For those who are unaware, I was responsible for chosing the semantics of statistics.mode. My choice was to treat mode() as it is taught in secondary schools here in Australia, namely that if there are two or more equally common values, there is no mode. Statistically, there is no one right answer to how to treat multiple modes. Sometimes you treat them as true multiple modes, sometimes you say there is no mode, and sometimes you treat the fact that there are multiple modes as an artifact of the sample and pick one or another as the actual mode. There's no particular statistical reason to choose the first over the second or the third. So following the Zen, I refused to guess, and raised an exception. (I toyed with returning None instead, but decided against it for reasons that don't matter here.) This seemed like a good decision up front, and I don't remember there being any objections to that behaviour when the PEP was discussed both here and on Python-Dev. But once we had a few years of real-world practice, it turns out that: (1) Raising an exception was an annoying choice that meant that every use of mode() outside of the interactive interpreter needed to be wrapped in a try...except block, making it painful to use. (2) There are at least one good use-case for returning the first mode, even though statistically there's no reason to prefer it over any other. Importantly, that use-case was something that neither I, nor anyone involved in the original PEP debate for this, had thought of. It took a few years of actual use in the wild before anyone came up with an important, definitive use-case -- and it turns out to be completely unrelated to the statistical use of mode! Raymond Hettinger persuaded me that this non-statistics use-case was important enough for mode to pick a behaviour which has no statistical justification. (Also, many other statistics packages do the same thing, so even if we're wrong, we're no worse than everyone else.) Had I ignored the Zen and, in the face of ambiguity, *guessed* which mode to return, I could have guessed wrongly and returned one of these: - the largest mode - or the smallest - the last seen mode - the mode closest to the mean - or median, or some other measure of central tendency - or some sort of special "multi-mode" object (perhaps a list). I would have missed a real use-case that I never imagined existed, as well as a good opportunity for optimization. Raymond's new version of mode is faster as well as more useful. Because I *refused to guess* and raised an exception: (1) mode was harder to use than it should have been; (2) but we were able to change its behaviour without a lengthy and annoying depreciation period, or introducing a "new_mode" function. Knowing what I know *now*, if I were designing mode() from scratch I'd go with Raymond's design. If it is statistically unjustified, its justified by other reasons, and if it is wrong, it's not so wrong as to be useless, and its wrong in a way that many other statistics libraries are also wrong. So we're in good company. But I didn't know that *then*, and I never would have guessed that there was a non-statistical use for mode. Lesson number one: Just because you have thought about your function for five minutes, or five months, doesn't mean you have thought of all the real-world uses. Lesson number two: A lot of the Zen is intended as a joke, but the koan about refusing to guess is very good advice. When possible, be conservative, take your time to make a decision, and base it on real-world experience, not gut feelings about what is "obviously" correct. In language design even more than personal code, You Ain't Gonna Need It (Yet) applies. Lesson number three: Sometimes, to not make a decision is itself a decision. In the case of mode, I had to deal with multiple modes *somehow*, I couldn't just ignore it. Fortunately I chose to raise an exception, which made it possible to change my mind later without a lengthy deprecation period. But that in turn made the function more annoying and difficult to use in practice. But in the case of the proposed str.cut_prefix and cut_suffix methods, we can avoid the decision of what to do with multiple affixes by just not supporting them! We don't have to make a decision to raise an exception, or return X (for whatever semantics of X we choose). There's no need to choose *anything* about the multiple affix case until we have more real-world experience to make a judgement. Lesson number four: Python is nearly 30 years old, and the str.replace() method still refuses to guess how to deal with the case of multiple target strings. That doesn't make replace useless. -- Steven From 2QdxY4RzWzUUiLuE at potatochowder.com Mon Apr 1 22:10:08 2019 From: 2QdxY4RzWzUUiLuE at potatochowder.com (Dan Sommers) Date: Mon, 1 Apr 2019 22:10:08 -0400 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> Message-ID: <2877460d-a160-dbda-82ed-4676a735bbfe@potatochowder.com> On 4/1/19 9:34 PM, David Mertz wrote: > On Mon, Apr 1, 2019, 8:54 PM Steven D'Aprano wrote: > >> The point I am making is not that we must not ever support multiple >> affixes, but that we shouldn't rush that decision. Let's pick the >> low-hanging fruit, and get some real-world experience with the function >> before deciding how to handle the multiple affix case. >> > > There are exactly two methods of strings that deal specifically with > affixes currently. Startswith and endswith. Both of those allow specifying > multiple affixes. That's pretty strong real-world experience, and breaking > the symmetry for no reason is merely confusing. Especially since the > consistency would be obviously as commonly useful. My imagination is failing me: for multiple affixes (affices?), what is a use case for removing one, but not having the function return which one? In other words, shouldn't a function that removes multiple affixes also return which one(s) were removed? I think I'm agreeing with Steven: take the low hanging fruit now, and worry about complexification later (because I'm not sure that the existing API is good when removing multiple affixes). Stemming is hard, because a lot of words begin/end with common affixes, but that string of letters isn't always an affix. For example, removing common prefixes from "relay" leaves "lay," but that's not the root; similarly with "relax" and "area." If my algorithm is "look for the word in a list of known words, if it's there then great, but if it's not then remove one affix and try again," then I don't want to remove all the affixes at once. When removing extensions from filenames, all of my use cases involve removing one at a time and acting on the one that was removed. For example, decompressing foo.tar.gz into foo.tar, and then untarring foo.tar into foo. I suppose I can imagine removing tar.gz and then decompressing and untarring in one step, but again, then I have to know which suffixes were removed. Or maybe I could process foo.tar.gz and want to end up with foo.norm (bonus points for recognizing the XKCD reference), but my personal preference would still be to produce foo.tar.gz.norm by default and let the user specify the ultimate filename if they want something else. So I've seen someone (likely David Mertz?) ask for something like filename.strip_suffix(('.png', '.jpg')). What is the context? Is it strictly a filename processing program? Do you subsequently have to determine the suffix(es) at hand? From steve at pearwood.info Mon Apr 1 22:15:17 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 2 Apr 2019 13:15:17 +1100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> Message-ID: <20190402021517.GX31406@ando.pearwood.info> On Mon, Apr 01, 2019 at 09:34:21PM -0400, David Mertz wrote: > On Mon, Apr 1, 2019, 8:54 PM Steven D'Aprano wrote: > > > The point I am making is not that we must not ever support multiple > > affixes, but that we shouldn't rush that decision. Let's pick the > > low-hanging fruit, and get some real-world experience with the function > > before deciding how to handle the multiple affix case. > > > > There are exactly two methods of strings that deal specifically with > affixes currently. Startswith and endswith. Both of those allow specifying > multiple affixes. When testing for the existence of a prefix (or suffix), there are no choices that need to be made for the multiple prefix case. If spam.startswith("contra"), then it also starts with "co", and we don't have to decide whether to delete six characters or two. If spam starts with one of ("de", "ex", "in", "mono"), then it doesn't matter what order we specify the tests, it will return True regardless. If you write a pure Python implementation of multiprefix startswith, there's one *obviously correct* version: def multi_startswith(astring, prefixes): return any(astring.prefix for prefix in prefixes) because it literally doesn't matter which of the prefixes triggered the match. I could randomize the order of the prefixes, and nothing would change. But if you delete the prefix, it matter a lot which prefix triggers the match. > That's pretty strong real-world experience But not as strong as str.replace, which is much older than starts/endswith and still refuses to guess what the user expects to do with multiple substrings. -- Steven From mertz at gnosis.cx Mon Apr 1 22:28:44 2019 From: mertz at gnosis.cx (David Mertz) Date: Mon, 1 Apr 2019 22:28:44 -0400 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <2877460d-a160-dbda-82ed-4676a735bbfe@potatochowder.com> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <2877460d-a160-dbda-82ed-4676a735bbfe@potatochowder.com> Message-ID: On Mon, Apr 1, 2019 at 10:11 PM Dan Sommers < 2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > So I've seen someone (likely David Mertz?) ask for something > like filename.strip_suffix(('.png', '.jpg')). What is the > context? Is it strictly a filename processing program? Do > you subsequently have to determine the suffix(es) at hand? > Yes, I've sometimes wanted something like "the basename of all the graphic files in that directory." But here's another example that is from my actually current job: I do machine-learning/data science for a living. As part of that, I generate a bunch of models that try to make predictions from the same dataset. So I name those models like: dataset1.KNN_distance_n10.gz dataset1.KNN_distance_n10_poly2_scaled.xz dataset2.KNN_manhattan_n6.zip dataset2.KNN_distance_n10_poly2_scaled.xz dataset1.KNN_minkowski_n5.gz dataset1.LinSVC_Poly3_Scaled.gz dataset2.LogReg.bz2 dataset2.LogReg_Poly.gz dataset1.NuSVC_poly2_scaled.gz I would like to answer the question "What types of models have I tried against the datasets?" Obviously, I *can* answer this question. But it would be pleasant to answer it like this: styles = {model.lstrip(('dataset1', 'dataset2')) .rstrip(('gz', 'xz', 'zip', 'bz2)) for model in models} That's something very close to code I actually have in production now. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Mon Apr 1 22:43:10 2019 From: mertz at gnosis.cx (David Mertz) Date: Mon, 1 Apr 2019 22:43:10 -0400 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <20190402005251.GU31406@ando.pearwood.info> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> Message-ID: On Mon, Apr 1, 2019 at 8:54 PM Steven D'Aprano wrote: > I can think of at least one English suffix pair that clash: -ify, -fy. > How about other languages? How comfortable are you to say that nobody > doing text processing in German or Hindi will need to deal with clashing > affixes? > Here are the 30 most common suffixes in a large list of Dutch words. For similar answers for other languages, see https://gist.github.com/DavidMertz/1a4aac0e889097d7bf80d8d41a3a644d. Note that there is absolutely nothing morphological here, simply dumb string literals: % head -30 suffix-frequency-nl.txt ('en', 55338) ('er', 14387) ('de', 12541) ('den', 11427) ('ten', 9402) ('te', 8263) ('ng', 7502) ('es', 7398) ('st', 7102) ('ing', 6949) ('gen', 6836) ('rs', 6592) ('ers', 5581) ('ren', 4842) ('el', 4602) ('ngen', 4451) ('rde', 4255) ('ken', 4203) ('re', 3870) ('je', 3868) ('len', 3784) ('ste', 3680) ('ie', 3658) ('nd', 3635) ('erde', 3620) ('rden', 3593) ('jes', 3307) ('eren', 3193) ('id', 3123) ('rd', 3083) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Apr 2 03:23:12 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 2 Apr 2019 16:23:12 +0900 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: <5CA28B15.8000105@canterbury.ac.nz> References:

<5CA28B15.8000105@canterbury.ac.nz> Message-ID: <23715.3552.729800.392240@turnbull.sk.tsukuba.ac.jp> Greg Ewing writes: > In light of that, I propose that the datetime module in Python 4 > be changed so that April 1 does not exist: > > >>> m31 = date(2019, 3, 31) > >>> m31 + timedelta(days = 1) > datetime.date(2019, 4, 2) > > This would remove a large amount of confusion from the world, and > ensure that Python never receives any more backwards incompatible > changes. > > Obviously, removing a whole day from the year will create problems It's much simpler to rename April 1 to February 29. (Bikeshed that 29 is already taken, and make it February 30 if you like. And note that February will maintain its uniqueness in an even more prominent way, as the only temporally disconnected month!) From jorropo.pgm at gmail.com Tue Apr 2 04:03:18 2019 From: jorropo.pgm at gmail.com (Jorropo .) Date: Tue, 2 Apr 2019 10:03:18 +0200 Subject: [Python-ideas] Backward-incompatible changes for Python 4 In-Reply-To: <23715.3552.729800.392240@turnbull.sk.tsukuba.ac.jp> References:

<5CA28B15.8000105@canterbury.ac.nz> <23715.3552.729800.392240@turnbull.sk.tsukuba.ac.jp> Message-ID: > It's much simpler to rename April 1 to February 29. That will create a gap of one 1 day beetween shity regular callendar and our new one for 2 month. An algorithm distributing leap seconds near 1 first april, is maybe more complicated but reduce side effect with the world. -------------- next part -------------- An HTML attachment was scrubbed... URL: From evrial at gmail.com Tue Apr 2 06:37:18 2019 From: evrial at gmail.com (Alex Grigoryev) Date: Tue, 2 Apr 2019 13:37:18 +0300 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: Message-ID: <32FA2A90-E1E7-43FB-9694-50424B9CDB31@getmailspring.com> In your case you probably should use [model.split(".")[1] for model in models] strip_prefix should not be used with file extensions, method for files already exist. On Apr 2 2019, at 5:43 am, David Mertz wrote: > On Mon, Apr 1, 2019 at 8:54 PM Steven D'Aprano wrote: > > > I can think of at least one English suffix pair that clash: -ify, -fy. > > How about other languages? How comfortable are you to say that nobody > > doing text processing in German or Hindi will need to deal with clashing > > affixes? > > > Here are the 30 most common suffixes in a large list of Dutch words. For similar answers for other languages, see https://gist.github.com/DavidMertz/1a4aac0e889097d7bf80d8d41a3a644d. Note that there is absolutely nothing morphological here, simply dumb string literals: > % head -30 suffix-frequency-nl.txt > ('en', 55338) > ('er', 14387) > ('de', 12541) > ('den', 11427) > ('ten', 9402) > ('te', 8263) > ('ng', 7502) > ('es', 7398) > ('st', 7102) > ('ing', 6949) > ('gen', 6836) > ('rs', 6592) > ('ers', 5581) > ('ren', 4842) > ('el', 4602) > ('ngen', 4451) > ('rde', 4255) > ('ken', 4203) > ('re', 3870) > ('je', 3868) > ('len', 3784) > ('ste', 3680) > ('ie', 3658) > ('nd', 3635) > ('erde', 3620) > ('rden', 3593) > ('jes', 3307) > ('eren', 3193) > ('id', 3123) > ('rd', 3083) > > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Tue Apr 2 07:06:33 2019 From: rhodri at kynesim.co.uk (Rhodri James) Date: Tue, 2 Apr 2019 12:06:33 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <20190402005251.GU31406@ando.pearwood.info> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> Message-ID: <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> On 02/04/2019 01:52, Steven D'Aprano wrote: > Here's a partial list of English prefixes that somebody doing text > processing might want to remove to get at the root word: > > a an ante anti auto circum co com con contra contro de dis > en ex extra hyper il im in ir inter intra intro macro micro > mono non omni post pre pro sub sym syn tele un uni up > > I count fourteen clashes: > > a: an ante anti > an: ante anti > co: com con contra contro > ex: extra > in: inter intra intro > un: uni > > (That's over a third of this admittedly incomplete list of prefixes.) > > I can think of at least one English suffix pair that clash: -ify, -fy. You're beginning to persuade me that cut/trim methods/functions aren't a good idea :-) So far we have two slightly dubious use-cases. 1. Stripping file extensions. Personally I find that treating filenames like filenames (i.e. using os.path or (nowadays) pathlib) results in me thinking more appropriately about what I'm doing. 2. Stripping prefixes and suffixes to get to root words. Python has been used for natural language work for over a decade, and I don't think I've heard any great call from linguists for the functionality. English isn't a girl who puts out like that on a first date :-) There are too many common exception cases for such a straightforward approach not to cause confusion. 3. My most common use case (not very common at that) is for stripping annoying prompts off text-based APIs. I'm happy using .startswith() and string slicing for that, though your point about the repeated use of the string to be stripped off (or worse, hard-coding its length) is well made. I am beginning to worry slightly that actually there are usually more appropriate things to do than simply cutting off affixes, and that in providing these particular batteries we might be encouraging poor practise. -- Rhodri James *-* Kynesim Ltd From p.f.moore at gmail.com Tue Apr 2 07:23:15 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 2 Apr 2019 12:23:15 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> Message-ID: On Tue, 2 Apr 2019 at 12:07, Rhodri James wrote: > So far we have two slightly dubious use-cases. > > 1. Stripping file extensions. Personally I find that treating filenames > like filenames (i.e. using os.path or (nowadays) pathlib) results in me > thinking more appropriately about what I'm doing. I'd go further and say that filename manipulation is a great example of a place where generic string functions should definitely *not* be used. > 2. Stripping prefixes and suffixes to get to root words. Python has > been used for natural language work for over a decade, and I don't think > I've heard any great call from linguists for the functionality. English > isn't a girl who puts out like that on a first date :-) There are too > many common exception cases for such a straightforward approach not to > cause confusion. Agreed, using prefix/suffix stripping on natural language is at best a "quick hack". For robust usage, one of the natural language processing packages from PyPI is likely a far better fit. But "quick hacks" using the stdlib are not an unrealistic use case, so I don't think we should completely discount this. It's certainly not *compelling*, though. > 3. My most common use case (not very common at that) is for stripping > annoying prompts off text-based APIs. I'm happy using .startswith() and > string slicing for that, though your point about the repeated use of the > string to be stripped off (or worse, hard-coding its length) is well made. > > I am beginning to worry slightly that actually there are usually more > appropriate things to do than simply cutting off affixes, and that in > providing these particular batteries we might be encouraging poor practise. It would be really helpful if someone could go through the various use cases presented in this thread and classify them - filename manipulation, natural language uses, and "other". We could then focus on the "other" category to get a better feel for what use cases might act as a good argument for the feature. To me, it's starting to feel like a proposal that looks deceptively valuable because it's a "natural", or "obvious", addition to make, and there's a weight of people thinking of cases where they "might find it useful", but the reality is that many of those cases are not actually as good a fit for the feature as it seems at first glance. It would help the people in favour of the proposal to make their case if they could dispel that impression by giving a clearer summary of the expected use cases... Paul From cspealma at redhat.com Tue Apr 2 08:00:59 2019 From: cspealma at redhat.com (Calvin Spealman) Date: Tue, 2 Apr 2019 08:00:59 -0400 Subject: [Python-ideas] PEP-582 and multiple Python versions In-Reply-To: References:

Message-ID: OK, I didn't know if this or -dev was more appropriate, so I opted on the safer side in terms of annoying people. I'll post to python-dev. On Mon, Apr 1, 2019 at 2:27 PM Brett Cannon wrote: > I just wanted to warn people that I don't know if any of the authors of > PEP 582 subscribe to python-ideas and they have not brought it forward for > discussion yet, so there's no guarantee of a response. > > On Mon, Apr 1, 2019 at 5:27 AM Calvin Spealman > wrote: > >> While the PEP does show the version number as part of the path to the >> actual packages, implying support for multiple versions, this doesn't seem >> to be spelled out in the actual text. Presumably __pypackages__/3.8/ might >> sit beside __pypackages__/3.9/, etc. to keep future versions capable of >> installing packages for each version, the way virtualenv today is bound to >> one version of Python. >> >> I'd like to raise a potential edge case that might be a problem, and >> likely an increasingly common one: users with multiple installations of the >> *same* version of Python. This is actually a common setup for Windows users >> who use WSL, Microsoft's Linux-on-Windows solution, as you could have both >> the Windows and Linux builds of a given Python version installed on the >> same machine. The currently implied support for multiple versions would not >> be able to separate these and could create problems if users pip install a >> Windows binary package through Powershell and then try to run a script in >> Bash from the same directory, causing the Linux version of Python to try to >> use Windows python packages. >> >> I'm not actually sure what the solution here is. Mostly I wanted to raise >> the concern, because I'm very keen on WSL being a great entry path for new >> developers and I want to make that a better experience, not a more >> confusing one. Maybe that version number could include some other unique >> identify, maybe based on Python's own executable. A hash maybe? I don't >> know if anything like that already exists to uniquely identify a Python >> build or installation. >> >> -- >> >> CALVIN SPEALMAN >> >> SENIOR QUALITY ENGINEER >> >> cspealma at redhat.com M: +1.336.210.5107 >> >> TRIED. TESTED. TRUSTED. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma at redhat.com M: +1.336.210.5107 TRIED. TESTED. TRUSTED. -------------- next part -------------- An HTML attachment was scrubbed... URL: From boxed at killingar.net Tue Apr 2 08:27:53 2019 From: boxed at killingar.net (=?utf-8?Q?Anders_Hovm=C3=B6ller?=) Date: Tue, 2 Apr 2019 14:27:53 +0200 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> Message-ID: > On 2 Apr 2019, at 13:23, Paul Moore wrote: > > It would be really helpful if someone could go through the various use > cases presented in this thread and classify them - filename > manipulation, natural language uses, and "other". We could then focus > on the "other" category to get a better feel for what use cases might > act as a good argument for the feature. To me, it's starting to feel > like a proposal that looks deceptively valuable because it's a > "natural", or "obvious", addition to make, and there's a weight of > people thinking of cases where they "might find it useful", but the > reality is that many of those cases are not actually as good a fit for > the feature as it seems at first glance. It would help the people in > favour of the proposal to make their case if they could dispel that > impression by giving a clearer summary of the expected use cases... I found two instances of strip_prefix in the code base I work on: stripping "origin/" from git branch names, and "Author:" to get the author from log output, again from git. A good place to look for examples is this: https://github.com/search?utf8=?&q=%22strip_prefix%28%22+extension%3Apy+language%3APython+language%3APython&type=Code&ref=advsearch&l=Python&l=Python A pattern that one sees quickly is that there are lots and lots of functions that strip a specific and hardcoded prefix. There's a lot of path manipulation too. And of course, there's an enormous amount of copy paste (jsfunctions.py is everywhere!). Some examples from the search above: Removing "file:" prefix: https://github.com/merijn/dotfiles/blob/43c736c73c5eda413dc7b4615bb679bd43a18d1a/dotfiles/hg-data/hooks/bitbucket.py#L16 This is a strange one, which seems to strip different things? https://github.com/imperodesign/paas-tools/blob/649372762a18acefed0a24a970b93eb494529df9/deis/prd/controller/registry/tests.py#L99 Removing "master.": https://github.com/mithro/chromium-build/blob/98d83e124dc08510756906171922a22ba27b87fa/scripts/tools/dump_master_cfg.py#L67 Also not path: https://github.com/BlissRoms-x86/platform_external_swiftshader/blob/01c0db17f511badb921efc53981849cdacb82793/third_party/subzero/bloat/bloat.py#L212 Removing "Re:" from email subject lines: https://github.com/emersion/python-emailthreads/blob/0a56af7fd6de16105c27b7c149eeb0282e95e587/emailthreads/util.py#L21 Removing "MAILER_": https://github.com/vitalk/flask-mailer/blob/c724643f13e51d2e57546164e3e4abf9eb5d8097/flask_mailer/util.py#L30 I'm giving up now, because I got tired :) / Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Tue Apr 2 11:42:39 2019 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 2 Apr 2019 18:42:39 +0300 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <061FBF45-02FD-4C3E-B792-EB09C05ED98E@getmailspring.com> <20190330124105.GC31406@ando.pearwood.info>

Message-ID: On Sun, Mar 31, 2019 at 3:28 AM Brandt Bucher wrote: > > > An idea worth considering: one can think of the ?strip? family of methods > as currently taking an iterable of strings as an argument (since a string > is itself an sequence of strings): > > It would not be a huge logical leap to allow them to take any iterable. > Backward compatible, no new methods: > > >>> fname.rstrip(('.jpg', '.png', '.gif')) > > It even, in my opinion, can clarify "classic" strip/rstrip/lstrip usage: > > >>> "abcd".rstrip(("d", "c")) > 'ab' > > Maybe I?m missing a breaking case though, or this isn?t as clear for others. Thoughts? Now with this syntax I would have to write: string.rstrip ([substring]) Which means I would have to remember to put extra brackets. Not that bad, but needs extra caution, since it would release the type of the argument and IMO in the end may become confusing. So you change the behavior by the type of the argument variable. And again - having an iterable with *multiple* elements needs useful and transparent behavior first. The only intuitive behavior seems to be the algorithm: - take first element - check if it's the string suffix - if yes, cut it off and STOP the iteration - if not, repeat check with next element So it guarantees that only one (first) found element will be cut off. I don't see other useful cases yet. Even this one seems to be odd. More complicated behavior would be just too hard to follow. IMO having separate method is more user-friendly. As for the name, I like "rcut", "lcut" Self-explanative enough and matches other similar existing methods names. From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Apr 2 13:55:02 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 3 Apr 2019 02:55:02 +0900 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> Message-ID: <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp> Rhodri James writes: Steven d'Aprano writes: > > (That's over a third of this admittedly incomplete list of prefixes.) > > > > I can think of at least one English suffix pair that clash: -ify, -fy. And worse: is "tries" the third person present tense of "try" or is it the plural of "trie"? Pure lexical manipulation can't tell you. > You're beginning to persuade me that cut/trim methods/functions aren't a > good idea :-) I don't think I would go there yet (well, I started there, but...). > So far we have two slightly dubious use-cases. > > 1. Stripping file extensions. Personally I find that treating filenames > like filenames (i.e. using os.path or (nowadays) pathlib) results in me > thinking more appropriately about what I'm doing. Very much agree. > 2. Stripping prefixes and suffixes to get to root words. for suffix in english_suffixes: root = word.cutsuffix(suffix) if lookup_in_dictionary(root): do_something_appropriate_with_each_root_found() is surely more flexible and accurate than a hard-coded slice, and significantly more readable than for suffix in english_suffixes: root = word[:-len(suffix)] if word.endswith(suffix) else word if lookup_in_dictionary(root): do_something_appropriate_with_each_root_found() I think enough so that I might use a local def for cutsuffix if the method doesn't exist. So my feeling is that the use case for "or"-ing multiple suffixes is a lot weaker than it is for .endswith, but .cutsuffix itself is plausible. That said, I wouldn't add it if it were up to me. Among other things, for this root-extracting application def extract_root(word, prefix, suffix): word = word[len(prefix):] if word.endswith(prefix) else word word = word[:-len(suffix)] if word.endswith(suffix) else word # perhaps try further transforms like tri -> try here? return word and a double loop for prefix in english_prefixes: # includes '' for suffix in english_suffixes: # includes '' root = extract_root(word, prefix, suffix) if lookup_in_dictionary(root): yield root (probably recursive, as well) seems most elegant. > 3. My most common use case (not very common at that) is for stripping > annoying prompts off text-based APIs. I'm happy using > .startswith() and string slicing for that, though your point about > the repeated use of the string to be stripped off (or worse, > hard-coding its length) is well made. I don't understand this use case, specifically the opposition to hard-coding the length. Although hard-coding the length wouldn't occur to me in many cases, since I'd use # remove my bash prompt prompt_re = re.compile(r'^[^\u0000-\u001f\u007f]+ \d\d:\d\d\$ ') lines = [prompt_re.sub('', line) for line in lines] if I understand the task correctly. Similarly, there's a lot of regexp-removable junk in MTA logs, timestamps and DNS lookups for example, that can't be handled with cutprefix. > I am beginning to worry slightly that actually there are usually > more appropriate things to do than simply cutting off affixes, and > that in providing these particular batteries we might be > encouraging poor practise. I don't think that's a worry, at least if restricted to the single-affix form, because simply cutting off affixes is surely part of most such algorithms. The harder part is remembering that you probably have to deal with multiplicities and further transformations, but that can't be incentivized by refusing to implement .cutsuffix. It's an independent consideration. Steve From rhodri at kynesim.co.uk Tue Apr 2 14:02:09 2019 From: rhodri at kynesim.co.uk (Rhodri James) Date: Tue, 2 Apr 2019 19:02:09 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp> Message-ID: On 02/04/2019 18:55, Stephen J. Turnbull wrote: >> = Me > > 3. My most common use case (not very common at that) is for stripping > > annoying prompts off text-based APIs. I'm happy using > > .startswith() and string slicing for that, though your point about > > the repeated use of the string to be stripped off (or worse, > > hard-coding its length) is well made. > > I don't understand this use case, specifically the opposition to > hard-coding the length. Although hard-coding the length wouldn't > occur to me in many cases, since I'd use > > # remove my bash prompt > prompt_re = re.compile(r'^[^\u0000-\u001f\u007f]+ \d\d:\d\d\$ ') > lines = [prompt_re.sub('', line) for line in lines] For me it's more often like input = get_line_from_UART() if input.startswith("INFO>"): input = input[5:] do_something_useful(input) which is error-prone when you cut and paste for a different prompt elsewhere and forget to change the slice to match. -- Rhodri James *-* Kynesim Ltd From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Apr 2 14:10:07 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 3 Apr 2019 03:10:07 +0900 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk>

Message-ID: <23715.42367.172126.972274@turnbull.sk.tsukuba.ac.jp> Anders Hovm?ller writes: > Removing "file:" prefix: > https://github.com/merijn/dotfiles/blob/43c736c73c5eda413dc7b4615bb679bd43a18d1a/dotfiles/hg-data/hooks/bitbucket.py#L16 This is interesting, because it shows the (so far standard) one-liner: word[len(prefix):] if word.startswith(prefix) else word can be improved (?!) to word[len(prefix) if word.startswith(prefix) else 0:] I don't know if this is more readable, but I think it's less so. Note that version 1 doesn't copy word if it doesn't start with prefix, while version 2 does. In many applications I can think of the results would be accumulated in a set, and version 1 equality tests will also be faster in the frequent case that the word doesn't start with the prefix. So that's the one I'd go with, as I can't think of any applications where multiple copies of the same string would be useful. Steve From python at mrabarnett.plus.com Tue Apr 2 14:28:01 2019 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 2 Apr 2019 19:28:01 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <23715.42367.172126.972274@turnbull.sk.tsukuba.ac.jp> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk>

<23715.42367.172126.972274@turnbull.sk.tsukuba.ac.jp> Message-ID: On 2019-04-02 19:10, Stephen J. Turnbull wrote: > Anders Hovm?ller writes: > > > Removing "file:" prefix: > > https://github.com/merijn/dotfiles/blob/43c736c73c5eda413dc7b4615bb679bd43a18d1a/dotfiles/hg-data/hooks/bitbucket.py#L16 > > This is interesting, because it shows the (so far standard) one-liner: > > word[len(prefix):] if word.startswith(prefix) else word > > can be improved (?!) to > > word[len(prefix) if word.startswith(prefix) else 0:] > It could be 'improved' more to: word[word.startswith(prefix) and len(prefix) : ] > I don't know if this is more readable, but I think it's less so. > > Note that version 1 doesn't copy word if it doesn't start with prefix, > while version 2 does. In many applications I can think of the results > would be accumulated in a set, and version 1 equality tests will also > be faster in the frequent case that the word doesn't start with the > prefix. So that's the one I'd go with, as I can't think of any > applications where multiple copies of the same string would be useful. > _Neither_ version copies if the word doesn't start with the prefix. If you won't believe me, test them! :-) From python at mrabarnett.plus.com Tue Apr 2 14:33:55 2019 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 2 Apr 2019 19:33:55 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp> Message-ID: <4305c9ce-394b-10de-0717-5a0a7663d3c8@mrabarnett.plus.com> On 2019-04-02 18:55, Stephen J. Turnbull wrote: > Rhodri James writes: > > Steven d'Aprano writes: > > > (That's over a third of this admittedly incomplete list of prefixes.) > > > > > > I can think of at least one English suffix pair that clash: -ify, -fy. > > And worse: is "tries" the third person present tense of "try" or is it > the plural of "trie"? Pure lexical manipulation can't tell you. > > > You're beginning to persuade me that cut/trim methods/functions aren't a > > good idea :-) > > I don't think I would go there yet (well, I started there, but...). > > > So far we have two slightly dubious use-cases. > > > > 1. Stripping file extensions. Personally I find that treating filenames > > like filenames (i.e. using os.path or (nowadays) pathlib) results in me > > thinking more appropriately about what I'm doing. > > Very much agree. > > > 2. Stripping prefixes and suffixes to get to root words. > > for suffix in english_suffixes: > root = word.cutsuffix(suffix) > if lookup_in_dictionary(root): > do_something_appropriate_with_each_root_found() > > is surely more flexible and accurate than a hard-coded slice, and > significantly more readable than > > for suffix in english_suffixes: > root = word[:-len(suffix)] if word.endswith(suffix) else word > if lookup_in_dictionary(root): > do_something_appropriate_with_each_root_found() > [snip] The code above contains a subtle bug. If suffix == '', then word.endswith(suffix) == True, and word[:-len(suffix)] == word[:-0] == ''. Each time I see someone do that, I see more evidence in support of adding the method. From rosuav at gmail.com Tue Apr 2 14:42:19 2019 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Apr 2019 05:42:19 +1100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <4305c9ce-394b-10de-0717-5a0a7663d3c8@mrabarnett.plus.com> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp> <4305c9ce-394b-10de-0717-5a0a7663d3c8@mrabarnett.plus.com> Message-ID: On Wed, Apr 3, 2019 at 5:34 AM MRAB wrote: > > The code above contains a subtle bug. > If suffix == '', then word.endswith(suffix) == True, and > word[:-len(suffix)] == word[:-0] == ''. > > Each time I see someone do that, I see more evidence in support of > adding the method. Either that, or it's evidence that negative indexing is only part of the story, and we need a real way to express "zero from the end" other than negative zero. For instance, word[:<0] might mean "zero from the end", and word[:<1] would be "one from the end". As a syntactic element rather than an arithmetic one, it would be safe against accidentally slicing from the front instead of the back. But that's an idea for another day. ChrisA From cs at cskk.id.au Tue Apr 2 18:58:07 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 3 Apr 2019 09:58:07 +1100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: Message-ID: <20190402225807.GA61774@cskk.homeip.net> On 02Apr2019 12:23, Paul Moore wrote: >On Tue, 2 Apr 2019 at 12:07, Rhodri James wrote: >> So far we have two slightly dubious use-cases. >> >> 1. Stripping file extensions. Personally I find that treating filenames >> like filenames (i.e. using os.path or (nowadays) pathlib) results in me >> thinking more appropriately about what I'm doing. > >I'd go further and say that filename manipulation is a great example >of a place where generic string functions should definitely *not* be >used. Filename manipulation on a path _component_ is generally pretty reliable (yes one can break things by, say, inserting os.sep). I do a fair bit of filename fiddling using string functions, and these fall into 3 categories off the top of my head: - file extensions, and here I do use splitext() - trimming extensions (only barely a second case), and it turns out the only case I could easily find using the endswith/[:-offset] incantation would probably go just as well with splitext() - normalising pathnames; as an example, for the home media library I routinely downcase filenames, convert whitespace into a dash, separate fields with "--" (eg episode designator vs title) and convert _ into a colon (hello Mac Finder and UI file save dialogues, a holdover compatibility mode from OS9) None of these seem to benefit directly from having a cutprefix/cutsuffix method. But splitext aside, I'm generally fiddling a pathname component (and usually a basename), and in that domain the general string functions are very handy and well used. So I think "filename" (basename) fiddling with str methods is actually pretty reasonable. It is _pathname_ fiddling that is hazardous, because the path separators often need to be treated specially. >> 2. Stripping prefixes and suffixes to get to root words. Python has >> been used for natural language work for over a decade, and I don't think >> I've heard any great call from linguists for the functionality. English >> isn't a girl who puts out like that on a first date :-) There are too >> many common exception cases for such a straightforward approach not to >> cause confusion. > >Agreed, using prefix/suffix stripping on natural language is at best a >"quick hack". Yeah. I was looking at the prefix list from a related article and seeing "intra" and thinking "intractable". Hacky indeed. _Unless_ the word has already been qualified as suitable for the action. And once it is, a cutprefix method would indeed be handy. >> 3. My most common use case (not very common at that) is for stripping >> annoying prompts off text-based APIs. I'm happy using .startswith() and >> string slicing for that, though your point about the repeated use of the >> string to be stripped off (or worse, hard-coding its length) is well made. In some ways the verbosity and bugproneness is my personal use case for cutprefix/cutsuffix (however spelt): - repeating the string is wordy and requires human eyeballing whenever I read it (to check for correctness); the same applies whenever I write such a piece of code - personally I'm quite prone to off-by-one errors when hand writing variations on this - a well named method is more readable and expresses intent better (the same argument holds for a standalone function, though a method is a bit better) - the anecdotally not uncommon misuse of .strip() where .cutsuffix() with be correct I confess being a little surprised at how few examples which could use cutsuffix I found in my own code, where I had expected it to be common. I find several bits line this: # parsing text which may have \r\n line endings if line.endswith('\r'): line = line[:-1] # parsing a UNIX network interface listing from ifconfig, # which varies platform to platform if ifname.endswith(':'): ifname = ifname[:-1] Here I DO NOT want rstrip() because I want to strip only one character, rather than as many as there are. So: the optional trailing marker in some input. But doing this for single character markers is much easier to get right than the broader case with longer suffixes, so I think this is not a very strong case. Fiddling the domain suffix on an email address: if not addr.endswith(old_domain): raise ValueError('addr does not end in old_domain') addr2 = addr[:-len(old_domain)] + new_domain which would be a good fit, _except_ for the sanity check. However, that sanity check is just one of a few preceeding the change, so in fact this is a good fit. I have a few classes which annotate their instances with some magic attributes. Here's a snippet from a class' __getattr__ for a db schema: if attr.endswith('_table'): # *_table ==> table "*" nickname = attr[:-6] if nickname in self.table_by_nickname: There's a little suite of "match attribute suffix, trim and do something specific with what's left" if statements. However, they are almost all of the form above, so rewriting it like this: if attr.endswith('_table'): # *_table ==> table "*" nickname = attr.cutsuffix('_table') if nickname in self.table_by_nickname: is a small improvement. Eevry magic number (the "6" above) is an opportunity for bugs. >> I am beginning to worry slightly that actually there are usually more >> appropriate things to do than simply cutting off affixes, and that in >> providing these particular batteries we might be encouraging poor practise. > >It would be really helpful if someone could go through the various use >cases presented in this thread and classify them - filename >manipulation, natural language uses, and "other". Surprisingly for me, the big subjective win is avoiding misuse of lstrip/rstrip by having obvious better named alternatives for affix trimming. Short summary: in my own code I find oportunities for an affix trim method less common than I had expected. But I still like the "might find it useful" argument. I think I find "might find it useful" more compelling than many do. Let me explain. I think a _well_ _defined_ battery is worth including in the kit (str methods) because: - the operation is simple and well defined: people won't be confused by its purpose, and when they want it there is a reliable debugged method sitting there ready for use - variations on this get written _all the time_, and writing those variations using the method is more readable and more reliable - the existing .strip battery is misused for this purpose by accident I have in the past found myself arguing for adding little tools like this in agile teams, and getting a lot of resistence. The resistence tended to take these forms: - YAGNI. While the tiny battery _can_ be written longhand, every time that happens makes for needlessly verbose code, is an opportunity for stupid bugs, and makes code whose purpose must be _deduced_ rather than doing what it says on the tin - not in this ticket: this leads to a starvation issue - the battery never goes in with any ticket, and a ticket just for the battery never gets chosen for a sprint - we've already got this other battery; subtext "not needed" or "we don't want 2 ways to do this", my subtext "does it worse, or does something which only _looks_ like this purpose". Classic example from the codebase I was in at the time was SQL parameter insertion. Eventually I said "... this" and wrote the battery anyway. My position on cut*affix is that (a) it is easy to implement (b) it can thus be debugged once (c) it makes code clearer when used (d) it reduces the liklihood of .strip() misuse. Cheers, Cameron Simpson From eric at trueblade.com Tue Apr 2 19:46:27 2019 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 2 Apr 2019 19:46:27 -0400 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp> Message-ID: On 4/2/2019 2:02 PM, Rhodri James wrote: > On 02/04/2019 18:55, Stephen J. Turnbull wrote: > >> = Me >> ? > 3. My most common use case (not very common at that) is for stripping >> ? > annoying prompts off text-based APIs.? I'm happy using >> ? > .startswith() and string slicing for that, though your point about >> ? > the repeated use of the string to be stripped off (or worse, >> ? > hard-coding its length) is well made. >> >> I don't understand this use case, specifically the opposition to >> hard-coding the length.? Although hard-coding the length wouldn't >> occur to me in many cases, since I'd use >> >> ???? # remove my bash prompt >> ???? prompt_re = re.compile(r'^[^\u0000-\u001f\u007f]+ \d\d:\d\d\$ ') >> ???? lines = [prompt_re.sub('', line) for line in lines] > > For me it's more often like > > ??? input = get_line_from_UART() > ??? if input.startswith("INFO>"): > ??????? input = input[5:] > ??? do_something_useful(input) > > which is error-prone when you cut and paste for a different prompt > elsewhere and forget to change the slice to match. I originally saw this, and I thought "Yeah, me, too!". But then I realize I rarely want to do this. I almost always want to know if the string began with the prefix. I'd normally use something like this: -------------------------- for line in ["INFO>rest-of-line", "not-INFO>more-text", "text", "INFO>", ""]: start, sep, rest = line.partition("INFO>") if not start and sep: print(f"control line {rest!r}") else: print(f"data line {line!r}") output: control line 'rest-of-line' data line 'not-INFO>more-text' data line 'text' control line '' data line '' -------------------------- Breaking it out as a function gives how I'd need to call this, if we made it a function (or method on str): -------------------------- def str_has_prefix(s, prefix): '''returns (True, rest-of-string) or (False, s)''' start, sep, rest = s.partition(prefix) if not start and sep: return True, rest else: return False, s for line in ["INFO>rest-of-line", "not-INFO>more-text", "text", "INFO>", ""]: has_prefix, line = str_has_prefix(line, "INFO>") if has_prefix: print(f"control line {line!r}") else: print(f"data line {line!r}") -------------------------- Now I'll admit it's not super-efficient to create the start, sep, and rest sub-strings all the time, and maybe the test "not start and sep" isn't so obvious at first glance, but for my work this is good enough. It's not super-important how the function (or method) is implemented, I'm more concerned about the interface. If it was done in C, it obviously wouldn't call .partition(). So while I was originally +1 on this proposal, now I'm not so sure, given how I normally need to check if the string starts with a prefix and get the rest of the string if it does start with the prefix. On the other hand, just this weekend I was helping (again) with someone who misunderstood str.strip() on the bug tracker: https://bugs.python.org/issue36480, so I know .strip() and friends confuses people. But I don't think we can use that fact to say that we need .lcut()/.rcut(). It's just that as it's being proposed here, I think lcut/rcut (of whatever names) just doesn't have a useful interface, for me. I don't think I've ever wanted to remove a prefix/suffix if it existed, else use the whole string, and not know which case occurred. Eric PS: I really tried to find a way to use := in this example so I could put the assignment inside the 'if' statement, but as I think Tim Peters pointed out, without C's comma operator, you can't. From shoyer at gmail.com Tue Apr 2 22:06:25 2019 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 2 Apr 2019 19:06:25 -0700 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp>

Message-ID: On Tue, Apr 2, 2019 at 5:43 PM Eric V. Smith wrote: > PS: I really tried to find a way to use := in this example so I could > put the assignment inside the 'if' statement, but as I think Tim Peters > pointed out, without C's comma operator, you can't. > Conceivably cut_prefix could return None if not found. Then you could write something like: if (stripped := cut_prefix(line, "INFO>")) is not None: print(f"control line {stripped!r}") else: print(f"data line {line!r}") You could even drop "is not None" in many circumstances, if you know the cut string will be non-empty. That's actually pretty readable: if stripped := cut_prefix(line, "INFO>"): print(f"control line {stripped!r}") else: print(f"data line {line!r}") -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Apr 2 22:44:11 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 3 Apr 2019 13:44:11 +1100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk>

<23715.42367.172126.972274@turnbull.sk.tsukuba.ac.jp> Message-ID: <20190403024410.GZ31406@ando.pearwood.info> On Tue, Apr 02, 2019 at 07:28:01PM +0100, MRAB wrote: [...] > > word[len(prefix) if word.startswith(prefix) else 0:] > > > It could be 'improved' more to: > > word[word.startswith(prefix) and len(prefix) : ] [...] > _Neither_ version copies if the word doesn't start with the prefix. If > you won't believe me, test them! :-) That slicing doesn't make a copy of the string is an implementation- dependent optimization, not a language guarantee. It's an obvious optimization to make (and in my testing, it does work all the way back to CPython 1.5) but if you want to write implementation-independent code, you shouldn't rely on it. By the letter of the language spec, an interpreter may make a copy of a string when doing a full slice string[0:len(string)]. -- Steven From steve at pearwood.info Tue Apr 2 23:54:10 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 3 Apr 2019 14:54:10 +1100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <20190402225807.GA61774@cskk.homeip.net> References: <20190402225807.GA61774@cskk.homeip.net> Message-ID: <20190403035409.GA31406@ando.pearwood.info> On Wed, Apr 03, 2019 at 09:58:07AM +1100, Cameron Simpson wrote: [...] > Yeah. I was looking at the prefix list from a related article and seeing > "intra" and thinking "intractable". Hacky indeed. That example supports my position that we ought to be cautious about allowing multiple prefixes. The correct prefix in that case is in- not intra-. Deciding which prefix ought to take precedence requires specific domain knowledge, not a simple rule like "first|last|shortest|longest wins". > _Unless_ the word has > already been qualified as suitable for the action. And once it is, a > cutprefix method would indeed be handy. Which is precisely the point. Of course stemming words in full generality is hard. It requires the nuclear reactor of something like NLTK, and even that sometimes gets it wrong. But this is not a proposal for a natural language stemmer, it is a proposal for simple battery which could be used any time you want to cut a known prefix or suffix. [...] > - the anecdotally not uncommon misuse of .strip() where .cutsuffix() > with be correct Anecdotal would be "I knew a guy who made this error", but the evidence presented is objectively verifiable posts on the bug tracker, mailing lists and especially stackoverflow showing that people need to cut affixes and misuse strip for that purpose. > I confess being a little surprised at how few examples which could use > cutsuffix I found in my own code, where I had expected it to be common. I don't expect it to be very common, just common enough to be a repeated source of pain. Its probably more common, and less specialised, than partition and zfill, but less common than startswith/endswith. [...] > if ifname.endswith(':'): > ifname = ifname[:-1] > > Here I DO NOT want rstrip() because I want to strip only one character, > rather than as many as there are. So: the optional trailing marker in > some input. But doing this for single character markers is much easier > to get right than the broader case with longer suffixes, so I think this > is not a very strong case. Imagine that these proposed methods had been added in Python 2.2. Would you be even a tiny bit tempted to write that code above, or would you use the string method? Now imagine it's five years from now, and you're using Python 3.11, and you came across code somebody (possibly even you!) wrote: ifname = ifname.cutsuffix(':') Would you say "Damn, I wish that method had never been added!" and replace it with the earlier code above? Those two questions are not so much aimed at you, Cameron, personally, they're more generic questions for any reader. -- Steven From p.f.moore at gmail.com Wed Apr 3 04:08:47 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 3 Apr 2019 09:08:47 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <20190402225807.GA61774@cskk.homeip.net> References: <20190402225807.GA61774@cskk.homeip.net> Message-ID: On Tue, 2 Apr 2019 at 23:58, Cameron Simpson wrote: > I think I find "might find it useful" more compelling than many do. Let > me explain. > > I think a _well_ _defined_ battery is worth including in the kit (str > methods) because: > > - the operation is simple and well defined: people won't be confused by > its purpose, and when they want it there is a reliable debugged method > sitting there ready for use > > - variations on this get written _all the time_, and writing those > variations using the method is more readable and more reliable > > - the existing .strip battery is misused for this purpose by accident > > I have in the past found myself arguing for adding little tools like > this in agile teams, and getting a lot of resistence. The resistence > tended to take these forms: > > - YAGNI. While the tiny battery _can_ be written longhand, every time > that happens makes for needlessly verbose code, is an opportunity for > stupid bugs, and makes code whose purpose must be _deduced_ rather > than doing what it says on the tin > > - not in this ticket: this leads to a starvation issue - the battery > never goes in with any ticket, and a ticket just for the battery never > gets chosen for a sprint > > - we've already got this other battery; subtext "not needed" or "we > don't want 2 ways to do this", my subtext "does it worse, or does > something which only _looks_ like this purpose". Classic example from > the codebase I was in at the time was SQL parameter insertion. > Eventually I said "... this" and wrote the battery anyway. These are very good arguments, and they aren't something I'd really thought about - they make a very good case (in general) for being sympathetic to proposals for small features that "might be useful", while also offering a couple of good tests for such proposals. "Simple and well defined" in particular strikes me as important (and it's often the one that gets lost when the bikeshedding about end cases starts ;-)) > My position on cut*affix is that (a) it is easy to implement (b) it can > thus be debugged once (c) it makes code clearer when used (d) it reduces > the liklihood of .strip() misuse. IMO, cut*fix at this point is mainly waiting on someone to actually put a feature request on bpo, and an implementation PR on github. At that point, whether it gets implemented will boil down to whether one of the core devs likes it enough to merge. I doubt more discussion here is going to make much difference, and the proposal isn't significant enough to warrant a PEP. Paul From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Apr 3 04:38:14 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 3 Apr 2019 17:38:14 +0900 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk>

<23715.42367.172126.972274@turnbull.sk.tsukuba.ac.jp> Message-ID: <23716.28918.309131.696660@turnbull.sk.tsukuba.ac.jp> MRAB writes: > On 2019-04-02 19:10, Stephen J. Turnbull wrote: > > word[len(prefix) if word.startswith(prefix) else 0:] > It could be 'improved' more to: > > word[word.startswith(prefix) and len(prefix) : ] Except that it would be asymmetric with suffix. That probably doesn't matter given the sequence[:-0] bug. BTW thank you for pointing out that bug (and not quoting the code where I deliberately explicitly introduced the null suffix! ;-) This works: word[:-len(suffix) or len(word)] if word.endswith(suffix) else word Do tutorials mention this pitfall with computed indicies (that -0 is treated as "beginning of sequence")? (I should check myself, but can't spend time this week and so probably won't. :-( ) > > prefix. So that's the one I'd go with, as I can't think of any > > applications where multiple copies of the same string would be useful. > _Neither_ version copies if the word doesn't start with the prefix. If > you won't believe me, test them! :-) Oh, I believe you. It just means somebody long ago thought more deeply about the need for copying immutable objects than I ever have. From python at mrabarnett.plus.com Wed Apr 3 13:50:37 2019 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 3 Apr 2019 18:50:37 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <23716.28918.309131.696660@turnbull.sk.tsukuba.ac.jp> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk>

<23715.42367.172126.972274@turnbull.sk.tsukuba.ac.jp> <23716.28918.309131.696660@turnbull.sk.tsukuba.ac.jp> Message-ID: <8e512886-df9e-af4b-c6ea-ccd146b6df25@mrabarnett.plus.com> On 2019-04-03 09:38, Stephen J. Turnbull wrote: > MRAB writes: >? > On 2019-04-02 19:10, Stephen J. Turnbull wrote: > >? > >????? word[len(prefix) if word.startswith(prefix) else 0:] > >? > It could be 'improved' more to: >? > >? >????? word[word.startswith(prefix) and len(prefix) : ] > > Except that it would be asymmetric with suffix.? That probably doesn't > matter given the sequence[:-0] bug. > > BTW thank you for pointing out that bug (and not quoting the code > where I deliberately explicitly introduced the null suffix! ;-)? This > works: > >???? word[:-len(suffix) or len(word)] if word.endswith(suffix) else word > I would've written it as: ??? word[: len(word) - len (suffix)] if word.endswith(suffix) else word > Do tutorials mention this pitfall with computed indicies (that -0 is > treated as "beginning of sequence")?? (I should check myself, but > can't spend time this week and so probably won't. :-( ) > >? > > prefix.? So that's the one I'd go with, as I can't think of any >? > > applications where multiple copies of the same string would be useful. > >? > _Neither_ version copies if the word doesn't start with the prefix. If >? > you won't believe me, test them! :-) > > Oh, I believe you.? It just means somebody long ago thought more > deeply about the need for copying immutable objects than I ever have. > From python at mrabarnett.plus.com Wed Apr 3 13:54:21 2019 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 3 Apr 2019 18:54:21 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk> <23715.41462.846192.286383@turnbull.sk.tsukuba.ac.jp>

Message-ID: <148ecb5c-3890-27f4-8313-f66b20a0b747@mrabarnett.plus.com> On 2019-04-03 03:06, Stephan Hoyer wrote: > On Tue, Apr 2, 2019 at 5:43 PM Eric V. Smith > wrote: > > PS: I really tried to find a way to use := in this example so I could > put the assignment inside the 'if' statement, but as I think Tim Peters > pointed out, without C's comma operator, you can't. > > > Conceivably cut_prefix could return None if not found. Then you could > write something like: > > if (stripped := cut_prefix(line, "INFO>")) is not None: > ? ? ?print(f"control line {stripped!r}") > else: > ? ? ?print(f"data line {line!r}") > > You could even drop "is not None" in many circumstances, if you know the > cut string will be non-empty. That's actually pretty readable: > > if stripped := cut_prefix(line, "INFO>"): > ? ? ?print(f"control line {stripped!r}") > else: > ? ? ?print(f"data line {line!r}") > -1 Sometimes you just want to remove it if present, otherwise leave the string as-is. I wouldn't want to have to write: line = line.lcut("INFO>") or line From barry at barrys-emacs.org Wed Apr 3 14:03:01 2019 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 3 Apr 2019 19:03:01 +0100 Subject: [Python-ideas] New explicit methods to trim strings In-Reply-To: <8e512886-df9e-af4b-c6ea-ccd146b6df25@mrabarnett.plus.com> References: <7D84D131-65B6-4EF7-9C43-51957F9DFAA9@getmailspring.com> <20190401000837.GK31406@ando.pearwood.info> <20190401013425.GC6059@ando.pearwood.info> <20190402005251.GU31406@ando.pearwood.info> <0e0f691b-ecba-72fa-1d2a-f734d6ad5154@kynesim.co.uk>

<23715.42367.172126.972274@turnbull.sk.tsukuba.ac.jp> <23716.28918.309131.696660@turnbull.sk.tsukuba.ac.jp> <8e512886-df9e-af4b-c6ea-ccd146b6df25@mrabarnett.plus.com> Message-ID: <13B67A27-F6DE-4057-B6ED-40909B390C4D@barrys-emacs.org> Use "without" as the action picking up on "with" as in startswith, endswith: new_string = a_string.withoutprefix( prefix ) new_string = a_sring.withoutsuffix( suffix ) And since we have "replace" "remove" would also seem obvious. new_string = a_string.removeprefix( prefix ) new_string = a_sring.removesuffix( suffix ) I know that some commented that remove sounds like its inplace. But then so does replace. Would "replacesuffix" and "replaceprefix" work? I'd default the "replacement" to the empty string "". new_string = a_string.replaceprefix( old_prefix, replacement ) new_string = a_sring.replacesuffix( old_suffix, replacement ) Barry From greg.ewing at canterbury.ac.nz Thu Apr 4 03:47:30 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Apr 2019 20:47:30 +1300 Subject: [Python-ideas] Add output() helper function to subprocess module Message-ID: <5CA5B692.2060407@canterbury.ac.nz> The check_output() function of the subprocess module raises an exception if the process returns a non-zero exit status. This is inconvenient for commands such as grep that use the return status to indicate something other than success or failure. The check_call() function has a companion call(), but here is currently no non-checking companion for check_call(). How about adding one with a signature such as output(args) --> (status, output) -- Greg From njs at pobox.com Thu Apr 4 04:08:14 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 4 Apr 2019 01:08:14 -0700 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: <5CA5B692.2060407@canterbury.ac.nz> References: <5CA5B692.2060407@canterbury.ac.nz> Message-ID: On Thu, Apr 4, 2019 at 12:48 AM Greg Ewing wrote: > > The check_output() function of the subprocess module raises an > exception if the process returns a non-zero exit status. This is > inconvenient for commands such as grep that use the return > status to indicate something other than success or failure. > > The check_call() function has a companion call(), but here is > currently no non-checking companion for check_call(). How > about adding one with a signature such as > > output(args) --> (status, output) Isn't this already available as: run(args, stdout=PIPE)? Is the object to the extra typing, or...? -n -- Nathaniel J. Smith -- https://vorpus.org From rosuav at gmail.com Thu Apr 4 04:44:29 2019 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Apr 2019 19:44:29 +1100 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: References: <5CA5B692.2060407@canterbury.ac.nz> Message-ID: On Thu, Apr 4, 2019 at 7:12 PM Nathaniel Smith wrote: > > On Thu, Apr 4, 2019 at 12:48 AM Greg Ewing wrote: > > > > The check_output() function of the subprocess module raises an > > exception if the process returns a non-zero exit status. This is > > inconvenient for commands such as grep that use the return > > status to indicate something other than success or failure. > > > > The check_call() function has a companion call(), but here is > > currently no non-checking companion for check_call(). How > > about adding one with a signature such as > > > > output(args) --> (status, output) > > Isn't this already available as: run(args, stdout=PIPE)? Is the object > to the extra typing, or...? > Or discoverability. If you want to run a subprocess and catch its output, you'll naturally reach for check_output, and it feels clunkier to have to use run() instead. +1 on adding a nice simple function, although I'm not 100% sold on the name "output". ChrisA From greg.ewing at canterbury.ac.nz Thu Apr 4 04:59:19 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Apr 2019 21:59:19 +1300 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: References: <5CA5B692.2060407@canterbury.ac.nz> Message-ID: <5CA5C767.2010903@canterbury.ac.nz> Nathaniel Smith wrote: > On Thu, Apr 4, 2019 at 12:48 AM Greg Ewing wrote: >>output(args) --> (status, output) > > Isn't this already available as: run(args, stdout=PIPE)? Yes, but you need to do more than that to get the output as a string. This is the relevant part of the implementation of check_output(): process = Popen(stdout=PIPE, *popenargs, **kwargs) output, unused_err = process.communicate() retcode = process.poll() -- Greg From greg.ewing at canterbury.ac.nz Thu Apr 4 05:01:33 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Apr 2019 22:01:33 +1300 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: References: <5CA5B692.2060407@canterbury.ac.nz>

Message-ID: <5CA5C7ED.8030008@canterbury.ac.nz> Chris Angelico wrote: > +1 on adding a nice simple function, although I'm not 100% sold on the > name "output". The idea is that output/check_output would go together like call/check_call. -- Greg From rosuav at gmail.com Thu Apr 4 05:04:09 2019 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Apr 2019 20:04:09 +1100 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: <5CA5C7ED.8030008@canterbury.ac.nz> References: <5CA5B692.2060407@canterbury.ac.nz>

<5CA5C7ED.8030008@canterbury.ac.nz> Message-ID: On Thu, Apr 4, 2019 at 8:02 PM Greg Ewing wrote: > > Chris Angelico wrote: > > +1 on adding a nice simple function, although I'm not 100% sold on the > > name "output". > > The idea is that output/check_output would go together like > call/check_call. > Yeah, so I think that on balance it's probably the best choice, but as its own thing, it's a bit odd. subprocess.output("...") I'm, let's say, +1 on the idea in general, and +0.9 on calling it "output". ChrisA From njs at pobox.com Thu Apr 4 05:10:50 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 4 Apr 2019 02:10:50 -0700 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: <5CA5C767.2010903@canterbury.ac.nz> References: <5CA5B692.2060407@canterbury.ac.nz> <5CA5C767.2010903@canterbury.ac.nz> Message-ID: On Thu, Apr 4, 2019 at 1:59 AM Greg Ewing wrote: > > Nathaniel Smith wrote: > > On Thu, Apr 4, 2019 at 12:48 AM Greg Ewing wrote: > >>output(args) --> (status, output) > > > > Isn't this already available as: run(args, stdout=PIPE)? > > Yes, but you need to do more than that to get the output > as a string. This is the relevant part of the implementation > of check_output(): > > process = Popen(stdout=PIPE, *popenargs, **kwargs) > output, unused_err = process.communicate() > retcode = process.poll() >>> from subprocess import run, pipe >>> p = run(["grep", "njs", "/etc/passwd"], stdout=PIPE) >>> p.returncode 0 >>> p.stdout b'njs:x:1000:1000:Nathaniel J. Smith,,,:/home/njs:/usr/bin/zsh\n' I do think it's a bit weird that you write 'stdout=PIPE' to mean 'please capture stdout' ? it's leaking an internal implementation detail across an abstraction boundary. But it's documented, and run() allows any combination of check=True/False, capturing stdout or not, and capturing stderr or not, without having to invent 8 different functions. -n -- Nathaniel J. Smith -- https://vorpus.org From storchaka at gmail.com Thu Apr 4 06:22:05 2019 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 4 Apr 2019 13:22:05 +0300 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: <5CA5C767.2010903@canterbury.ac.nz> References: <5CA5B692.2060407@canterbury.ac.nz> <5CA5C767.2010903@canterbury.ac.nz> Message-ID: 04.04.19 11:59, Greg Ewing ????: > Nathaniel Smith wrote: >> On Thu, Apr 4, 2019 at 12:48 AM Greg Ewing >> wrote: >>> output(args) --> (status, output) >> >> Isn't this already available as: run(args, stdout=PIPE)? > > Yes, but you need to do more than that to get the output > as a string. This is the relevant part of the implementation > of check_output(): > > ??? process = Popen(stdout=PIPE, *popenargs, **kwargs) > ??? output, unused_err = process.communicate() > ??? retcode = process.poll() > check_output() is currently implemented (besides arguments checks and legacy support) as just run(..., stdout=PIPE, check=True).stdout For getting unchecked output you need just to omit check=True. run(..., stdout=PIPE).stdout I think that after adding run() there is no need in output(). From phd at phdru.name Thu Apr 4 05:46:32 2019 From: phd at phdru.name (Oleg Broytman) Date: Thu, 4 Apr 2019 11:46:32 +0200 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: References: <5CA5B692.2060407@canterbury.ac.nz>

Message-ID: <20190404094632.ytboieqzcooz6ax3@phdru.name> On Thu, Apr 04, 2019 at 07:44:29PM +1100, Chris Angelico wrote: > On Thu, Apr 4, 2019 at 7:12 PM Nathaniel Smith wrote: > > > > On Thu, Apr 4, 2019 at 12:48 AM Greg Ewing wrote: > > > > > > The check_output() function of the subprocess module raises an > > > exception if the process returns a non-zero exit status. This is > > > inconvenient for commands such as grep that use the return > > > status to indicate something other than success or failure. > > > > > > The check_call() function has a companion call(), but here is > > > currently no non-checking companion for check_call(). How > > > about adding one with a signature such as > > > > > > output(args) --> (status, output) > > > > Isn't this already available as: run(args, stdout=PIPE)? Is the object > > to the extra typing, or...? > > > > Or discoverability. If you want to run a subprocess and catch its > output, you'll naturally reach for check_output, and it feels clunkier > to have to use run() instead. > > +1 on adding a nice simple function, although I'm not 100% sold on the > name "output". get_output ? > ChrisA Oleg. -- Oleg Broytman https://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From guido at python.org Thu Apr 4 13:14:34 2019 From: guido at python.org (Guido van Rossum) Date: Thu, 4 Apr 2019 10:14:34 -0700 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: <20190404094632.ytboieqzcooz6ax3@phdru.name> References: <5CA5B692.2060407@canterbury.ac.nz>

<20190404094632.ytboieqzcooz6ax3@phdru.name> Message-ID: Let?s please leave this alone. As Serhiy says run() covers everything. On Thu, Apr 4, 2019 at 3:03 AM Oleg Broytman wrote: > On Thu, Apr 04, 2019 at 07:44:29PM +1100, Chris Angelico > wrote: > > On Thu, Apr 4, 2019 at 7:12 PM Nathaniel Smith wrote: > > > > > > On Thu, Apr 4, 2019 at 12:48 AM Greg Ewing < > greg.ewing at canterbury.ac.nz> wrote: > > > > > > > > The check_output() function of the subprocess module raises an > > > > exception if the process returns a non-zero exit status. This is > > > > inconvenient for commands such as grep that use the return > > > > status to indicate something other than success or failure. > > > > > > > > The check_call() function has a companion call(), but here is > > > > currently no non-checking companion for check_call(). How > > > > about adding one with a signature such as > > > > > > > > output(args) --> (status, output) > > > > > > Isn't this already available as: run(args, stdout=PIPE)? Is the object > > > to the extra typing, or...? > > > > > > > Or discoverability. If you want to run a subprocess and catch its > > output, you'll naturally reach for check_output, and it feels clunkier > > to have to use run() instead. > > > > +1 on adding a nice simple function, although I'm not 100% sold on the > > name "output". > > get_output ? > > > ChrisA > > Oleg. > -- > Oleg Broytman https://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cspealma at redhat.com Fri Apr 5 08:20:32 2019 From: cspealma at redhat.com (Calvin Spealman) Date: Fri, 5 Apr 2019 08:20:32 -0400 Subject: [Python-ideas] Add output() helper function to subprocess module In-Reply-To: References: <5CA5B692.2060407@canterbury.ac.nz>

<20190404094632.ytboieqzcooz6ax3@phdru.name> Message-ID: This is probably the most common first use case someone has when trying to use subprocess for the first time and I think it has always been a bit of a wart that, given all the helpers and wrappers the subprocess module already has, it lacks one for that very obvious and common need. Yes, run() covers everything, but so does subprocess.Popen() itself. Helpers still help. On Thu, Apr 4, 2019 at 1:15 PM Guido van Rossum wrote: > Let?s please leave this alone. As Serhiy says run() covers everything. > > On Thu, Apr 4, 2019 at 3:03 AM Oleg Broytman wrote: > >> On Thu, Apr 04, 2019 at 07:44:29PM +1100, Chris Angelico < >> rosuav at gmail.com> wrote: >> > On Thu, Apr 4, 2019 at 7:12 PM Nathaniel Smith wrote: >> > > >> > > On Thu, Apr 4, 2019 at 12:48 AM Greg Ewing < >> greg.ewing at canterbury.ac.nz> wrote: >> > > > >> > > > The check_output() function of the subprocess module raises an >> > > > exception if the process returns a non-zero exit status. This is >> > > > inconvenient for commands such as grep that use the return >> > > > status to indicate something other than success or failure. >> > > > >> > > > The check_call() function has a companion call(), but here is >> > > > currently no non-checking companion for check_call(). How >> > > > about adding one with a signature such as >> > > > >> > > > output(args) --> (status, output) >> > > >> > > Isn't this already available as: run(args, stdout=PIPE)? Is the object >> > > to the extra typing, or...? >> > > >> > >> > Or discoverability. If you want to run a subprocess and catch its >> > output, you'll naturally reach for check_output, and it feels clunkier >> > to have to use run() instead. >> > >> > +1 on adding a nice simple function, although I'm not 100% sold on the >> > name "output". >> >> get_output ? >> >> > ChrisA >> >> Oleg. >> -- >> Oleg Broytman https://phdru.name/ >> phd at phdru.name >> Programmers don't die, they just GOSUB without RETURN. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- > --Guido (mobile) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma at redhat.com M: +1.336.210.5107 TRIED. TESTED. TRUSTED. -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Fri Apr 5 22:05:11 2019 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Fri, 5 Apr 2019 22:05:11 -0400 Subject: [Python-ideas] Python list of list matrixes without NumPy Message-ID: The semantics of list operators are congruent, but unintuitive sometimes. For example, I've fallen over this many times: >>> truth_matrix = [[False] * n ] * m It doesn't create what you'd want. The only safe way I know to creat an NxM matrix is: >>> truth_matrix = [ [False for _ in range(n) for _ in range(m)] There could be a library to deal with that common case, and those of checking if a list of lists (of lists?) is properly rectangular, square, etc., and several other common cases. It takes experience to be aware that `truth_matrix[i][j]` will change more than one "cell" if the initialization is the former. `numpy` is not part of stdlib. A standard library for list of list would be usefu, specially to newcomersl. I don't remember such being discussed lately. >>> truth_matrix = lol(n, m, init=bool) >>> other_matrix = lol(n, m, o, p, init=float) Maybe it could also be done with syntax, but I don't have any ideas in that regard (I don't think "lol()" is overloaded). Regards, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From pythonchb at gmail.com Sat Apr 6 14:48:36 2019 From: pythonchb at gmail.com (Christopher Barker) Date: Sat, 6 Apr 2019 11:48:36 -0700 Subject: [Python-ideas] Python list of list matrixes without NumPy In-Reply-To: References:

Message-ID: By the way, your reply isn't go to the list --bringing it back on. On Sat, Apr 6, 2019 at 5:03 AM Juancarlo A?ez wrote: > Or maybe there is one in PyPi ? have you looked? >> > > There's a bunch of "matrix" packages in PyPi, none of them dealing with > the simple use cases I'm talking about. > Be careful about terminology -- "matrix" is a term from mathematics (linear algebra) that has a specific meaning -- I don't think that's what you mean. If it is what you mean, then what you want is numpy (though it fact, numpy is not a matrix system -- it is general purpose n-dimensional arrays designed for numerical computation that includes some features to support linear algebra ). > It could be static methods in `list`? > > x = list.matrix(n, m, default=0.0) > y = list.matrix(n, m, o, p, default=False) > i = list.eye(n, False, True) > If all you want are constructors for common 2-d or n-d arrays that would create lists of lists, then make a handful of utility functions, put them on PyPi, and see if anyone finds them useful -- if so, then offer it up as an addition the standard list object. But for my part, I think simple constructors are of limited utility -- sure, it's an easy source of errors to do it "wrong", e.g.: [[None] * 5] * 6 appears to create a 2 dimensional array, but really has all the rows as references to the same object. But users will discover this pretty quickly, and there is something to be said for folks needing to understand how python works with regard to multiple references to the same mutable object. But what might be more useful is a 2D or ND array class that would provide more than just constructors, but would provide nifty things like 2-dimension indexing: arr = NdArray((3,4), fill=None) arr: [[None, None, None, None], [None, None, None, None], [None, None, None, None]] and nifty things like 2-d indexing: arr[:, 2] to get columns, for instance. Also some controls on resizing -- you really don't want folks able to append arbitrarily to any of this. THAT might gain some traction. (or not, as you get this with numpy, and so much more, anyway. But either way, write it and show its utility first. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Sat Apr 6 15:49:58 2019 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sat, 6 Apr 2019 15:49:58 -0400 Subject: [Python-ideas] Python list of list matrixes without NumPy In-Reply-To: References: Message-ID: On Sat, Apr 6, 2019 at 4:11 AM Steve Barnes wrote: > > ipython > > Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit > (AMD64)] > > Type 'copyright', 'credits' or 'license' for more information > > IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help. > > > > In [1]: [[False]*2]*5 > > Out[1]: > > [[False, False], > > [False, False], > > [False, False], > > [False, False], > > [False, False]] > > # Looks like exactly what was wanted > [ins] In [1]: x = [[False]*2]*5 [ins] In [2]: x Out[2]: [[False, False], [False, False], [False, False], [False, False], [False, False]] [ins] In [3]: x[1][1] = True [ins] In [4]: x Out[4]: [[False, True], [False, True], [False, True], [False, True], [False, True]] -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From t_glaessle at gmx.de Sun Apr 7 07:42:46 2019 From: t_glaessle at gmx.de (=?UTF-8?B?VGhvbWFzIEdsw6TDn2xl?=) Date: Sun, 7 Apr 2019 13:42:46 +0200 Subject: [Python-ideas] cli tool to print value, similar to pydoc Message-ID: <5ff9f31c-4701-6d75-f08a-eeda095ebd32@gmx.de> Hi, what do you think about adding a tool to print a specified object, similar to pydoc shows its help? I regularly find myself checking the contents of variables to get a quick grasp on what the value looks like during development, and then type something like this: ??? python -c 'import os; print(os.pathsep)' ??? # or even ??? python ??? >>> import os ??? >>> os.pathsep other examples may be MODULE.__version__, sys.path, or sys.platform, but really this happens a lot for a lot of different things. It would be nice to just type, e.g. any of: ??? pyval os.pathsep ??? python -m pprint os.pathsep ??? python -p os.pathsep ??? pydoc --value os.pathsep There is already a tool like this on PyPI [1] (sadly py2 only atm), but if you agree that this is a common pattern, I believe it would be a lot more useful to have it in the stdlib. [1] https://pypi.org/project/pyeval/ Additional considerations: It might be useful to add an option to pass a format specification, which would then do the equivalent of: ??? print(format(value, spec)) Note that [1] is even more powerful, because it can directly evaluate expressions, e.g.: ??? pyeval 'math.sin(math.pi/5)' This can be really useful at times, but maybe the base version would be enough for a first step. What do you think? Best, Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: OpenPGP digital signature URL: From bitsink at gmail.com Sun Apr 7 21:53:20 2019 From: bitsink at gmail.com (Nam Nguyen) Date: Sun, 7 Apr 2019 18:53:20 -0700 Subject: [Python-ideas] Built-in parsing library In-Reply-To: References:

Message-ID: On Mon, Apr 1, 2019 at 3:13 PM Terry Reedy