Thanks for looking at other languages do. It gives some hope that this won't end-up being a usability fiasco.

End-user experience isn't something that can just be argued away. Steve and I are reporting a recurring annoyance. The point of a beta release is to elicit these kinds of reports so they can be addressed before it is too late. ISTM you are choosing not to believe the early feedback and don't want to provide a mitigation. This decision reverses 25+ years of Python practice and is the software equivalent of telling users "you're holding it wrong". Instead of an awareness campaign to use the silent-by-default warnings, we're going directly towards breaking working code. That seems pretty user hostile to me. Chris's language survey one shows only language, Lua, that treated this an error. For compiled languages that emit warnings, the end-user will never see those warning so there is no end-user consequence. In our case though, end-users will see the messages and may not have an ability to do anything about it. I wish people with more product management experience would chime in; otherwise, 3.8 is going to ship with an intentional hard-to-ignore annoyance on the premise that we don't like the way people have been programming and that they need to change their code even if it was working just fine.

On Mon, Aug 5, 2019 at 6:47 PM wrote:

I wish people with more product management experience would chime in; otherwise, 3.8 is going to ship with an intentional hard-to-ignore annoyance on the premise that we don't like the way people have been programming and that they need to change their code even if it was working just fine.

I was resisting weighing in since I don't know the discussion around deprecating this language feature in the first place (other than what's given in this thread). However, in the product I work on we made a very similar change in our last release so I'll throw it out there for people to take what they will from it. We have a long standing feature which allows people to define groups of hosts and give them a name. In the past that name could include dashes, dots, and other characters which are not legal as Python identifiers. When users use those group names in our "DSL" (not truly a DSL but close enough), they can do it using either dictionary-lookup syntax (groupvars['groupname']) or using dotted attribute notation groupvars.groupname. We also have a longstanding problem where users will try to do something like groupvars.group-name.... using the dotted attribute notation with group names that aren't proper python identifiers. This causes problems as the name then gets split on the characters that aren't legal in identifiers and results in something unexpected (undefined variable, an actual subtraction operation, etc). In our last release we decided to deprecate and eventually make it illegal to use non-python-identifiers for the group names. At first, product management *did* let us get away with this. But after some time and usage of the pre-releases, they came to realize that this was a major problem. User's had gotten used to being able to use these characters in their group names. They had defined their group names and gotten used to typing their group names and built up a whole body of playbooks that used these group names.... Product management still let us get away with this.. sort of. The scope of the change was definitely modified. Users were now allowed to select whether invalid group names were disallowed (so they could port their installations), allowed with a warning (presumably so they could do work but also see that they were affected) or allow without a warning (presumably because they knew not to use these group names with dotted attribute notation) . This feature was also no longer allowed to be deprecated... We could have a warning that said "Don't do this" but not remove the feature in the future. Now... I said this was a config option.... So what we do have in the release is that the config option allows but warns by default and *the config option* has a deprecation warning. You see... we're planning on changing from warn by default now to disallowing by default in the future so the deprecation is flagging the change in config value. And you know what? User's absolutely hate this. They don't like the warning. They don't like the implication that they're doing something wrong by using a long-standing feature. They don't like that we're going to change the default so that they're current group names will break. They dislike that it's being warned about because of attribute-lookup-notation which they can just learn not to use with their group names. They dislike this so much that some of us have talked about abandoning this idea... instead, having a public group name that users use when they write in the "DSL" and an internal group name that we use when evaluating the group names. Perhaps that works, perhaps it doesn't, but I think that's where my story starts being specific to our feature and no longer applicable to Python and escape sequences.... Now like I said, I don't know the discussions that lead to invalid escape sequences being deprecated so I don't know whether there's more compelling reasons for doing it but it seems to me that there's even less to gain by doing this than what we did in Ansible. The thing Ansible is complaining about can do the wrong thing when used in conjunction with certain other features of our "DSL". The thing that the python escape sequences is complaining about are never invalid (As was pointed out, it's complaining when a sequence of two characters will do what the user intended rather than complaining when a sequence of two characters will do something that the user did not intend). Like the Ansible feature, though, the problem is that over time we've discovered that it is hard to educate users about the exact characteristic of the feature (\k == k but \n == newline; groupvars['group-name'] works but groupvars.group-name does not) so we've both given up on continuing to educate the users in favor of attempting to nanny the user into not using the feature. That most emphatically has not worked for us and has spent a bunch of goodwill with our users but the python userbase is not necessarily the same as ours so the audience may not be as tough to please. -Toshio P.S. One entry point to "discussions" about the Ansible feature: https://github.com/ansible/ansible/issues/56930 That bug starts off just asking for better documentation but becomes embroiled with other asks and just general dislike of the idea as a whole... getting linked to other issues, PRs, and referenced by mailing list threads..... The dislike of this feature really is a many headed hydra).

Making it an error so soon would be mistake, IMHO. That will break currently working code for small benefit. When Python was a young language with a few thousand users, it was easier to make these kinds of changes. Now, we should be much more conservative and give people a long time and a lot of warning. Ideally, we should provide tools to fix code if possible. Could PyPI and pip gain the ability to warn and even fix these issues? Having a warning from pip at install time could be better than a warning at import time. If linting was built into PyPI, we could even do a census to see how many packages would be affected by turning it into an error. On 2019-08-05, raymond.hettinger@gmail.com wrote:

P.S. In the world of C compilers, I suspect that if the relatively new compiler warnings were treated as errors, the breakage would be widespread. Presumably that's why they haven't gone down this road.

The comparision with C compilers is relevant. C and C++ represent a fairly extreme position on not breaking working code. E.g. K & R style functional declarations were supported for decades. I don't think we need to go quite that far but also one or two releases is not enough time. Regards, Neil

On Tue, Aug 6, 2019 at 10:06 AM Neil Schemenauer wrote:

Making it an error so soon would be mistake, IMHO. That will break currently working code for small benefit. When Python was a young language with a few thousand users, it was easier to make these kinds of changes. Now, we should be much more conservative and give people a long time and a lot of warning. Ideally, we should provide tools to fix code if possible.

Could PyPI and pip gain the ability to warn and even fix these issues? Having a warning from pip at install time could be better than a warning at import time. If linting was built into PyPI, we could even do a census to see how many packages would be affected by turning it into an error.

PyPI could warn on or reject packages at upload time when they contain these kinds code compileall -> pyc time of issues. that'd force the issue to be in front of the package owners rather than anyone else. Similarly, automation could proactively scan existing pypi packages claiming py3 compatibility for the issue and reach out to owners. But the issue becomes one of versions: we shouldn't be complaining about older versions of packages to the package owners. yet people are free to list old versions or < constraints in their own packages requirements regardless of what python version they're running on. -gps

On 2019-08-05, raymond.hettinger@gmail.com wrote:

...

P.S. In the world of C compilers, I suspect that if the relatively new compiler warnings were treated as errors, the breakage would be widespread. Presumably that's why they haven't gone down this road.

The comparision with C compilers is relevant. C and C++ represent a fairly extreme position on not breaking working code. E.g. K & R style functional declarations were supported for decades. I don't think we need to go quite that far but also one or two releases is not enough time.

Regards,

Neil _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/V2EDFDJG...

On 8/5/2019 4:30 PM, raymond.hettinger@gmail.com wrote:

Thanks for weighing in. I think this is an important usability discussion. IMO it is the number one issue affecting the end user experience with this release. If we could get more people to actively use the beta release, the issue would stand-out front and center. But if people don't use the beta in earnest, we won't have confirmation until it is too late.

We really don't have to go this path. Arguably, the implicit conversion of '\latex' to '\\latex' is a feature that has existed for three decades, and now we're deciding to turn it off to define existing practices as errors. I don't think any commercial product manager would allow this to occur without a lot of end user testing.

As much as I'd love to force this change through [0], it really does seem like we're forcing it. Especially given Nathaniel's point about the discoverability problems with compile-time warnings, I think we should delay a visible warning about this. Possibly in 3.9 we can do something about making these warnings visible at run time, not just compile time. I had a similar problems with f-strings (can't recall the details now, since resolved), and the compile-time-only nature made it difficult to notice. I realize a run-time warning for this would require a fair bit of work that might not be worth it. I think Raymond's point goes beyond this. I think he's proposing that we never make this change. I'm sympathetic to that, too. But the first step is to change 3.8's behavior to not make this visible. That is, we should restore the 3.7 warning behavior. Eric [0] And the real reason I'd like this is so we can add \e

Rob Cliffe via Python-Dev wrote:

Sorry, that won't work. Strings are parsed at compile time, open() is executed at run-time.

It could check for control characters, which are probably the result of a backslash accident. Maybe even auto-correct them... -- Greg

On 06/08/2019 23:41:25, Greg Ewing wrote:

Rob Cliffe via Python-Dev wrote:

...

Sorry, that won't work.  Strings are parsed at compile time, open() is executed at run-time.

It could check for control characters, which are probably the result of a backslash accident. Maybe even auto-correct them...

By "It", do you mean open() ?  If so: It already checks for control characters, at least with Python 2.7 on Windows:

...

...

open('mydir\test') Traceback (most recent call last):   File "<stdin>", line 1, in <module> IOError: [Errno 22] invalid mode ('r') or filename: 'mydir\test'

As for auto-correct (presumably "\a" to "\\a", "\b" to "\\b" etc.), I hope you're not serious. "In the face of gibberish, refuse the temptation to show how smart your guessing is."

On Tue, 6 Aug 2019 at 17:39, Matt Billenstein wrote:

On Mon, Aug 05, 2019 at 04:22:50AM -0000, raymond.hettinger@gmail.com wrote:

...

This once seemed like a reasonable and innocuous idea to me; however, I've been using the 3.8 beta heavily for a month and no longer think it is a good idea. The warning crops up frequently, often due to third-party packages (such as docutils and bottle) that users can't easily do anything about.

Perhaps those packages could be flagged now via pylint and problems raised with the respective package maintainers before the actual 3.8 release? Checking the top 100 or top 1000 packages on PyPI?

I don't see issues reported in the bug trackers for docutils and bottle. Maybe as a start, someone could raise issues there? And any other projects Raymond encountered issues with? If nothing else, it's polite to give these projects a warning now that they should be stricter about how they use escape sequences, because the core devs intend to deprecate and ultimately remove the current permissive behaviour. That's what the 3.8 betas are for, after all. If the feedback from bug reports like this is that projects consider it an unacceptable burden to change, then maybe we would then rethink the timescales of the deprecation, or even whether we should do it at all. If they just release a quick fix, maybe we're worrying over the wrong thing here? I remain ambivalent about the change myself. The point that it's the false positives we *want* to address, but this change hits the false negatives, is a telling one for me. I don't think I'd support the change now if we were discussing it for the first time. But on the other hand, the discussion has already happened, and the decision was made, and while I'm OK with responding to the evidence that having loud user-visible warnings is a bad UX, I don't think there's any new evidence here that significantly changes the facts on which the decision to eventually make invalid escapes an error was made - just evidence that the way we chose to introduce that change may be a problem. (And honestly, to my knowledge, we've never yet found a case where there *hasn't* been controversy about warnings from libraries being triggered by user code, so it's not like this is a new problem). Paul

On Tue, Aug 6, 2019 at 3:44 PM Brett Cannon wrote:

I think this is a good example of how the community is not running tests with warnings on and making sure that their code is warnings-free. This warning has existed for at least one full release and fixing it doesn't require some crazy work-around for backwards compatibility, and so this tells me people are simply either ignoring the warnings or they are not aware of them.

If it's the case that people are choosing to ignore warnings then that's on them and there's not much we can do there.

But my suspicion is it's the latter case of people simply not thinking about running with warnings on and making sure to check for them. For instance, are people running their CI with warnings turned on? How about making sure to check the output of their CI to make sure there are no warnings? Or even better, how many people are running CI with warnings turned into exceptions? My guess is all of this is rather low because people are probably just doing `pytest` without thinking of turning on warnings as exceptions to trigger a CI failure and are only looking for CI passing versus checking its output.

There's an important point here that I think has been missed. These days deprecation warnings are much more visible in general, because all the major test systems enable them by default. BUT, this SPECIFIC warning almost completely circumvented all those systems, so almost no-one saw it. For example, all my projects run tests with deprecation warnings enabled and warnings turned into errors, but I never saw any of these warnings. What happens is: the warning is issued when the .py file is byte-compiled; but at this point, deprecation warnings probably aren't visible. Later on, when pytest imports the file, it has warnings enabled... but now the warning isn't issued. Quoting Aaron Meurer from the bpo thread:

As an anecdote, for SymPy's CI, we went through five (if I am counting correctly) iterations of trying to test this. Each of the first four were subtly incorrect, until we finally managed to find the correct one (for reference, 'python -We:invalid -m compileall -f -q module/'). So most library authors who will attempt to add tests against this will get it wrong.

Since folks don't seem to be reading that thread, I'll re-post my comment from it as well:

I think we haven't *actually* done a proper DeprecationWarning period for this. We tried, but because of the issue with byte-compiling, the warnings were unconditionally suppressed for most users -- even the users who are diligent enough to enable warnings and look at warnings in their test suites. I can see a good argument for making the change, but if we're going to do it then it's obviously the kind of change that requires a proper deprecation period, and that hasn't happened. Maybe .pyc files need to be extended to store a list of syntax-related DeprecationWarnings and SyntaxWarnings, that are re-issued every time the .pyc is loaded? Then we'd at least have the technical capability to deprecate this properly.

-n -- Nathaniel J. Smith -- https://vorpus.org

07.08.19 01:37, Brett Cannon пише:

I think this is a good example of how the community is not running tests with warnings on and making sure that their code is warnings-free. This warning has existed for at least one full release and fixing it doesn't require some crazy work-around for backwards compatibility, and so this tells me people are simply either ignoring the warnings or they are not aware of them.

There are several PRs for fixing warnings on GitHub every month. And seems a deprecation warning about importing ABCs from collections is at least so common (if not more) as a warning about "invalid escape sequences". The former is more visible to end users because is emitted at every run, not only at the first bytecode compilation.

On Tue, Aug 06, 2019 at 04:32:04PM +0000, Matt Billenstein wrote:

Perhaps those packages could be flagged now via pylint and problems raised with the respective package maintainers before the actual 3.8 release? Checking the top 100 or top 1000 packages on PyPI?

fwiw, ran pylint on the top 100 pypi pkgs from: https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.json The list of packages is pretty small: https://gist.github.com/mattbillenstein/ad862d032b8575f8d6e08384850f2223 but some have quite a few errors... m -- Matt Billenstein matt@vazor.com http://www.vazor.com/

On Wed, Aug 7, 2019 at 10:03 AM Steven D'Aprano wrote:

- Keep the SyntaxWarning silent by default for 3.8. That gives us another year or more to gently pressure third-party libraries to fix their code, and to find ways to encourage developers to run with warnings enabled.

How do you propose to apply this pressure? How about: whenever a third-party library uses a potentially-wrong escape sequence, it creates a message on the console. Then when someone sees that message, they can post a bug report against the package. In other words, a non-silent warning. ChrisA

People distribute code via pypi. if we reject uploads of packages with these problems and link to fixers (modernize can be taught what to do), we prevent them from spreading further. A few years after doing that, we can revisit how much pain and for whom making this a SyntaxWarning or even SyntaxError would actually be. it isn't useful to tell *users* of packages to spend time figuring out who to complain to that some packages code that they somehow depend on (often transitively) is not modern enough. On Tue, Aug 6, 2019 at 5:39 PM Rob Cliffe via Python-Dev < python-dev@python.org> wrote:

On 07/08/2019 01:14:08, Chris Angelico wrote:

...

On Wed, Aug 7, 2019 at 10:03 AM Steven D'Aprano wrote:

...

- Keep the SyntaxWarning silent by default for 3.8. That gives us another year or more to gently pressure third-party libraries to fix their code, and to find ways to encourage developers to run with warnings enabled. How do you propose to apply this pressure?

How about: whenever a third-party library uses a potentially-wrong escape sequence, it creates a message on the console. Then when someone sees that message, they can post a bug report against the package.

In other words, a non-silent warning.

ChrisA _______________________________________________

The interpreter knows which module contains the questionable string. So: is it feasible for the warning message to include something like "... If you are not the maintainer of xxxmodule.py, please contact them, or post a bug report on ..." _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JJULDRHO...

07.08.19 03:57, Gregory P. Smith пише:

People distribute code via pypi.  if we reject uploads of packages with these problems and link to fixers (modernize can be taught what to do), we prevent them from spreading further.

How can we check that there are such problems in the package? Pass all *.py files through a linter? But the package can contain "incorrect" files, for example files for Python 2 or earlier Python 3 versions. Even the CPython testsuite contains bad Python files for testing purpose.

On Wed, Aug 07, 2019 at 10:14:08AM +1000, Chris Angelico wrote:

On Wed, Aug 7, 2019 at 10:03 AM Steven D'Aprano wrote:

...

- Keep the SyntaxWarning silent by default for 3.8. That gives us another year or more to gently pressure third-party libraries to fix their code, and to find ways to encourage developers to run with warnings enabled.

How do you propose to apply this pressure?

We already have some good information about the offending libraries. (Remember, the libraries here haven't done anything wrong. They were using a documented feature. We've just changed our mind about that feature.) Raymond mentioned two, docutils and bottle, and Matt did a scan of the top 100 downloads on PyPI. We can start by reporting this as a bug to them.

How about: whenever a third-party library uses a potentially-wrong escape sequence, it creates a message on the console. Then when someone sees that message, they can post a bug report against the package.

You're right, of course, and if we were talking about one or two warnings a week, affecting a handful of users, I don't think Raymond would have said anything. But apparently this is a widespread problem with common third party libraries. That means its going to affect lots of people. We have a few problems: - The people affected will mostly be the end users, not the developers. - These sorts of SyntaxWarnings are scary and intimdating to beginners, even when they are harmless. Many of them will not know how to silence warnings, or who to report it as a bug to. - Since end users rarely search for existing bug reports before adding a new one, or upgrade to the latest version, we're effectively sentencing the third-party library authors to be flooded with potentially dozens of identical bug reports long after they have fixed the issue. - I expect that many end users will report it as a *Python* bug, so we're going to share some of that pain too. - The benefit of the desired change is relatively low. The intention was for the developers, not end users, to see the warning. If end users see more than a tiny number of these warnings, our plan failed. That's okay: since the benefit of the breaking change is small, we can rethink the plan, delay the breaking change, and try to come up with a better system that ensures developers see these warnings before their users do. We're not fixing a major security issue here, or adding a new feature that will make people's code enormously better. We're breaking people's code to force them to write "better" code, so that *maybe* some day in the future we can add new escape sequences. That's a really small benefit for breaking backwards compatibility. We don't break backwards compatibility lightly because of the knock on effects of code churn, libraries that stop working, frustrated users, obsoleted blog posts and books, questions asked on Stackoverflow etc. When the benefit is small, we require the pain to be correspondingly small. That's not going to be the case if we continue with the plan. Don't think of this as a failure. Think of it as an opportunity: we've identified a weakness in our deprecation process. Let's fix that process, make sure that *developers* will see the warning in 3.8 or 3.9, and not raise an exception until 4.0 or 4.1. I know people just want to get it over and done with, I do too. But we have responsibilities to the community, and we've lived with the current behaviour for 25+ years, another 2-3 years won't kill us. -- Steven

On Wed, Aug 07, 2019 at 02:33:51PM +1000, Chris Angelico wrote:

On Wed, Aug 7, 2019 at 1:54 PM Steven D'Aprano wrote:

...

Don't think of this as a failure. Think of it as an opportunity: we've identified a weakness in our deprecation process. Let's fix that process, make sure that *developers* will see the warning in 3.8 or 3.9, and not raise an exception until 4.0 or 4.1.

So HOW are you going to make sure developers see it?

I've only just started thinking about it, give me a couple of minutes! *wink* What's the rush? Let's be objective here: what benefit are we going to get from this change? Is there anyone hanging out desperately for "\d" and "\-" to become SyntaxErrors, so they can... do what? Because our processes don't work the way we assumed, it turns out that in practice we haven't given developers the deprecation period we thought we had. Read Nathaniel's post, if you haven't already done so: https://mail.python.org/archives/list/python-dev@python.org/message/E7QCC74O... He makes a compelling case that while we might have had the promised deprecation period by the letter of the law, in practice most developers will have never seen it, and we will be breaking the spirit of the promise if we continue with the unmodified plan. Quite frankly, if we continue with the unmodified plan, third-party devs who are affected will have the right to feel mightly pissed off at us. We make an implicit, if not explicit, promise that we won't break backswards compatibility lightly, but if we do, we will give them plenty of notice except under the most dire circumstances (such as a serious security vulnerability). And yet here we are rushing through a breaking change in an accelerated manner, for a change of marginal benefit. Sure, we can say that *technically* we gave them all the notice promised, it was at the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of The Leopard". https://www.goodreads.com/quotes/40705-but-the-plans-were-on-display-on-disp... I'm sure that the affected devs will understand why it was *their* fault they couldn't see the warnings, when even people from a first-class library like SymPy took four iterations to do it right.

Currently it requires some extra steps or flags, which are not well known. What change are you proposing for 3.8 that will ensure that this actually gets solved?

Absolutely nothing. I don't have to: we're an entire community, this doesn't have to fall only on my shoulders. I'm not even the messenger: that's Raymond. I'm just (partly) agreeing with him. Just because I don't have a solution for this problem doesn't mean the problem doesn't exist.

Otherwise, all you're doing is saying "I wish this problem would just go away".

No, I'm saying we don't have to rush this into 3.8. Let's keep the warning silent and push everything back a release. Now is better than never. Although never is often better than *right* now. Right now, we're looking at a seriously compromised user-experience for 3.8. People are going to hate these warnings, many of them won't know what to do with them and will be sure that Python is buggy, and for very little benefit. Let's slow down and put it off for another release, giving us time to solve the warnings problem, and library authors the deprecation period promised.

Library authors can start _right now_ fixing their code so it's more 3.8 compatible.

Provided that (1) they are aware that this is a problem that needs to be fixed, and (2) they have the round tuits to actually fix it by 3.8.0. Neither are guaranteed. Its not a big fix, but people have other priorities, like work, family, a life, etc. That's why we normally give developers *multiple years* of warnings to fix problems, not weeks. This change is not so important that we have to push it through in an accelerated time frame.

("More" because 3.8 doesn't actually break anything.) What is actually gained by waiting longer

We gain the avoidance of a painful experience in 3.8 for a significant number of users and third-party devs. The question we haven't had answered is what we gain by pushing through with the original plan. Plenty of people have said "Let's just do it" but as far as I can see not one has explained *why* we should put end- users and library developers through this frustrating and annoying rushed deprecation period. -- Steven

On Wed, Aug 7, 2019 at 7:33 PM Steven D'Aprano wrote:

What's the rush? Let's be objective here: what benefit are we going to get from this change? Is there anyone hanging out desperately for "\d" and "\-" to become SyntaxErrors, so they can... do what?

So that problems can start to be detected. Time and again, Python users on Windows get EXTREMELY confused by the way their code worked perfectly with one path, then bizarrely fails with another. That is a very real problem, and the problem is that it appeared to work when actually it was wrong. Python has a history of fixing these problems. It used to be that b"\x61\x62\x63\x64" was equal to u"abcd", but now Python sees these as fundamentally different. Data-dependent bugs caused by a syntactic oddity are a language flaw that needs to be fixed.

Because our processes don't work the way we assumed, it turns out that in practice we haven't given developers the deprecation period we thought we had. Read Nathaniel's post, if you haven't already done so:

https://mail.python.org/archives/list/python-dev@python.org/message/E7QCC74O...

He makes a compelling case that while we might have had the promised deprecation period by the letter of the law, in practice most developers will have never seen it, and we will be breaking the spirit of the promise if we continue with the unmodified plan.

Yes, that's a fair complaint. But merely pushing the deprecation back by a version is not solving it. There has to be SOMETHING done differently.

And yet here we are rushing through a breaking change in an accelerated manner, for a change of marginal benefit.

It's not a marginal benefit. For people who try to teach Python on multiple operating systems, this is a very very real benefit. Just because YOU don't see the benefit doesn't mean it isn't there.

...

Otherwise, all you're doing is saying "I wish this problem would just go away".

No, I'm saying we don't have to rush this into 3.8. Let's keep the warning silent and push everything back a release.

Now is better than never. Although never is often better than *right* now.

Not sure how the Zen supports what you're saying there, since you're specifically saying "not never, not now, just later". But what do you actually mean by not rushing this into 3.8?

Right now, we're looking at a seriously compromised user-experience for 3.8. People are going to hate these warnings, many of them won't know what to do with them and will be sure that Python is buggy, and for very little benefit.

Then the problem is that people blame Python for these warnings. That is a problem to be solved; we need people to understand that a warning emitted by a library is a *library bug* not a language flaw.

...

Library authors can start _right now_ fixing their code so it's more 3.8 compatible.

Provided that (1) they are aware that this is a problem that needs to be fixed, and (2) they have the round tuits to actually fix it by 3.8.0. Neither are guaranteed.

(1) Yes it is, see above; (2) fair point, but this is restricted to string literals and can be detected simply by compiling the code, so it's a reasonably findable problem.

...

("More" because 3.8 doesn't actually break anything.) What is actually gained by waiting longer

We gain the avoidance of a painful experience in 3.8 for a significant number of users and third-party devs.

The question we haven't had answered is what we gain by pushing through with the original plan. Plenty of people have said "Let's just do it" but as far as I can see not one has explained *why* we should put end- users and library developers through this frustrating and annoying rushed deprecation period.

And unless you have a plan to do something different in 3.8 that ensures that library devs see the warnings, there's no justification for the delay. All you'll do is defer the exact same problem by another eighteen months. If the warning remains silent in 3.8, how will library devs get any indication that they need to fix something? If you can offer a better plan, then by all means, do so. But deferring without a change is of no real value, and it means ANOTHER eighteen months added onto the time before novice programmers get to be told about string literal problems. ChrisA

On 07Aug2019 0247, Chris Angelico wrote:

On Wed, Aug 7, 2019 at 7:33 PM Steven D'Aprano wrote:

...

What's the rush? Let's be objective here: what benefit are we going to get from this change? Is there anyone hanging out desperately for "\d" and "\-" to become SyntaxErrors, so they can... do what?

So that problems can start to be detected. Time and again, Python users on Windows get EXTREMELY confused by the way their code worked perfectly with one path, then bizarrely fails with another. That is a very real problem, and the problem is that it appeared to work when actually it was wrong. [...] If you can offer a better plan, then by all means, do so. But deferring without a change is of no real value, and it means ANOTHER eighteen months added onto the time before novice programmers get to be told about string literal problems.

Allow me to offer one: * change the SyntaxWarning into a default-silenced one that fires every time a .pyc is loaded (this is the hard part, but it's doable) * change pathlib.PureWindowsPath, os.fsencode and os.fsdecode to explicitly warn when the path contains control characters * change the PyErr_SetExcFromWindowsErrWithFilenameObjects function to append (or chain) an extra message when either of the filenames contains control characters (or change OSError to do it, or the default sys.excepthook) I don't care whether the changes are applied to all platforms rather than just Windows, but since Windows developers hit the problem and (some) Linux developers like to use control characters in filenames, I can see a justification for only warning on Windows. Long term we can still deprecate and eventually block unrecognized escape sequences, but the long standing behaviour can stand for a few more years without creating more harm. Cheers, Steve

This whole thread would be an excellent justification for following 3.9 with 4.0. It's as near as we ever want to get to a breaking change, and a major version number would indicate the need to review. If increasing strictness of escape code interpretation in string literals is the only incompatibility there would surely be general delight. Kind regards, Steve Holden On Wed, Aug 7, 2019 at 8:19 PM eryk sun wrote:

On 8/7/19, Steve Dower wrote:

...

* change the PyErr_SetExcFromWindowsErrWithFilenameObjects function to append (or chain) an extra message when either of the filenames contains

c

...

control characters (or change OSError to do it, or the default sys.excepthook)

On a related note for Windows, if the error is specifically ERROR_INVALID_NAME, we could extend this to look for and warn about the five reserved wildcard characters (asterisk, question mark, double quote, less than, greater than), pipe, and colon. It's only sometimes the case for colon because it's allowed in device names and used as the name and type delimiter for stream names.

Kernel object names don't reserve wildcard characters, pipe, and colon. So I wouldn't want anything but the control-character warning if it's say ERROR_FILE_NOT_FOUND. An example would be SharedMemory(name='Global\test'), or a similar error for registry key and value names such as OpenKey(hkey, 'spam\test'), that is if winreg were updated to include the name in the exception. Note that forward slash is just a name character in these cases, not a path separator, so we have to use backslash, even if just via replace('/', '\\'). _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UFMVFL4Q...

Steve Holden writes:

This whole thread would be an excellent justification for following 3.9 with 4.0. It's as near as we ever want to get to a breaking change, and a major version number would indicate the need to review. If increasing strictness of escape code interpretation in string literals is the only incompatibility there would surely be general delight.

This should be the first chapter in the Beautiful Version Numbering book! I love it!

This discussion looks like there's no end in sight. Maybe the Steering Council should take a vote? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On 09Aug2019 0905, Serhiy Storchaka wrote:

09.08.19 18:30, Guido van Rossum пише:

...

This discussion looks like there's no end in sight. Maybe the Steering Council should take a vote?

Possible options:

1. SyntaxWarning in 3.8+ (the current status). 2. DeprecationWarning in 3.8, SyntaxWarning in 3.9+ (revert changes in 3.8 only). 3. DeprecationWarning in 3.8 and 3.9 (revert changes in master and 3.8). 4. No warnings at all.

I also posted another possible option that helps solve the real problem faced by users, and not just the "we want to have a warning" problem that is purely ours.

* change the SyntaxWarning into a default-silenced one that fires every time a .pyc is loaded (this is the hard part, but it's doable) * change pathlib.PureWindowsPath, os.fsencode and os.fsdecode to explicitly warn when the path contains control characters * change the PyErr_SetExcFromWindowsErrWithFilenameObjects function to append (or chain) an extra message when either of the filenames contains control characters (or change OSError to do it, or the default sys.excepthook)

Cheers, Steve

09.08.19 19:39, Steve Dower пише:

I also posted another possible option that helps solve the real problem faced by users, and not just the "we want to have a warning" problem that is purely ours.

Warnings solve two problems: * Teaching users that a backslash has special meaning and should be escaped unless it is used for special meaning. * Avoid breaking or introducing bugs if we add new escape sequences (like \e).

...

* change the SyntaxWarning into a default-silenced one that fires every time a .pyc is loaded (this is the hard part, but it's doable)

It was considered an advantage that these warnings are shown only once at compile time. So they will be shown to the author of the code, but the user of the code will not see them (except of installation time). Actually we need to distinguish the the author and the user of the code and show warnings only to the author. Using .pyc files was just an heuristic: the author compiles the Python code, and the user uses compiled .pyc files. Would be nice to have more reliable way to determine the owning of the code. It is related not only to SyntaxWarnings, but to runtime DeprecationWarnings. Maybe silence warnings only for readonly files and make files installed by PIP readonly?

...

* change pathlib.PureWindowsPath, os.fsencode and os.fsdecode to explicitly warn when the path contains control characters

This can cause additional harm. Currently you get expected FileNotFound when use user specified bad path, it can be caught and handled. But with warnings you will either get a noise on the output or an unexpected unhandled error.

...

* change the PyErr_SetExcFromWindowsErrWithFilenameObjects function to append (or chain) an extra message when either of the filenames contains control characters (or change OSError to do it, or the default sys.excepthook)

I do not understand what goal will be achieved by this.

On Sat, 10 Aug 2019 at 01:44, Guido van Rossum wrote:

This discussion looks like there's no end in sight. Maybe the Steering Council should take a vote?

I find the "Our deprecation warnings were even less visible than normal" argument for extending the deprecation period compelling. I also think the UX of the warning itself could be reviewed to provide a more explicit nudge towards using raw strings when folks want to allow arbitrary embedded backslashes. Consider: SyntaxWarning: invalid escape sequence \, vs something like: SyntaxWarning: invalid escape sequence \, (Note: adding the raw string literal prefix, r, will accept all non-trailing backslashes) After all, the habit we're trying to encourage is "If I want to include backslashes without escaping them all, I should use a raw string", not "I should memorize the entire set of valid escape sequences" or even "I should always escape backslashes". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 8/9/2019 2:28 PM, Jonathan Goble wrote:

On Fri, Aug 9, 2019 at 12:34 PM Nick Coghlan wrote:

...

I find the "Our deprecation warnings were even less visible than normal" argument for extending the deprecation period compelling. Outsider's 2 cents from reading this discussion (with no personal experience with this warning):

I am perplexed at the opinion, seemingly espoused by multiple people in this thread, that because a major part of the problem is that the warnings were not visible enough, somehow the proposed solution is making them not visible enough again? It's too late, in my understanding, in the 3.8 cycle to add a new feature like a change to how these warnings are produced (it seems a significant change to the .pyc structure is needed to emit them at runtime), so this supposed "solution" is nothing but kicking the can down the road. When 3.9 rolls around, public exposure to the problem of invalid escape sequences will still be approximately what it is now (because if nobody saw the warnings in 3.7, they certainly won't see them in 3.8 with this "fix"), so you'll end up with the same complaints about SyntaxWarning that started this discussion, end up back on DeprecationWarning for 3.9 (hopefully with support for emitting them at runtime instead of just compile-time), then have to wait until 3.10/4.0 for SyntaxWarning and eventually the next version to actually make them errors.

Yes, I think that's the idea: Deprecation warning in 3.9, but more visible that what 3.7 has. That is, not just at compile time but at run time. What's required to make that happen is an open question.

It seems to me, in my humble but uneducated opinion, that if people are not seeing the warnings, then continuing to give them warnings they won't see isn't a solution to anything. Put the warning front and center. The argument of third-party packages will always be an issue, even if we wait ten years. So put these warnings front and center now so package and code maintainers actually see it, and I'll bet the problematic escape sequences get fixed rather quickly.

What am I missing here?

Hopefully the warnings in 3.9 would be more visible that what we saw in 3.7, so that library authors can take notice and do something about it before 3.10 rolls around. Eric

Eric V. Smith wrote:

Hopefully the warnings in 3.9 would be more visible that what we saw in 3.7, so that library authors can take notice and do something about it before 3.10 rolls around. Eric

Apologies for the ~double-post on the thread, but: the SymPy team has figured out the right pytest incantation to expose these warnings. Given the extensive adoption of pytest, perhaps it would be good to combine (1) a FR on pytest to add a convenience flag enabling this mix of options with (2) an aggressive "marketing push", encouraging library maintainers to add it to their testing/CI.

Nathaniel Smith wrote:

Unfortunately, their solution isn't a pytest incantation, it's a separate 'compileall' invocation they run on their source tree. I'm not sure how you'd convert this into a pytest feature, because I don't think pytest always know which parts of your code are your code versus which parts are supporting libraries. -n

Ahh, did not appreciate this. :-( Nevermind, then!

On 8/9/2019 4:07 PM, Gregory P. Smith wrote:

On Fri, Aug 9, 2019 at 11:37 AM Eric V. Smith mailto:eric@trueblade.com> wrote:

On 8/9/2019 2:28 PM, Jonathan Goble wrote: > On Fri, Aug 9, 2019 at 12:34 PM Nick Coghlan mailto:ncoghlan@gmail.com> wrote: >> I find the "Our deprecation warnings were even less visible than >> normal" argument for extending the deprecation period compelling. > Outsider's 2 cents from reading this discussion (with no personal > experience with this warning): > > I am perplexed at the opinion, seemingly espoused by multiple people > in this thread, that because a major part of the problem is that the > warnings were not visible enough, somehow the proposed solution is > making them not visible enough again? It's too late, in my > understanding, in the 3.8 cycle to add a new feature like a change to > how these warnings are produced (it seems a significant change to the > .pyc structure is needed to emit them at runtime), so this supposed > "solution" is nothing but kicking the can down the road. When 3.9 > rolls around, public exposure to the problem of invalid escape > sequences will still be approximately what it is now (because if > nobody saw the warnings in 3.7, they certainly won't see them in 3.8 > with this "fix"), so you'll end up with the same complaints about > SyntaxWarning that started this discussion, end up back on > DeprecationWarning for 3.9 (hopefully with support for emitting them > at runtime instead of just compile-time), then have to wait until > 3.10/4.0 for SyntaxWarning and eventually the next version to actually > make them errors.

Yes, I think that's the idea: Deprecation warning in 3.9, but more visible that what 3.7 has. That is, not just at compile time but at run time. What's required to make that happen is an open question.

i've lost track of who suggested what in this thread, but yes, that concept has been rolling over in my mind as a potentially good idea after someone suggested it.  Compile time warnings should turn into bytecode for a warnings.warn call in the generated pyc.  I haven't spent time trying to reason if that actually addresses the real issues we're having moving forward with a syntax warning change though. A reasonable feature to ask for as a feature in 3.9 or later perhaps.

The documentation actually claims it was deprecated in version 3.6. So it has already been 2 releases worth of deprecation, visible warning or not. Ship it.

On Sat, 10 Aug 2019 at 00:36, Steven D'Aprano wrote:

2. To strongly discourage newbie Windows developers from hard-coding paths using backslashes, but to use forward slashes instead.

(Side issue) As a Windows developer, who has seen far too many cases where use of slashes in filenames implies a Unix-based developer not thinking sufficiently about Windows compatibility, or where it leads to people hard coding '/' rather than using os.sep (or better, pathlib), I strongly object to this characterisation. Rather, I would simply say "to make Windows users more aware of the clash in usage between backslashes in filenames and backslashes as string escapes". There are *many* valid ways to write Windows pathnames in your code: 1. Raw strings 2. Doubling the backslashes 3. Using pathlib (possibly with slash as a directory separator, where it's explicitly noted as a portable option) 4. Using slashes IMO, using slashes is the *worst* of these. But this latter is a matter of opinion - I've no objection to others believing differently, but I *do* object to slashes being presented as the only option, or the recommended option without qualification. Paul

On 8/10/19 7:05 AM, Chris Angelico wrote:

On Sat, Aug 10, 2019 at 6:39 PM Paul Moore wrote:

...

There are *many* valid ways to write Windows pathnames in your code:

1. Raw strings 2. Doubling the backslashes 3. Using pathlib (possibly with slash as a directory separator, where it's explicitly noted as a portable option) 4. Using slashes

IMO, using slashes is the *worst* of these. But this latter is a matter of opinion - I've no objection to others believing differently, but I *do* object to slashes being presented as the only option, or the recommended option without qualification. Please expand on why this is the worst?

ChrisA

One big issue with trying to get use to using / on windows for the directory separator is that it doesn't work for many windows programs because on Windows the / character is defined to be the option character (instead of - for *nix) Yes, you can write your program to use the foreign convention of using - for options, and because the system calls accept either \ or / as the directory separator, paths which use the 'wrong' separator will work, but your program will be violating the conventions of the host environment. -- Richard Damon

...

On Sat, Aug 10, 2019 at 6:39 PM Paul Moore wrote:

...

There are *many* valid ways to write Windows pathnames in your code:

1. Raw strings 2. Doubling the backslashes 3. Using pathlib (possibly with slash as a directory separator, where it's explicitly noted as a portable option) 4. Using slashes

IMO, using slashes is the *worst* of these. But this latter is a matter of opinion - I've no objection to others believing differently, but I *do* object to slashes being presented as the only option, or the recommended option without qualification. Please expand on why this is the worst? I did say it was a matter of opinion, so I'm not going to respond if

On Sat, 10 Aug 2019 at 12:06, Chris Angelico wrote: people say that any of the following is "wrong", but since you asked:

1. Backslash is the native separator, whereas slash is not (see eryk sun's post for *way* more detail). 2. People who routinely use slash have a tendency to forget to use os.sep rather than a literal slash in places where it *does* matter. 3. Using slash, in my experience, ends up with paths with "mixed" separators (os.path.join("C:/work/apps", "foo") -> 'C:/work/apps\\foo') which are messy to deal with, and ugly for the user. 4. If a path with slashes is displayed directly to the user without normalisation, it looks incorrect and can confuse users who are only used to "native" Windows programs.

Etc. Not to mention the problem of passing paths with / to other windows

On 8/10/2019 7:03 AM, Paul Moore wrote: programs via system or subprocess.

On 8/10/2019 11:16 AM, Terry Reedy wrote:

On 8/10/2019 4:33 AM, Paul Moore wrote:

...

(Side issue)

This deserves its own thread.

...

As a Windows developer, who has seen far too many cases where use of slashes in filenames implies a Unix-based developer not thinking sufficiently about Windows compatibility, or where it leads to people hard coding '/' rather than using os.sep (or better, pathlib), I strongly object to this characterisation. Rather, I would simply say "to make Windows users more aware of the clash in usage between backslashes in filenames and backslashes as string escapes".

There are *many* valid ways to write Windows pathnames in your code:

1. Raw strings

As pointed out elsewhere, Raw strings have limitations, paths ending in \ cannot be represented, and such do exist in various situations, not all of which can be easily avoided... except by the "extra character contortion" of   "C:\directory\ "[:-1]  (does someone know a better way?) It would be useful to make a "really raw" string that doesn't treat \ special in any way. With 4 different quoting possibilities ( ' " ''' """ ) there isn't really a reason to treat \ special at the end of a raw string, except for backward compatibility. I wonder how many raw strings actually use the \"  escape productively? Maybe that should be deprecated too! ?  I can't think of a good and necessary use for it, can anyone? Or invent "really raw" in some spelling, such as rr"c:\directory\" or e for exact, or x for exact, or <your favorite character here>"c:\directory\" And that brings me to the thought that if   \e  wants to become an escape for escape, that maybe there should be an "extended escape" prefix... if you want to use more escapes, define   ee"string where \\ can only be used as an escape or escaped character, \e means the ASCII escape character, and \ followed by a character with no escape definition would be an error." Of course "extended escape" could be spelled lots of different ways too, but not the same way as "really raw" :)

...

2. Doubling the backslashes 3. Using pathlib (possibly with slash as a directory separator, where it's explicitly noted as a portable option) 4. Using slashes

IMO, using slashes is the *worst* of these. But this latter is a matter of opinion - I've no objection to others believing differently, but I *do* object to slashes being presented as the only option, or the recommended option without qualification.

Perhaps Python Setup and Usage, 3. Using Python on Windows, should have a section of file paths, at most x.y.z, so visible in the TOC listed by https://docs.python.org/3/using/index.html

On 8/10/2019 12:19 PM, Guido van Rossum wrote:

Regular expressions.

I assume that is in response to the "good use for \" escape" question? But can't you just surround them with ' instead of " ?  Or  ''' ?

On Sat, Aug 10, 2019 at 12:12 Glenn Linderman mailto:v%2Bpython@g.nevcal.com> wrote:

On 8/10/2019 11:16 AM, Terry Reedy wrote:

...

On 8/10/2019 4:33 AM, Paul Moore wrote:

...

(Side issue)

This deserves its own thread.

...

As a Windows developer, who has seen far too many cases where use of slashes in filenames implies a Unix-based developer not thinking sufficiently about Windows compatibility, or where it leads to people hard coding '/' rather than using os.sep (or better, pathlib), I strongly object to this characterisation. Rather, I would simply say "to make Windows users more aware of the clash in usage between backslashes in filenames and backslashes as string escapes".

There are *many* valid ways to write Windows pathnames in your code:

1. Raw strings

As pointed out elsewhere, Raw strings have limitations, paths ending in \ cannot be represented, and such do exist in various situations, not all of which can be easily avoided... except by the "extra character contortion" of "C:\directory\ "[:-1]  (does someone know a better way?)

It would be useful to make a "really raw" string that doesn't treat \ special in any way. With 4 different quoting possibilities ( ' " ''' """ ) there isn't really a reason to treat \ special at the end of a raw string, except for backward compatibility.

I wonder how many raw strings actually use the \"  escape productively? Maybe that should be deprecated too! ?  I can't think of a good and necessary use for it, can anyone?

Or invent "really raw" in some spelling, such as rr"c:\directory\" or e for exact, or x for exact, or <your favorite character here>"c:\directory\"

And that brings me to the thought that if   \e  wants to become an escape for escape, that maybe there should be an "extended escape" prefix... if you want to use more escapes, define   ee"string where \\ can only be used as an escape or escaped character, \e means the ASCII escape character, and \ followed by a character with no escape definition would be an error."

Of course "extended escape" could be spelled lots of different ways too, but not the same way as "really raw" :)

...

...

2. Doubling the backslashes 3. Using pathlib (possibly with slash as a directory separator, where it's explicitly noted as a portable option) 4. Using slashes

IMO, using slashes is the *worst* of these. But this latter is a matter of opinion - I've no objection to others believing differently, but I *do* object to slashes being presented as the only option, or the recommended option without qualification.

Perhaps Python Setup and Usage, 3. Using Python on Windows, should have a section of file paths, at most x.y.z, so visible in the TOC listed by https://docs.python.org/3/using/index.html

_______________________________________________ Python-Dev mailing list -- python-dev@python.org mailto:python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org mailto:python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5MZAXJJY...

-- --Guido (mobile)

On 8/10/2019 3:36 PM, Greg Ewing wrote:

Glenn Linderman wrote:

...

I wonder how many raw strings actually use the \"  escape productively? Maybe that should be deprecated too! ?  I can't think of a good and necessary use for it, can anyone?

Quite rare, I expect, but it's bound to break someone's code. It might be better to introduce a new string prefix, e.g. 'v' for 'verbatim':

   v"C:\Users\Fred\"

Which is why I suggested  rr"C:\directory\", but allowed as how there might be better spellings.... I like your  v for verbatim !

10.08.19 22:10, Glenn Linderman пише:

As pointed out elsewhere, Raw strings have limitations, paths ending in \ cannot be represented, and such do exist in various situations, not all of which can be easily avoided... except by the "extra character contortion" of   "C:\directory\ "[:-1]  (does someone know a better way?)

Other common idiom is r"C:\directory" "\\"

I wonder how many raw strings actually use the \"  escape productively? Maybe that should be deprecated too! ?  I can't think of a good and necessary use for it, can anyone?

This is an interesting question. I have performed some experiments. 15 files in the stdlib (not counting the tokenizer) use \' or \" in raw strings. And one test (test_venv) is failed because of using them in third-party code. All cases are in regular expressions. It is possible to rewrite them, but it is less trivial task than fixing invalid escape sequences. So changing this will require much much more long deprecation period.

11.08.19 23:07, Glenn Linderman пише:

On 8/11/2019 1:26 AM, Serhiy Storchaka wrote:

...

10.08.19 22:10, Glenn Linderman пише:

...

I wonder how many raw strings actually use the \"  escape productively? Maybe that should be deprecated too! ?  I can't think of a good and necessary use for it, can anyone?

This is an interesting question. I have performed some experiments. 15 files in the stdlib (not counting the tokenizer) use \' or \" in raw strings. And one test (test_venv) is failed because of using them in third-party code. All cases are in regular expressions. It is possible to rewrite them, but it is less trivial task than fixing invalid escape sequences. So changing this will require much much more long deprecation period.

Couldn't they be rewritten using the above idiom? Why would that be less trivial? Or by using triple quotes, so the \" could be written as " ? That seems trivial.

Yes, they could. You can use different quote character, triple quotes, string literal concatenation. There are many options, and you should choose what is applicable in any particular case and what is optimal. You need to analyze the whole string literal and code transformation usually is more complex than just duplicating a backslash or adding the `r` prefix. For example, in many cases `\"` can be replaced with `"'"'r"`, but it does not look pretty readable. See https://github.com/python/cpython/pull/15217.

12.08.19 22:51, Glenn Linderman пише:

On 8/12/2019 12:11 AM, Serhiy Storchaka wrote:

...

For example, in many cases `\"` can be replaced with `"'"'r"`, but it does not look pretty readable.

No, that is not readable.  But neither does it seem to be valid syntax, or else I'm not sure what you are saying. Ah, maybe you were saying that a seqence like the '\"' that is already embedded in a raw string can be converted to the sequence `"'"'r"` also embedded in the raw string. That makes the syntax work, but if that is what you were saying, your translation dropped the \ from before the ", since the raw string preserves both the \ and the ".

Yes, this is what I meant. Thank you for correction. I dropped the `\` because in context of regular expression `\"` and `"` is the same, and a backslash is only used to prevent `"` to end a string literal. This is why `\"` is so rarely used in other strings: because only in regular expressions `\` before `"` does not matter.

Regarding the readability, I think any use of implicitly concatenated strings should have at least two spaces or a newline between them to make the implicit concatenation clearer.

Agree. I have wrote it without spaces for dramatic effect.

On 8/11/2019 2:50 AM, Steven D'Aprano wrote:

On Sat, Aug 10, 2019 at 12:10:55PM -0700, Glenn Linderman wrote:

...

Or invent "really raw" in some spelling, such as rr"c:\directory\" or e for exact, or x for exact, or <your favorite character here>"c:\directory\"

And that brings me to the thought that if   \e  wants to become an escape for escape, that maybe there should be an "extended escape" prefix... if you want to use more escapes, define   ee"string where \\ can only be used as an escape or escaped character, \e means the ASCII escape character, and \ followed by a character with no escape definition would be an error." Please no.

We already have b-strings, r-strings, u-strings, f-strings, br-strings, rb-strings, fr-strings, rf-strings, each of which comes in four varieties (single quote, double quote, triple single quote and triple double quote). Now you're talking about adding rr-strings, v-strings (Greg suggested that) and ee-strings, presumably some or all of which will need b*- and *b- or f*- and *f- varieties too.

Don't forget the upper & lower case varieties :)

If the plan to deprecate unrecognised escapes and then make them an exception goes ahead, and I expect that it will, in a few more releases this "extended escape" ee-string will be completely redundent. If \e is required, we will be able to add it to regular strings as needed, likewise for any future new escapes we might want. (If any.) So unrecognized escapes were deprecated in 3.6. And didn't get removed in 3.7. And from all indications, aren't going to be removed in 3.8. What makes you think the same arguments won't happen again for 3.9?

And if we end up keeping the existing behaviour, oh well, we can always write \x1B instead. New escapes are a Nice To Have, not a Must Have.

"Really raw" rr'' versus "nearly raw" r'' is a source of confusion just waiting to happen, when people use the wrong numbers of r's, or are simply unclear which they should use. I agree that Greg's v is far better than rr, especially if someone tried to write rfr or rbr. It's not like we have no other options:

location = r'C:\directory\subdirectory' '\\'

works fine. But I never thought of that, until Serhiy mentioned it in his reply, so there are probably lots of other stupid people that didn't think of it either. It's not like it is even suggested in the documentation as a way to work around the non-rawness of raw strings. And it still requires doubling one of the \, so it is more consistent and understandable to just double them all.

So does this:

location = 'directory/subdirectory/'.replace('/', os.sep)

This is a far greater run-time cost with the need to scan the string. Granted the total cost isn't huge, unless it is done repeatedly.

Even better, instead of hard-coding our paths in the source code, we can read them from a config file or database. Yep, I do that sometimes. But hard-coded paths make good defaults in many circumstances.

It is unfortunate that Windows is so tricky with backslashes and forwards slashes, and that it clashes with the escape character, but I'm sure that other languages which use \ for escaping haven't proliferated a four or more kinds of strings with different escaping rules in response.

I agree with this. But Bill didn't consult Guido about the matter.

On 8/11/2019 8:40 PM, Eric V. Smith wrote:

On 8/11/2019 4:18 PM, Glenn Linderman wrote:

...

On 8/11/2019 2:50 AM, Steven D'Aprano wrote:

...

On Sat, Aug 10, 2019 at 12:10:55PM -0700, Glenn Linderman wrote:

...

Or invent "really raw" in some spelling, such as rr"c:\directory\" or e for exact, or x for exact, or <your favorite character here>"c:\directory\"

And that brings me to the thought that if   \e  wants to become an escape for escape, that maybe there should be an "extended escape" prefix... if you want to use more escapes, define ee"string where \\ can only be used as an escape or escaped character, \e means the ASCII escape character, and \ followed by a character with no escape definition would be an error." Please no.

We already have b-strings, r-strings, u-strings, f-strings, br-strings, rb-strings, fr-strings, rf-strings, each of which comes in four varieties (single quote, double quote, triple single quote and triple double quote). Now you're talking about adding rr-strings, v-strings (Greg suggested that) and ee-strings, presumably some or all of which will need b*- and *b- or f*- and *f- varieties too.

Don't forget the upper & lower case varieties :)

And all orders!

...

...

...

_all_string_prefixes() {'', 'b', 'BR', 'bR', 'B', 'rb', 'F', 'RF', 'rB', 'FR', 'Rf', 'Fr', 'RB', 'f', 'r', 'rf', 'rF', 'R', 'u', 'fR', 'U', 'Br', 'Rb', 'fr', 'br'} len(_all_string_prefixes()) 25

And if you add just 'bv' and 'fv', it's 41:

{'', 'fr', 'Bv', 'BR', 'F', 'rb', 'Fv', 'VB', 'vb', 'vF', 'br', 'FV', 'vf', 'FR', 'fV', 'bV', 'Br', 'Vb', 'Rb', 'RF', 'bR', 'r', 'R', 'Vf', 'fv', 'U', 'RB', 'B', 'rB', 'vB', 'Fr', 'rF', 'fR', 'Rf', 'BV', 'VF', 'bv', 'b', 'u', 'f', 'rf'}

There would be no need for 'uv' (not needed for backward compatibility) or 'rv' (can't be both raw and verbatim).

I'm not in any way serious about this. I just want people to realize how many wacky combinations there would be. And heaven forbid we ever add some combination of 3 characters. If 'rfv' were actually also valid, you get to 89:

{'', 'br', 'vb', 'fR', 'F', 'rFV', 'fRv', 'fV', 'rVF', 'Rfv', 'u', 'vRf', 'fVR', 'rfV', 'Fvr', 'vrf', 'fVr', 'vB', 'Vb', 'Rvf', 'Fv', 'Fr', 'FVr', 'B', 'rVf', 'FVR', 'vfr', 'VB', 'VrF', 'BR', 'VRf', 'vfR', 'FR', 'Br', 'RFV', 'Rf', 'fvR', 'f', 'rb', 'VfR', 'VFR', 'fr', 'vFR', 'VRF', 'frV', 'bR', 'b', 'FrV', 'r', 'R', 'RVF', 'FV', 'rvF', 'FRV', 'Vrf', 'rvf', 'FRv', 'Frv', 'vF', 'bV', 'VF', 'fv', 'RF', 'RB', 'rB', 'vRF', 'RFv', 'RVf', 'Rb', 'Vfr', 'vrF', 'rf', 'Bv', 'vf', 'rF', 'U', 'bv', 'FvR', 'RfV', 'Vf', 'VFr', 'vFr', 'fvr', 'BV', 'rFv', 'rfv', 'fRV', 'frv', 'RvF'}

If only we could deprecate upper case prefixes!

Eric

Yes. Happily while there is a combinatorial explosion in spellings and casings, there is no cognitive overload: each character has an independent effect on the interpretation and use of the string, so once you understand the 5 existing types (b r u f and plain) you understand them all. Should we add one or two more, it would be with the realization (hopefully realized in the documentation also) that v and e would effectively be replacements for r and plain, rather than being combined with them. Were I to design a new language with similar string syntax, I think I would use plain quotes for verbatim strings only, and have the following prefixes, in only a single case: (no prefix) - verbatim UTF-8 (at this point, I see no reason not to require UTF-8 for the encoding of source files) b - for verbatim bytes e - allow (only explicitly documented) escapes f - format strings Actually, the above could be done as a preprocessor for python, or a future import. In other words, what you see is what you get, until you add a prefix to add additional processing.  The only combinations that seem useful are  eb  and  ef.  I don't know that constraining the order of the prefixes would be helpful or not, if it is helpful, I have no problem with a canonical ordering being prescribed. As a future import, one could code modules to either the current combinatorial explosion with all its gotchas, special cases, and passing of undefined escapes; or one could code to the clean limited cases above. Another thing that seems awkward about the current strings is that {{ and }} become "special escapes".  If it were not for the permissive usage of \{ and \} in the current plain string processing, \{ and \} could have been used to escape the non-format-expression uses of { and }, which would be far more consistent with other escapes.  Perhaps the future import could regularize that, also. A future import would have no backward compatibility issues to disrupt a simplified, more regular syntax. Does anyone know of an existing feature that couldn't be expressed in a straightforward manner with only the above capabilities? The only other thing that I have heard about regarding strings is that multi-line strings have their first line indented, and other lines not. Some have recommended making the first line blank, and just chopping off the first \n, others have recommended indenting all lines, and replacing "\n" followed by the number of indented spaces by "\n", so the text can be aligned in the code like it will be aligned for use. Both techniques seem to have their place in aiding code readability. Both techniques could be used together, in practice, using one more prefix character for triple quotes only:     longstring = l""" The traditional first blank line form could be used at it has.""" If the first character of a long-string is a newline character, then it will be removed. If the string wants to have an initial newline character, a second one can be provided, which would not be removed.      longstring = l"""The traditional indented form                       could be used as it has, also.""" This would be contracted by removing up to the number of space characters to reach the first character of the first line of the string (if the lexer can provide that) after newlines within the string. If fewer space characters are available after a newline, only the number available would be removed. If there are more, they would be retained. A new form would also be permitted:     longstring = l"""         An indented form that isn't pushed as far right as the traditional indented form could also be used.""" If the first character of an l-string is a newline and the second character is a space character, this form would count the number of space characters in the second line, and remove up to that many space characters from all lines, as well as removing the initial newline character. If l-strings were implemented (l for layout), they could be combined with f and/or e. Are there any other string feature workarounds in common use that could be codified in a future import scenario? Glenn

On 8/12/2019 2:52 AM, Greg Ewing wrote:

Eric V. Smith wrote:

...

I'm not in any way serious about this. I just want people to realize how many wacky combinations there would be.

It doesn't matter how many combinations there are, as long as multiple prefixes combine in the way you would expect, which they do as far as I can see.

In general I agree, although there's some cognitive overhead to which combinations are valid or not. There's no "fu" strings, for example. But for reading code that doesn't matter, so your point stands. Eric

On Mon, Aug 12, 2019, at 15:15, Terry Reedy wrote:

Please no more combinations. The presence of both legal and illegal combinations is already a mild nightmare for processing and testing. idlelib.colorizer has the following re to detest legal combinations

stringprefix = r"(?i:r|u|f|fr|rf|b|br|rb)?"

More advanced syntax highlighting editors have to handle each string type separately anyway, because they highlight (valid) backslash-escapes and f-string formatters. The proposed 'v-string' type would need separate handling even in a simplistic editor like IDLE, because it's different at the basic level of \" not ending the string (whereas, for better or worse, all current string types have exactly the same rules for how to find the end delimiter)

On Mon, Aug 12, 2019, at 15:15, Terry Reedy wrote:

...

Please no more combinations. The presence of both legal and illegal combinations is already a mild nightmare for processing and testing. idlelib.colorizer has the following re to detest legal combinations

stringprefix = r"(?i:r|u|f|fr|rf|b|br|rb)?" More advanced syntax highlighting editors have to handle each string type separately anyway, because they highlight (valid) backslash-escapes and f-string formatters. The proposed 'v-string' type would need separate handling even in a simplistic editor like IDLE, because it's different at the basic level of \" not ending the string (whereas, for better or worse, all current string types have exactly the same rules for how to find the end delimiter) I had to read this several times, and then only after reading Eric's reply, it finally hit me that what you are saying is that \" doesn't end

Currently a raw literal cannot end in a single backslash (e.g. in r"C:\User\"). Although there are reasons for this. It is an old gotcha, and there are many closed issues about it. This question is even included in FAQ. which indicates that I am not the only one that has been tripped up by

On 8/14/2019 8:02 AM, Random832 wrote: the string in any other form of string, but that sequence would end a v-string. It seems that also explains why Serhiy, in describing his experiment really raw string literals mentioned having to change the tokenizer as well as the parser (proving that it isn't impossible to deal with truly raw strings). \" not ending a raw string was certainly a gotcha for me when I started using Python (with a background in C and Perl among other languages), and it convinced me not to raw strings, that that gotcha was not worth the other benefits of raw strings. Serhiy said: that over the years. Trying to look at it from the eyes of a beginning programmer, the whole idea of backslash being an escape character is an unnatural artifice. I'm unaware (but willing to be educated) of any natural language, when using quotations, that has such a  concept. Nested quotations exist, in various forms:  use of a different quotation mark for the inner and outer quotations, and block quotations (which in English, have increased margin on both sides, and have a blank line before and after). Python actually supports constructs very similar to the natural language formats, allowing both  " and ' for quotations and nested quotations, and the triple-quoted string with either " or ' is very similar in concept to a block quotation. But _all_ the strings forms are burdened with surprises for the beginning programmer: escape sequences of one sort or another must be learned and understood to avoid surprises when using the \ character. Programming languages certainly need an escape character mechanism to deal with characters that cannot easily be typed on a keyboard (such as ¤ ¶ etc.), or which are visually indistinguishable from other characters or character sequences (various widths of white space), or which would be disruptive to the flow of code or syntax if represented by the usual character (newline, carriage return, formfeed, maybe others). But these are programming concepts, not natural language concept.  The basic concept of a quoted string should best be borrowed directly from natural language, and then enhancements to that made to deal with programming concepts. In Python, as in C, the escape characters are built in the basic string syntax, one must learn the quirks of the escaping mechanism in order to write In Perl, " strings include escapes, and ' strings do not. So there is a basic string syntax that is similar to natural language, and one that is extended to include programming concepts. [N.B. There are lots of reasons I switched from Perl to Python, and don't have any desire to go back, but I have to admit, that the lack of a truly raw string in Python was a disappointment.] So that, together with the desire for new escape sequences, and the creation of a new escape mechanism in the f-string {} (which adds both { and } as escape characters by requiring them to be doubled to be treated as literal inside an f-string, instead of using \{ and \} as the escapes [which would have been possible, due to the addition of the f prefix]), and the issue that because every current \-escape is defined to do something, is why I suggested elsewhere in this thread https://mail.python.org/archives/list/python-dev@python.org/message/XJNS45JG... that perhaps the whole irregular string syntax should be rebooted with a future import, and it seems it could both be simpler, more regular, and more powerful as a result. And by using a future import, there are no backward incompatibility issues, and migration can be module by module. The more I think about this, the more tempting it is to attempt to fork Python just to have a better string syntax! But alas! So many other time commitments, and a lack of in-depth internals knowledge make that an impossibility. I daresay, though, that if I get a free week, I might well write a preprocessor that converts my suggested future syntax to C-Python, so that I can use it in my own projects!

On 8/15/19 10:40 AM, Greg Ewing wrote:

If we want a truly raw string format that allows all characters, including any kind of quote, we could take a tip from Fortran:

    s = 31HThis is a "totally raw" string!

Or from Rust: let s = r"Here's a raw string"; let s = r#"Here's a raw string with "quotes" in it"#; let s = r##"Here's r#"the raw string syntax"# in raw string"##; let s = r###"and here's a '##"' as well"###;

On 15/08/2019 12:17:36, Petr Viktorin wrote:

On 8/15/19 10:40 AM, Greg Ewing wrote:

...

If we want a truly raw string format that allows all characters, including any kind of quote, we could take a tip from Fortran:

     s = 31HThis is a "totally raw" string!

Or from Rust:

let s = r"Here's a raw string"; let s = r#"Here's a raw string with "quotes" in it"#; let s = r##"Here's r#"the raw string syntax"# in raw string"##; let s = r###"and here's a '##"' as well"###; _______________________________________________ I rather like the idea!  (Even though it would add to the proliferation of string types.) Obviously Python can't use # as the special character since that introduces a comment, and a lot of other possibilities are excluded because they would lead to ambiguous syntax. Say for the sake of argument we used "!" (exclamation mark). Possible variations include: (1) Like Rust:     s = r"Here's a raw string";     s = r!"Here's a raw string with "quotes" in it"!;     s = r!!"Here's r!"the raw string syntax"! in raw string"!!;     s = r!!!"and here's a '!!"' as well"!!!; (2) Same, but omit the leading 'r' when using !:     s = r"Here's a raw string";     s = !"Here's a raw string with "quotes" in it"!;     s = !!"Here's a raw string with "quotes" and !exclamation marks! in it"!!;     s = !!!"and here's a '!!"' as well"!!!;     # Cons: Would conflict with adding ! as an operator (or at minimum, as a unary operator) for some other purpose in future.     #            Makes it less obvious that a !string! is a raw string. (3) Allow the user to specify his own delimiting character:     s = r!|This raw string can't contain a "bar".| (4) As above, but the "!" is not required:     s = r|This raw string can't contain a "bar".|     # In this case the delimiter ought not to be a letter     # (it might conflict with current or future string prefixes);     # this could be forbidden. (5) Similar, but allow the user to specify his own delimiting *string* (specified between "!"s) (as long as it doesn't contain !):     let s = r!?@!Could this string could contain almost anything? Yes!?@     # The text in this string is:     #                    Could this string could contain almost anything?  Yes! (6) Same except the first "!" is not required.  In this case the first character of the delimiting string should not be a letter:     let s = r?@!Could this string could contain almost anything? Yes!?@     # The text in this string is:     #                   Could this string could contain almost anything?  Yes!

I can dream ... A point about the current syntax: It is not true that a raw string can't end in a backslash, as https://en.wikipedia.org/wiki/String_literal points out.  It can't end in an *odd number* of backslashes.  42 is fine, 43 is no good.  Which makes it seem even more of a language wart (think of program-generated art). Rob Cliffe

On Fri, Aug 09, 2019 at 01:12:59PM -0700, Glenn Linderman wrote:

The reason I never use raw strings is in the documentation, it is because \ still has a special meaning, and the first several times I felt the need for raw strings, it was for directory names that wanted to end with \ and couldn't.

Can you elaborate? I find it unlikely that I would ever want a docstring that ends with a backslash: def func(): r"""Documentation goes here... more documentation... ending with a Windows path that needs a trailing backslash like this C:\directory\""" That seems horribly contrived. Why use backslashes in the path when the strong recommendation is to use forward slashes? And why not solve the problem by simply moving the closing quotes to the next line, as PEP 8 recommends? r"""Documentation ... C:\directory\ """ [...]

...

Even in a raw literal, quotes can be escaped with a backslash

Indeed, they're not so much "raw" strings as only slightly blanched strings. -- Steven

I'm not trying to be confrontational, I'm trying to understand your use-case(s) and see if it would be broken by the planned change to string escapes. On Fri, Aug 09, 2019 at 03:18:29PM -0700, Glenn Linderman wrote:

On 8/9/2019 2:53 PM, Steven D'Aprano wrote:

...

On Fri, Aug 09, 2019 at 01:12:59PM -0700, Glenn Linderman wrote:

...

The reason I never use raw strings is in the documentation, it is because \ still has a special meaning, and the first several times I felt the need for raw strings, it was for directory names that wanted to end with \ and couldn't. Can you elaborate? I find it unlikely that I would ever want a docstring

I didn't mention docstring.  I just wanted a string with a path name ending in \.

You said you never used raw strings in the documentation. I read that as doc strings. What sort of documentation are you writing that isn't a doc string but is inside your .py files where the difference between raw and regular strings is meaningful?

Windows users are used to seeing backslashes in paths, I don't care to be the one to explain why my program uses / and all the rest use \.

If you don't use raw strings for paths, you get to explain why your program uses \\ and all the rest use \ *wink* If they're Windows end users, they won't be reading your source code and will never know how you represent hard-coded paths in the source code. If they're Windows developers, they ought to be aware that the Windows file system API allows / anywhere you can use \ and it is the common convention in Python to use forward slashes. I'm also curious why the string needs to *end* with a backslash. Both of these are the same path: C:\foo\bar\baz\ C:\foo\bar\baz -- Steven

On 8/9/2019 4:08 PM, MRAB wrote:

On 2019-08-09 23:56, Steven D'Aprano wrote:

...

I'm not trying to be confrontational, I'm trying to understand your use-case(s) and see if it would be broken by the planned change to string escapes.

On Fri, Aug 09, 2019 at 03:18:29PM -0700, Glenn Linderman wrote:

...

On 8/9/2019 2:53 PM, Steven D'Aprano wrote:

...

On Fri, Aug 09, 2019 at 01:12:59PM -0700, Glenn Linderman wrote:

...

The reason I never use raw strings is in the documentation, it is because \ still has a special meaning, and the first several times I felt the need for raw strings, it was for directory names that wanted to end with \ and couldn't. Can you elaborate? I find it unlikely that I would ever want a docstring

I didn't mention docstring.  I just wanted a string with a path name ending in \.

You said you never used raw strings in the documentation. I read that as doc strings. What sort of documentation are you writing that isn't a doc string but is inside your .py files where the difference between raw and regular strings is meaningful?

...

Windows users are used to seeing backslashes in paths, I don't care to be the one to explain why my program uses / and all the rest use \.

If you don't use raw strings for paths, you get to explain why your program uses \\ and all the rest use \ *wink*

If they're Windows end users, they won't be reading your source code and will never know how you represent hard-coded paths in the source code.

If they're Windows developers, they ought to be aware that the Windows file system API allows / anywhere you can use \ and it is the common convention in Python to use forward slashes.

I'm also curious why the string needs to *end* with a backslash. Both of these are the same path:

     C:\foo\bar\baz\      C:\foo\bar\baz

The only time it's required is for the root directory of a drive:

C:\

That's not the only time it's required, but it is a case that is far harder to specify in other ways.  It's required any time you  want to say   + filename without writing + "\\" + filename, or os.path.join( 'C:\\", filename )

On 8/9/19, Steven D'Aprano wrote:

I'm also curious why the string needs to *end* with a backslash. Both of these are the same path:

C:\foo\bar\baz\ C:\foo\bar\baz

The above two cases are equivalent. But that's not the case for the root directory. Unlike Unix, filesystem namespaces are implemented directly on devices. For example, "//./C:" might resolve to a volume device such as "\\Device\\HarddiskVolume2". With a trailing slash added, "//./C:/" resolves to "\\Device\\HarddiskVolume2\\", which is the root directory of the mounted filesystem on the volume. Also, as a classic DOS path, "C:" without a trailing slash expands to the working directory on drive "C:". The system runtime library looks for this path in a hidden environment variable named "=C:". The Windows API never sets these hidden "=X:" drive variables. The C runtime sets them, as does Python's os.chdir. Some volume-management functions require a trailing slash or backslash, such as GetVolumeInformationW [1]. GetVolumeNameForVolumeMountPointW [2] actually requires it to be a trailing backslash. It will not accept a trailing forward slash such as "C:\\Mount\\Volume/" (a bug since Windows 2000). The volume name (e.g. "\\\\?\\Volume{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}\\") returned by the latter includes a trailing backslash, which must be present in the target path in order for a mountpoint to function properly as a directory, else it would resolve to the volume device instead of the root directory. [1] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getvol... [2] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getvol...

If they're Windows developers, they ought to be aware that the Windows file system API allows / anywhere you can use \ and it is the common convention in Python to use forward slashes.

The Windows file API actually does not allow slash to be used anywhere that we can use backslash. It's usually allowed, but not always. For the most part, the conditions where forward slash is not supported are intentional. Windows replaces forward slash with backslash in normal DOS paths and normal device paths. But sometimes we have to use a special form of device path that bypasses normalization. A path that isn't normalized can only use backslash as the path separator. For example, the most common case is that the process doesn't have long paths enabled. In this case we're limited to MAX_PATH, which limits file paths to a paltry 259 characters (sans the terminating null); the current directory to 258 characters (sans a trailing backslash and null); and the path of a new directory to 247 characters (subtract 12 from 259 to leave space for an 8.3 filename). By skipping DOS normalization, we can access a path with up to about 32,750 characters (i.e. 32,767 sans the length of the device name in the final NT path under "\\Device\\"). (Long normalized paths are available starting in Windows 10, but the system policy that allows this is disabled by default, and even if enabled, each application has to declare itself to be long-path aware in its manifest. This is declared for python[w].exe in Python 3.6+.) A device path is an explicit reference to a user's local device directory (in the object namespace), which shadows the global device directory. In NT, this directory is aliased to a special "\\??\\" prefix (backslash only). A local device directory is created for each logon session (not terminal session) by the security system that runs in terminal session 0 (i.e. the system services session). The per-logon directory is located at "\\Sessions\\0\\DosDevices\\<Logon Session ID>". In the Windows API, it's accessible as "//?/" or "//./", or with any mix of forward slashes or backslashes, but only the all-backslash form is special-cased to bypass the normalization step.

On 10/08/2019 11:50:35, eryk sun wrote:

...

I'm also curious why the string needs to *end* with a backslash. Both of these are the same path:

C:\foo\bar\baz\ C:\foo\bar\baz Also, the former is simply more *informative* - it tells the reader that baz is expected to be a directory, not a file. Rob Cliffe The above two cases are equivalent. But that's not the case for the root directory. Unlike Unix, filesystem namespaces are implemented

On 8/9/19, Steven D'Aprano wrote: directly on devices. For example, "//./C:" might resolve to a volume device such as "\\Device\\HarddiskVolume2". With a trailing slash added, "//./C:/" resolves to "\\Device\\HarddiskVolume2\\", which is the root directory of the mounted filesystem on the volume.

Also, as a classic DOS path, "C:" without a trailing slash expands to the working directory on drive "C:". The system runtime library looks for this path in a hidden environment variable named "=C:". The Windows API never sets these hidden "=X:" drive variables. The C runtime sets them, as does Python's os.chdir.

Some volume-management functions require a trailing slash or backslash, such as GetVolumeInformationW [1]. GetVolumeNameForVolumeMountPointW [2] actually requires it to be a trailing backslash. It will not accept a trailing forward slash such as "C:\\Mount\\Volume/" (a bug since Windows 2000). The volume name (e.g. "\\\\?\\Volume{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}\\") returned by the latter includes a trailing backslash, which must be present in the target path in order for a mountpoint to function properly as a directory, else it would resolve to the volume device instead of the root directory.

[1] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getvol... [2] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getvol...

...

If they're Windows developers, they ought to be aware that the Windows file system API allows / anywhere you can use \ and it is the common convention in Python to use forward slashes. The Windows file API actually does not allow slash to be used anywhere that we can use backslash. It's usually allowed, but not always. For the most part, the conditions where forward slash is not supported are intentional.

Windows replaces forward slash with backslash in normal DOS paths and normal device paths. But sometimes we have to use a special form of device path that bypasses normalization. A path that isn't normalized can only use backslash as the path separator. For example, the most common case is that the process doesn't have long paths enabled. In this case we're limited to MAX_PATH, which limits file paths to a paltry 259 characters (sans the terminating null); the current directory to 258 characters (sans a trailing backslash and null); and the path of a new directory to 247 characters (subtract 12 from 259 to leave space for an 8.3 filename). By skipping DOS normalization, we can access a path with up to about 32,750 characters (i.e. 32,767 sans the length of the device name in the final NT path under "\\Device\\").

(Long normalized paths are available starting in Windows 10, but the system policy that allows this is disabled by default, and even if enabled, each application has to declare itself to be long-path aware in its manifest. This is declared for python[w].exe in Python 3.6+.)

A device path is an explicit reference to a user's local device directory (in the object namespace), which shadows the global device directory. In NT, this directory is aliased to a special "\\??\\" prefix (backslash only). A local device directory is created for each logon session (not terminal session) by the security system that runs in terminal session 0 (i.e. the system services session). The per-logon directory is located at "\\Sessions\\0\\DosDevices\\<Logon Session ID>". In the Windows API, it's accessible as "//?/" or "//./", or with any mix of forward slashes or backslashes, but only the all-backslash form is special-cased to bypass the normalization step. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3SDFM2EK...

--- This email has been checked for viruses by AVG. https://www.avg.com

Rob Cliffe via Python-Dev wrote:

Also, the former is simply more *informative* - it tells the reader that baz is expected to be a directory, not a file.

On Windows you can usually tell that from the fact that filenames almost always have an extension, and directory names almost never do. -- Greg

On 8/10/2019 10:30 PM, Rob Cliffe via Python-Dev wrote:

On 10/08/2019 23:30:18, Greg Ewing wrote:

...

Rob Cliffe via Python-Dev wrote:

...

Also, the former is simply more *informative* - it tells the reader that baz is expected to be a directory, not a file.

On Windows you can usually tell that from the fact that filenames almost always have an extension, and directory names almost never do.

Usually, but not always.  I have not infrequently used files with a blank extension. I can't recall using a directory name with an extension (but I can't swear that I never have).

I most commonly see this with bare git repositories <reponame>.git. And I've created directory names with "extensions" for my own use. Eric

Because of the "invalid escape sequence" and "raw string" discussion, when looking at the documentation, I also noticed the following description for f-strings:

Escape sequences are decoded like in ordinary string literals (except when a literal is also marked as a raw string). After decoding, the grammar for the contents of the string is: followed by lots of stuff, followed by Backslashes are not allowed in format expressions and will raise an error: f"newline: {ord('\n')}" # raises SyntaxError

What I don't understand is how, if f-strings are processed AS DESCRIBED, how the \n is ever seen by the format expression. The description is that they are first decoded like ordinary strings, and then parsed for the internal grammar containing {} expressions to be expanded.  If that were true, the \n in the above example would already be a newline character, and the parsing of the format expression would not see the backslash. And if it were true, that would actually be far more useful for this situation. So given that it is not true, why not? And why go to the extra work of prohibiting \ in the format expressions? The PEP 498, of course, has an apparently more accurate description, that the {} parsing actually happens before the escape processing. Perhaps this avoids making multiple passes over the string to do the work, as the literal pieces and format expression pieces have to be separate in the generated code, but that is just my speculation: I'd like to know the real reason. Should the documentation be fixed to make the description more accurate? If so, I'd be glad to open an issue. The PEP further contains the inaccurate statement:

Like all raw strings in Python, no escape processing is done for raw f-strings:

not mentioning the actual escape processing that is done for raw strings, regarding \" and \'.

Glenn Linderman wrote:

If that were true, the \n in the above example would already be a newline character, and the parsing of the format expression would not see the backslash. And if it were true, that would actually be far more useful for this situation.

But then it would fail for a different reason -- the same reason that this is a syntax error: 'hello world'

Why go to the extra work of prohibiting \ in the format expressions?

Maybe to avoid problems like the above? Or maybe because it would be confusing -- there are two levels of string literal processing going on, one on the outer f-string and one on the embedded string literal in the expression. What level is the backslash expansion done in? Is it done in both? To get a backslash in the embedded string, do I need two backslashes or four? Banning backslashes altogether sidesteps all these issues.

not mentioning the actual escape processing that is done for raw strings, regarding \" and \'.

Technically that's not part of "escape processing", since it takes place during lexical analysis -- it has to, because it affects how the input stream is divided into tokens. However, the backslash prohibition seems to apply even to this use in f-strings:

...

...

f"quote: {ord('\"')}" File "<stdin>", line 1 SyntaxError: f-string expression part cannot include a backslash

So it seems that f-strings are even more special than r-strings when it comes to the treatment of backslashes. -- Greg

On Sat, Aug 10, 2019, at 19:54, Glenn Linderman wrote:

Because of the "invalid escape sequence" and "raw string" discussion, when looking at the documentation, I also noticed the following description for f-strings:

...

Escape sequences are decoded like in ordinary string literals (except when a literal is also marked as a raw string). After decoding, the grammar for the contents of the string is: followed by lots of stuff, followed by Backslashes are not allowed in format expressions and will raise an error: f"newline: {ord('\n')}" # raises SyntaxError What I don't understand is how, if f-strings are processed AS DESCRIBED, how the \n is ever seen by the format expression.

The description is that they are first decoded like ordinary strings, and then parsed for the internal grammar containing {} expressions to be expanded. If that were true, the \n in the above example would already be a newline character, and the parsing of the format expression would not see the backslash. And if it were true, that would actually be far more useful for this situation.

So given that it is not true, why not? And why go to the extra work of prohibiting \ in the format expressions?

AIUI there were strong objections to the "AS DESCRIBED" process (which would require almost all valid uses of backslashes inside to be doubled, and would incidentally leave your example *still* a syntax error), and disallowing backslashes is a way to pretend that it doesn't work that way and leave open the possibility of changing how it works in the future without breaking compatibility. The only dubious benefit to the described process with backslashes allowed would be that f-strings (or other strings, in the innermost level) could be infinitely nested as f'{f\'{f\\\'{...}\\\'}\'}', rather than being hard-limited to four levels as f'''{f"""{f'{"..."}'}"""}'''

On Fri, Aug 9, 2019 at 8:43 AM Guido van Rossum wrote:

This discussion looks like there's no end in sight. Maybe the Steering Council should take a vote?

I've merged the PR reverting the behavior in 3.8 and am doing the same in the master branch. The sheer volume of email this is generating shows that we're not ready to do this to our users. Recall the nightmare caused by md5.py and sha.py DeprecationWarning's in 2.5... this would be similar. We need owners of code to see the problems, not end users of other peoples code. FWIW, lest people think I don't like this change and just pushed the revert buttons as a result, wrong. I agree with the ultimate SyntaxError and believe we should move the language there (it is better for long term code quality). But it needs to be done in a way that disrupts the *right* people in the process, not disrupting an exponentially higher number of users of other peoples code. If the steering council does anything it should be deciding if we're still going to do this at all and, if so, planning how we do it without repeating past mistakes. -gps

While not a total solution, it seems like it might be worthwhile forcing flake8 or similar checks when uploading PyPI modules. That would catch the illegal escape sequences where it really matters - before they enter the ecosystem. (general) fathead:pyxll-www sholden$ cat t.py "Docstring with illegal \escape sequence" (general) fathead:pyxll-www sholden$ flake8 t.py t.py:1:25: W605 invalid escape sequence '\e' while this won't mitigate the case for existing packages, it should reduce the number of packages containing potentially erroneous string constants, preparing the ground for the eventual introduction of the syntax error. Steve Holden On Sat, Aug 10, 2019 at 8:07 AM Serhiy Storchaka wrote:

10.08.19 02:04, Gregory P. Smith пише:

...

I've merged the PR reverting the behavior in 3.8 and am doing the same in the master branch.

I was going to rebase it to master and go in normal backporting process if we decide that DeprecationWarning should be in master. I waited the end of the discussion.

...

Recall the nightmare caused by md5.py and sha.py DeprecationWarning's in 2.5... this would be similar.

It is very different because DeprecationWarning for md5.py and sha.py is emitted at runtime. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H5VXWS6U...

On Fri, Aug 9, 2019 at 11:22 PM Steven D'Aprano wrote:

And this change won't fix that, because *good* paths that currently work today will fail in the future, but *bad* paths that silently do the wrong thing will continue to silently do the wrong thing.

Except that many paths can be both "good" and "bad", because paths have multiple components. So the warning has a VERY high probability of happening. But I've given up on this debate. No more posts from me. Some things aren't worth fighting for. With the number of words posted in this thread saying "we need convenience, not correctness!!!!", I'm done arguing. ChrisA

This isn't about me. As a heavy user of the 3.8 beta, I'm just the canary in the coal mine. After many encounters with these warnings, I'm starting to believe that Python's long-standing behavior was convenient for users. Effectively, "\-" wasn't an error, it was just a way of writing "\-". For the most part, that worked out fine. Sure, we all seen interactive prompt errors from having \t in a pathname but not in production (likely because a FileNotFoundError would surface immediately).

When you take a text string and create a string literal to represent it, sometimes you have to modify it to become syntactically valid.

Even simpler: use r""" instead of """ The only case where that won't work is when you need actual escape sequences. But I find this very rare in practice for triple-quoted strings.

On 8/8/2019 5:31 AM, Dima Tisnek wrote:

These two ought to be converted to raw strings, shouldn't they?

For the first example, yes or no. It depends ;-) See below. The problem is that string literals in python code are, by default, half-baked. The interpretation of '\' by the python parser, and the resulting string object, depends on the next char. I can see how this is sometimes a convenience, but I consider it a design bug. There is no way for a user to say "I intend for this string to be fully baked, so if it cannot be, I goofed." And the convenience gets used when it must not be.

On Thu, 8 Aug 2019 at 08:04, wrote:

...

For me, these warnings are continuing to arise almost daily. See two recent examples below. In both cases, the code previously had always worked without complaint.

----- Example from yesterday's class ----

''' How old-style formatting works with positional placeholders

print('The answer is %d today, but was %d yesterday' % (new, old)) \--------------------o \------------------------------------o '''

SyntaxWarning: invalid escape sequence \-

For true ascii-only character art, where one will never want '\' baked, an 'r' prefix is appropriate. It is in fact mandatory when '\' may be followed by a legal escape code. If one is making unicode art, with '\u' and '\U' escapes used, one must not use the 'r' prefix, but should instead use '\\' for unbaked backslashes. The unicode escapes have already thrown off column alignments.

...

----- Example from today's class ----

# Cut and pasted from: # https://en.wikipedia.org/wiki/VCard#vCard_2.1 vcard = ''' BEGIN:VCARD VERSION:2.1 N:Gump;Forrest;;Mr. FN:Forrest Gump ORG:Bubba Gump Shrimp Co. TITLE:Shrimp Man PHOTO;GIF:http://www.example.com/dir_photos/my_photo.gif TEL;WORK;VOICE:(111) 555-1212 TEL;HOME;VOICE:(404) 555-1212 ADR;WORK;PREF:;;100 Waters Edge;Baytown;LA;30314;United States of America LABEL;WORK;PREF;ENCODING=QUOTED-PRINTABLE;CHARSET=UTF-8:100 Waters Edge=0D= =0ABaytown\, LA 30314=0D=0AUnited States of America ADR;HOME:;;42 Plantation St.;Baytown;LA;30314;United States of America LABEL;HOME;ENCODING=QUOTED-PRINTABLE;CHARSET=UTF-8:42 Plantation St.=0D=0A= Baytown, LA 30314=0D=0AUnited States of America EMAIL:forrestgump@example.com REV:20080424T195243Z END:VCARD '''

SyntaxWarning: invalid escape sequence \,

Based on my reading of the Wikipedia vCard page linked above, the vCard protocol mandates use of '\' chars that must be passed through unbaked to a vCard processor. (I don't know why '\,', but it does not matter.) So vCard strings using '\' should generally have 'r' prefixes, just as for regex and latex strings. For version 2.1, it appears that one can currently, in 3.7-, get away with omitting 'r'. In versions 3.0 and 4.0, embedded 'newline' is represented by '\n' instead of '=0D=0A'. It must not be baked by python, but passed on as is. So omitting 'r' becomes a bug for those versions. To me, this one of the major problems with the half-baked default. People who want string literals left as is sometimes get away with omitting explicit mention of that fact, but sometimes don't. Note: when we added '\u' and '\U' escapes, we broke working code that had Windows paths like "C:\Users\Terry". But we did it anyway. -- Terry Jan Reedy

FWIW, the web archive https://mail.python.org/archives/list/python-dev@python.org/thread/ZX2JLOZDO... does not seem to display the problems ... apparently the individual messages are not included in view source, and are cleaned up for chrome's inspect. I'm not sure whether that counts as a bug in the archiving or not.