On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote:

On Tue, Feb 24, 2015 at 2:44 AM, Guido van Rossum wrote:

...

I think that's a bit too strong. This has been unquestionably valid, correct Python -- it was an intentional feature from the start. It may not have turned out great, but I think that before warning loudly about every instance of this we should have a silent deprecation (which you can turn into a visible warning with a command-line flag or a warnings filter). And we should have agreement that we're eventually going to make it a syntax error.

Is it at all possible for this to be introduced in the 2.x line, or is the entire concept of a deprecation period one that has to start with a minor version?

Starts with a minor version.

If it's never going to happen in 2.x, I'll raise this as yet another

reason to get the course and all our students migrated to 3.x, but on

the flip side, it means that we absolutely can't get the benefit until that jump is made.

Never going to happen in 2.x..

On 02/23/2015 08:12 AM, Brett Cannon wrote:

On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote:

...

Is it at all possible for this to be introduced in the 2.x line [...]

Starts with a minor version.

Isn't there a -3 switch or something like that in 2.7.x to trigger warnings/errors to help port to 3.x? Seems like this kind of warning could go there. -- ~Ethan~

On Feb 23, 2015 9:30 AM, "Guido van Rossum" wrote:

On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote:

...

On 02/23/2015 08:12 AM, Brett Cannon wrote:

...

On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote:

...

...

Is it at all possible for this to be introduced in the 2.x line [...]

Starts with a minor version.

Isn't there a -3 switch or something like that in 2.7.x to trigger

warnings/errors to help port to 3.x? Seems like this

...

kind of warning could go there.

If we agree it will be a syntax error in 3.x eventually.

+1 to that FWIW. -n

On 23 February 2015 at 16:47, Antoine Pitrou wrote:

On Mon, 23 Feb 2015 09:29:09 -0800 Guido van Rossum wrote:

...

On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote:

...

On 02/23/2015 08:12 AM, Brett Cannon wrote:

...

On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote:

...

...

Is it at all possible for this to be introduced in the 2.x line [...]

Starts with a minor version.

Isn't there a -3 switch or something like that in 2.7.x to trigger warnings/errors to help port to 3.x? Seems like this kind of warning could go there.

If we agree it will be a syntax error in 3.x eventually.

That sounds frankly like a pedantic change.

The real problem there, even motivating the request, seems to be new users in Windows platforms using "\" as file path separator. That happens all the time, and is this use case that should possibly be addressed here - maybe something as simple as adding a couple of paragraphs to different places in the documentation could mitigate the issue. (in contrast to make a tons of otherwise valid code to become deprecated in a couple releases).

Regards

Antoine.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br

On 23.02.15 21:58, Joao S. O. Bueno wrote:

That happens all the time, and is this use case that should possibly be addressed here - maybe something as simple as adding a couple of paragraphs to different places in the documentation could mitigate the issue. (in contrast to make a tons of otherwise valid code to become deprecated in a couple releases).

The problem is that the user don't know that he should read the documentation. It just find that his script works with "C:\sample.txt", but doesn't work with "D:\test.txt". He has no ideas what happen.

On Tue, Feb 24, 2015 at 8:39 AM, Greg Ewing wrote:

Serhiy Storchaka wrote:

...

The problem is that the user don't know that he should read the documentation. It just find that his script works with "C:\sample.txt", but doesn't work with "D:\test.txt". He has no ideas what happen.

Even with the syntax error, there are filenames that will mysteriously fail, e.g. "C:\ninjamoves.txt".

Yes, there are; but they will fail, rather than randomly happening to work. With the proposed syntax warning, there will be no cases which silently succeed; everything will either be interpreted as an escape sequence (as in your example), fail with a hard error ("C:\Users" in a Unicode string, or "C:\x-do-not-track"), or produce a warning (everything else). The most tricky escape sequence that I can envision is \134, so you might find that an all-numeric path gets beheaded:

...

...

print("C:\134689") C:\689

The chances that someone will accidentally get something like this are very low, and the main thing is, "\X" for any X will never be interpreted the same way as "\\X", so the backslash will _always_ be seen as special. ChrisA

On Tue, Feb 24, 2015 at 1:00 PM, Joao S. O. Bueno wrote:

On 23 February 2015 at 18:39, Greg Ewing wrote:

...

Serhiy Storchaka wrote:

...

The problem is that the user don't know that he should read the documentation. It just find that his script works with "C:\sample.txt",

but

...

doesn't work with "D:\test.txt". He has no ideas what happen.

Even with the syntax error, there are filenames that will mysteriously fail, e.g. "C:\ninjamoves.txt".

That is the reason I urge folks to think about the root cause, and check if this proposal is the best way to tackle it:

The failing cases are the ones that won't be affected by this change at all

- as they are valid escaped strings!

I would argue that some cases working is the root cause of the confusion. If *none* of the cases work, it would be a lot less confusing, even if the error message is different. Trying to make the error messages more similar, or more identifying, may be a good idea (as long as they aren't misleading when people *meant* to use escape sequences in a string) but fortunately that's separate from making the cases that current silently and misleadingly work, emit a warning (or error.) Emitting a warning does not address the question 'why does "C:\test" not exist when I see it in Explorer.' It does address the question 'why does my program break when I change "C:\Test" to "C:\test". It also addresses 'why does my regexp not work when I change "\d+" to "\b\d+", and unlike changing the error message for opening files it is much closer, in time and space, to the actual problem. It's a helpful thing regardless of what other changes are made. I do expect there to need to be a much longer deprecation period than one or even two releases, though: there is a lot of code that uses the misfeature, mostly by accident (not realizing they should be using r'', etc.)

There will be no error for c:\tmp\myfile.txt - it will ever be just "file not found" - unless the warning comes to using "\" as file path separator (and no, I don think just doing that would properly address the issue as well).

Could we just use Guido's time machine and go back to the point in time where some MS head decided to go with "\" instead of "/", and deliver a well placed punch? :-)

Or maybe have IOError for "file not found" to get some fuzzy logic and display a more verbose error message if there are "\n"s and "\t"s on the filename and platform is Windows? I think that could go along the proposal for deprecating non-escaping "\<x>" sequences .

By the way, although I am writing to get the root issue covered, I think deprecating them would be a good thing, yes. Despite my previous comment on deprecating code that works, on a deeper though it _is_ worth the issue.

js -><-

...

-- Greg

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/thomas%40python.org

-- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread!

Chris Angelico writes:

I don't mind how long the deprecation period is, as long as there can be an option to Python that makes a noisy warning.

If that's OK with you and for the use case you explicitly described, though, a .bat file that runs a linter first might be the better option since (a) you don't have to wait and (b) you can put any bugaboo you like in there and (c) it can warn about syntacticly correct strings that "just fail" and may be hard to find. I think the Zen is right on, here: Now is better than never. Although never is often better than *right* now. I don't have a good sense for which it is, though, partly because (a) I don't program on Windows, but more importantly, (b) many of the dangerous strings actually used won't generate warnings or errors ever.

Nick Coghlan writes:

The linter developers don't have a decision making process that lets them pursue things like this on their own.

I'm not suggesting that the linter developers do any such thing. I'm suggesting that users (specifically Chris) customize their preferred linters (most permit that). What bothers me is that the hardest to understand failures cannot possibly be helped, because they are legal, meaningful syntax that conflicts with syntax common in external contexts. Replacing mnemonics like '\t' with '\x09' (or even '\u0009' in Python 3) in debugging output (including error messages) seems like a much better idea to me.

On Wed, Feb 25, 2015 at 5:00 PM, Ben Finney wrote:

There is a wider context here, too: semantics of the backslash escape URL:https://en.wikipedia.org/wiki/Backslash#Usage commonly include “backslash followed by a character not otherwise mentioned will produce that character, verbatim”.

Proposals to change that for Python's string literals must account for the fact this will make Python's rules for backslash escape surprisingly different in this regard from many other usages of backslash escape.

That's different from Python's rule, which is that "backslash followed by a character not otherwise mentioned will produce *a backslash followed by* that character, verbatim". ChrisA

On 23/02/2015 21:27, Serhiy Storchaka wrote:

On 23.02.15 21:58, Joao S. O. Bueno wrote:

...

That happens all the time, and is this use case that should possibly be addressed here - maybe something as simple as adding a couple of paragraphs to different places in the documentation could mitigate the issue. (in contrast to make a tons of otherwise valid code to become deprecated in a couple releases).

The problem is that the user don't know that he should read the documentation. It just find that his script works with "C:\sample.txt", but doesn't work with "D:\test.txt". He has no ideas what happen.

Isn't this why users have help desks? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

On 24 February 2015 at 08:40, Nick Coghlan wrote:

On 24 February 2015 at 07:39, Mark Lawrence wrote:

...

On 23/02/2015 21:27, Serhiy Storchaka wrote:

...

On 23.02.15 21:58, Joao S. O. Bueno wrote:

...

That happens all the time, and is this use case that should possibly be addressed here - maybe something as simple as adding a couple of paragraphs to different places in the documentation could mitigate the issue. (in contrast to make a tons of otherwise valid code to become deprecated in a couple releases).

The problem is that the user don't know that he should read the documentation. It just find that his script works with "C:\sample.txt", but doesn't work with "D:\test.txt". He has no ideas what happen.

Isn't this why users have help desks?

Most don't, and cases like "\n" or "\t" in a Windows path name being converted to whitespace are utterly impossible to look up in an internet search when they fail, so a user learning on their own gets left with a broken program and no particularly effective ways to ask for help figuring it out.

Like Unicode encoding errors they may appear a long way from the source of the offending data value (in this case, likely to be a file name copy and pasted from elsewhere on their system), and they don't give a particularly helpful error message (especially when the escape sequences are for whitespace).

While I originally disliked the idea, I think this is a genuine usability issue on Windows that would be worth addressing. However, it's a significant enough change that I believe it needs a PEP and a reasonably long transition period before anything actually breaks. For example:

- pep8 and pylint warnings as soon as a patch can be accepted - Py3kWarning in Python 2.7.x - DeprecationWarning in Python 3.5 - SyntaxWarning in Python 3.6 - SyntaxError in Python 3.7

Another suggestion: we may want to turn this particular deprecation warning on by default in the interactive interpreter, and recommend that other interactive interpreter developers (such as the IPython folks) consider doing the same. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 23/02/2015 23:32, Serhiy Storchaka wrote:

On 24.02.15 00:50, Nick Coghlan wrote:

...

...

- SyntaxError in Python 3.7

I think that SyntaxError could wait to 4.0.

...

Another suggestion: we may want to turn this particular deprecation warning on by default in the interactive interpreter, and recommend that other interactive interpreter developers (such as the IPython folks) consider doing the same.

Why not turn all deprecation warning on by default in the interactive interpreter?

In any given release how many of these would you see? I wouldn't mind seeing a few lines whenever I ran the II, but if the number started getting into tens or hundreds, at best I'd find it extremely irritating. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

On 23.02.2015 23:50, Nick Coghlan wrote:

On 24 February 2015 at 08:40, Nick Coghlan wrote:

...

On 24 February 2015 at 07:39, Mark Lawrence wrote:

...

On 23/02/2015 21:27, Serhiy Storchaka wrote:

...

On 23.02.15 21:58, Joao S. O. Bueno wrote:

...

That happens all the time, and is this use case that should possibly be addressed here - maybe something as simple as adding a couple of paragraphs to different places in the documentation could mitigate the issue. (in contrast to make a tons of otherwise valid code to become deprecated in a couple releases).

The problem is that the user don't know that he should read the documentation. It just find that his script works with "C:\sample.txt", but doesn't work with "D:\test.txt". He has no ideas what happen.

Isn't this why users have help desks?

Most don't, and cases like "\n" or "\t" in a Windows path name being converted to whitespace are utterly impossible to look up in an internet search when they fail, so a user learning on their own gets left with a broken program and no particularly effective ways to ask for help figuring it out.

I think the easiest way would be to tweak the error message output to indicate the real problem. At the moment, you get:

...

...

open('c:\test.txt') Traceback (most recent call last): File "<stdin>", line 1, in <module> FileNotFoundError: [Errno 2] No such file or directory: 'c:\test.txt'

which isn't helpful. It would be better to display the escapes as what they are: FileNotFoundError: [Errno 2] No such file or directory: 'c: est.txt' or use a special representation function to highlight the possible problem: FileNotFoundError: [Errno 2] No such file or directory: 'c:[\t]est.txt' - warning: embedded escapes with the representation being: "'c:[\t]est.txt' - warning: embedded escapes" Such a change doesn't require a deprecation phase or a PEP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 24 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 24.02.15 11:09, Nick Coghlan wrote:

Oh, I like that idea, especially the variant that includes a search friendly phrase like "embedded escapes" in the error message. And unlike changing the handling of string literals generally, the impact would be isolated to a few exception __repr__ implementations.

It can require changing code in tens of places like ``PyErr_Format(FooError, "File %R is not valid FOO", filename)`` or ``raise FooError("File %r is not valid FOO" % (filename,))``. And of course 'c:\test.txt' is valid name on Unix.

On 02/24/2015 02:58 AM, Chris Angelico wrote:

On Tue, Feb 24, 2015 at 7:40 PM, M.-A. Lemburg wrote:

...

I think the easiest way would be to tweak the error message output to indicate the real problem.

At the moment, you get:

...

...

...

open('c:\test.txt') Traceback (most recent call last): File "<stdin>", line 1, in <module> FileNotFoundError: [Errno 2] No such file or directory: 'c:\test.txt'

which isn't helpful.

The problem isn't the cases where the file can't be found. Those can be dealt with fairly easily, one way or another. The problem is much, much earlier, when the student was using "c:\sample.txt" and everything was working fine. Then he changed the code over to use his own file instead of the provided sample, and at the same time, switched from using open() to using csv.reader(open()), and moved all the code into a function, and fixed three other bugs, and now it isn't working. And he can't figure out why.

That's why I'd like "c:\sample.txt" to raise a warning.

Should I start writing up a PEP? Is that the way forward with this?

Go for it. I think the solution will be two pieces: - deprecating unknown backslash-escapes, with the goal of eventually being a SyntaxError (this removes a wart from the language, as well as allowing a warning backport to 2.7) - M.A.Lemburg's idea of changing the exception message in key places to make a successful backslash replace obvious (FileNotFoundError: [Errno 2] No such file or directory: 'c:[\t]est.txt' - warning: embedded escapes) -- ~Ethan~

TBH I think this will be a tough sell. I worry that an meme will emerge to make all string literals use raw mode, or worse, to deprecate non-raw string literals. I'm not sure how we'll get out of the whole conundrum, and I'm not in a hurry to see this in 3.5. -- --Guido van Rossum (python.org/~guido)

On Tue, Feb 24, 2015 at 10:00 AM, Ethan Furman wrote:

On 02/24/2015 09:44 AM, Guido van Rossum wrote:

...

TBH I think this will be a tough sell. I worry that an meme will emerge

...

string literals use raw mode, or worse, to deprecate non-raw string

to make all literals. I'm

...

not sure how we'll get out of the whole conundrum, and I'm not in a hurry to see this in 3.5.

So should we move forward with just the better error messages? It's a good first step, and even if it ends up being the only step it would be a huge improvement.

This is about messages from failing file-open operations if the filename contains an escaped character? I'd go slow there too: here are a lot of places where files are opened and messages are printed, both in the C code and in the stdlib. I'm not sure I buy the argument that just echoing the repr() of the name back doesn't help -- the escapes in there are actually useful in case a filename containing garbage chars (or even a trailing space) was read from some other source. And I'd weigh the needs of users who know what they are doing somewhat higher than educating newbies through error messages. While newbies are most likely to try out open() with a string literal, in "real" programs that is rare, and filenames are typically taken from the command line or from some other source where the peculiarities of Python string literals are irrelevant. -- --Guido van Rossum (python.org/~guido)

I like the \x07 solution. On Tue, Feb 24, 2015 at 10:47 AM, Ethan Furman wrote:

On 02/24/2015 10:14 AM, Guido van Rossum wrote:

...

This is about messages from failing file-open operations if the filename contains an escaped character? I'd go slow there too: here are a lot of places where files are opened and messages are printed, both in the C code and in the stdlib. I'm not sure I buy the argument that just echoing the repr() of the name back doesn't help -- the escapes in there are actually useful in case a filename containing garbage chars (or even a trailing space) was read from some other source.

I can attest from my impoverished Windows programming days that looking at

--> os.listdir('c:\temp') SomeErrorMessage about syntax 'c:\temp'

is not very helpful. There is zero visual indication that the \ and the t are one character, not two. Changing that error message to:

SomeErrorMessage about syntax 'c:[\t]emp'

or

SomeErrorMessage about syntax 'c:\x07emp'

or something that shouts out, "hey! one character in this location!" would be a good thing.

-- ~Ethan~

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

On Tue, 24 Feb 2015, Ethan Furman wrote:

On 02/24/2015 10:49 AM, Guido van Rossum wrote:

...

On Tue, Feb 24, 2015 at 10:47 AM, Ethan Furman wrote:

...

I can attest from my impoverished Windows programming days that looking at

--> os.listdir('c:\temp') SomeErrorMessage about syntax 'c:\temp'

is not very helpful. There is zero visual indication that the \ and the t are one character, not two. Changing that error message to:

SomeErrorMessage about syntax 'c:[\t]emp'

or

SomeErrorMessage about syntax 'c:\x07emp'

or something that shouts out, "hey! one character in this location!" would be a good thing.

I like the \x07 solution.

So final question is do we take the easy road and change the repr for str to always use hex or unicode escapes instead of simple backslash-escapes (and then run for our lives), or do we just hunt down and change the appropriate error messages for files (and possibly re) ?

Just a data point from a random programmer: I like the \x07 solution for the error message as it draws attention to the character at issue, but I also like to see \n, \t etc. in the result of repr because it is more readable for the common cases. Isaac Morland CSCF Web Guru DC 2619, x36650 WWW Software Specialist

On 02/24/2015 09:18 AM, Thomas Wouters wrote:

On Tue, Feb 24, 2015 at 5:37 PM, Ethan Furman wrote:

...

- M.A.Lemburg's idea of changing the exception message in key places to make a successful backslash replace obvious (FileNotFoundError: [Errno 2] No such file or directory: 'c:[\t]est.txt' - warning: embedded escapes)

Any ideas on how this check could be implemented? How would it distinguish \t from an actual tab in the string literal, or "C:\x64-python" from "C:\d-python"...?

--> 'a tab' 'a\ttab' --> 'a\ttab' 'a\ttab' So the phrase "embedded escapes" may not always be accurate, but will be something to search on, and will help explain what is being seen. -- ~Ethan~

On Wed, Feb 25, 2015 at 6:54 AM, Greg Ewing wrote:

Chris Angelico wrote:

...

Then he changed the code over to use his own file instead of the provided sample, and at the same time, switched from using open() to using csv.reader(open()), and moved all the code into a function, and fixed three other bugs, and now it isn't working. And he can't figure out why.

There's probably a useful meta-lesson in there: test after making every change!

Of course! But when you're working with students remotely, you don't necessarily even know that that's what's happened. All you get is "It works with sample.txt, but it doesn't work with my file". You may or may not even get to see the code at first. Firm believer in "commit often" policy... ChrisA

On Tue, Feb 24, 2015 at 9:40 AM, Nick Coghlan wrote:

Most don't, and cases like "\n" or "\t" in a Windows path name being converted to whitespace are utterly impossible to look up in an internet search when they fail, so a user learning on their own gets left with a broken program and no particularly effective ways to ask for help figuring it out.

Like Unicode encoding errors they may appear a long way from the source of the offending data value (in this case, likely to be a file name copy and pasted from elsewhere on their system), and they don't give a particularly helpful error message (especially when the escape sequences are for whitespace).

And on the subject of Unicode errors, imagine a path name of "C:\Users\YourName\Desktop\FileName.txt". Under Python 2, that'll work fine; under Python 3, it'll fail with an error that, in all probability, will be interpreted as "Unicode is hard :(" rather than "loose backslash escapes are a problem". ChrisA

Hi, On Mon, Feb 23, 2015 at 9:58 PM, Joao S. O. Bueno wrote:

On 23 February 2015 at 16:47, Antoine Pitrou wrote:

...

On Mon, 23 Feb 2015 09:29:09 -0800 Guido van Rossum wrote:

...

On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote:

...

On 02/23/2015 08:12 AM, Brett Cannon wrote:

...

On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote:

...

...

Is it at all possible for this to be introduced in the 2.x line [...]

Starts with a minor version.

Isn't there a -3 switch or something like that in 2.7.x to trigger warnings/errors to help port to 3.x? Seems like this kind of warning could go there.

If we agree it will be a syntax error in 3.x eventually.

That sounds frankly like a pedantic change.

The real problem there, even motivating the request, seems to be new users in Windows platforms using "\" as file path separator.

That happens all the time, and is this use case that should possibly be addressed here - maybe something as simple as adding a couple of paragraphs to different places in the documentation could mitigate the issue. (in contrast to make a tons of otherwise valid code to become deprecated in a couple releases).

Note that in the introduction page of the tutorial[0], there is already the following snippet: """ If you don’t want characters prefaced by \ to be interpreted as special characters, you can use raw strings by adding an r before the first quote:

...

...

...

...

print('C:\some\name') # here \n means newline! C:\some ame print(r'C:\some\name') # note the r before the quote C:\some\name """"

Best Regards, Ezio Melotti [0]: https://docs.python.org/3/tutorial/introduction.html

...

Regards

Antoine.