<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body><div><div style="font-family: Calibri,sans-serif; font-size: 11pt;">"You consistently ignore Makefiles, .ini, etc."<br><br>Do people really do open('makefile', 'rb'), extract filenames and try to use them without ever decoding the file contents?<br><br>I've honestly never seen that, and it certainly looks like the sort of thing Python 3 was intended to discourage. (As soon as you open(..., 'r') you're only affected by this change if you explicitly encode again with mbcs.)<br><br>Top-posted from my Windows Phone</div></div><div dir="ltr"><hr><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">From: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;"><a href="mailto:turnbull.stephen.fw@u.tsukuba.ac.jp">Stephen J. Turnbull</a></span><br><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">Sent: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;">8/17/2016 19:43</span><br><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">To: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;"><a href="mailto:steve.dower@python.org">Steve Dower</a></span><br><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">Cc: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;"><a href="mailto:p.f.moore@gmail.com">Paul Moore</a>; <a href="mailto:python-ideas@python.org">Python-Ideas</a></span><br><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">Subject: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;">Re: [Python-ideas] Fix default encodings on Windows</span><br><br></div>Steve Dower writes:<br> > On 17Aug2016 0235, Stephen J. Turnbull wrote:<br><br> > > So a full statement is, "How do we best represent Windows file<br> > > system paths in bytes for interoperability with systems that<br> > > natively represent paths in bytes?" ("Other systems" refers to<br> > > both other platforms and existing programs on Windows.)<br> > <br> > That's incorrect, or at least possible to interpret correctly as<br> > the wrong thing. The goal is "code compatibility with systems ...",<br> > not interoperability.<br><br>You're right, I stated that incorrectly. I don't have anything to add<br>to your corrected version.<br><br> > > In a properly set up POSIX locale[1], it Just Works by design,<br> > > especially if you use UTF-8 as the preferred encoding. It's<br> > > Windows developers and users who suffer, not those who wrote the<br> > > code, nor their primary audience which uses POSIX platforms.<br> > <br> > You mentioned "locale", "preferred" and "encoding" in the same sentence, <br> > so I hope you're not thinking of locale.getpreferredencoding()? Changing <br> > that function is orthogonal to this discussion,<br><br>You consistently ignore Makefiles, .ini, etc. It is *not* orthogonal,<br>it is *the* reason for all opposition to your proposal or request that<br>it be delayed. Filesystem names *are* text in part because they are<br>*used as filenames in text*.<br><br> > When Windows developers and users suffer, I see it as my responsibility <br> > to reduce that suffering. Changing Python on Windows should do that <br> > without affecting developers on Linux, even though the Right Way is to <br> > change all the developers on Linux to use str for paths.<br><br>I resent that. If I were a partisan Linux fanboy, I'd be cheering you<br>on because I think your proposal is going to hurt an identifiable and<br>large class of *Windows* users. I know about and fear this possiblity<br>because they use a language I love (Japanese) and an encoding I hate<br>but have achieved a state of peaceful coexistence with (Shift JIS).<br><br>And on the general principle, *I* don't disagree. I mentioned earlier<br>that I use only the str interfaces in my own code on Linux and Mac OS<br>X, and that I suspect that there are no real efficiency implications<br>to using str rather than bytes for those interfaces.<br><br>On the other hand, the programming convenience of reading the<br>occasional "text" filename (or other text, such as XML tags) out of a<br>binary stream and passing it directly to filesystem APIs cannot be<br>denied. I think that the kind of usage you propose (a fixed,<br>universal codec, universally accepted; ie, 'utf-8') is the best way to<br>handle that in the long run. But as Grandmaster Lasker said, "Before<br>the end game, the gods have placed the middle game." (Lord Keynes<br>isn't relevant here, Python will outlive all of us. :-)<br><br> > I don't think there's any reasonable way to noisily deprecate these<br> > functions within Python, but certainly the docs can be made<br> > clearer. People who explicitly encode with<br> > sys.getfilesystemencoding() should not get the deprecation message,<br> > but we can't tell whether they got their bytes from the right<br> > encoding or a RNG, so there's no way to discriminate.<br><br>I agree with you within Python; the custom is for DeprecationWarnings<br>to be silent by default.<br><br>As for "making noise", how about announcing the deprecation as like<br>the top headline for 3.6, postponing the actual change to 3.7, and in<br>the meantime you and Nick do a keynote duet at PyCon? (Your partner<br>could be Guido, too, but Nick has been the most articulate proponent<br>for this particular aspect of "inclusion". I think having a<br>representative from the POSIX world explaining the importance of this<br>for "all of us" would greatly multiply the impact.) Perhaps, given my<br>proposed timing, a discussion at the language summit in '17 and the<br>keynote in '18 would be the best timing.<br><br>(OT, political: I've been strongly influenced in this proposal by<br>recently reading http://blog.aurynn.com/contempt-culture. There's not<br>as much of it in Python as in other communities I'm involved in, but I<br>think this would be a good symbolic opportunity to express our<br>oppostion to it. "Inclusion" isn't just about gender and race!)<br><br> > I'm going to put together a summary post here (hopefully today) and get <br> > those who have been contributing to basically sign off on it, then I'll <br> > take it to python-dev. The possible outcomes I'll propose will basically <br> > be "do we keep the status quo, undeprecate and change the functionality, <br> > deprecate the deprecation and undeprecate/change in a couple releases, <br> > or say that it wasn't a real deprecation so we can deprecate and then <br> > change functionality in a couple releases".<br><br>FWIW, of those four, I dislike 'status quo' the most, and like 'say it<br>wasn't real, deprecate and change' the best. Although I lean toward<br>phrasing that as "we deprecated it, but we realize that practitioners<br>are by and large not aware of the deprecation, and nobody expects the<br>Spanish Inquisition".<br><br>@Nick, if you're watching: I wonder if it would be possible to expand<br>the "in the file system, bytes are UTF-8" proposal to POSIX as well,<br>perhaps for 3.8?<br></body></html>