<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Aug 21, 2014 at 7:42 PM, Oleg Broytman <span dir="ltr"><<a href="mailto:phd@phdru.name" target="_blank">phd@phdru.name</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>On Thu, Aug 21, 2014 at 05:30:14PM -0700, Chris Barker - NOAA Federal <<a href="mailto:chris.barker@noaa.gov" target="_blank">chris.barker@noaa.gov</a>> wrote:<br>
> This brings up the other key problem. If file names are (almost)<br>
> arbitrary bytes, how do you write one to/read one from a text file<br>
> with a particular encoding? ( or for that matter display it on a<br>
> terminal)<br>
<br>
</div> There is no such thing as an encoding of text files. So we just<br>
write those bytes to the file</blockquote><div><br></div><div>So I write bytes that are encoded one way into a text file that's encoded another way, and expect to be abel to read that later? you're kidding, right? Only if that's he only thing in the file -- usually not the case with my text files.</div>
<div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">or output them to the terminal. I often do<br>
that. My filesystems are full of files with names and content in<br>at least 3 different encodings - koi8-r, utf-8 and cp1251. So I open a<br>terminal with koi8 or utf-8 locale and fonts and some file always look<br>weird. But however weird they are it's possible to work with them.<br>
</blockquote><div><br></div><div>Not for me (or many other users) -- terminals are sometimes set with ascii-only encoding, so non-ascii barfs -- or you get some weird control characters that mess up your terminal -- dumping arbitrary bytes to a terminal does not always "just work". </div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>> And people still want to say posix isn't broken in this regard?<br>
<br></div> Not at all! And broken or not broken it's what I (for many different<br>reasons) prefer to use for my desktops, servers, notebooks, routers and<br>smartphones,</blockquote><div><br></div><div>Sorry -- that's a Red Herring -- I agree, "broken" or "simple and consistent" is irrelevant, we all want Python to work as well as it can on such systems.</div>
<div><br></div></div><div>The point is that if you are reading a file name from the system, and then passing it back to the system, then you can treat it as just bytes -- who cares? And if you add the byte value of 47 thing, then you can even do basic path manipulations. But once you want to do other things with your file name, then you need to know the encoding. And it is very, very common for users to need to do other things with filenames, and they almost always want them as text that they can read and understand.</div>
<div><br></div><div>Python3 supports this case very well. But it does indeed make it hard to work with filenames when you don't know the encoding they are in. And apparently that's pretty common -- or common enough that it would be nice for Python to support it well. This trick is how -- we'd like the "just pass it around and do path manipulations" case to work with (almost) arbitrary bytes, but everything else to work naturally with text (unicode text).</div>
<div><br></div><div>Which brings us to the "what APIs should accept bytes" question. I think that's been pretty much answered: All the low-level ones, so that protocol and library programmers can write code that works on systems with undefined filename encodings.</div>
<div><br></div><div>But: casual users still need to do the normal things with file names and paths, and ideally those should work the same way on all systems.</div><div><br></div><div>I think the way to do this is to abstract the path concept, like pathlib does. Back in the day, paths were "just strings", and that worked OK with py2 strings, because you could put arbitrary bytes in them. But the "py2 strings were perfect" folks seem to not acknowledge that while they are nice for matching the posix filename model, they were a pain in the neck when you needed to do somethign else like write them in to a JSON file or something. From my personal experience, non-ascii filenames are much easier to deal with if I use unicode for filenames everywhere (py2). Somehow, I have yet to be bitten by mixed encoding in filenames.</div>
<div><br></div><div>So will using a surrogate-escape error handling with pathlib make all this just work?</div><div><br></div><div>-Chris</div><div><br></div></div>-- <br>
<br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R <a href="tel:%28206%29%20526-6959" value="+12065266959" target="_blank">(206) 526-6959</a> voice<br>7600 Sand Point Way NE <a href="tel:%28206%29%20526-6329" value="+12065266329" target="_blank">(206) 526-6329</a> fax<br>
Seattle, WA 98115 <a href="tel:%28206%29%20526-6317" value="+12065266317" target="_blank">(206) 526-6317</a> main reception<br>
<br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a>
</div></div>