<div dir="ltr">On Fri, Jan 17, 2014 at 12:36 PM, <span dir="ltr"><<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><div class="h5">> ('S' ?) -- which is probably not what you want particularly if you specify<br>
> an encoding. Though I can't figure out at the moment why the previous one<br>
> failed -- where did the bytes object come from when the encoding was<br>
> specified?<br>
<br>
</div></div>Yes, it's a utf-8 file with nonascii.<br>
<br>
I don't know what I **should** want.<br></blockquote><div><br></div><div style>well, you **should** want:</div><div style><br></div><div style>The numbers parsed out for you (Other wise, why use recfromtxt), and the text as properly decoded unicode strings.<br>
</div><div style><br></div><div style>Python does very well with unicode -- and you are MUCH happier if you do the encoding/decoding as close to I/O as possible. recfromtxt is, in a way, decoding already, converting ascii representation of numbers to an internal binary representation -- why not handle the text at the same time.<br>
</div><div style><br></div><div style>There certainly are use cases for keeping the text as encoded bytes, but I'd say those fall into the categories of:</div><div style><br></div><div style>1) Special case</div><div style>
2) You should know what you are doing.</div><div style><br></div><div style>So having recfromtxt auto-determine that for you makes little sense.</div><div style><br></div><div style>Note that if you don't know the file encoding, this is tricky. My thoughts:</div>
<div style><br></div><div style>1) don't use the system default encoding!!! (see my other note on that!)</div><div style><br></div><div style>2) Either:</div><div style> a) open as a binary file and use bytes for anything that doesn't parse as text -- this means that the user will need to do the conversion to text themselves</div>
<div style><br></div><div style> b) decode as latin-1: this would work well for ascii and _some_ non-ascii text, and would be recoverable for ALL text.</div><div style><br></div><div style>I prefer (b). The point here is that if the user gets bytes, then they will either have to assume ascii, or need to hand-decode it, and if they just want assume ascii, they have a bytes object with limited text functionality so will probably need to decode it anyway (unless they are just passing it through)</div>
<div style><br></div><div style>If the user gets unicode objects that are may not properly decoded, they can either assume it was ascii, and if they only do ascii-compatible things with it, it will work, or they can encode/decode it and get the proper stuff back, but only if they know the encoding, and if that's the case, why did they not specify that in the first place?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">For now I do want bytes, because that's how I changed statsmodels in<br>
the py3 conversion.<br>
<br>
This was just based on the fact that recfromtxt doesn't work with<br>
strings on python 3, so I switched to using bytes following the lead<br>
of numpy.<br></blockquote><div><br></div><div style>Well, that's really too bad -- it doesn't sound like you wanted bytes, it sounds like you wanted something that didn't crash -- fair enough. But the "proper" solution is for recfromtext to support text....</div>
<div style><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I'm mainly worried about backwards compatibility, since we have been<br>
using this for 2 or 3 years. It would be easy to change in statsmodels<br>
when gen/recfromtxt is fixed, but I assume there is lots of other code<br>
using similar interpretation of S/bytes in numpy.<br></blockquote><div><br></div><div style>well, it does sound like enough folks are using 'S' to mean bytes -- too bad, but what can we do now about that?</div><div style>
<br></div><div style>I'd like a 's' for ascii-stings though.</div><div style><br></div><div style>-Chris</div><div style><br></div></div>-- <br><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>
NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a>
</div></div>