<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Apr 26, 2017 at 11:31 AM, Nathaniel Smith <span dir="ltr"><<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><span class="gmail-"><div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail-m_251411978938451708quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">UTF-8 does not match the character-oriented Python text model. Plenty<br>
of people argue that that isn't the "correct" model for Unicode text<br>
-- maybe so, but it is the model python 3 has chosen. I wrote a much<br>
longer rant about that earlier.<br>
<br>
So I think the easy to access, and particularly defaults, numpy string<br>
dtypes should match it.<br></blockquote></div></div></div><div dir="auto"><br></div></span><div dir="auto">This seems a little vague?</div></div></blockquote><div><br></div><div>sorry -- that's what I get for trying to be concise...</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto"> The "character-oriented Python text model" is just that str supports O(1) indexing of characters. </div></div></blockquote><div><br></div><div>not really -- I think the performance characteristics are an implementation detail (though it did influence the design, I'm sure)</div><div><br></div><div>I'm referring to the fact that a python string appears (to the user -- also under the hood, but again, implementation detail) to be a sequence of characters, not a sequence of bytes, not a sequence of glyphs, or graphemes, or anything else. Every Python string has a length, and that length is the number of characters, and if you index you get a string of length-1, and it has one character it it, and that character matches to a code point of a single value.</div><div><br></div><div>Someone could implement a python string using utf-8 under the hood, and none of that would change (and I think micropython may have done that...)</div><div><br></div><div>Sure, you might get two characters when you really expect a single grapheme, but it's at least a consistent oddity. (well, not always, as some graphemes can be represented by either a single code point or two combined -- human language really sucks!)</div><div><br></div><div>The UTF-8 Manifesto (<a href="http://utf8everywhere.org/">http://utf8everywhere.org/</a>) makes the very good point that a character-oriented interface is not the only one that makes sense, and may not make sense at all. However:</div><div><br></div><div>1) Python has chosen that interface</div><div><br></div><div>2) It is a good interface (probably the best for computer use) if you need to choose only one</div><div><br></div><div>utf8everywhere is mostly arguing for utf-8 over utf16 -- and secondarily for utf-8 everywhere as the best option for working at the C level. That's probably true.</div><div><br></div><div>(I also think the utf-8 fans are in a bit of a fantasy world -- this would all be easier, yes, if one encoding was used for everything, all the time, but other than that, utf-8 is not a Pancea -- we are still going to have encoding headaches no matter how you slice it)</div><div><br></div><div>So where does numpy fit? well, it does operate at the C level, but people work with it from python, so exposing the details of the encoding to the user should be strictly opt-in.</div><div><br></div><div>When a numpy user wants to put a string into a numpy array, they should know how long a string they can fit -- with "length" defined how python strings define it. </div><div><br></div><div>Using utf-8 for the default string in numpy would be like using float16 for default float--not a good idea!</div><div><br></div><div>I believe Julian said there would be no default -- you would need to specify, but I think there does need to be one:</div><div><br></div><div>np.array(["a string", "another string"]) </div><div><br></div><div>needs to do something.</div><div><br></div><div>if we make a parameterized dtype that accepts any encoding, then we could do:</div><div><br></div><div>np.array(["a string", "another string"], dtype=no.stringtype["utf-8"]) <br></div><div><br></div><div>If folks really want that.</div><div><br></div><div>I'm afraid that that would lead to errors -- cool,. utf-8 is just like ascii, but with full Unicode support!</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto">But... Numpy doesn't. If you want to access individual characters inside a string inside an array, you have to pull out the scalar first, at which point the data is copied and boxed into a Python object anyway, using whatever representation the interpreter prefers. </div></div></blockquote><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto">So AFAICT it makes literally no difference to the user whether numpy's internal representation allows for fast character access.</div></div></blockquote><div><br></div><div>agreed - unless someone wants to do a view that makes a N-D array for strings look like a 1-D array of characters.... Which seems odd, but there was recently a big debate on the netcdf CF conventions list about that very issue...</div><div><br></div><div>-CHB</div><div><br></div></div><div><br></div>-- <br><div class="gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div>