<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Apr 26, 2017 at 4:30 PM, Stephan Hoyer <span dir="ltr"><<a href="mailto:shoyer@gmail.com" target="_blank">shoyer@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-"><div><br></div></span><div>Sorry, I remain unconvinced (for the reasons that Robert, Nathaniel and myself have already given), but we seem to be talking past each other here.</div></div></div></div></blockquote><div><br></div><div>yeah -- I think it's not clear what the use cases we are talking about are.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I am still -1 on any new string encoding support unless that includes at least UTF-8, with length indicated by the number of bytes.</div></div></div></div></blockquote><div><br></div><div>I've said multiple times that utf-8 support is key to any "exchange binary data" use case (memory mapping?) -- so yes, absolutely.</div><div><br></div><div>I _think_ this may be some of the source for the confusion:</div><div><br></div><div>The name of this thread is: "proposal: smaller representation of string arrays".</div><div><br></div><div>And I got the impression, maybe mistaken, that folks were suggesting that internally encoding strings in numpy as "UTF-8, with length indicated by the number of bytes." was THE solution to the</div><div><br></div><div>" the 'U' dtype takes up way too much memory, particularly for mostly-ascii data" problem.</div><div><br></div><div>I do not think it is a good solution to that problem.</div><div><br></div><div>I think a good solution to that problem is latin-1 encoding. (bear with me here...)</div><div> </div><div>But a bunch of folks have brought up that while we're messing around with string encoding, let's solve another problem:</div><div><br></div><div>* Exchanging unicode text at the binary level with other systems that generally don't use UCS-4.</div><div><br></div><div>For THAT -- utf-8 is critical.</div><div><br></div><div>But if I understand Julian's proposal -- he wants to create a parameterized text dtype that you can set the encoding on, and then numpy will use the encoding (and python's machinery) to encode / decode when passing to/from python strings.</div><div><br></div><div>It seems this would support all our desires:</div><div><br></div><div>I'd get a latin-1 encoded type for compact representation of mostly-ascii data.</div><div><br></div><div>Thomas would get latin-1 for binary interchange with mostly-ascii data</div><div><br></div><div>The HDF-5 folks would get utf-8 for binary interchange (If we can workout the null-padding issue)</div><div><br></div><div>Even folks that had weird JAVA or Windows-generated UTF-16 data files could do the binary interchange thing....</div><div><br></div><div>I'm now lost as to what the hang-up is.</div><div><br></div><div>-CHB</div></div><div class="gmail_extra"><br></div><div class="gmail_extra">PS: null padding is a pain, python strings seem to preserve the zeros, whic is odd -- is thre a unicode code-point at x00?</div><div class="gmail_extra"><br></div><div class="gmail_extra">But you can use it to strip properly with the unicode sandwich:</div><div class="gmail_extra"><br></div><font face="monospace, monospace">In [63]: ut16 = text.encode('utf-16') + b'\x00\x00\x00\x00\x00\x00'<br><br>In [64]: ut16.decode('utf-16')<br>Out[64]: 'some text\x00\x00\x00'<br><br>In [65]: ut16.decode('utf-16').strip('\x00')<br>Out[65]: 'some text'<br><br></font></div><div class="gmail_extra"><font face="monospace, monospace">In [66]: ut16.decode('utf-16').strip('\x00').encode('utf-16')<br>Out[66]: b'\xff\xfes\x00o\x00m\x00e\x00 \x00t\x00e\x00x\x00t\x00'</font><div class="gmail_extra"><br></div><div class="gmail_extra">-CHB</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div>-- <br><div class="gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div>