<div dir="ltr">Folks,<div><br></div><div>I've been blathering away on the related threads a lot -- sorry if it's too much. It's gotten a bit tangled up, so I thought I'd start a new one to address this one question (i.e. dont bring up genfromtext here):</div>
<div><br></div><div>Would it be a good thing for numpy to have a one-byte--per-character string type?</div><div><br></div><div style>We did have that with the 'S' type in py2, but the changes in py3 have made it not quite the right thing. And it appears that enough people use 'S' in py3 to mean 'bytes', so that we can't change that now.</div>
<div style><br></div><div style>The only difference may be that 'S' currently auto translates to a bytes object, resulting in things like:<br></div><div style><br></div><div style><div>np.array(['some text',], dtype='S')[0] == 'some text'</div>
<div><br></div><div style>yielding False on Py3. And you can't do all the usual text stuff with the resulting bytes object, either. (and it probably used the default encoding to generate the bytes, so will barf on some inputs, though that may be unavoidable.) So you need to decode the bytes that are given back, and now that I think about it, I have no idea what encoding you'd need to use in the general case.</div>
<div style><br></div><div style>So the correct solution is (particularly on py3) to use the 'U' (unicode) dtype for text in numpy arrays.</div><div style><br></div><div style>However, the 'U' dtype is 4 bytes per character, and that may be "too big" for some use-cases. And there is a lot of text in scientific data sets that are pure ascii, or at least some 1-byte-per-character encoding.</div>
<div style><br></div><div style>So, in the spirit of having multiple numeric types that use different amounts of memory, and can hold different ranges of values, a one-byte-per character dtype would be nice:</div><div style>
<br></div><div style>(note, this opens the door for a 2-byte per (UCS-2) dtype too, I personally don't think that's worth it, but maybe that's because I'm an english speaker...)</div><div style><br></div>
<div style>
It could use the 's' (lower-case s) type identifier.</div><div style><br></div><div style>For passing to/from python built-in objects, it would</div><div style><br></div><div style>* Allow either Python bytes objects or Python unicode objects as input</div>
<div style> a) bytes objects would be passed through as-is<br></div><div style> b) unicode objects would be encoded as latin-1</div><div style> </div><div style>[note: I'm not entirely sure that bytes objects should be allowed, but it would provide an nice efficiency in a fairly common case]</div>
<div style><br></div><div style>* It would create python unicode text objects, decoded as latin-1.</div><div style><br></div><div style>Could we have a way to specify another encoding? I'm not sure how that would fit into the dtype system.</div>
<div style><br></div><div style>I've explained the latin-1 thing on other threads, but the short version is:</div><div style><br></div><div style> - It will work perfectly for ascii text</div><div style> - It will work perfectly for latin-1 text (natch)</div>
<div style> - It will never give you an UnicodeEncodeError regardless of what arbitrary bytes you pass in.</div><div style> - It will preserve those arbitrary bytes through a encoding/decoding operation.</div><div style>
<br>
</div><div style>(it still wouldn't allow you to store arbitrary unicode -- but that's the limitation of one-byte per character...)</div><div style><br></div><div style>So:</div><div style><br></div><div style>Bad idea all around: shut up already!</div>
<div style><br></div><div style>or</div><div style><br></div><div style>Fine idea, but who's going to write the code? not me!</div><div style><br></div><div style>or</div><div style><br></div><div style>We really should do this.</div>
<div style><br></div><div style>(of course, with the options of amending the above not-very-fleshed out proposal)</div><div style><br></div><div style>-Chris</div><div style><br></div></div><div style>-- <br><br>Christopher Barker, Ph.D.<br>
Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br>
<a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a>
</div></div>