<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 24, 2017 at 4:08 PM, Robert Kern <span dir="ltr"><<a href="mailto:robert.kern@gmail.com" target="_blank">robert.kern@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Chris, you've mashed all of my emails together, some of them are in reply to you, some in reply to others. Unfortunately, this dropped a lot of the context from each of them, and appears to be creating some misunderstandings about what each person is advocating.</div></div></blockquote><div><br></div><div>Sorry about that -- I was trying to keep an already really long thread from getting eve3n longer....</div><div><br></div><div>And I'm not sure it matters who's doing the advocating, but rather *what* is being advocated -- I hope I didn't screw that up too badly.</div><div><br></div><div>Anyway, I think I made the mistake of mingling possible solutions in with the use-cases, so I'm not sure if there is any consensus on the use cases -- which I think we really do need to nail down first -- as Robert has made clear.</div><div><br></div><div>So I'll try again -- use-case only! we'll keep the possible solutions separate.</div><div><br></div><div>Do we need to write up a NEP for this? it seems we are going a bit in circles, and we really do want to capture the final decision process.</div><div><br></div>1) The default behaviour for numpy arrays of strings is compatible with Python3's string model: i.e. fully unicode supporting, and with a character oriented interface. i.e. if you do::<br><br> arr = np.array(("this", "that",))<br><br></div><div class="gmail_quote">you get an array that can store ANY unicode string with 4 or less characters.<br><br>and arr[1] will return a native Python3 string object.<div><br></div><div>This is the use-case for "casual" numpy users -- not the folks writing H5py and the like, or the ones writing Cython bindings to C++ libs.</div><div><br></div><div> </div>2) There be some way to store mostly ascii-compatible strings in a single byte-per-character array -- so not to be wasting space for "typical european-language-oriented data". Note: this should ALSO be compatible with Python's character-oriented string model. i.e. a Python String with length N will fit into a dtype of size N.<br><br></div><div class="gmail_quote">arr = np.array(("this", "that",), dtype=np.single_byte_string)<br><br>(name TBD)<br><br>and arr[1] would return a python string.<br><br></div><div class="gmail_quote">attempting to put in a not-compatible with the encoding String would raise an EncodingError.<br><br>This is also a use-case primarily for "casual" users -- but ones concerned with the size of the data storage and know that are using european text.<br><div><br></div>3) dtypes that support storage in particular encodings:</div><div class="gmail_quote"><br></div><div class="gmail_quote"> Python strings would be encoded appropriately when put into the array. A Python string would be returned when indexing.</div><div><br></div><div class="gmail_quote"> a) There be a dtype that could store strings in null-terminated utf-8 binary format -- for interchange with other systems (netcdf, HDF, others???) at the binary level.</div><div class="gmail_quote"><br></div><div class="gmail_quote"><div> b) There be a dtype that could store data in any encoding supported by Python -- to facilitate bytes-level interchange with other systems. If we need more than utf-8, then we might as well have the full set.</div><div> <br></div>4) a fixed length bytes dtype -- pretty much what 'S' is now under python three -- settable from a bytes or bytearray object (or other memoryview?), and returns a bytes object.<br><br></div><div class="gmail_quote">You could use astype() to convert between bytes and a specified encoding with no change in binary representation. This could be used to store any binary data, including encoded text or anything else. this should map directly to the Python bytes model -- thus NOT null-terminted.</div><div class="gmail_quote"><br></div><div class="gmail_quote">This is a little different than 'S' behaviour on py3 -- it appears that with 'S', a if ALL the trailing bytes are null, then it is truncated, but if there is a null byte in the middle, then it is preserved. I suspect that this is a legacy from Py2's use of "strings" as both text and binary data. But in py3, a "bytes" type should be about bytes, and not text, and thus null-values bytes are simply another value a byte can hold.<br><br><div>There are multiple ways to address these use cases -- please try to make your comments clear about whether you think the use-case is unimportant, or ill-defined, or if you think a given solution is a poor choice.</div><div><br></div><div>To facilitate that, I will put my comments on possible solutions in a separate note, too.</div><div><br></div></div><div>-CHB</div><div><br></div><div><br></div>-- <br><div class="gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div>