<div dir="ltr">So while compression+ucs-4 might be OK for out-of-core representation, what about in-core? blosc+ucs-4? I don't think that works for mmap, does it?</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Apr 27, 2017 at 7:11 AM Francesc Alted <<a href="mailto:faltet@gmail.com">faltet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif">2017-04-27 3:34 GMT+02:00 Stephan Hoyer </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:shoyer@gmail.com" target="_blank">shoyer@gmail.com</a>></span><span style="font-family:arial,sans-serif">:</span><br></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Wed, Apr 26, 2017 at 4:49 PM, Nathaniel Smith <span dir="ltr"><<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>></span> wrote:<br></span><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="auto"><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="m_8355645822660463935m_7815591276373255554gmail-m_-6233911898642044880gmail-m_834704364764283414m_-7914534151798526079quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>It's worthwhile enough that both major HDF5 bindings don't support Unicode arrays, despite user requests for years. The sticking point seems to be the difference between HDF5's view of a Unicode string array (defined in size by the bytes of UTF-8 data) and numpy's current view of a Unicode string array (because of UCS-4, defined by the number of characters/codepoints/whatever). So there are HDF5 files out there that none of our HDF5 bindings can read, and it is impossible to write certain data efficiently.<br></div></div></blockquote></div></div><div class="gmail_extra"><br>I would really like to hear more from the authors of these libraries about what exactly it is they feel they're missing. Is it that they want numpy to enforce the length limit early, to catch errors when the array is modified instead of when they go to write it to the file? Is it that they really want an O(1) way to look at a array and know the maximum number of bytes needed to represent it in utf-8? Is it that utf8<->utf-32 conversion is really annoying and files that need it are rare so they haven't had the motivation to implement it? My impression is similar to Julian's: you *could* implement HDF5 fixed-length utf-8 <-> numpy U arrays with a few dozen lines of code, which is nothing compared to all the other hoops these libraries are already jumping through, so if this is really the roadblock then I must be missing something.</div></div></div></div></blockquote><div><br></div></span><div>I actually agree with you. I think it's mostly a matter of convenience that h5py matched up HDF5 dtypes with numpy dtypes:</div><div>fixed width ASCII -> np.string_/bytes</div><div>variable length ASCII -> object arrays of np.string_/bytes</div><div>variable length UTF-8 -> object arrays of unicode</div><div><br></div><div>This was tenable in a Python 2 world, but on Python 3 it's broken and there's not an easy fix.</div><div><br></div><div>We absolutely could fix h5py by mapping everything to object arrays of Python unicode strings, as has been discussed (<a href="https://github.com/h5py/h5py/pull/871" target="_blank">https://github.com/h5py/h5py/pull/871</a>). For fixed width UTF-8, this would be a fine but non-ideal solution, since there is currently no fixed width UTF-8 support.</div><div><br></div><div>For fixed width ASCII arrays, this would mean increased convenience for Python 3 users, at the price of decreased convenience for Python 2 users (arrays now contain boxed Python objects), unless we made the h5py behavior dependent on the version of Python. Hence, we're back here, waiting for better dtypes for encoded strings.</div><div><br></div><div>So for HDF5, I see good use cases for ASCII-with-surrogateescape (for handling ASCII arrays as strings) and UTF-8 with length equal to the number of bytes.</div></div></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Well, I'll say upfront that I have not read this discussion in the fully, but apparently some opinions from developers of HDF5 Python packages would be welcome here, so here I go :) </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">As a long-time developer of one of the Python HDF5 packages (PyTables), I have always been of the opinion that plain ASCII (for byte strings) and UCS-4 (for Unicode) encoding would be the appropriate dtypes for storing large amounts of data, most specially for disk storage (but also using compressed in-memory containers). My rational is that, although UCS-4 may require way too much space, compression would reduce that to basically the space that is required by compressed UTF-8 (I won't go into detail, but basically this is possible by using the shuffle filter).</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I remember advocating for UCS-4 adoption in the HDF5 library many years ago (2007?), but I had no success and UTF-8 was decided to be the best candidate. So, the boat with HDF5 using UTF-8 sailed many years ago, and I don't think there is a go back (not even adding UCS-4 support on it, although I continue to think it would be a good idea). So, I suppose that if HDF5 is found to be an important format for NumPy users (and I think this is the case), a solution for representing Unicode characters by using UTF-8 in NumPy would be desirable (at the risk of making the implementation more complex).</div><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Francesc</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div></div></div><div dir="ltr"><div class="gmail_extra"><br><br clear="all"><div><br></div>-- <br><div class="m_8355645822660463935gmail_signature" data-smartmail="gmail_signature">Francesc Alted</div>
</div></div>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
</blockquote></div>