<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi all,<br>
<br>
I originally posted this to the issue tracker
(<a class="moz-txt-link-freetext" href="https://github.com/numpy/numpy/issues/5686">https://github.com/numpy/numpy/issues/5686</a>), and am posting here as
well at the request of charris.<br>
<br>
Currently,
<meta http-equiv="content-type" content="text/html; charset=utf-8">
np.genfromtxt uses a numpy.lib._iotools.NameValidator which mangles
field names by replacing spaces and stripping out certain
non-alphanumeric characters etc.:<br>
<br>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<tt> import numpy as np</tt><tt><br>
</tt><tt> from io import BytesIO</tt><tt><br>
</tt><tt><br>
</tt><tt> s = 'name,name with spaces,2*(x-1)!\n1,2,3\n4,5,6'</tt><tt><br>
</tt><tt> x = np.genfromtxt(BytesIO(s), delimiter=',',
names=True)</tt><tt><br>
</tt><tt> print(repr(x))</tt><tt><br>
</tt><tt> # array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], </tt><tt><br>
</tt><tt> # dtype=[('name', '<f8'), ('name_with_spaces',
'<f8'), ('2x1', '<f8')])</tt><br>
<br>
This behaviour has been the cause of some confusion in the past,
e.g. <a class="moz-txt-link-freetext" href="http://stackoverflow.com/q/29097917/1461210">http://stackoverflow.com/q/29097917/1461210</a>,
<a class="moz-txt-link-freetext" href="http://stackoverflow.com/q/16020137/1461210">http://stackoverflow.com/q/16020137/1461210</a>. Part of the issue is
that it's currently not very well covered by the documentation for
np.genfromtext - at best, it's alluded to in the descriptions for
some of the keyword arguments ('deletechars', 'autostrip',
'replace_space' etc.).<br>
<br>
However, I think the more fundamental problem is that this behaviour
seems to be inconsistent with the rules for naming the fields in
structured arrays. In the example above, all of the original field
names are perfectly legal:<br>
<br>
<tt> names = ['name', 'name with spaces', '2*(x-1)!']</tt><tt><br>
</tt><tt> types = ('f',) * 3</tt><tt><br>
</tt><tt> dtype = zip(names, types)</tt><tt><br>
</tt><tt><br>
</tt><tt> x2 = np.empty(2, dtype=dtype)</tt><tt><br>
</tt><tt> x2[:] = [(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)]</tt><tt><br>
</tt><tt> print(repr(x2))</tt><tt><br>
</tt><tt> # array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], </tt><tt><br>
</tt><tt> # dtype=[('name', '<f4'), ('name with spaces',
'<f4'), ('2*(x-1)!', '<f4')])</tt><tt><br>
</tt><tt> print(x2['2*(x-1)!'])</tt><tt><br>
</tt><tt> # [3. 6.]</tt><br>
<br>
What is the rationale behind the use of NameValidator here? One
possible reason would be to ensure that the field names would also
be legal for an np.recarray. However, this doesn't make sense for
several reasons:<br>
<br>
Firstly, the names above also seem to be legal field names for a
recarray:<br>
<br>
<tt> xr = x2.view(np.recarray)<br>
</tt><tt><tt>print(repr(x2))</tt><tt><br>
</tt># rec.array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], <br>
# dtype=[('name', '<f4'), ('name with spaces',
'<f4'), ('2*(x-1)!', '<f4')])<br>
<br>
</tt>Obviously if the field names aren't valid Python identifiers
then it won't be possible to access them via 'xr.fieldname' syntax,
but dict-style indexing is still fine, e.g. xr['2*(x-1)!']. Also, if
the goal of NameValidator were to ensure that the field names were
always valid Python identifiers then it currently fails at this
anyway, since in my first example, '2x1' is not a valid Python
identifier.<br>
<br>
What is perhaps most confusing is the fact that np.genfromtxt will
even mangle field names that you pass in directly via the 'names'
keyword argument. Suppose you wanted to specify field names that
NameValidator doesn't like. You might try something like this:<tt><br>
</tt><br>
<tt> print(repr(np.genfromtxt(BytesIO(s), delimiter=',',
names=names, skip_header=1)</tt><tt>)</tt><tt>)</tt><tt><br>
</tt><tt> # array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], </tt><tt><br>
</tt><tt> # dtype=[('name', '<f8'), ('name_with_spaces',
'<f8'), ('2x1', '<f8')])<br>
<br>
</tt>Or even this:<tt><br>
<br>
</tt><tt> print(repr(np.genfromtxt(BytesIO(s), delimiter=',',
names=names, skip_header=1,<br>
deletechars=[], replace_space=False,
excludelist=[], autostrip=False))</tt><tt>)</tt><tt><br>
</tt><tt> # array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], </tt><tt><br>
</tt><tt> # dtype=[('name', '<f8'), ('name_with_spaces',
'<f8'), ('2x1', '<f8')])</tt><br>
<br>
Still no luck! As far as I can tell, there is no option in
np.genfromtxt that allows you to preserve field names that don't
conform to NameValidator's seemingly arbitrary rules.<br>
<br>
What should be done about this?
<meta http-equiv="content-type" content="text/html; charset=utf-8">
Personally, I think that either things like spaces and
non-alphanumeric characters should be disallowed in structured array
field names altogether (and my second example should raise an
exception), or np.genfromtxt should leave field names alone by
default.<br>
<br>
It would also be a good idea to raise a SyntaxWarning in a case
where the user creates a recarray containing field names that are
not valid Python identifiers (and are therefore incompatible with
the dot indexing syntax). This is essentially what PyTables does for
non-conforming HDF5 node names:
<a class="moz-txt-link-freetext" href="https://github.com/PyTables/PyTables/blob/13047c897d28b7278cbeab732f12feadbfef3f22/tables/exceptions.py#L285-L294">https://github.com/PyTables/PyTables/blob/13047c897d28b7278cbeab732f12feadbfef3f22/tables/exceptions.py#L285-L294</a>.<br>
<br>
Any thoughts on this?<br>
<br>
Alistair<br>
<br>
</body>
</html>