[Numpy-discussion] using loadtxt to load a text file in to a numpy array

Chris Barker chris.barker at noaa.gov
Wed Jan 22 15:07:28 EST 2014


On Wed, Jan 22, 2014 at 2:46 AM, Oscar Benjamin
<oscar.j.benjamin at gmail.com>wrote:

> BTW, as much as the fixed-width 'S' dtype doesn't really work for str in
>  Python 3 it's also a poor fit for bytes since it strips trailing nulls:
>
> >>> a = np.array(['a\0s\0', 'qwert'], dtype='S')
> >>> a
> array([b'a\x00s', b'qwert'],
>       dtype='|S5')
> >>> a[0]
> b'a\x00s'


WHOOA!  Good catch, Oscar.

This conversation started with me suggesting that 'S' on py3 should mean
"ascii string" (or latin-1 string).

Then it was pointed out that it was already being used for arbitrary bytes,
and thus could not be changed to mean a string without breaking already
working code.

However,  if 'S' is assigning meaning to null bytes, and doing something
with that, then it is, indeed being treated as an ANSI string (or the old c
string "type", anyway). And any code that is expecting it to be arbitrary
bytes is already broken, and in a way that could result in pretty subtle,
hard to find bugs in the future.

I think we really need a proper bytes dtype (which could be 'S' with the
null byte thing removed), and a proper one-byte-per-character string type.

Though I still don't know the use case for the fixed-length bytes type that
can't be satisfied with the other numeric types, maybe:

In [58]: bytes_15 = np.dtype(('B', 15))


though that doesn't in fact do what I expect:

In [59]: arr = np.zeros((5,), dtype = bytes_15)

In [60]: arr
Out[60]:
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=uint8)

shouldn't I get a shape (5,) array, with each element a compound dtype with
15 bytes in it???

How would I spell that?

By the way, from the docs for dtypes:

http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html

"""
The first character specifies the kind of data and the remaining characters
specify how many bytes of data. The supported kinds are

'b'  Boolean
'i'  (signed) integer
'u'  unsigned integer
'f'  floating-point
'c'  complex-floating point
'S', 'a',  string
'U'  unicode
'V'  raw data (void)
"""
Could we use the 'a' for ascii string? (even though in now mapps directly
to 'S')

And by the way, the docs clearly say "string" there -- not bytes, so at the
very least we need to update the docs...

-Chris


Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140122/8145c83f/attachment.html>


More information about the NumPy-Discussion mailing list