[Numpy-discussion] Question about dtype

[Nathaniel Smith]

On 12 Dec 2014 23:22, "Valentin Haenel" <valentin at haenel.co> wrote:
>
> Dear Nathaniel,
>
> thanks very much for your response.
>
> * Nathaniel Smith <njs at pobox.com> [2014-12-11]:
> > On Wed, Dec 10, 2014 at 8:26 PM, Valentin Haenel <valentin at haenel.co>
wrote:
> > > I am using numpy version 1.9.0 and Python 2.7.9 and have a question
> > > about the dtype:
> > >
> > > In [14]: np.dtype("<f8")
> > > Out[14]: dtype('float64')
> > >
> > > In [15]: np.dtype(u"<f8")
> > > Out[15]: dtype('float64')
> > >
> > > In [16]: np.dtype([("<f8", "<f8")])
> > > Out[16]: dtype([('<f8', '<f8')])
> > >
> > > So far so good. Now what happens if I use unicode?
> > >
> > > In [17]: np.dtype([(u"<f8", "<f8")])
> > >
---------------------------------------------------------------------------
> > > TypeError                                 Traceback (most recent call
> > > last)
> > > <ipython-input-17-ce004acab7f5> in <module>()
> > > ----> 1 np.dtype([(u"<f8", "<f8")])
> > >
> > > TypeError: data type not understood
> >
> > Yep, looks like a bug to me. (I guess this is particularly relevant
> > when __future__.unicode_literals is in effect.)
>
> If you could point me in the approximate direction, I'll give it a
> shot. (my best guess would be numpy/core/_internal.py)

I'm not sure and I'm on my phone - maybe someone else will pipe up. Or you
could just start grepping, I guess - that's all I'd be doing :-).

> > > Also, it really does need to be a tuple?
> > >
> > > In [18]: np.dtype([["<f8", "<f8"]])
> > >
---------------------------------------------------------------------------
> > > TypeError                                 Traceback (most recent call
> > > last)
> > > <ipython-input-18-c82761d7306d> in <module>()
> > > ----> 1 np.dtype([["<f8", "<f8"]])
> > >
> > > TypeError: data type not understood
> >
> > Lists and tuples are both valid inputs to np.dtype, but they're
> > interpreted differently -- the problem here isn't that you used a
> > list, it's that if you use a list then numpy expects different
> > contents. See:
> >     http://docs.scipy.org/doc/numpy/user/basics.rec.html
>
> Ok, let me ask a different question---to give you some perspective of what
> I actually need. I'm trying to roundtrip a dtype through JSON and I'm
> having some trouble since the tuples are converted into lists( hence my
> example above.) Those then can't be converted into a dtype instance
> anymore... The utf-8 thing above is also part of the Problem since JSON
> gives back unicode objects for strings, but should
> probably be fixed upstream (i.e. in numpy).
>
> Here is how far I get:
>
>   In [2]: import json
>
> The following goes in
>
>   In [3]: dt = [("<f8", "<f8")]
>
>   In [4]: inst = np.dtype(dt)
>
>   In [5]: inst
>   Out[5]: dtype([('<f8', '<f8')])
>
>   In [6]: jd = json.dumps(dt)
>
>   In [7]: dtud = json.loads(jd)
>
> And this is what comes back out:
>
>  In [8]: dtud
>  Out[8]: [[u'<f8', u'<f8']]
>
>  In [9]: jd
>  Out[9]: '[["<f8", "<f8"]]'
>
>  In [10]: inst.descr
>  Out[10]: [('<f8', '<f8')]
>
>  In [11]: np.dtype(dtud)
>
 ---------------------------------------------------------------------------
>  TypeError                                 Traceback (most recent call
>  last)
>  <ipython-input-11-85f434de7ebe> in <module>()
>  ----> 1 np.dtype(dtud)
>
>  TypeError: data type not understood
>
>
> Maybe there is a better way to roundtrip a dtype through JSON? Perhaps
> this is a known and solved problem?

Ah, so your question is about how to serialize dtypes.

The simplest approach would be to use pickle and shove the resulting string
into your json. However, this is very dangerous if you need to process
untrusted files, because if I can convince you to unpickle an arbitrary
string, then I can run arbitrary code on your computer.

I believe .npy file format has a safe method for (un)serializing drypes.
I'd look up what it does.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20141213/94f35a41/attachment.html>