[Numpy-discussion] converting list of int16 values to bitmask and back to

Nissim Derdiger NissimD at elspec-ltd.com
Mon Oct 16 08:23:19 EDT 2017


Thomas, Thanks for your answers!
Just for the closer of this issue, here are the 2 np solutions that I used with Thomas help:
(both are MUCH faster than my original solution, func2 slightly more)

def func1():
    Endian = '<I' # Little endian
    ParameterFormat = 'f' # float32
    RawDataList = np.array([17252, 26334, 16141, 58057,17252, 15478, 16144, 43257]) # list of int32 registers
	w1 = np.array(RawDataList, dtype='<u2').reshape((-1, 2))
    w2 = w1[:, ::-1].ravel()
    w3 = np.frombuffer(w2.data,dtype='f4')
      print(w3)

def func2():
    Endian = '<I' # Little endian
    ParameterFormat = 'f' # float32
    RawDataList = np.array([17252, 26334, 16141, 58057,17252, 15478, 16144, 43257]) # list of int32 registers
	x1 = np.array(RawDataList, dtype='>u2')
    x2 = np.frombuffer(x1.data, dtype='>u4')
    x3 = np.array(x2, dtype='<u4')
    x4 = np.frombuffer(x3.data, dtype='f4')
	print(x4)

Nissim.

-----Original Message-----
From: NumPy-Discussion [mailto:numpy-discussion-bounces+nissimd=elspec-ltd.com at python.org] On Behalf Of numpy-discussion-request at python.org
Sent: Monday, October 09, 2017 7:00 PM
To: numpy-discussion at python.org
Subject: NumPy-Discussion Digest, Vol 133, Issue 7

Send NumPy-Discussion mailing list submissions to
	numpy-discussion at python.org

To subscribe or unsubscribe via the World Wide Web, visit
	https://mail.python.org/mailman/listinfo/numpy-discussion
or, via email, send a message with subject or body 'help' to
	numpy-discussion-request at python.org

You can reach the person managing the list at
	numpy-discussion-owner at python.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of NumPy-Discussion digest..."


Today's Topics:

   1. Re: converting list of int16 values to bitmask and back to
      bitmask and back to list of int32\float values (Thomas Jollans)
   2. Re: converting list of int16 values to bitmask and back to
      bitmask and back to list of int32\float values (Thomas Jollans)


----------------------------------------------------------------------

Message: 1
Date: Sun, 8 Oct 2017 22:50:02 +0200
From: Thomas Jollans <tjol at tjol.eu>
To: numpy-discussion at python.org
Subject: Re: [Numpy-discussion] converting list of int16 values to
	bitmask and back to bitmask and back to list of int32\float values
Message-ID: <78197fb1-3f0f-06a6-1385-a310fd556d38 at tjol.eu>
Content-Type: text/plain; charset=utf-8

On 08/10/17 09:12, Nissim Derdiger wrote:
> Hi again,
> I realize that my question was not clear enough, so I've refined it 
> into one runnable function (attached below) My question is basically - 
> is there a way to perform the same operation, but faster using NumPy (or even just by using Python better..) Thanks again and sorry for the unclearness..
> Nissim.
>
> import struct
>
> def Convert():
>     Endian = '<I' # Big endian
< is little endian. Make sure you're getting out the right values!
>     ParameterFormat = 'f' # float32
>     RawDataList = [17252, 26334, 16141, 58057,17252, 15478, 16144, 43257] # list of int32 registers
>     NumOfParametersInRawData = int(len(RawDataList)/2)
>     Result = []
>     for i in range(NumOfParametersInRawData):
Iterating over indices is not very Pythonic, and there's usually a better way. In this case: for int1, int2 in zip(RawDataList[::2],
RawDataList[1::2])

>         # pack every 2 registers, take only the first 2 bytes from each one, change their endianess than unpack them back to the Parameter format
>         
> Result.append((struct.unpack(ParameterFormat,(struct.pack(Endian,RawDa
> taList[(i*2)+1])[0:2] + struct.pack('<I',RawDataList[i*2])[0:2])))[0])

You can do this a little more elegantly (and probably faster) with struct by putting it in a list comprehension:

[struct.unpack('f', struct.pack('<HH', i1 & 0xffff, i2 & 0xffff))[0] for i1, i2 in zip(raw_data[::2], raw_data[1::2])]

Numpy can also do it. You can get your array of little-endian shorts with


le_shorts = np.array(raw_data, dtype='<u2')

and then reinterpret the bytes backing it as float32 with np.frombuffer:

np.frombuffer(le_shorts.data, dtype='f4')

For small lists like the one in your example, the two approaches are equally fast. For long ones, numpy is much faster:

In [82]: raw_data Out[82]: [17252, 26334, 16141, 58057, 17252, 15478, 16144, 43257] In [83]: raw_data2 = np.random.randint(0, 2**32, size=10**6, dtype='u4') # 1 million random integers In [84]: %timeit np.frombuffer(np.array(raw_data, dtype='<u2').data, dtype='f4') 6.45 ?s ? 60.9 ns per loop (mean ? std. dev. of 7 runs, 100000 loops each) In
[85]: %timeit np.frombuffer(np.array(raw_data2, dtype='<u2').data,
dtype='f4') 854 ?s ? 37.3 ?s per loop (mean ? std. dev. of 7 runs, 1000 loops each) In [86]: %timeit [struct.unpack('f', struct.pack('<HH', i1 & 0xffff, i2 & 0xffff))[0] for i1, i2 in zip(raw_data[::2], raw_data[1::2])] 4.87 ?s ? 17.3 ns per loop (mean ? std. dev. of 7 runs,
100000 loops each) In [87]: %timeit [struct.unpack('f', struct.pack('<HH', i1 & 0xffff, i2 & 0xffff))[0] for i1, i2 in zip(raw_data2[::2], raw_data2[1::2])] 3.6 s ? 9.78 ms per loop (mean ?
std. dev. of 7 runs, 1 loop each)

-- Thomas



------------------------------

Message: 2
Date: Sun, 8 Oct 2017 22:58:41 +0200
From: Thomas Jollans <tjol at tjol.eu>
To: numpy-discussion at python.org
Subject: Re: [Numpy-discussion] converting list of int16 values to
	bitmask and back to bitmask and back to list of int32\float values
Message-ID: <a1ac7020-857a-366f-43cb-6ea2b4f248e8 at tjol.eu>
Content-Type: text/plain; charset=utf-8

On 08/10/17 22:50, Thomas Jollans wrote:
> On 08/10/17 09:12, Nissim Derdiger wrote:
>> Hi again,
>> I realize that my question was not clear enough, so I've refined it 
>> into one runnable function (attached below) My question is basically 
>> - is there a way to perform the same operation, but faster using NumPy (or even just by using Python better..) Thanks again and sorry for the unclearness..
>> Nissim.
>>
>> import struct
>>
>> def Convert():
>>     Endian = '<I' # Big endian
> < is little endian. Make sure you're getting out the right values!
>>     ParameterFormat = 'f' # float32
>>     RawDataList = [17252, 26334, 16141, 58057,17252, 15478, 16144, 43257] # list of int32 registers
>>     NumOfParametersInRawData = int(len(RawDataList)/2)
>>     Result = []
>>     for i in range(NumOfParametersInRawData):
> Iterating over indices is not very Pythonic, and there's usually a 
> better way. In this case: for int1, int2 in zip(RawDataList[::2],
> RawDataList[1::2])
> 
>>         # pack every 2 registers, take only the first 2 bytes from each one, change their endianess than unpack them back to the Parameter format
>>         
>> Result.append((struct.unpack(ParameterFormat,(struct.pack(Endian,RawD
>> ataList[(i*2)+1])[0:2] + 
>> struct.pack('<I',RawDataList[i*2])[0:2])))[0])
> 
> You can do this a little more elegantly (and probably faster) with 
> struct by putting it in a list comprehension:
> 
> [struct.unpack('f', struct.pack('<HH', i1 & 0xffff, i2 & 0xffff))[0] 
> for i1, i2 in zip(raw_data[::2], raw_data[1::2])]
> 
> Numpy can also do it. You can get your array of little-endian shorts 
> with
> 
> 
> le_shorts = np.array(raw_data, dtype='<u2')
> 
> and then reinterpret the bytes backing it as float32 with np.frombuffer:
> 
> np.frombuffer(le_shorts.data, dtype='f4')
> 
> For small lists like the one in your example, the two approaches are 
> equally fast. For long ones, numpy is much faster:

*sigh* let's try that again:

In [82]: raw_data
Out[82]: [17252, 26334, 16141, 58057, 17252, 15478, 16144, 43257]

In [83]: raw_data2 = np.random.randint(0, 2**32, size=10**6, dtype='u4')

In [84]: %timeit np.frombuffer(np.array(raw_data, dtype='<u2').data,
dtype='f4')
6.45 ?s ? 60.9 ns per loop (mean ? std. dev. of 7 runs, 100000 loops each)

In [85]: %timeit np.frombuffer(np.array(raw_data2, dtype='<u2').data,
dtype='f4')
854 ?s ? 37.3 ?s per loop (mean ? std. dev. of 7 runs, 1000 loops each)

In [86]: %timeit [struct.unpack('f', struct.pack('<HH', i1 & 0xffff, i2 & 0xffff))[0] for i1, i2 in zip(raw_data[::2], raw_data[1::2])]
4.87 ?s ? 17.3 ns per loop (mean ? std. dev. of 7 runs, 100000 loops each)

In [87]: %timeit [struct.unpack('f', struct.pack('<HH', i1 & 0xffff, i2 & 0xffff))[0] for i1, i2 in zip(raw_data2[::2], raw_data2[1::2])]
3.6 s ? 9.78 ms per loop (mean ? std. dev. of 7 runs, 1 loop each)

> 
> In [82]: raw_data Out[82]: [17252, 26334, 16141, 58057, 17252, 15478, 
> 16144, 43257] In [83]: raw_data2 = np.random.randint(0, 2**32, 
> size=10**6, dtype='u4') # 1 million random integers In [84]: %timeit 
> np.frombuffer(np.array(raw_data, dtype='<u2').data, dtype='f4') 6.45 
> ?s ? 60.9 ns per loop (mean ? std. dev. of 7 runs, 100000 loops each) 
> In
> [85]: %timeit np.frombuffer(np.array(raw_data2, dtype='<u2').data,
> dtype='f4') 854 ?s ? 37.3 ?s per loop (mean ? std. dev. of 7 runs, 
> 1000 loops each) In [86]: %timeit [struct.unpack('f', 
> struct.pack('<HH', i1 & 0xffff, i2 & 0xffff))[0] for i1, i2 in 
> zip(raw_data[::2], raw_data[1::2])] 4.87 ?s ? 17.3 ns per loop (mean ? 
> std. dev. of 7 runs,
> 100000 loops each) In [87]: %timeit [struct.unpack('f', 
> struct.pack('<HH', i1 & 0xffff, i2 & 0xffff))[0] for i1, i2 in 
> zip(raw_data2[::2], raw_data2[1::2])] 3.6 s ? 9.78 ms per loop (mean ?
> std. dev. of 7 runs, 1 loop each)
> 
> -- Thomas
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
> 



------------------------------

Subject: Digest Footer

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion at python.org
https://mail.python.org/mailman/listinfo/numpy-discussion


------------------------------

End of NumPy-Discussion Digest, Vol 133, Issue 7
************************************************


More information about the NumPy-Discussion mailing list