I've got a module that can send array data to ds9. It screws up on byteswapped data and I'm trying to fix it.
The code sends the data to ds9 via a pipe using arr.tofile(). To send byteswapped data I think I have two choices (given that I don't want to modify the input array):
1) Figure out the byte order and specify a suitable flag to ds9 (it accepts "bigendian" or "littleendian" flags).
This avoids copying the data, so it sounds attractive, but can I do it without relying on "internals"? Numarrays have an "isbyteswapped" method, but all this says is "not native machine order"; it doesn't actually tell me the order.
I need to know if the order is bigendian or littleendian, and I can't find a documented way to get that, just an undocumented attribute _byteorder.
2) If the array is not in native byte order, make a native byte ordered copy before sending it.
The following seems to work: if arr.isbyteswapped(): arr = arr.copy()
but is this the recommended way to get native-order copy?
- Is "copy" guaranteed to return a copy in native byte order? The documentation doesn't say. - I was tempted to use num.array(arr) to make the copy, but the documentation doesn't say what "array" does if the input is a already a numarray array.
As an aside, I tried to use the byteswapped method, but it has a totally different effect:
d
array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], ...type=Float32)
d.isbyteswapped()
1
dcopy = d.copy() d
array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], ...type=Float32)
dcopy.isbyteswapped()
0
dswap = d.byteswapped() dswap
array([[ 1.91063654e-38, 1.91063654e-38, 1.91063654e-38, ..., ...type=Float32)
which I guess means I'd have to byteswap the byteswapped data. Aargh.
-- Russell
El dl 31 de 10 del 2005 a les 16:53 -0800, en/na Russell E. Owen va escriure:
I've got a module that can send array data to ds9. It screws up on byteswapped data and I'm trying to fix it.
I don't exactly know what a ds9 is, but I'm supposing here that it is kind of an exotic computer.
I need to know if the order is bigendian or littleendian, and I can't find a documented way to get that, just an undocumented attribute _byteorder.
Yes, it's undocumented, but as far as I can tell, this works flawlessly in numarray.
- If the array is not in native byte order, make a native byte ordered
copy before sending it.
The following seems to work: if arr.isbyteswapped(): arr = arr.copy()
but is this the recommended way to get native-order copy?
If you can afford doing a memory copy, yes, I think this is a good way to do it.
- Is "copy" guaranteed to return a copy in native byte order? The
documentation doesn't say.
It does:
In [23]:a._byteorder Out[23]:'big'
In [24]:a.copy()._byteorder Out[24]:'little'
In general, copy() will return you a well-behaved (native-byte ordered, non-strided and non-offsetted) array.
- I was tempted to use num.array(arr) to make the copy, but the
documentation doesn't say what "array" does if the input is a already a numarray array.
This also works:
In [25]:array(a)._byteorder Out[25]:'little'
and it is very similar to copy() in performance:
In [34]:t1=Timer("b=a.copy()","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'")
In [35]:t1.repeat(3,10000) Out[35]:[1.0613389015197754, 1.0313379764556885, 1.0303771495819092]
In [36]:t2=Timer("b=numarray.array(a)","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'")
In [37]:t2.repeat(3,10000) Out[37]:[1.2250277996063232, 1.2067301273345947, 1.2336349487304688]
(perhaps copy is just a little bit faster)
As an aside, I tried to use the byteswapped method, but it has a totally different effect:
d
array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], ...type=Float32)
d.isbyteswapped()
1
dcopy = d.copy() d
array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], ...type=Float32)
dcopy.isbyteswapped()
0
dswap = d.byteswapped() dswap
array([[ 1.91063654e-38, 1.91063654e-38, 1.91063654e-38, ..., ...type=Float32)
which I guess means I'd have to byteswap the byteswapped data. Aargh.
This is the intended behaviour. From the docstrings:
In [59]:a.byteswap? [snip...] Docstring: Byteswap data in place, leaving the _byteorder attribute untouched.
So, you already have done a byteswapping, but you still need to tell the object that its byteorder has actually changed (if don't, numarray will convert the data to you local native order before printing it on the flight). Just issue a:
dswap._byteorder = {"little":"big","big":"little"}[dswap._byteorder]
and you are done. In fact, using byteswap() does the conversion in-place, and doesn't need a memory copy. This is actually faster than doing a copy() much of the times:
In [60]:t3=Timer("b=a.byteswap();a._byteorder='little'","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'")
In [61]:t3.repeat(3,10000) Out[61]:[0.23770284652709961, 0.23195695877075195, 0.23380589485168457]
Cheers,
In article 1130838789.2968.23.camel@localhost.localdomain, Francesc Altet faltet@carabos.com wrote:
El dl 31 de 10 del 2005 a les 16:53 -0800, en/na Russell E. Owen va escriure:
I've got a module that can send array data to ds9. It screws up on byteswapped data and I'm trying to fix it.
I don't exactly know what a ds9 is, but I'm supposing here that it is kind of an exotic computer.
It's an image display program.
I need to know if the order is bigendian or littleendian, and I can't find a documented way to get that, just an undocumented attribute byteorder.
Yes, it's undocumented, but as far as I can tell, this works flawlessly in numarray.
I'm sure it works now. I was hoping to know if it would continue working in future versions. It'd be nice to have a documented way to get at the info.
... In general, copy() will return you a well-behaved (native-byte ordered, non-strided and non-offsetted) array.
The manual promises that a copy will be contiguous but I didn't see a promise of native byte order. I have submitted a documentation PR on sourceforge.
... Thanks for the explanation of what byteswapped was doing. That was very helpful.
To use documented interfaces (i.e. not arra._byteorder) and to avoid byteswapping the input array, I think I'm going to be stuck doing something like:
import sys if sys.byteorder == 'big': isBigendian = not arr.isbyteswapped() else: isBigendian = arr.isbyteswapped()
I also submitted a request for a documented direct way to get the info on sourceforge.
Regards,
-- Russell
On Wed, 2 Nov 2005, Russell E. Owen wrote:
To use documented interfaces (i.e. not arra._byteorder) and to avoid byteswapping the input array, I think I'm going to be stuck doing something like:
import sys if sys.byteorder == 'big': isBigendian = not arr.isbyteswapped() else: isBigendian = arr.isbyteswapped()
A simpler version:
isBigendian = arr.isbyteswapped() != numarray.isBigEndian
In article Pine.OSX.4.58.0511030849170.5996@catnip.local, Rick White rlw@stsci.edu wrote:
On Wed, 2 Nov 2005, Russell E. Owen wrote:
To use documented interfaces (i.e. not arra._byteorder) and to avoid byteswapping the input array, I think I'm going to be stuck doing something like:
import sys if sys.byteorder == 'big': isBigendian = not arr.isbyteswapped() else: isBigendian = arr.isbyteswapped()
A simpler version:
isBigendian = arr.isbyteswapped() != numarray.isBigEndian
Cute. isBigEndian doesn't seem to be present in Numeric 23.8. I wonder if it'll be in scipy.core.
Based on stunningly fast responses by Jay T Miller to most of the PRs I submitted to sourceforge:
- The byteswap and byteswapped methods do just affect the data and not the byteorder flag. The doc string for byteswapped said otherwise and that is fixed in the repository. I don't know if the manual has been clarified.
- To toggle the byteorder flag, use the togglebyteorder method (duh, I wish I'd seen that one).
- An official way to figure out if an array is byteswapped is to use the __array_typestr__ method and look at the first character of the returned string. "<" means little-endian, ">" means big-endian. This is apparently also part of the array interface for scipy.core. I sure hope there'll be a more obvious method to call than __array_typestr__ at some point.
-- Russell
Russell E. Owen wrote:
Cute. isBigEndian doesn't seem to be present in Numeric 23.8. I wonder if it'll be in scipy.core.
No, in Numeric the arrays are always assumed to be in machine byte order and so sys.byteorder tells all (Numeric.LittleEndian is a Boolean that is also defined).
In scipy core, arr.flags.swapped tells you if the bytes are in swapped order.
Then, sys.byteorder or scipy.LittleEndian indicates the machine order.
Or you use the array_interface descriptor.
is_big_endian = (arr.dtypestr[0] == '>')
-Travis
participants (4)
-
Francesc Altet -
Rick White -
Russell E. Owen -
Travis Oliphant