Mailman 3 June 2011 - NumPy-Discussion

what python module to handle csv?
by Chao YUE 22 Jun '11

22 Jun '11

Dear all pythoners, what do you use python module to handle csv file (for reading we can use numpy.genfromtxt)? is there anyone we can do with csv file very convinient as that in R? can numpy.genfromtxt be used as writing? (I didn't try this yet because on our server we have only numpy 1.0.1...). This really make me struggling since csv is a very important interface file (I think so...). Thanks a lot, Sincerely, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 77 30; Fax:01.69.08.77.16 ************************************************************************************

3 3

ANN: Numpy 1.6.1 release candidate 1
by Ralf Gommers 21 Jun '11

21 Jun '11

Hi, I am pleased to announce the availability of the first release candidate of NumPy 1.6.1. This is a bugfix release, list of fixed bugs: #1834 einsum fails for specific shapes #1837 einsum throws nan or freezes python for specific array shapes #1838 object <-> structured type arrays regression #1851 regression for SWIG based code in 1.6.0 #1863 Buggy results when operating on array copied with astype() If no problems are reported, the final release will be in one week. Sources and binaries can be found at<http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc1/> https://sourceforge.net/projects/numpy/files/NumPy/1.6.1rc1/ Enjoy, Ralf

9 33

ANN: ETS 4.0 released
by Ilan Schnell 21 Jun '11

21 Jun '11

Hello, I am happy to announce the release of ETS 4.0. This is the first major release of the Enthought Tool Suite in almost three years. This release removes the 'enthought' namespace from all projects. For example: from enthought.traits.api import HasTraits is now simply:: from traits.api import HasTraits For backwards compatibility, a proxy package 'etsproxy' has been added, which should permit existing code to work. For convenience this package also contains a refactor tool 'ets3to4' to convert projects to the new namespace (so that they don't rely on the 'etsproxy' package). If you want to download the source code of all ETS projects, you can download http://www.enthought.com/repo/ets/ALLETS-4.0.0.tar (41MB). The projects themselves are now hosted on: https://github.com/enthought We understand that the namespace refactor (which prompted this major release in the first place) is a big change, and even though we have tested examples and some of our own code against this ETS version, we expect there to be little glitches. We are therefore already planning a 4.0.1 bug-fix release in about 2-3 weeks. We are looking forward to your feedback (the development mailing list is enthought-dev(a)enthought.com), and hope you enjoy ETS 4. - Ilan

1 0

fast SSD
by Alex Flint 21 Jun '11

21 Jun '11

Is there a fast way to compute an array of sum-of-squared-differences between a (small) K x K array and all K x K sub-arrays of a larger array? (i.e. each element x,y in the output array is the SSD between the small array and the sub-array (x:x+K, y:y+K) My current implementation loops over each sub-array and computes the SSD with something like ((A-B)**2).sum(). Cheers, Alex

3 2

(cumsum, broadcast) in (numexpr, weave)
by srean 21 Jun '11

21 Jun '11

Hi All, is there a fast way to do cumsum with numexpr ? I could not find it, but the functions available in numexpr does not seem to be exhaustively documented, so it is possible that I missed it. Do not know if 'sum' takes special arguments that can be used. To try another track, does numexpr operators have something like the 'out' parameter for ufuncs ? If it is so, one could perhaps use add( a[0:-1], a[1,:], out = a[1,:) provided it is possible to preserve the sequential semantics. Another option is to use weave which does have cumsum. However my code requires expressions which implement broadcast. That leads to my next question, does repeat or concat return a copy or a view. If they avoid copying, I could perhaps use repeat to simulate efficient broadcasting. Or will it make a copy of that array anyway ?. I would ideally like to use numexpr because I make heavy use of transcendental functions and was hoping to exploit the VML library. Thanks for the help -- srean

1 1

poor performance of sum with sub-machine-word integer types
by Zachary Pincus 21 Jun '11

21 Jun '11

Hello all, As a result of the "fast greyscale conversion" thread, I noticed an anomaly with numpy.ndararray.sum(): summing along certain axes is much slower with sum() than versus doing it explicitly, but only with integer dtypes and when the size of the dtype is less than the machine word. I checked in 32-bit and 64-bit modes and in both cases only once the dtype got as large as that did the speed difference go away. See below... Is this something to do with numpy or something inexorable about machine / memory architecture? Zach Timings -- 64-bit mode: ---------------------- In [2]: i = numpy.ones((1024,1024,4), numpy.int8) In [3]: timeit i.sum(axis=-1) 10 loops, best of 3: 131 ms per loop In [4]: timeit i[...,0]+i[...,1]+i[...,2]+i[...,3] 100 loops, best of 3: 2.57 ms per loop In [5]: i = numpy.ones((1024,1024,4), numpy.int16) In [6]: timeit i.sum(axis=-1) 10 loops, best of 3: 131 ms per loop In [7]: timeit i[...,0]+i[...,1]+i[...,2]+i[...,3] 100 loops, best of 3: 4.75 ms per loop In [8]: i = numpy.ones((1024,1024,4), numpy.int32) In [9]: timeit i.sum(axis=-1) 10 loops, best of 3: 131 ms per loop In [10]: timeit i[...,0]+i[...,1]+i[...,2]+i[...,3] 100 loops, best of 3: 6.37 ms per loop In [11]: i = numpy.ones((1024,1024,4), numpy.int64) In [12]: timeit i.sum(axis=-1) 100 loops, best of 3: 16.6 ms per loop In [13]: timeit i[...,0]+i[...,1]+i[...,2]+i[...,3] 100 loops, best of 3: 15.1 ms per loop Timings -- 32-bit mode: ---------------------- In [2]: i = numpy.ones((1024,1024,4), numpy.int8) In [3]: timeit i.sum(axis=-1) 10 loops, best of 3: 138 ms per loop In [4]: timeit i[...,0]+i[...,1]+i[...,2]+i[...,3] 100 loops, best of 3: 3.68 ms per loop In [5]: i = numpy.ones((1024,1024,4), numpy.int16) In [6]: timeit i.sum(axis=-1) 10 loops, best of 3: 140 ms per loop In [7]: timeit i[...,0]+i[...,1]+i[...,2]+i[...,3] 100 loops, best of 3: 4.17 ms per loop In [8]: i = numpy.ones((1024,1024,4), numpy.int32) In [9]: timeit i.sum(axis=-1) 10 loops, best of 3: 22.4 ms per loop In [10]: timeit i[...,0]+i[...,1]+i[...,2]+i[...,3] 100 loops, best of 3: 12.2 ms per loop In [11]: i = numpy.ones((1024,1024,4), numpy.int64) In [12]: timeit i.sum(axis=-1) 10 loops, best of 3: 29.2 ms per loop In [13]: timeit i[...,0]+i[...,1]+i[...,2]+i[...,3] 10 loops, best of 3: 23.8 ms per loop

3 4

Controlling endianness of ndarray.tofile()
by Ben Forbes 21 Jun '11

21 Jun '11

Hi, On my system (Intel Xeon, Windows 7 64-bit), ndarray.tofile() outputs in little-endian. This is a bit inconvenient, since everything else I do is in big-endian. Unfortunately, scipy.io.write_arrray() is deprecated, and I can't find any other routines that write pure raw binary. Are there any other options, or perhaps could tofile() be modified to allow control over endianness? Cheers, Ben -- Benjamin D. Forbes School of Physics The University of Melbourne Parkville, VIC 3010, Australia

2 2

numpy submodules missing in intersphinx objects.inv?
by Skipper Seabold 20 Jun '11

20 Jun '11

I was just trying to link to the numpy.testing module but can't using intersphinx. Does anyone know why certain submodules aren't included in objects.inv? It looks as though it has something to do either with having a reference at the top of the rst file (so you can :ref: link to it) or having that module autodoc'd (then you can :mod: link to it), but I can't find any actual documentation about this actually being the case. This is the relevant ticket http://projects.scipy.org/numpy/ticket/1432 Skipper

1 0

Re: [Numpy-discussion] npyio -> gzip 271 Error -3 while decompressing ?
by Derek Homeier 20 Jun '11

20 Jun '11

Moin Denis, On 20 Jun 2011, at 19:04, denis wrote: > a separate question, have you run genfromtxt( "xx.csv.gz" ) lately ? I haven't, and I was not particularly involved with it before this patch, so this would possibly be better addressed to the list. > On on .../scikits.learn-0.8/scikits/learn/datasets/data.digits.csv.gz > numpy 1.6.0, py 2.6 mac I get > > X = np.genfromtxt( filename, delimiter="," ) > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ > python2.6/site-packages/numpy/lib/npyio.py", line 1271, in genfromtxt > first_line = fhd.next() > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ > python2.6/gzip.py", line 438, in next > line = self.readline() > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ > python2.6/gzip.py", line 393, in readline > c = self.read(readsize) > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ > python2.6/gzip.py", line 219, in read > self._read(readsize) > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ > python2.6/gzip.py", line 271, in _read > uncompress = self.decompress.decompress(buf) > zlib.error: Error -3 while decompressing: invalid distance too far > back > > It would be nice to fix this too, if it hasn't been already. > Btw the file gunzips fine. I could reproduce that error for the gzip'ed csv files in that directory; it can be isolated to the underlying gzip call above - fhd = gzip.open('digits.csv.gz', 'rbU'); fhd.next() produces the same error for these files with all python2.x versions on my Mac, but not with python3.x. Also only with the 'U' mode specified, yet the same mode is parsing other .gz files just fine. I could not really track down what the 'U' flag is doing in gzip.py, but I assume it is specifying some kind of unbuffered read. Also it's a mystery to me what is different in those files that would trigger the error. I even read them in with loadtxt() and wrote them back using constant line width and/or spaces as separators, still producing the same exception. The obvious place to fix this (or work around a bug in python2's gzip.py, whatever), would be changing the open command in genfromtxt fhd = iter(np.lib._datasource.open(fname, 'rbU')) to omit the 'U' at least with python2. Alternatively one could do a test read and catch the exception, to then re-open the file with mode 'rb'... Pierre, if you are reading this, can you comment how important the 'U' is for performance considerations or such? HTH, Derek

1 0

fast grayscale conversion
by Alex Flint 20 Jun '11

20 Jun '11

At the moment I'm using numpy.dot to convert a WxHx3 RGB image to a grayscale image: src_mono = np.dot(src_rgb.astype(np.float), np.ones(3)/3.); This seems quite slow though (several seconds for a 3 megapixel image) - is there a more specialized routine better suited to this? Cheers, Alex

4 4