Mailman 3 June 2011 - NumPy-Discussion

Multiply along axis
by Robert Elsner 29 Jun '11

29 Jun '11

Hello everyone, I would like to solve the following problem (preferably without reshaping / flipping the array a). Assume I have a vector v of length x and an n-dimensional array a where one dimension has length x as well. Now I would like to multiply the vector v along a given axis of a. Some example code a = np.random.random((2,3)) x = np.zeros(2) a * x # Fails because not broadcastable So how do I multiply x with the columns of a so that for each column j a[:,j] = a[:,j] * x without using a loop. Is there some (fast) fast way to accomplish that with numpy/scipy? Thanks for your help Robert Elsner

3 4

Missing/accumulating data
by Joe Harrington 29 Jun '11

29 Jun '11

As with Travis, I have not had time to wade through the 150+ messages on masked arrays, but I'd like to raise a concept I've mentioned in the past that would enable a broader use if done slightly differently. That is, the "masked array" problem is a subset of this more-general problem. Please respond on this thread if you want me to see the replies. The problem is of data accumulation and averaging, and it comes up all the time in astronomy, medical imaging, and other fields that handle stacks of images. Say that I observe a planet, taking images at, say, 30 Hz for several hours, as with many video cameras. I want to register or map those images to a common coordinate system, stack them, and average them. I must recognize: 1. transient there are bad pixels due to cosmic ray hits on the CCD array 2. some pixels are permanently dead and always produce bad data 3. some data in the image is not part of the planet 4. some images have too much atmospheric blurring to be used 5. sometimes moons of planets like Jupiter pass in front 6. etc. For each original image: Make a mask where 1 is good and 0 is bad. This trims both bad pixels and pixels that are not part of the planet. Evaluate the images and zero the entire image (or regions of an image) if quality is low (or there is an obstruction). Shift (or map, in the case of lon-lat mapping) the image AND THE MASK to a common coordinate grid. Append the image and the mask to the ends of the stacks of prior images and masks. For each x and y position in the image/mask stacks: Perform some averaging of all pixels in the stack that are not masked, or otherwise use the mask info to decide how to use the image info. The most common of these averages is to multiply the stack of images by the stack of masks, zeroing the bad data, then sum both images and masks down the third axis. This produces an accumulator image and a count image, where the count image says how many data points went into the accumulator at each pixel position. Divide the accumulator by the count. This produces an image with no missing data and the highest SNR per pixel possible. Two things differ from the current ma package: the sense of TRUE/FALSE and the ability to have numbers other than 0 or 1. For example, if data quality were part of the assessment, each pixel could get a data quality number and the accumulation would only include pixels with better than a certain number, or there could be weighted averaging. Currently, we just implement this as described, without using the masked array capability at all. However, it's cumbersome to hand around pairs of objects, and making a new object to handle them together means breaking it apart when calling standard routines that don't handle masks. There are probably some very fast ways to do interesting kinds of averages that we haven't explored but would if this kind of capability were in the infrastructure. So, I hope that if some kind of masking goes into the array object, it allows TRUE=good, FALSE=bad, and at least an integer count if not a floating quality number. One bit is not enough. Of course, these could be options. Certainly adding 7 or 31 more bits to the mask will make space problems for some, as would flipping the sense of the mask. However, speed matters. Some of those doing this kind of processing are handling hours of staring video data, which is 1e6 frames a night from each camera. --jh--

3 2

pull request for testing/review: ufunc 'where=' parameter
by Mark Wiebe 29 Jun '11

29 Jun '11

https://github.com/numpy/numpy/pull/99 In [24]: a = np.ones((1000,1000)) In [25]: b = np.ones((1000,1000)) In [26]: c = np.zeros((1000, 1000)) In [27]: m = np.random.rand(1000,1000) > 0.5 In [28]: timeit c[m] = a[m] + b[m] 1 loops, best of 3: 246 ms per loop In [29]: timeit np.add(a, b, out=c, where=m) 10 loops, best of 3: 20.3 ms per loop -Mark

1 0

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 57, Issue 155
by RadimRehurek 28 Jun '11

28 Jun '11

Hello, > From: santhu kumar <mesanthu(a)gmail.com> > > After looking at the suggestions I have checked my matrix again and found > that the matrix was wrong.(Has some NAN's and other values which are wrong). > After correcting it it works fine. Not to beat a dead horse, but here's an (ancient) related numpy ticket: http://projects.scipy.org/numpy/ticket/706 It may not be your case here, but this bug bit me in the past as well (OS X with vecLib). For what it's worth, my solution was what Lou suggested (squaring the input). Best, Radim

1 0

Re: [Numpy-discussion] SVD does not converge
by santhu kumar 28 Jun '11

28 Jun '11

Hello After looking at the suggestions I have checked my matrix again and found that the matrix was wrong.(Has some NAN's and other values which are wrong). After correcting it it works fine. Sorry for the mistake and thanks for your time, Santhosh On Tue, Jun 28, 2011 at 11:43 AM, <numpy-discussion-request(a)scipy.org>wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. SVD does not converge (santhu kumar) > 2. Re: SVD does not converge (Charles R Harris) > 3. Re: SVD does not converge (Sebastian Berg) > 4. Re: missing data discussion round 2 (Charles R Harris) > 5. Re: SVD does not converge (Lou Pecora) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 28 Jun 2011 10:56:48 -0500 > From: santhu kumar <mesanthu(a)gmail.com> > Subject: [Numpy-discussion] SVD does not converge > To: numpy-discussion(a)scipy.org > Message-ID: <BANLkTinYtcT_AiA1rJ-s_kx5itpe3W2sqw(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hello, > > I have a 380X5 matrix and when I am calculating pseudo-inverse of the > matrix > using pinv(numpy.linalg) I get the following error message: > > raise LinAlgError, 'SVD did not converge' > numpy.linalg.linalg.LinAlgError: SVD did not converge > > I have looked in the list that it is a recurring issue but I was unable to > find any solution. Can somebody please guide me on how to fix that issue? > > Thanks > Santhosh >

1 0

SVD does not converge
by santhu kumar 28 Jun '11

28 Jun '11

Hello, I have a 380X5 matrix and when I am calculating pseudo-inverse of the matrix using pinv(numpy.linalg) I get the following error message: raise LinAlgError, 'SVD did not converge' numpy.linalg.linalg.LinAlgError: SVD did not converge I have looked in the list that it is a recurring issue but I was unable to find any solution. Can somebody please guide me on how to fix that issue? Thanks Santhosh

5 5

Recommndations for an easy GUI
by Nadav Horesh 28 Jun '11

28 Jun '11

I have an application which generates and displays RGB images as rate of several frames/seconds (5-15). Currently I use Tkinter+PIL, but I have a problem that it slows down the rate significantly. I am looking for a fast and easy alternative. Platform: Linux I prefer tools that would work also with python3 I looked at the following: 1. matplotlib's imshow: Easy but too slow. 2. pyqwt: Easy, fast, does not support rgb images (only grayscale) 3. pygame: Nice but lacking widgets like buttons etc. 4. PyQT: Can do, I would like something simpler (I'll rather focus on computations, not GUI) Nadav.

3 3

Re: [Numpy-discussion] data type specification when using numpy.genfromtxt
by Chao YUE 28 Jun '11

28 Jun '11

Hi Derek! I tried with the lastest version of python(x,y) package with numpy version of 1.6.0. I gave the data to you with reduced columns (10 column) and rows. b=np.genfromtxt('99Burn2003all_new.csv',delimiter=';',names=True,usecols=tuple(range(10)),dtype=['S10'] + [ float for n in range(9)]) works. if you change usecols=tuple(range(10)) to usecols=range(10), it still works. b=np.genfromtxt('99Burn2003all_new.csv',delimiter=';',names=True,dtype=None) works. but b=np.genfromtxt('99Burn2003all_new.csv',delimiter=';',names=True,dtype=['S10'] + [ float for n in range(9)]) didn't work. I use Python(x,y)-2.6.6.1 with numpy version as 1.6.0, I use windows 32-bit system. Please don't spend too much time on this if it's not a potential problem. the final thing is, when I try to do this (I want to try the missing_values in numpy 1.6.0), it gives error: In [33]: import StringIO as StringIO In [34]: data = "1, 2, 3\n4, 5, 6" In [35]: np.genfromtxt(StringIO(data), delimiter=",",dtype="int,int,int",missing_values=2) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) D:\data\LaThuile_ancillary\Jim_Randerson_data\<ipython console> in <module>() TypeError: 'module' object is not callable I think it must be some problem of my own python configuration? Much thanks again, cheers, Chao 2011/6/27 Derek Homeier <derek(a)astro.physik.uni-goettingen.de> > Hi Chao, > > this seems to have become quite a number of different issues! > But let's make sure I understand what's going on... > > > Thanks very much for your quick reply. I make a short summary of what > I've tried. Actually the ['S10'] + [ float for n in range(48) ] only works > when you explicitly specify the columns to be read, and genfromtxt cannot > automatically determine the type if you don't specify the type.... > > > > > In [164]: > b=np.genfromtxt('99Burn2003all.csv',delimiter=';',names=True,usecols=tuple(range(49)),dtype=['S10'] > + [ float for n in range(48)]) > ... > > But if I use the following, it gives error: > > > > In [171]: > b=np.genfromtxt('99Burn2003all.csv',delimiter=';',names=True,dtype=['S > > 10'] + [ float for n in range(48)]) > > > --------------------------------------------------------------------------- > > ValueError Traceback (most recent call > last) > > > And the above (without the usecols) did work if you explicitly typed > dtype=('S10', float, float....)? That by itself would be quite weird, > because the two should be completely equivalent. > What happens if you cast the generated list to a tuple - > dtype=tuple(['S10'] + [ float for n in range(48)])? > If you are using a recent numpy version (1.6.0 or 1.6.1rc1), could you > please file a bug report with complete machine info etc.? But I suspect this > might be an older version, you should also be able to simply use > 'usecols=range(49)' (without the tuple()). Either way, I cannot reproduce > this behaviour with the current numpy version. > > > If I don't specify the dtype, it will not recognize the type of the first > column (it displays as nan): > > > > In [172]: > b=np.genfromtxt('99Burn2003all.csv',delimiter=';',names=True,usecols=(0,1,2)) > > > > In [173]: b > > Out[173]: > > array([(nan, -999.0, -1.028), (nan, -999.0, -0.40899999999999997), > > (nan, -999.0, 0.16700000000000001), ..., (nan, -999.0, -999.0), > > (nan, -999.0, -999.0), (nan, -999.0, -999.0)], > > dtype=[('TIMESTAMP', '<f8'), ('CO2_flux', '<f8'), ('Net_radiation', > '<f8') > > ]) > > > You _do_ have to specify 'dtype=None', since the default is 'dtype=float', > as I have remarked in my previous mail. If this does not work, it could be a > matter of the numpy version gain - there were a number of type conversion > issues fixed between 1.5.1 and 1.6.0. > > > > Then the final question is, actually the '-999.0' in the data is missing > value, but I cannot display it as 'nan' by specifying the missing_values as > '-999.0': > > but either I set the missing_values as -999.0 or using a dictionary, it > neither work... > ... > > > > Even this doesn't work (suppose 2 is our missing_value), > > In [184]: data = "1, 2, 3\n4, 5, 6" > > > > In [185]: np.genfromtxt(StringIO(data), > delimiter=",",dtype="int,int,int",missin > > g_values=2) > > Out[185]: > > array([(1, 2, 3), (4, 5, 6)], > > dtype=[('f0', '<i4'), ('f1', '<i4'), ('f2', '<i4')]) > > OK, same behaviour here - I found the only tests involving 'valid numbers' > as missing_values use masked arrays; for regular ndarrays they seem to be > ignored. I don't know if this is by design - the question is, what do you > need to do with the data if you know ' -999' always means a missing value? > You could certainly manipulate them after reading in... > If you have to convert them already on reading in, and using np.mafromtxt > is not an option, your best bet may be to define a custom converter like > (note you have to include any blanks, if present) > > conv = dict(((n, lambda s: s==' -999' and np.nan or float(s)) for n in > range(1,49))) > > Cheers, > Derek > > -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 77 30; Fax:01.69.08.77.16 ************************************************************************************

2 2

[ANN] Theano 0.4.0 released
by David Warde-Farley 27 Jun '11

27 Jun '11

========================== Announcing Theano 0.4.0 =========================== This is a major release, with lots of new features, bug fixes, and some interface changes (deprecated or potentially misleading features were removed). The upgrade is recommended for everybody, unless you rely on deprecated features that have been removed. For those using the bleeding edge version in the mercurial repository, we encourage you to update to the `0.4.0` tag. Deleting old cache ------------------ The caching mechanism for compiled C modules has been updated. In some cases, using previously-compiled modules with the new version of Theano can lead to high memory usage and code slow-down. If you experience these symptoms, we encourage you to clear your cache. The easiest way to do that is to execute: theano-cache clear (The theano-cache executable is in Theano/bin.) What's New ---------- [Include the content of NEWS.txt here] Change in output memory storage for Ops: If you implemented custom Ops, with either C or Python implementation, this will concern you. The contract for memory storage of Ops has been changed. In particular, it is no longer guaranteed that output memory buffers are either empty, or allocated by a previous execution of the same Op. Right now, here is the situation: * For Python implementation (perform), what is inside output_storage may have been allocated from outside the perform() function, for instance by another node (e.g., Scan) or the Mode. If that was the case, the memory can be assumed to be C-contiguous (for the moment). * For C implementations (c_code), nothing has changed yet. In a future version, the content of the output storage, both for Python and C versions, will either be NULL, or have the following guarantees: * It will be a Python object of the appropriate Type (for a Tensor variable, a numpy.ndarray, for a GPU variable, a CudaNdarray, for instance) * It will have the correct number of dimensions, and correct dtype However, its shape and memory layout (strides) will not be guaranteed. When that change is made, the config flag DebugMode.check_preallocated_output will help you find implementations that are not up-to-date. Deprecation: * tag.shape attribute deprecated (#633) * CudaNdarray_new_null is deprecated in favour of CudaNdarray_New * Dividing integers with / is deprecated: use // for integer division, or cast one of the integers to a float type if you want a float result (you may also change this behavior with config.int_division). * Removed (already deprecated) sandbox/compile module * Removed (already deprecated) incsubtensor and setsubtensor functions, inc_subtensor and set_subtensor are to be used instead. Bugs fixed: * In CudaNdarray.__{iadd,idiv}__, when it is not implemented, return the error. * THEANO_FLAGS='optimizer=None' now works as expected * Fixed memory leak in error handling on GPU-to-host copy * Fix relating specifically to Python 2.7 on Mac OS X * infer_shape can now handle Python longs * Trying to compute x % y with one or more arguments being complex now raises an error. * The output of random samples computed with uniform(..., dtype=...) is guaranteed to be of the specified dtype instead of potentially being of a higher-precision dtype. * The perform() method of DownsampleFactorMax did not give the right result when reusing output storage. This happen only if you use the Theano flags 'linker=c|py_nogc' or manually specify the mode to be 'c|py_nogc'. Crash fixed: * Work around a bug in gcc 4.3.0 that make the compilation of 2d convolution crash. * Some optimizations crashed when the "ShapeOpt" optimization was disabled. Optimization: * Optimize all subtensor followed by subtensor. GPU: * Move to the gpu fused elemwise that have other dtype then float32 in them (except float64) if the input and output are float32. * This allow to move elemwise comparisons to the GPU if we cast it to float32 after that. * Implemented CudaNdarray.ndim to have the same interface in ndarray. * Fixed slowdown caused by multiple chained views on CudaNdarray objects * CudaNdarray_alloc_contiguous changed so as to never try to free memory on a view: new "base" property * Safer decref behaviour in CudaNdarray in case of failed allocations * New GPU implementation of tensor.basic.outer * Multinomial random variates now available on GPU New features: * ProfileMode * profile the scan overhead * simple hook system to add profiler * reordered the output to be in the order of more general to more specific * DebugMode now checks Ops with different patterns of preallocated memory, configured by config.DebugMode.check_preallocated_output. * var[vector of index] now work, (grad work recursively, the direct grad work inplace, gpu work) * limitation: work only of the outer most dimensions. * New way to test the graph as we build it. Allow to easily find the source of shape mismatch error: `http://deeplearning.net/software/theano/tutorial/debug_faq.html#interactive-debugger`__ * cuda.root inferred if nvcc is on the path, otherwise defaults to /usr/local/cuda * Better graph printing for graphs involving a scan subgraph * Casting behavior can be controlled through config.cast_policy, new (experimental) mode. * Smarter C module cache, avoiding erroneous usage of the wrong C implementation when some options change, and avoiding recompiling the same module multiple times in some situations. * The "theano-cache clear" command now clears the cache more thoroughly. * More extensive linear algebra ops (CPU only) that wrap scipy.linalg now available in the sandbox. * CUDA devices 4 - 16 should now be available if present. * infer_shape support for the View op, better infer_shape support in Scan * infer_shape supported in all case of subtensor * tensor.grad now gives an error by default when computing the gradient wrt a node that is disconnected from the cost (not in the graph, or no continuous path from that op to the cost). * New tensor.isnan and isinf functions. Documentation: * Better commenting of cuda_ndarray.cu * Fixes in the scan documentation: add missing declarations/print statements * Better error message on failed __getitem__ * Updated documentation on profile mode * Better documentation of testing on Windows * Better documentation of the 'run_individual_tests' script Unit tests: * More strict float comparaison by default * Reuse test for subtensor of tensor for gpu tensor(more gpu test) * Tests that check for aliased function inputs and assure appropriate copying (#374) * Better test of copies in CudaNdarray * New tests relating to the new base pointer requirements * Better scripts to run tests individually or in batches * Some tests are now run whenever cuda is available and not just when it has been enabled before * Tests display less pointless warnings. Other: * Correctly put the broadcast flag to True in the output var of a Reshape op when we receive an int 1 in the new shape. * pydotprint: high contrast mode is now the default, option to print more compact node names. * pydotprint: How trunk label that are too long. * More compact printing (ignore leading "Composite" in op names) Download -------- You can download Theano from http://pypi.python.org/pypi/Theano. Description ----------- Theano is a Python library that allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays. It is built on top of NumPy. Theano features: * tight integration with NumPy: a similar interface to NumPy's. numpy.ndarrays are also used internally in Theano-compiled functions. * transparent use of a GPU: perform data-intensive computations up to 140x faster than on a CPU (support for float32 only). * efficient symbolic differentiation: Theano can compute derivatives for functions of one or many inputs. * speed and stability optimizations: avoid nasty bugs when computing expressions such as log(1+ exp(x)) for large values of x. * dynamic C code generation: evaluate expressions faster. * extensive unit-testing and self-verification: includes tools for detecting and diagnosing bugs and/or potential problems. Theano has been powering large-scale computationally intensive scientific research since 2007, but it is also approachable enough to be used in the classroom (IFT6266 at the University of Montreal). Resources --------- About Theano: http://deeplearning.net/software/theano/ About NumPy: http://numpy.scipy.org/ About SciPy: http://www.scipy.org/ Machine Learning Tutorial with Theano on Deep Architectures: http://deeplearning.net/tutorial/ Acknowledgments --------------- I would like to thank all contributors of Theano. For this particular release, many people have helped during the release sprint: (in alphabetical order) Frederic Bastien, James Bergstra, Nicolas Boulanger-Lewandowski, Raul Chandias Ferrari, Olivier Delalleau, Guillaume Desjardins, Philippe Hamel, Pascal Lamblin, Razvan Pascanu and David Warde-Farley. Also, thank you to all NumPy and SciPy developers as Theano builds on its strength. All questions/comments are always welcome on the Theano mailing-lists ( http://deeplearning.net/software/theano/ )

1 0

Re: [Numpy-discussion] fast numpy i/o
by Nils Becker 27 Jun '11

27 Jun '11

Hi, >> Finally, the former Scientific.IO NetCDF interface is now part of >> scipy.io, but I assume it only supports netCDF 3 (the documentation >> is not specific about that). This might be the easiest option for a >> portable data format (if Matlab supports it). > Yes, it is NetCDF 3. In recent versions of Scientific, NetCDF 4 can be written and read, see http://dirac.cnrs-orleans.fr/ScientificPython/ScientificPythonManual/Scient… N.

2 1