Mailman 3 March 2008 - SciPy-User

Monte Carlo & sparse.linsolve for medium size arrays - any strategies?
by Christopher Mutel 26 Mar '08

26 Mar '08

Hello- I am working with medium size sparse matrices (4000 by 4000, coverage of ~1%), and need to do uncertainty analysis using a Monte Carlo approach. For each element in the matrix, I have an average value, the distribution (mostly lognormal, some normal and uniform), and the relevant uncertainty parameters. I am solving the classic Ax = B matrix problem, where the large matrix is A; B is a constant array. I anticipate needing to do on the order of 1000 iterations. SciPy is a fantastic library, but I am not creative enough to come up with an efficient way to store the relevant uncertainty information so that I am not iteratively generating the large matrix for each Monte Carlo run. Is there a clever way to construct or sublcass a sparse matrix as a generator, so that each time it is referenced a new matrix is generated? Or is there a better approach (i.e. I am sure there is a better approach that I haven't thought of)? I know that this is a rather general question, but I have been thinking about this off and on for quite a while, and have had great luck in the past getting help on this list. Thanks in advance! -Chris -- ############################ Chris Mutel Ökologisches Systemdesign - Ecological Systems Design Institut f.Umweltingenieurwissenschaften - Institute for Environmental Engineering ETH Zürich - HIF C 42 - Schafmattstr. 6 8093 Zürich Telefon: +41 44 633 71 45 - Fax: +41 44 633 10 61 ############################

3 2

import scipy.io error in debian
by Angus McMorland 26 Mar '08

26 Mar '08

Hi all, I'm trying to load a .mat file (my new lab is MatLab-crazy) using debian-testing (numpy 1.0.4 and scipy 0.6.0. I'm getting an import error for scipy.io: "No module named numpyio". Is this a known issue with these versions or a bug? If the former, do I need to update to svn, or is this fixed in the debian-unstable version? Thanks, Angus. -- AJC McMorland, PhD candidate Physiology, University of Auckland Post-doctoral research fellow Neurobiology, University of Pittsburgh

1 0

intepolate.interp2d why X==Y?[SEC=UNCLASSIFIED]
by Scott Collis 26 Mar '08

26 Mar '08

Hello Scipy-users, I've often scanned the mailing lists for tidbits and answers.. But now I have a question of my own. I am regridding data from various sources onto a common grid. These grids are non square and interpolate.interp2d requires X==Y to create a interplolated object. I have modified the code and removed the condition: self.x, self.y, self.z = map(ravel, map(array, [x, y, z])) if not map(rank, [self.x, self.y, self.z]) == [1,1,1]: raise ValueError("One of the input arrays is not 1-d.") --> #if len(self.x) != len(self.y): --> # raise ValueError("x and y must have equal lengths") if len(self.z) == len(self.x) * len(self.y): self.x, self.y = meshgrid(x,y) self.x, self.y = map(ravel, [self.x, self.y]) if len(self.z) != len(self.x): raise ValueError("Invalid length for input z") and it -seems- to be working Is the check there only for the case when X and Y are declared pointwise (ie len(x)=len(y)=len(z)) rather than gridwise (len(x)*len(y)=len(x)) Also... It seems grids over 100x100 seem to kill it.. if anyone has any other suggestions for re-gridding please let me know.. (I may just end up writing my own quick nearest neighbour algorithm instead of using splines like interp2d) Cheers, Scott -- Dr Scott Collis Meteorologist National Meteorological & Oceanographic Centre Australian Bureau of Meteorology Mb: 0412177550

1 0

aggregation of long-term time series
by Marco Tuckner 25 Mar '08

25 Mar '08

Hello, I'd like to aggregate several years of a long term data set into a year-long data set (365 days). The below listed sample data which contains some random value for each month of the years 2005, 2006, 2007. Although the frequency shouldn't really matter in this case I use a monthly frequency to simplify our example. I would like a kind of design year which will be generated by calculating the average over all years for each data point. month 1; 829.67=247+485+292 month 2; 575.67=889+655+183 month 3; X ... Instead of average one could equally calculate sums, square roots, standard deviation, etc... What would be the most efficient way to generate such a mean values data set? Which appraoch would you use? Is such a function already implemented in the timeseries package and I simply overlooked it? Kind regards and thanks in advance for your help, Marco Tuckner ### ##sample data: ### year;month;value 2005;1;247 2005;2;889 2005;3;914 2005;4;292 2005;5;183 2005;6;251 2005;7;953 2005;8;156 2005;9;991 2005;10;557 2005;11;581 2005;12;354 2006;1;485 2006;2;655 2006;3;862 2006;4;399 2006;5;598 2006;6;744 2006;7;445 2006;8;374 2006;9;168 2006;10;995 2006;11;943 2006;12;326 2007;1;292 2007;2;183 2007;3;251 2007;4;953 2007;5;156 2007;6;991 2007;7;557 2007;8;581 2007;9;354 2007;10;399 2007;11;598 2007;12;744 ###

2 2

Re: [SciPy-user] iPython: How to reload & run
by Allen Fowler 25 Mar '08

25 Mar '08

> > New iPython user here.... > > > > Typing %run only reloads the actually file... any of my modules are not > reloaded even if I've chenged them. > > > > Is there short-cut command to %run a file and forcefully reload any modules > that its uses? > > No, unfortunately no. I typically just put a few judiciously chosen > > reload(foo) > reload(bar) > > atop the script itself. You should keep in mind that generically, > reloading is tricky business: it's order-dependent, and what to do > with already in-memory objects when their supporting modules change > isn't clear. > > Python still has a way to go before we catch the fabled lisp machines > from the 70's in terms of interactive modifications of live code, I'm > afraid. > > Hmm.. This being the case.. is there a way to simply do a quick "reset everything, and then run this file"? -- Thanks ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

What's the equivalent of list.pop in numpy?
by Armando Serrano Lombillo 25 Mar '08

25 Mar '08

Hello: >>> a = [[1,2,3],[4,5,6],[7,8,9]] >>> print a [[1, 2, 3], [4, 5, 6], [7, 8, 9]] >>> b = a.pop(1) >>> print a [[1, 2, 3], [7, 8, 9]] >>> print b [4, 5, 6] >>> from numpy import array >>> a = array([[1,2,3],[4,5,6],[7,8,9]]) >>> print a [[1 2 3] [4 5 6] [7 8 9]] >>> b = a.pop(1) Traceback (most recent call last): File "<pyshell#14>", line 1, in <module> b = a.pop(1) AttributeError: 'numpy.ndarray' object has no attribute 'pop' >>> So, what's the "right" way of doing this in numpy? Armando.

3 2

Sobel filter, magnitude and gradient direction
by Daniele Di Mauro 25 Mar '08

25 Mar '08

Hi folks, it's the first time i write on this mailing list, i hope i chose the right one. Yesterday i started a small project for university and i need some help: I've to calculate the sobel edge magnitude and gradient direction of an image, and i haven't undestood if i'm doing right, btw this is the code: import Image import scipy import scipy.ndimage data = Image.open("image.jpg") image_magnitude = scipy.ndimage.filters.generic_gradient_magnitude(data, scipy.ndimage.filters.sobel) output = Image.fromstring("RGB",(320,210),image_magnitude.tostring()) output.save("imagem.jpg","JPEG") I've not look at the gradient direction part yet, but the result it's quite strange, i cannot see edges. I tried also this: import Image import scipy import scipy.ndimage data = Image.open("image.jpg") image_magnitude = scipy.ndimage.filters.sobel(data) output = Image.fromstring("RGB",(320,210),image_magnitude.tostring()) output.save("imagem.jpg","JPEG") but it doesn't convice me either coz it looks quite different from the result of gimp built-in version or using gimp matrix convolution. Thanx in advance for your help Daniele

4 4

Re: [SciPy-user] to openopt users
by dmitrey 25 Mar '08

25 Mar '08

Sorry, I just pressed "reply" and hadn't noticed that it was replied to Alan G. Isaac only Alan G Isaac wrote: > On Wed, 19 Mar 2008, dmitrey apparently wrote: > >> could anyone provide some info like the one from here? >> http://scipy.org/scipy/scikits/wiki/OpenOptUsers > > Do you mean that you would like OpenOpt users to > let you know by email if you can list them as > users on the above page? Yes > Do you want individual > users or just institutions? > Any > Cheers, > Alan Isaac > Regards, D.

2 1

Monte Carlo & sparse.linsolve for medium size arrays - any strategies?
by Christopher Mutel 24 Mar '08

24 Mar '08

Hello- I am working with medium size sparse matrices (4000 by 4000, coverage of ~1%), and need to do uncertainty analysis using a Monte Carlo approach. For each element in the matrix, I have an average value, the distribution (mostly lognormal, some normal and uniform), and the relevant uncertainty parameters. I am solving the classic Ax = B matrix problem, where the large matrix is A; B is a constant array. I anticipate needing to do on the order of 1000 iterations. SciPy is a fantastic library, but I am not creative enough to come up with an efficient way to store the relevant uncertainty information so that I am not iteratively generating the large matrix for each Monte Carlo run. Is there a clever way to construct or sublcass a sparse matrix as a generator, so that each time it is referenced a new matrix is generated? Or is there a better approach (i.e. I am sure there is a better approach that I haven't thought of)? I know that this is a rather general question, but I have been thinking about this off and on for quite a while, and have had great luck in the past getting help on this list. Thanks in advance! -Chris -- ############################ Chris Mutel Ökologisches Systemdesign - Ecological Systems Design Institut f.Umweltingenieurwissenschaften - Institute for Environmental Engineering ETH Zürich - HIF C 42 - Schafmattstr. 6 8093 Zürich Telefon: +41 44 633 71 45 - Fax: +41 44 633 10 61 ############################

1 0

How to read legacy scipy.Pickler arrays?
by Mark Coletti 24 Mar '08

24 Mar '08

I have some old python code that used scipy.Pickler to save arrays to disk. After installing the new scipy, I get this error: data_unpickler = scipy.Unpickler( data_file ) AttributeError: 'module' object has no attribute 'Unpickler' So I went looking for it: mcoletti@sapient:/usr/local/lib/python2.5/site-packages/scipy> find . -name "*.py" | xargs grep -H Pickler ./special/tests/Test.py: p=Numeric.Pickler(f) Hmm. Ok, so mebbe I can directly use the pickle module instead? time_min_data = data_unpickler.load() File "/usr/local/lib/python2.5/pickle.py", line 858, in load dispatch[key](self) KeyError: 'A' Well, ok. Maybe not. Does numpy have a Pickler? mcoletti@sapient:/usr/local/lib/python2.5/site-packages/numpy> find . -name "*.py" | xargs grep -H Pickler ./oldnumeric/misc.py: 'Pickler', 'dot', 'outerproduct', 'innerproduct', 'insert'] ./oldnumeric/misc.py:class Pickler(pickle.Pickler): ./numarray/session.py: p = pickle.Pickler(file, protocol=2) Well, that has potential. Let's try that: data_unpickler = numpy.oldnumeric.Unpickler( data_file ) File "/usr/local/lib/python2.5/site-packages/numpy/oldnumeric/misc.py", line 34, in __init__ raise NotImplemented TypeError: exceptions must be classes, instances, or strings (deprecated), not NotImplementedType Ok, I give up. What do I need to do to read legacy scipy pickled data? Cheers, Mark -- I'm taking reality in small doses to build immunity.

2 1