Mailman 3 April 2008 - SciPy-User

Python Student Internship - National Renewable Energy Laboratory
by Chris Perkins April 28, 2008

April 28, 2008

Student Intern – Scientific Computing Group 5900 5900-7259 A student internship is available in the National Renewable Energy Laboratory's (NREL) Scientific Computing Group. NREL is the nation's primary laboratory for research, development and deployment of renewable energy and energy efficiency technologies. The intern will be supporting work concerning management of scientific and technical data. Our data group is cutting-edge with respect to capturing rapidly changing scientific metadata and allowing the scientists to relate different kinds of data in a meaningful way. We have an immediate opening for a summer student internship with possible extension to one year in our Golden, Colorado office. The position would be part-time (15 - 25 hours per week) during the school year and/or full time during the summer. DUTIES: Will include working with researchers on techniques to enable the capture and storage of technical data in a scientific setting. Your role in our development team would be to support data harvesting using existing software, and develop new visualization techniques for existing data sets. DESIRED QUALIFICATIONS: Undergraduate or graduate student in computer science or related field, with demonstrated experience in programming, databases and software development. Experience using agile techniques and test-driven development. Demonstrated of Unit Testing. Experience with major dynamic languages like Python, Ruby, or C#. PREFERRED: Demonstrated good writing skills and computer skills, specifically including programming in python and database use. Experience with systems related to management of scientific data. Candidate must be a US citizen. Qualified candidates should e-mail their resume to: Laura Davis NREL, Human Resources Office Reference: Req. #5900-7259 E-Mail: laura_da...<http://groups.google.com/groups/unlock?msg=4f53e5a7a50c49c5&_done=/group/tu…> @nrel.gov ========================================= feel free to email me with any questions. cheers. -chris

1 0

__setstate__ of TimeSeries subclass
by Pijus Virketis April 28, 2008

April 28, 2008

Hi, I am trying to get the __setstate__ method working on a subclass of scikits.TimeSeries. The setup: from scikits.timeseries import * class Test(TimeSeries): def __new__(cls, *args, **kwargs): return(TimeSeries.__new__(cls, [], date_array([]))) test = Test() # works fine print(repr(test)) The problem: # one-observation TimeSeries instance ts = time_series([1], date_array([Date(string="2000-01-01",freq="U")])) # would like to transfer state from ts to test test.__setstate__(ts.__getstate__()) # hm, seems to have gone well? print(repr(test)) # --------------------- Traceback (most recent call last): File "<stdin>", line 1, in <module> File "c:\python25\lib\site-packages\scikits\timeseries\tseries.py", line 527, in __repr__ timestr = str(self.dates) File "C:\Python25\lib\site-packages\numpy-1.0.5.dev4749.0013-py2.5-win32.egg\numpy\core\numeric.py", line 499, in array_str return array2string(a, max_line_width, precision, suppress_small, ' ', "", str) File "C:\Python25\lib\site-packages\numpy-1.0.5.dev4749.0013-py2.5-win32.egg\numpy\core\arrayprint.py", line 240, in array2string separator, prefix) File "C:\Python25\lib\site-packages\numpy-1.0.5.dev4749.0013-py2.5-win32.egg\numpy\core\arrayprint.py", line 177, in _array2string _summaryEdgeItems, summary_insert)[:-1] File "C:\Python25\lib\site-packages\numpy-1.0.5.dev4749.0013-py2.5-win32.egg\numpy\core\arrayprint.py", line 286, in _formatArray word = format_function(a[-1]) File "c:\python25\lib\site-packages\scikits\timeseries\tdates.py", line 219, in __getitem__ unsorted = self._unsorted[indx] IndexError: index out of bounds # --------------------- Tracing through with pdb showed that while the _data attribute is set nicely, something seems to go wrong with _dates, which are turned into None. I am continuing up the stack with DateArray.__setstate__, but I figure this is a reasonable time to ask for help. I am using an approximatley month-old version of numpy 1.0.5 and today's version of scikits.TimeSeries. Thanks for taking a look! Pijus

3 4

Matlab file reading and writing
by Fabrice Silva April 28, 2008

April 28, 2008

Hi here! Are you aware of the libmatio library. It intends to write and read mat files : http://sourceforge.net/projects/matio It provides a C library and a limited Fortran interface. Are there comparisons with scipy mio Python-written functions ?

2 1

Re: [SciPy-user] Large matrix support
by Dinesh B Vadhia April 28, 2008

April 28, 2008

Nathan / Alan Here is how I'm constructing the sparse matrices: > import numpy > import scipy > from scipy import sparse > # create matrix A using the scipy.sparse coo_matrix method. A is very sparse <10% > ij = numpy.array(numpy.zeros((nnz, 2), dtype=numpy.int)) <snip ... read data into ij> > row = ij[:,0] > column = ij[:,1] > n = ij.shape[0] > data = scipy.ones(n, dtype=numpy.int8) > A = sparse.coo_matrix((data, (row, column)), shape=(I,J)) # changed from dims to shape for new svn > A = sparse.csr_matrix(X) > f = open('A.pkl', 'wb') # pickle the sparse matrix > pickle.dump(A, f, 2) > f.close() > f = open('A.pkl', 'rb') # unpickle the sparse matrix > A = pickle.load(f) > f.close() > I = A.shape[0] > J = A.shape[1] <snip ... a and b are two scalar vectors> > b = A*x ------------------------------ Message: 4 Date: Fri, 25 Apr 2008 19:16:04 -0400 From: "Alan McIntyre" <alan.mcintyre(a)gmail.com> Subject: Re: [SciPy-user] Large matrix support To: "SciPy Users List" <scipy-user(a)scipy.org> Message-ID: <1d36917a0804251616q4babd57crf8acc2b47c6b9072(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 On Fri, Apr 25, 2008 at 5:01 PM, Dinesh B Vadhia <dineshbvadhia(a)hotmail.com> wrote: > My matrix sizes can be large ie. > minimum 20,000 rows x 1m columns. > > When just creating these data sets using Python/Numpy/Scipy, I get > pythonw.exe errors and the program aborts. I guess the Microsoft 32K limit > has been reached somewhere. > > I haven't got any Python program to run with these data sets because of 'not > enough memory' errors. I'm waiting for the int8 support in the sparse > library to hopefully resolve that issue. Dinesh, I apologize if this is too basic a question to ask, but are the matrices sparse? Could you post some sample code to show how you're trying to construct these matrices? Thanks, Alan

2 1

Re: [SciPy-user] Sparse csr_matrix and column sum
by Dinesh B Vadhia April 28, 2008

April 28, 2008

Thanks Nathan. Sorry, I wasn't being imprecise as A.sum(0) didn't work and still doesn't - I've just tried again. However, A.todense().sum(0) does work - but takes a performance hit. My import statements are: > import numpy > import scipy > from scipy import sparse Which means that I have to qualify each function/operation with a numpy. or scipy. - is the problem that I haven't qualified the statement: > colSum = A.sum(0) correctly? Anyway, A.todense().sum() works for small sized matrices but unfortunately, because of the large matrices being used the A.todense().sum(0) results in a memory error. For I = 20000 and J = 66000, here is the Traceback: Traceback (most recent call last): File "C:\... sparseTest.py", line 42 colSum = A.todense().sum(0) # sum of each column of A File "C:\Python25\Lib\site-packages\scipy\sparse\base.py", line 416, in todense return asmatrix(self.toarray()) File "C:\Python25\Lib\site-packages\scipy\sparse\compressed.py", line 627, in toarray M = zeros(self.shape, dtype=self.dtype) MemoryError Is there a way around this? Cheers Dinesh -------------------------------------------------------------------------------- From: Nathan Bell <wnbell <at> gmail.com> Subject: Re: Sparse csr_matrix and column sum Newsgroups: gmane.comp.python.scientific.user Date: 2008-04-28 00:08:23 GMT (1 hour and 38 minutes ago) On Sun, Apr 27, 2008 at 6:41 PM, Dinesh B Vadhia <dineshbvadhia <at> hotmail.com> wrote: > > > If A is a sparse csr_matrix and you want to calculate the sum of each column > then the 'normal' method is: > > import numpy > import scipy > from scipy import sparse > > colSum = scipy.asmatrix(scipy.zeros((1,J), dtype=numpy.float)) > colSum = A.mean(0) > > This isn't working. Do we have to do something else (eg. a todense()) for a > sparse matrix? If so, how? What do you mean by "isn't working"? In [1]: from scipy import * In [2]: from scipy.sparse import * In [3]: A = csr_matrix(rand(3,3)) In [4]: A.todense() Out[4]: matrix([[ 0.95297535, 0.81029421, 0.79146232], [ 0.88477059, 0.9025494 , 0.80259054], [ 0.06691343, 0.76691617, 0.68518027]]) In [5]: A.mean(0) Out[5]: matrix([[ 0.63488646, 0.82658659, 0.75974438]]) In [6]: A.mean(1) Out[6]: matrix([[ 0.85157729], [ 0.86330351], [ 0.50633662]]) In [7]: A.todense().mean(0) Out[7]: matrix([[ 0.63488646, 0.82658659, 0.75974438]]) In [8]: A.todense().mean(1) Out[8]: matrix([[ 0.85157729], [ 0.86330351], [ 0.50633662]]) Dinesh, as a courtesy, would you provide specific details when reporting your problems with SciPy? I'd rather not have to speculate on the precise nature of each issue raised. -- Nathan Bell wnbell <at> gmail.com http://graphics.cs.uiuc.edu/~wnbell/

2 1

Sparse csr_matrix and column sum
by Dinesh B Vadhia April 28, 2008

April 28, 2008

If A is a sparse csr_matrix and you want to calculate the sum of each column then the 'normal' method is: import numpy import scipy from scipy import sparse colSum = scipy.asmatrix(scipy.zeros((1,J), dtype=numpy.float)) colSum = A.mean(0) This isn't working. Do we have to do something else (eg. a todense()) for a sparse matrix? If so, how? Dinesh

2 1

[ Python(x,y) ] New release : 1.1.3
by Python(x,y) April 27, 2008

April 27, 2008

Hi all, Python(x,y) 1.1.3 is now available on http://www.pythonxy.com. Changes history 04 -28 -2008 - Version 1.1.3 : * Added: o "Welcome to Python(x,y)": new GUI to launch useful scripts or consoles, and to find help quickly on some of the installed packages (see screenshot on http://www.pythonxy.com/screenshots.php) o Eclipse: workspace will now refresh automatically * Updated: o SymPy 0.5.14 (see changes on http://code.google.com/p/sympy/) o Enthought Tool Suite 2.7.1 o Interactive consoles with matplotlib, Qt4 or wxPython threading support : Console 2 configuration has again been updated (appearance, mostly) * Corrected: o Eclipse workspace existing settings will no longer be erased during installation -- P. Raybaut Python(x,y) http://www.pythonxy.com

1 0

gammaincinv test hangs?
by Johann Cohen-Tanugi April 26, 2008

April 26, 2008

hello, I am testing scipy on my FC8 box with gcc 4.1 and current revision seems to hang at the gammaincinv test. Can anyone tell me whether it is normal that this test takes a lot of time, or else what issue I might have? thanks, Johann

2 1

test failures on os x
by Gideon Simpson April 26, 2008

April 26, 2008

finally got scipy running on os x 10.5.2, with gfortran 4.3, Apple Python 2.5.1, and Apple gcc 4.0.1, but I'm getting the following test errors: ====================================================================== FAIL: check_dot (scipy.lib.tests.test_blas.test_fblas1_simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/lib/python2.5/site-packages/scipy/lib/blas/tests/ test_blas.py", line 76, in check_dot assert_almost_equal(f([3j,-4,3-4j],[2,3,1]),-9+2j) File "/opt/lib/python2.5/site-packages/numpy/testing/utils.py", line 158, in assert_almost_equal assert round(abs(desired - actual),decimal) == 0, msg AssertionError: Items are not equal: ACTUAL: 1.1081566907292542e-37j DESIRED: (-9+2j) ====================================================================== FAIL: Solve: single precision complex ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/lib/python2.5/site-packages/scipy/linsolve/umfpack/tests/ test_umfpack.py", line 32, in check_solve_complex_without_umfpack assert_array_almost_equal(a*x, b) File "/opt/lib/python2.5/site-packages/numpy/testing/utils.py", line 232, in assert_array_almost_equal header='Arrays are not almost equal') File "/opt/lib/python2.5/site-packages/numpy/testing/utils.py", line 217, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 20.0%) x: array([ 1.00000000+0.j, 1.99999809+0.j, 3.00000000+0.j, 4.00000048+0.j, 5.00000000+0.j], dtype=complex64) y: array([1, 2, 3, 4, 5]) ---------------------------------------------------------------------- Ran 1725 tests in 4.014s FAILED (failures=2) <unittest._TextTestResult run=1725 errors=0 failures=2>

4 3

Re: [SciPy-user] problem building svn scipy
by Damian Eads April 26, 2008

April 26, 2008

Hi there, The hierarchy_wrap.c file is code I wrote and Chris Burns checked in. I did not try to compile it with an earlier version of gcc. Sorry about that. $ gcc --version gcc (GCC) 4.1.2 20070925 (Red Hat 4.1.2-33) $ I will log-in to a machine with gcc-3.4.1 and try recompiling. I hope to have a fix soon. Robert: interesting catch. I could try renaming CS, _X, and _dm. Let me try reproducing this compiler error on a machine with the gcc version John is using. Damian On Fri, Apr 25, 2008 at 9:28 AM, John Hunter <jdh2358(a)gmail.com> wrote: > With numpy r5083 and scipy r4176 I am getting the following build error > > building 'scipy.cluster._hierarchy_wrap' extension > compiling C sources > C compiler: /opt/app/g++lib6/gcc-3.4/bin/gcc -fno-strict-aliasing > -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC > > compile options: > '-I/home/titan/johnh/dev/lib/python2.4/site-packages/numpy/core/include > -I/home/titan/johnh/dev/lib/python2.4/site-packages/numpy/core/include > -I/opt/app/g++lib6/python-2.4/include/python2.4 -c' > gcc: scipy/cluster/src/hierarchy_wrap.c > scipy/cluster/src/hierarchy_wrap.c: In function `calculate_cluster_sizes_wrap': > scipy/cluster/src/hierarchy_wrap.c:117: error: parse error before > numeric constant > scipy/cluster/src/hierarchy_wrap.c:120: error: invalid lvalue in unary `&' > scipy/cluster/src/hierarchy_wrap.c:124: error: invalid type argument of `->' It looks like something is #defining CS to be a number. Search through the headers for this. > scipy/cluster/src/hierarchy_wrap.c: In function > `cluster_maxclust_monocrit_wrap': > scipy/cluster/src/hierarchy_wrap.c:251: warning: unused variable `cutoff' > scipy/cluster/src/hierarchy_wrap.c: In function `pdist_euclidean_wrap': > scipy/cluster/src/hierarchy_wrap.c:387: error: parse error before > numeric constant Possibly the same thing with _X or _dm. Are there any more errors following this? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

2 5