Mailman 3 January 2009 - SciPy-Dev

Inheritance from poly1d (orthogonal.py)
by Tom Grydeland Jan. 16, 2009

Jan. 16, 2009

Hello, developers. In reference to the ticket I just submitted: http://scipy.org/scipy/scipy/ticket/844 I think the problem is with the class 'poly1d' in numpy.lib.polynomial. Several of the methods of this class (such as __mul__, __rmul__, integ, deriv) refer explicitly to 'poly1d' instead of self's class. In the same vein, I expect that __repr__ should use self's class instead of 'poly1d'. -- Tom Grydeland <Tom.Grydeland(a)(gmail.com)>

1 1

cut-and-paste error in scipy\io\matlab\mio5.py
by josef.pktd＠gmail.com Jan. 15, 2009

Jan. 15, 2009

File "C:\Programs\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 502, in get_raw_array result = super(Mat5ObjectMatrixGetter, self).get_raw_array() TypeError: super(type, obj): obj must be an instance or subtype of type class Mat5FunctionMatrixGetter(Mat5CellMatrixGetter): def get_raw_array(self): result = super(Mat5ObjectMatrixGetter, self).get_raw_array() return MatlabFunction(result) super(Mat5ObjectMatrixGetter should be: super(Mat5FunctionMatrixGetter I guess this is covered by a test Josef

3 4

SciPy 0.7.x branch frozen
by Jarrod Millman Jan. 15, 2009

Jan. 15, 2009

The SciPy 0.7.x branch is now frozen. The trunk remains open for development. Please do *not* make any commits to the branch from now until 0.7.0 final is released. If you have a bug fix you want included in the 0.7.x series, please commit it to the trunk and send an email to the dev-list asking for it to be reviewed and backported. I will tag 0.7.0rc1 tomorrow and, unless something unexpected turns up, the 0.7.0 final release will be tagged a little over one week later. Here is the release schedule: 1/13 0.7.x branch frozen 1/14 0.7.0rc1 tagged 1/15 0.7.0rc1 announced, binaries posted to sourceforge 1/24 0.7.0 tagged 1/26 0.7.0 announced, binaries posted to sourceforge A 0.7.1 release is scheduled for sometime in February (the exact date will depend on exactly what issues are reported and fixed with 0.7.0). Thanks to everyone who has helped get this release ready. 0.7 is a very significant release in terms of features, bug-fixes, testing, documentation, license compliance, and release process. I am looking forward to the 0.8 release and will send a separate email to start discussing the 0.8 development roadmap. Best, Jarrod

2 1

No module named latexwriter
by Nils Wagner Jan. 14, 2009

Jan. 14, 2009

Hi all, I tried to build the documentation by make html in numpy/doc I am using OpenSuse11.1 on a x86_64 box. ~/svn/numpy/doc> make html mkdir -p build ./sphinxext/autosummary_generate.py source/reference/*.rst \ -p dump.xml -o source/reference/generated /home/nwagner/svn/numpy/doc/sphinxext/autosummary.py:59: DeprecationWarning: The sphinx.builder module is deprecated; please import builders from the respective sphinx.builders submodules. import sphinx.addnodes, sphinx.roles, sphinx.builder Failed to import 'numpy.__array_priority__': Failed to import 'numpy.distutils.misc_util.get_numarray_include_dirs': Failed to import 'numpy.generic.__squeeze__': touch build/generate-stamp mkdir -p build/html build/doctrees LANG=C sphinx-build -b html -d build/doctrees source build/html /home/nwagner/svn/numpy/doc/sphinxext/autosummary.py:59: DeprecationWarning: The sphinx.builder module is deprecated; please import builders from the respective sphinx.builders submodules. import sphinx.addnodes, sphinx.roles, sphinx.builder Extension error: Could not import extension only_directives (exception: No module named latexwriter) make: *** [html] Fehler 1 How can I fix the problem ? Nils

2 2

IOError in fftpack/tests
by Nils Wagner Jan. 13, 2009

Jan. 13, 2009

>>> scipy.__version__ '0.8.0.dev5461' ====================================================================== ERROR: Failure: IOError ([Errno 2] No such file or directory: '/data/home/nwagner/local/lib/python2.5/site-packages/scipy/fftpack/tests/test.mat') ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/data/home/nwagner/local/lib/python2.5/site-packages/scipy/fftpack/tests/test_real_transforms.py", line 11, in <module> squeeze_me=True, struct_as_record=True, mat_dtype=True) File "/data/home/nwagner/local/lib/python2.5/site-packages/scipy/io/matlab/mio.py", line 110, in loadmat MR = mat_reader_factory(file_name, appendmat, **kwargs) File "/data/home/nwagner/local/lib/python2.5/site-packages/scipy/io/matlab/mio.py", line 67, in mat_reader_factory byte_stream = open(full_name, 'rb') IOError: [Errno 2] No such file or directory: '/data/home/nwagner/local/lib/python2.5/site-packages/scipy/fftpack/tests/test.mat'

2 1

2009 Google Summer of Code
by Jarrod Millman Jan. 12, 2009

Jan. 12, 2009

Google has announced that it will be running Summer of Code and Google Highly Open Participation Contest again this summer. Due to the current economic climate, the SoC will be smaller this year. But the participant stipends will remain the same amount as last year. SciPy has participated in the SoC for the last several years. Our involvement in this program has been extremely productive. Last year there were several NumPy/SciPy related projects including: NumPy test framework enhancement Alan McIntyre Developing Cython for Easy NumPy Integration Dag Sverre Seljebotn Moving OpenOpt Forward Dmitrey Kroshko While it isn't certain that we will get any student proposals accepted, it is important to start developing proposals now. Please feel free to send ideas to the list or use the wiki: http://scipy.org/scipy/scipy/wiki/SummerofCodeIdeas For instance, here are a few ideas for SoC projects that I have been thinking about: - Port NumPy/SciPy to Python 3.0 - Date/time types http://projects.scipy.org/scipy/numpy/browser/trunk/doc/neps/datetime-propo… - integrate Jonathan Taylor's statistical models into scipy.stats. - improve datasource and integrate it into all the numpy/scipy io http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/lib/_datasource.py - clean up, refactor scipy package structure: scipy.lib, scipy.misc, scipy.stsci - modernize, clean-up scipy.weave/numpy.f2py, integrate fast-vectorize Thanks, Jarrod

2 1

Re: [SciPy-dev] [SciPy-user] Scipy 0.7, weave, windows
by Daniel Wheeler Jan. 12, 2009

Jan. 12, 2009

Hi David, I ran all the weave tests in FiPy (on windows) with the binary posted and I get absolutley no errors. I was a little confused by this because we generally get a mountain of doctest errors the first time we run the weave tests. These are associated with the "weave compiling" output noise. I deleted the ".python25_compiled" directory and I only get one doctest error associated with the creation of that directory. Anyway, the long and the short of it is that everything seems cool and as an addition weave is no longer puking that annoying "weave compiling" noise that breaks all the doctests. Cheers On Fri, Jan 9, 2009 at 7:31 AM, David Cournapeau <cournape(a)gmail.com> wrote: > On Fri, Jan 9, 2009 at 5:00 AM, Daniel Wheeler > <daniel.wheeler2(a)gmail.com> wrote: >> On Thu, Jan 8, 2009 at 11:15 AM, David Cournapeau <cournape(a)gmail.com> wrote: >>> On Fri, Jan 9, 2009 at 12:49 AM, Daniel Wheeler >>> <daniel.wheeler2(a)gmail.com> wrote: >>>> On Thu, Jan 8, 2009 at 10:19 AM, David Cournapeau >>>> <david(a)ar.media.kyoto-u.ac.jp> wrote: >>>>> Hi, >>>>> >>>>> I just did a full build/install/test dance of scipy 0.7 on windows, >>>>> and things look good - except weave, which brings 205 errors when the >>>>> full test suite is run. Do people use weave on windows ? >>>> >>>> Yes. Our test suite for fipy currently passes all it's weave tests on >>>> windows with python 2.5 and scipy version 0.6.0 and that includes a >>>> lot of auto generated weave code. >>> >>> Thanks for the info. Would you mind testing it with scipy 0.7.x branch >>> ? There are some test failures which showed some old code which could >>> not have worked (like using python code which was removed from python >>> svn 5 years ago), but as I am not a weave user myself, I can't really >>> assess what's significant and what's not. >>> >>> I could make a binary installer if that makes it easier for you to test, >> >> That would be great if you have it set to build quickly and easily. >> Don't fancy figuring out how to build scipy on windows. Cheers. > > No need to worry, I am the one who coded the tools for the windows > binary installer, so hopefully I am still familiar with it :) > > Here we are: > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/scipy/scipy-0.7.0.… > > David > _______________________________________________ > SciPy-user mailing list > SciPy-user(a)scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-user > -- Daniel Wheeler

1 0

ERROR: Failure: ImportError (No module named convolve)
by Nils Wagner Jan. 12, 2009

Jan. 12, 2009

>>> numpy.__version__ '1.3.0.dev6316' >>> scipy.__version__ '0.8.0.dev5437' ====================================================================== ERROR: Failure: ImportError (No module named convolve) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/data/home/nwagner/local/lib/python2.5/site-packages/scipy/stsci/__init__.py", line 2, in <module> import image File "/data/home/nwagner/local/lib/python2.5/site-packages/scipy/stsci/image/__init__.py", line 2, in <module> from _image import * File "/data/home/nwagner/local/lib/python2.5/site-packages/scipy/stsci/image/_image.py", line 2, in <module> import convolve ImportError: No module named convolve ---------------------------------------------------------------------- Ran 3572 tests in 58.251s FAILED (KNOWNFAIL=2, SKIP=17, errors=1)

2 2

Numerical Recipes (was tagging 0.7rc1 this weekend?)
by Jarrod Millman Jan. 11, 2009

Jan. 11, 2009

Given the discussion about Numerical Recipes derived-code, I did a little grepping and found 7 instances where code is referencing Numerical Recipes. Before branching for 0.7, I would like to see this resolved. I am not sure what we should do, but the license for using Numerical Recipes derived code clearly does not permit us to include it in scipy: http://www.nr.com/licenses/redistribute.html Below I include all the instances I could easily find. Thoughts? -------------------------------------------------- In scipy/optimize/zeros.py: def ridder(f, a, b, args=(), xtol=_xtol, rtol=_rtol, maxiter=_iter, full_output=False, disp=False): """Find root of f in [a,b] Ridder routine to find a zero of the function f between the arguments a and b. f(a) and f(b) can not have the same signs. Faster than bisection, but not generaly as fast as the brent rountines. A description may be found in a recent edition of Numerical Recipes. The routine here is modified a bit to be more careful of tolerance. def brentq(f, a, b, args=(), xtol=_xtol, rtol=_rtol, maxiter=_iter, full_output=False, disp=False): """Find root of f in [a,b] The classic Brent routine to find a zero of the function f between the arguments a and b. f(a) and f(b) can not have the same signs. Generally the best of the routines here. It is a safe version of the secant method that uses inverse quadratic extrapolation. The version here is a slight modification that uses a different formula in the extrapolation step. A description may be found in Numerical Recipes, but the code here is probably easier to understand. In scipy/signal/medianfilter.c: /* * This Quickselect routine is based on the algorithm described in * "Numerical recipies in C", Second Edition, * Cambridge University Press, 1992, Section 8.5, ISBN 0-521-43108-5 */ In scipy/special/orthogonal.py: """ A collection of functions to find the weights and abscissas for Gaussian Quadrature. These calculations are done by finding the eigenvalues of a tridiagonal matrix whose entries are dependent on the coefficients in the recursion formula for the orthogonal polynomials with the corresponding weighting function over the interval. Many recursion relations for orthogonal polynomials are given: a1n f_n+1 (x) = (a2n + a3n x ) f_n (x) - a4n f_n-1 (x) The recursion relation of interest is P_n+1 (x) = (x - A_n) P_n (x) - B_n P_n-1 (x) where P has a different normalization than f. The coefficients can be found as: A_n = -a2n / a3n B_n = ( a4n / a3n sqrt(h_n-1 / h_n))**2 where h_n = int_a^b w(x) f_n(x)^2 assume: P_0(x) = 1 P_-1(x) == 0 See Numerical Recipies in C, page 156 and Abramowitz and Stegun p. 774, 782 In scipy/stats/stats.py: def ttest_ind(a, b, axis=0): """Calculates the t-obtained T-test on TWO INDEPENDENT samples of scores a, and b. From Numerical Recipies, p.483. Axis can equal None (ravel array first), or an integer (the axis over which to operate on a and b). def ttest_rel(a,b,axis=None): """Calculates the t-obtained T-test on TWO RELATED samples of scores, a and b. From Numerical Recipies, p.483. Axis can equal None (ravel array first), or an integer (the axis over which to operate on a and b). def ks_2samp(data1, data2): """ Computes the Kolmogorov-Smirnof statistic on 2 samples. Modified from Numerical Recipies in C, page 493. Returns KS D-value, prob. Not ufunc- like.

15 39

denoising spatial point process data
by Sturla Molden Jan. 9, 2009

Jan. 9, 2009

If it is of interest I'll donate a useful algorithm of denoising spatial point process data to scipy.spatial. It works by fitting a mixture of two Poisson processes (signal + noise) using an EM algorithm. It was originally developed for US DoD to detect minefields using reconnaissance aircraft images. I use it on neurophysiological spike data to remove artifact waveforms. It seems to be very efficient. Regards, Sturla Molden # Written by Sturla Molden import numpy from scipy.special import gamma def nnclean(K, knn, ndim, convergence=0.01): """ K-th nearest neighbour clutter removal Algorithm from: Byers, S & Raftery, A (1998). Nearest-Neighbor Clutter Removal for estimating Features in Spatial Point Processes. J. Am. Stat. Assoc., 93: 577-585. Input: K: neighbour to use (K=5 often works well) Dk: distance to K-th nearest neighbour. Obtain by searching a scipy.spatial.cKDTree with p=2 ndim: number of dimensions in the data set from which Dk was computed. Output: signal: indices of signal points noise: indices of clutter points delta: likelihood that a point is signal loglike: total log-likelihood of the fit """ # size of data d, n = ndim, len(Dk) # calculate volume of N-D hypersphere ad = (2*numpy.pi**(d/2.0))/(d*gamma(d/2.0)) #select a random starting point delta = numpy.random.rand(n) l1 = K * delta.sum()/(ad*numpy.sum((Dk**d)*delta)) l2 = K*numpy.sum(1.0-delta)/(ad*numpy.sum((Dk**d)*(1.0-delta))) #l1[numpy.where(l1<1E-12)[0]] = 1E-12 #l2[numpy.where(l2<1E-12)[0]] = 1E-12 p = numpy.sum(delta)/n old_loglike = numpy.sum( -p*l1*ad*((Dk**d)*delta) - (1.0-p)*l2*ad*((Dk**d)*(1.0-delta)) + delta*K*numpy.log(l1*ad) + (1.0-delta)*K*numpy.log(l2*ad) ) # estimate Poisson mixture by EM updates term = 100 loglike = [old_loglike] counter = 0 run = 1 old_classification = numpy.round(delta) while run: new_delta, l1, l2, p = EM_update(Dk,K,n,d,ad,delta,l1,l2,p) #l1[numpy.where(l1<1E-12)[0]] = 1E-12 #l2[numpy.where(l2<1E-12)[0]] = 1E-12 new_loglike = numpy.sum( -p*l1*ad*((Dk**d)*new_delta) - (1-p)*l2*ad*((Dk**d)*(1.0-new_delta)) + new_delta*K*numpy.log(l1*ad) + (1.0-new_delta)*K*numpy.log(l2*ad) ) update = 100*(new_loglike - old_loglike)/new_loglike # Force at least 3 EM updates, quit after 50. # Quit if EM starts to oscillate. counter = counter + 1 if counter < 3: run = 1 elif counter > 50: run = 0 elif old_loglike > new_loglike: run = 0 classification = old_classification new_loglike = old_loglike; new_delta = delta elif update < convergence: run = 0 old_loglike = new_loglike loglike.append(new_loglike) delta = new_delta if l2 > l1: delta = 1.0 - delta idx, = numpy.where(delta < 1E-6) delta[idx] = 1E-6 signal = numpy.where(delta > .5)[0] noise = numpy.where(delta <= .5)[0] # done return signal, noise, delta, loglike #Single EM update def EM_update(Dk,K,n,d,ad,delta,l1,l2,p): factorial = lambda K : numpy.arange(1.0,K+1.0).prod() if K > 1 else 1.0 pdf = lambda x,K,l,d: numpy.exp(-l*(x**d)) * 2 * (l**K) * x**(d*K - 1) / factorial(K-1) # E-step P1 = pdf(Dk,K,l1*ad,d) P2 = pdf(Dk,K,l2*ad,d) index, = numpy.where((P1 == 0.0) * (P2 == 0.0)) delta[index] = .5 index, = numpy.where((P1 != 0.0) + (P2 != 0.0)) delta[index] = p*P1[index]/(p*P1[index] + (1.0-p)*P2[index]) # M-step l1 = K*numpy.sum(delta)/(ad*numpy.sum((Dk**d)*delta)) l2 = K*numpy.sum(1.0-delta)/(ad*numpy.sum((Dk**d)*(1.0-delta))) p = numpy.sum(delta)/n return delta, l1, l2, p

3 4