Mailman 3 April 2006 - SciPy-Dev

some statistical models / formulas
by Jonathan Taylor May 4, 2006

May 4, 2006

i have made a numpy/scipy package for some linear statistical models http://www-stat.stanford.edu/~jtaylo/scipy_stats_models-0.01a.tar.gz i was hoping that it might someday get into scipy.stats, maybe as scipy.stats.models? anyways, i am sure the code needs work and more docs with examples, but right now there is basic functionality for the following (the tests give some examples): - model formulae as in R (to some extent) - OLS (ordinary least square regression) - WLS (weighted least square regression) - AR1 regression (non-diagonal covariance -- right now just AR1 but easy to extend to ARp) - generalized linear models (all of R's links and variance functions but extensible as well -- not everything has been rigorously tested but logistic agrees with R, for instance) - robust linear models using M estimators (with a number of standard default robust norms as in R's rlm) - robust scale estimates (MAD, Huber's proposal 2). it would be nice to add a few things over time, too, like: - mixed effects models - generalized additive models (gam), generalized estimating equations (gee).... - nonlinear regression (i have some quasi working code for this, too, but it is not yet included). + anything else people want to add. -- jonathan -- ------------------------------------------------------------------------ I'm part of the Team in Training: please support our efforts for the Leukemia and Lymphoma Society! http://www.active.com/donate/tntsvmb/tntsvmbJTaylor GO TEAM !!! ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305

3 2

Re: [SciPy-dev] scipy.optimize
by Robert Cimrman May 1, 2006

May 1, 2006

Hi all, outside the list, I have proposed some enhancements to scipy.optimize, which are now below. Opinions welcome. Travis Oliphant wrote: > Robert Cimrman wrote: > >> Hi Travis, >> >> I write first to you as you are the author of scipy.optimize. >> >> I have been using scipy.optimize recently quite a lot and also coded the >> steepest descent method (named fmin_sd) myself, which has some features >> I would like to see e.g. in fmin_cg too. These are: >> 1. supporting hooks for a (user-provided?) logging class to log and >> possibly plot the convergence progress; >> 2. time statistics (via modified wrap_function) >> 3. passing configuration parameters in a structure (e,g. conf.gtol, >> conf.maxiter) >> >> Do you think that some points would be interesting to others? I attach >> my code (outside of scipy now, so you cannot run it as it is) and an >> example log plot. > > > Yes. logging is particularly requested quite often. As for > configuration parameters, I suspect that it is best done by making a > nice object-based interface to the optimization routines. I'm not sure > what the benefit is of passing in a configuration object over simply > using keyword arguments. Well, I have nothing against keyword arguments. But with a configuration object, the function argument list would be smaller, all fmin_* functions could have the same syntax and new configuration options would not influence the argument list. I am willing to work on the object-based interface. Not too much time right now, but... > But, make your suggestion on the list. It would also be nice to have > the fmin_sd code in SciPy. Yes, I am going to include it to SciPy. > In the near future, I'd like to put a trust_region optimization method > as well. Yes. that would be very usefull. r.

5 8

-Qnew option breaks scipy
by Eric Nodwell April 30, 2006

April 30, 2006

Using the -Qnew switch with python breaks scipy (version 0.4.8 and previous versions I have tested). The lines below demonstrate this. In summary, I run: (1) python without -Q option, and without "from __future__ import division". This works. (2) python without -Q option, and with "from __future__ import division". This works. (2) python with the -Qnew option, and without "from __future__ import division". This breaks scipy. $ /sw/bin/python Python 2.4.2 (#1, Mar 5 2006, 22:35:36) [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 3/2 # verify old style division 1 >>> import scipy >>> scipy.version.version '0.4.8' >>> $ /sw/bin/python Python 2.4.2 (#1, Mar 5 2006, 22:35:36) [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from __future__ import division >>> 3/2 # verify new style division 1.5 >>> import scipy >>> $ /sw/bin/python -Qnew Python 2.4.2 (#1, Mar 5 2006, 22:35:36) [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 3/2 # verify new style division 1.5 >>> import scipy Traceback (most recent call last): File "<stdin>", line 1, in ? File "/sw/lib/python2.4/site-packages/scipy/__init__.py", line 32, in ? from numpy import * File "/sw/lib/python2.4/site-packages/numpy/f2py/__init__.py", line 12, in ? import f2py2e File "/sw/lib/python2.4/site-packages/numpy/f2py/f2py2e.py", line 27, in ? import rules File "/sw/lib/python2.4/site-packages/numpy/f2py/rules.py", line 96, in ? module_rules={ File "/sw/lib/python2.4/site-packages/numpy/f2py/auxfuncs.py", line 395, in gentitle return '/*%s %s %s*/'%(l*'*',name,l*'*') TypeError: can't multiply sequence by non-int >>>

2 5

sinm and cosm for matrices with real coefficients
by Vanuxem Grégory April 30, 2006

April 30, 2006

Hi, The code of cosm and sinm for matrices with real coefficients can be improved. The actual code is something like: sinm : (expm(j * M) - expm(- j*M)) / (2 * j) cosm : (expm(j * M) + expm(- j* M)) / 2 But this can be changed to (only for matrices with real coefficients): sinm : imaginary_part(expm(j * M)) cosm : real_part(expm(j * M)) Sorry, as you can see I'm not a python programmer, just reading the scipy source code... Cheers, Greg

3 2

Error compiling delaunay
by Bernhard Reimar Hoefle April 29, 2006

April 29, 2006

Hi! I updated my subversion repository today. It get the following error when I try to compile scipy with the sandbox/delaunay module. I have gcc-Version 4.1.0, Python 2.4.1 and Numpy 0.9.7. creating build/temp.linux-i686-2.4/Lib/sandbox creating build/temp.linux-i686-2.4/Lib/sandbox/delaunay compile options: '-ILib/sandbox/delaunay -I/usr/lib/python2.4/site-packages/numpy/core/include -I/usr/include/python2.4 -c' c++: Lib/sandbox/delaunay/_delaunay.cpp Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:208: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »ELgethash« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:231: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »openpl« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:232: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »line« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:233: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »circle« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:234: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »range« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:208: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »ELgethash« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:231: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »openpl« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:232: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »line« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:233: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »circle« Lib/sandbox/delaunay/VoronoiDiagramGenerator.h:234: Fehler: extra qualification »VoronoiDiagramGenerator::« on member »range« error: Command "c++ -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -m32 -march=i386 -mtune=pentium4 -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -fPIC -ILib/sandbox/delaunay -I/usr/lib/python2.4/site-packages/numpy/core/include -I/usr/include/python2.4 -c Lib/sandbox/delaunay/_delaunay.cpp -o build/temp.linux-i686-2.4/Lib/sandbox/delaunay/_delaunay.o" failed with exit status 1

2 1

SciPy SVN site
by Travis Oliphant April 28, 2006

April 28, 2006

Is anybody else having trouble accessing the scipy SVN site? I can't seem to update the tree on my work computer... -Travis

3 4

Porting (some of?) Netlab to SciPy (Google Soc)
by Fred Mailhot April 27, 2006

April 27, 2006

Hello, The Python Software Foundation's page of potential projects for this year's Google Summer of Code contains the following entry: "Conduct a review of one chunk of functionality in scipy similar to the one currently in progress about the statistics package [...] Recently discussed requests have been [...], and porting the Netlab neural network code to scipy." I am highly interested in carrying out the Netlab project in the context of Google's Summer of Code (fulltime from June-August) and would like to know whether this is something that is sufficiently of interest to SciPy's developers and if so whether there is someone available and willing to serve as mentor (I envision weekly online meetings plus emergency contact as needed). A bit about myself: I am a graduate student in Cognitive Science with about 2 years of experience with Python in particular and about 6 years total of programming experience. I have implemented several neural network models (feedforward, recurrent, Hopfield) as well as many other machine learning algorithms so I am quite comfortable with the requisite concepts. If someone is interested in pursuing this please contact me as soon as possible so that we can work on putting a great proposal together. I am happy to pass along my CV or answer any other questions Regards, Fred Mailhot -- Institute of Cognitive Science Carleton University Ottawa, ON, Canada

3 3

Segfault
by Nils Wagner April 27, 2006

April 27, 2006

Hi all, I am going to visualize the sparsity pattern of large matrices using numpy/scipy/matplotlib. I received a segfault running the following script from pylab import show, plot, axis, xlabel, subplot, title, spy from scipy import * K = io.mmread('k.mtx') M = io.mmread('m.mtx') print 'Reading finished' K = K.todense() spy(K) show() Starting program: /usr/bin/python matrixstructure.py [Thread debugging using libthread_db enabled] [New Thread 16384 (LWP 13423)] Reading finished Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 13423)] array_from_pyobj (type_num=12, dims=0x7fffffffd270, rank=2, intent=0, obj=<value optimized out>) at fortranobject.c:547 547 PyArray_FILLWBYTE(arr, 0); Current language: auto; currently c (gdb) bt #0 array_from_pyobj (type_num=12, dims=0x7fffffffd270, rank=2, intent=0, obj=<value optimized out>) at fortranobject.c:547 #1 0x00002aaab37606ab in f2py_rout_sparsetools_dcsctofull (capi_self=<value optimized out>, capi_args=<value optimized out>, capi_keywds=<value optimized out>, f2py_func=0x2aaab3778760 <dcsctofull_>) at sparsetoolsmodule.c:12056 #2 0x00002aaab376f25b in fortran_call (fp=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at fortranobject.c:289 #3 0x00002aaaaabf7f20 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #4 0x00002aaaaac4ebaa in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #5 0x00002aaaaac510af in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #6 0x00002aaaaac510af in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #7 0x00002aaaaac510af in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #8 0x00002aaaaac51705 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #9 0x00002aaaaac51992 in PyEval_EvalCode () from /usr/lib64/libpython2.4.so.1.0 #10 0x00002aaaaac6aeb9 in run_node () from /usr/lib64/libpython2.4.so.1.0 #11 0x00002aaaaac6c3c9 in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.4.so.1.0 #12 0x00002aaaaac6c921 in PyRun_AnyFileExFlags () from /usr/lib64/libpython2.4.so.1.0 #13 0x00002aaaaac72023 in Py_Main () from /usr/lib64/libpython2.4.so.1.0 #14 0x00000000004008d9 in main (argc=<value optimized out>, argv=<value optimized out>) at ccpython.cc:10 Any idea how to fix this problem ? Nils

4 4

sparse matrix comparisons
by Nils Wagner April 27, 2006

April 27, 2006

Hi all, The current status wrt comparision of sparse matrices is File "/usr/lib64/python2.4/site-packages/scipy/sparse/sparse.py", line 155, in __cmp__ raise TypeError, "comparison of sparse matrices not implemented" TypeError: comparison of sparse matrices not implemented http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2240626 Shall I submit a ticket for this feature ? Nils

1 0

interest in Time series functionality?
by Matt Knox April 26, 2006

April 26, 2006

I hope this is the correct mailing list to post this kind of question to... and if not, my apologies. I work in the quantitative finance division of a financial services company in Canada and the last year or so we have been doing a lot more python based work. Most of the data we work with is time series data (stock price data, etc) and we have traditionally used FAME (a product of sungard) to store and manipulate this data. We have developed a python api on top of the included FAME c api to access FAME data from python, but the problem now is that there aren't any available python libraries (to my knowledge) for manipulating time series data in any way that comes close to the power of FAME. Our motivation for being able to manipulate this data in Python is primarily for web-based applications. Although being able to bring this data into the python world opens up many other possibilities for us as well (FAME has no matrix capabilities whatsoever, which is pretty sad really given their target market). We have done some preliminary work in developing a time series class built on top of the numpy array class, and it has gotten to the point where it works reasonably well for what we are using it for, although I'm certain it could be optimized a great deal. The key features of this module are: - works with different frequencies of data (currently supports monthly, daily, business days, and secondly frequencies) - able to index the time series directly by date objects (from a custom date class we have created) - handle missing values (along the lines of masked arrays) - global module settings to dictate how certain scenarios are handled - perform operations on time series that do not necessarily have the same start/end dates (+,-,*,/) (and handle missing values appropriately in the operation according to certain global option settings). This involves an implicit resizing of the arrays. - perform operations on time series that do not have the same frequency and perform implicit frequency conversions (according to certain global option settings). Again, this involves implicitly resizing the arrays We have basically attempted to model the time series functionality to be similar to how FAME handles it since that works reasonably well. I'm wondering if there is any kind of interest in this? Our group consists mostly of financial practitioners and engineers, not really pure software developers, so if somebody is interested in taking this to the next level I would be willing to release the code (both the FAME api, and the time series module) if someone wanted to improve upon this and share their improvements in the future. The code is definitely not a polished product right now, but it is functional. If you have any thoughts on this (positive or negative) I would love to hear them. Thanks, - Matt Knox _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

7 12