Mailman 3 October 2002 - SciPy-Dev

Re: [SciPy-dev] Testing stats.py - Error in stats.kurtosis
by Travis Oliphant 31 Oct '02

31 Oct '02

> Travis > > 1. check_kurtosis changed > 2. check_2d error > Thanks for the help. I'm including these tests in the CVS tree, today. But, I first have to correct errors I'm getting with them... Regarding your questions. > > where b2 = array((1.0,2.0,2.64575131106)) > When I enter stats.std(a) I get: > array([ 4.52769257, 2.12132034, 7.21110255]) I don't get these answers regardless of what I put for the second argument. What is a for you? > When I enter stats.std(a,1) I get: > array([ 1. , 2. , 2.64575131]) I do get this result, though. > Finaly, stats.std(ravel(a)) gives me: > 3.5707142142714252 I get this too. > > So, > A) Is the intent of the check_2d to find the population > standard deviation of each row in the array, in which case > check_2d should be changed from > assert_array_almost_equal(stats.std(a),b2,11) My results don't agree with yours. When I say stats.std(a) it should find the standard deviation along the last dimension of the array (axis == 1) or (axis == -1). So, it should give equivalent answers to stats.std(a,1). I don't know why your version is not doing this. Mine is. Are you sure you are getting the right stats module from the right place? > > B) When no axis parameter is entered into stats.std() [and by > extention stats.var() ] When no axis is given, the default is to use the last dimension of the array. There is still some debate on what the proper default behavior should be, but currently all of the stats module (and all of scipy) should behave like this (and does as far as I know). The None argument to axis will ravel the array prior to applying the std operation. > > I'm not much of a stats expert but I'll help where I can once > we've decided what we are trying to do. Thanks for your help. It is appreciated. -Travis

3 3

Testing stats.py - Error in stats.kurtosis [ATTN: Travis O.]
by Louis Luangkesorn 29 Oct '02

29 Oct '02

[ATTN: Travis Oliphant] I've written some more tests for the stats module. Since the stats.py module groups functions in the introductory pydoc comments, I organized the tests I have written into classes based on these groupings. I don't know of any standard problem sets, so I used a basic test set [1,2,3,4] as well as a test set from the Mathworks web site (Matlab documentation). The return value for each test was determined by using R v1.5.1 on Windows, and the R input used was copied into the comments for each test. I have one test that is failing, kurtosis. For the two test cases, I get using R [testcase] Kurtosis = 1.64 [Mathworks] Kurtosis = 2.1658856803 [Mathworks has 2.1615] http://www.mathworks.com/access/helpdesk/help/toolbox/stats/kurtosis.shtml Using stats.py I get [testcase] Kurtosis = -1.36 [Mathworks] Kurtosis = -0.834114319703 Attached are the additional test classes. The classes test_moments and test_variability can be added directly before the function test_suite(level=1) in test_stats.py. The lines in test_suite in the attached file should be added to the current test_suite in test_stats.py. Let me know if this works. I almost put it directly in CVS, but I think Travis O. has someone working on the stats module so I thought that it should be added from there. Louis ===== lluang(a)northwestern.edu http://pubweb.nwu.edu/~kll560 GPG Key:0xF3E2D362 Y!M, AOL, jabber.com: lluang Whatsoever things are true, ... honest, ... just, ... pure, ... lovely, ... of good report; if there be any virtue, and if there be any praise, think on these things. - motto - Northwestern University __________________________________________________ Do you Yahoo!? HotJobs - Search new jobs daily now http://hotjobs.yahoo.com/ class test_variability(TestCase): """ Comparison numbers are found using R v.1.5.1 note that length(testcase) = 4 """ testcase = [1,2,3,4] def check_std(self): y = scipy.stats.std(self.testcase) assert_approx_equal(y,1.290994449) def check_var(self): """ var(testcase) = 1.666666667 """ #y = scipy.stats.var(self.shoes[0]) #assert_approx_equal(y,6.009) y = scipy.stats.var(self.testcase) assert_approx_equal(y,1.666666667) def check_samplevar(self): """ R does not have 'samplevar' so the following was used var(testcase)*(4-1)/4 where 4 = length(testcase) """ #y = scipy.stats.samplevar(self.shoes[0]) #assert_approx_equal(y,5.4081) y = scipy.stats.samplevar(self.testcase) assert_approx_equal(y,1.25) def check_samplestd(self): #y = scipy.stats.samplestd(self.shoes[0]) #assert_approx_equal(y,2.325532197) y = scipy.stats.samplestd(self.testcase) assert_approx_equal(y,1.118033989) def check_signaltonoise(self): """ this is not in R, so used mean(testcase)/(sqrt(var(testcase)*3/4)) """ #y = scipy.stats.signaltonoise(self.shoes[0]) #assert_approx_equal(y,4.5709967) y = scipy.stats.signaltonoise(self.testcase) assert_approx_equal(y,2.236067977) def check_stderr(self): """ this is not in R, so used sqrt(var(testcase))/sqrt(4) """ ## y = scipy.stats.stderr(self.shoes[0]) ## assert_approx_equal(y,0.775177399) y = scipy.stats.stderr(self.testcase) assert_approx_equal(y,0.6454972244) def check_sem(self): """ this is not in R, so used sqrt(var(testcase)*3/4)/sqrt(3) """ #y = scipy.stats.sem(self.shoes[0]) #assert_approx_equal(y,0.775177399) y = scipy.stats.sem(self.testcase) assert_approx_equal(y,0.6454972244) def check_z(self): """ not in R, so used (10-mean(testcase))/sqrt(var(testcase)*3/4) """ y = scipy.stats.z(self.testcase,scipy.stats.mean(self.testcase)) assert_almost_equal(y,0.0) def check_zs(self): """ not in R, so tested by using (testcase[i]-mean(testcase))/sqrt(var(testcase)*3/4) """ y = scipy.stats.zs(self.testcase) desired = ([-1.3416407864999, -0.44721359549996 , 0.44721359549996 , 1.3416407864999]) assert_array_almost_equal(desired,y,decimal=13) class test_moments(TestCase): """ Comparison numbers are found using R v.1.5.1 note that length(testcase) = 4 testmathworks comes from documentation for the Statistics Toolbox for Matlab and can be found at both http://www.mathworks.com/access/helpdesk/help/toolbox/stats/kurtosis.shtml http://www.mathworks.com/access/helpdesk/help/toolbox/stats/skewness.shtml Note that both test cases came from here. """ testcase = [1,2,3,4] testmathworks = [1.165 , 0.6268, 0.0751, 0.3516, -0.6965] def check_moment(self): """ mean((testcase-mean(testcase))**power))""" y = scipy.stats.moment(self.testcase,1) assert_approx_equal(y,0.0,10) y = scipy.stats.moment(self.testcase,2) assert_approx_equal(y,1.25) y = scipy.stats.moment(self.testcase,3) assert_approx_equal(y,0.0) y = scipy.stats.moment(self.testcase,4) assert_approx_equal(y,2.5625) def check_variation(self): """ variation = samplestd/mean """ ## y = scipy.stats.variation(self.shoes[0]) ## assert_approx_equal(y,21.8770668) y = scipy.stats.variation(self.testcase) assert_approx_equal(y,0.44721359549996, 13) def check_skewness(self): """ sum((testmathworks-mean(testmathworks))**3)/((sqrt(var(testmathworks)*4/5))**3)/5 """ y = scipy.stats.skew(self.testmathworks) assert_approx_equal(y,-0.29322304336607,10) y = scipy.stats.skew(self.testcase) assert_approx_equal(y,0.0,10) def check_kurtosis(self): """ sum((testcase-mean(testcase))**4)/((sqrt(var(testcase)*3/4))**4)/4 sum((test2-mean(testmathworks))**4)/((sqrt(var(testmathworks)*4/5))**4)/5 """ y = scipy.stats.kurtosis(self.testmathworks) assert_approx_equal(y, 2.1658856802973,10) y = scipy.stats.kurtosis(self.testcase) assert_approx_equal(y,1.64) def test_suite(level=1): suites = [] if level > 0: suites.append( unittest.makeSuite(test_variability, 'check_') ) suites.append( unittest.makeSuite(test_moments, 'check_') )

2 1

Replacing scipy.fftpack with fftpack2
by Pearu Peterson 26 Oct '02

26 Oct '02

Greetings scipy-devels! The latest version of fftpack2 is available at http://cens.ioc.ee/~pearu/misc/fftpack2-0.2.tar.gz It includes the following changes: * Renamed freqshift to fftshift * Renamed dftfreq to fftfreq * For even signal length, fftshift(f)[0] corresponds to Nyquist mode (like in Matlab). I believe that fftpack2 is ready to replace scipy.fftpack and I would like commit fftpack2 to Scipy CVS by the end of next week. I am actually ready to do that now but the week is provided for review. Let me know if you need more time for review. After the commit, Scipy will require F2PY version 2.23.190-1367 or newer. Regards, Pearu

1 0

possible pilutils.py bug + patch
by Hayden John Callow 26 Oct '02

26 Oct '02

with slightly old scipy cvs scipy.imshow() segfaults with complex arrays (I presume that complex arrays are not a desired input) I have included a patch to trap complex array arguments System info --> RH8.0 + python 2.2.1 >>> scipy.__cvs_version__.cvs_version (1, 144, 1520, 4373) >>> Image.VERSION '1.1.2' >>> sys.version '2.2.1 (#1, Aug 30 2002, 12:15:30) \n[GCC 3.2 20020822 (Red Hat Linux Rawhide 3.2-4)]' cheers Hayden

1 0

Re: [Scipy-cvs] world/scipy/special/tests test_cephes.py,1.1,1.2
by Travis Oliphant 26 Oct '02

26 Oct '02

> Update of /home/cvsroot/world/scipy/special/tests > In directory shaft:/tmp/cvs-serv14246/tests > > Modified Files: > test_cephes.py > Log Message: > (1) Introduced testing hooks and few simple test calls. Test methods starting with _check_.. give segfault with ifc but pass with gcc. (2) Renamed isnan to cephes_isnan and using platform isnan macro if available. Now builds with ifc. (3) Made special locally buildable and testable (temporarily, at least, as orthogonal.py uses linalg.eig) > I have a student working on a large set of test cases for special, so I would not worry too much about testing this right now. -Travis

1 0

Progress with Intel Fortran Compiler support
by Pearu Peterson 21 Oct '02

21 Oct '02

Hi, It took a whole weekend to get Scipy working with the Intel compiler. Now all scipy tests pass (except test_shapiro) also with ifc. However, few functions from special.cephes still cause python to crash (see test methods starting with _check). I'll get back to it later when special module has more tests that actually test the correctness cephes functions results. Currently when cephes functions (built with gcc) return results then I am not always sure that they are correct, in same cases I have even been unable to call these functions without getting domain errors (see test methods starting with __check). Anyway, guide lines how to build scipy with ifc can be found in INSTALL.txt. Unfortunately, the building process is not as straightforward as one would wish: 1) there is binary incompability between ifc and g77 generated codes so that ATLAS and LAPACK libraries *must* be compiled with ifc. 2) in some cases ifc optimization is too good so that LAPACK functions ?lamch fail for that; see INSTALL.txt for a workaround. Finally, Intel Fortran Compiler 5.0 is not supported because scipy build fails with internal compiler error when debugging flags are enabled. Regards, Pearu

2 3

scipy.org
by Sarah Williams 20 Oct '02

20 Oct '02

Hi I visited scipy.org, and noticed that you're not listed on some search engines! I think we can offer you a service which can help you increase traffic and the number of visitors to your website. I would like to introduce you to TrafficMagnet.com. We offer a unique technology that will submit your website to over 300,000 search engines and directories every month. You'll be surprised by the low cost, and by how effective this website promotion method can be. To find out more about TrafficMagnet and the cost for submitting your website to over 300,000 search engines and directories, visit us at: http://p1j2m3a4.pdhost.com/pdsvr/www/r?1000043532.623.23.XBv+4lSCqhDYEf I would love to hear from you. Best Regards, Sarah Williams Sales and Marketing E-mail: Sarah_Williams(a)trafficmagnet.com http://www.TrafficMagnet.com This email was sent to scipy-dev(a)scipy.org. We apologize if this email has reached you in error. We honor all removal requests. Please go to the link below to be removed from our mailing list. http://p1j2m3a4.pdhost.com/pdsvr/www/optoutredirect?UC=Lead&UI=10519279

1 0

Failure installing SciPy
by Johnny boi 19 Oct '02

19 Oct '02

Hi all I've install all the Prerequisits for SciPy, (python2.1, Numerical Python, ATLAS, f2py). Then I went on and did a 'python2.1 setup.py install' on SciPy and it gave me all these errors: atlas_info: NOT AVAILABLE linalg/setup_linalg.py:86: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the scipy_distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: NOT AVAILABLE linalg/setup_linalg.py:90: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the scipy_distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) blas_src_info: NOT AVAILABLE Traceback (most recent call last): File "setup.py", line 130, in ? install_package() File "setup.py", line 95, in install_package config.extend([get_package_config(x,parent_package)for x in standard_packages]) File "setup.py", line 45, in get_package_config config = mod.configuration(parent) File "linalg/setup_linalg.py", line 93, in configuration raise BlasSrcNotFoundError,BlasSrcNotFoundError.__doc__ scipy_distutils.system_info.BlasSrcNotFoundError: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the scipy_distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. I've done sanity_test for atlas and everything passed. So what could be the problem?? Regards Johnny http://careers.yahoo.com.au - Yahoo! Careers - 1,000's of jobs waiting online for you!

2 1

Re: [SciPy-user] Install failure of SciPy
by Pearu Peterson 19 Oct '02

19 Oct '02

On Fri, 18 Oct 2002, Jeffrey B. Layton wrote: > Good afternoon, > > After some suggestions Pearu made, I rebuilt gcc-3.1.1 in > my home account. I found the g2c libs and linked them > appropriately. I then downloaded the CVS version of Scipy > and start building again. Unfortunately, I got another error, > > ifc -shared build/temp.linux-i686-2.2/cephesmodule.o > build/temp.linux-i686-2.2/amos_wrappers.o > build/temp.linux-i686-2.2/specfun_wrappers.o > build/temp.linux-i686-2.2/toms_wrappers.o > build/temp.linux-i686-2.2/cdf_wrappers.o > build/temp.linux-i686-2.2/ufunc_extras.o -Lbuild/temp.linux-i686-2.2 > -Lbuild/temp.linux-i686-2.2 > -L/home/laytonj/local/gcc-3.1.1/lib/gcc-lib/i686-pc-linux-gnu/3.1.1 > -lamos -ltoms -lc_misc -lcephes -lmach -lcdf -lspecfun -lg2c -o > build/lib.linux-i686-2.2/scipy/special/cephes.so > /opt/intel/compiler60/ia32/lib/libF90.a(int8.o): In function `isnan': > int8.o(.text+0x4c40): multiple definition of `isnan' > build/temp.linux-i686-2.2/libcephes.a(isnan.o):/home/laytonj/TEMP/scipy/special/cephes/isnan.c:125: > first defined here > ld: Warning: size of symbol `isnan' changed from 51 to 80 in > /opt/intel/compiler60/ia32/lib/libF90.a(int8.o) > error: command 'ifc' failed with exit status 1 > > > I'm not sure the problem is, but I'll hazard to guess that > isnan is being redefined. I hope that after all the trouble, you'll not give up on SciPy, especially when I have another bad news about the Intel compiler support. First, to fix the above problem, insert the following line #define isnan cephes_isnan to files special/cephes/mconf_LE.h special/cephes/mconf_BE.h and re-run the build command that should finish now successfully. The bad news is that when running scipy tests, it will segfault after .Testing ncf in the stats module (I have tested it both on suse and debian with Intel 5.0 compiler). All other tests from other modules seem to pass fine. I am not sure yet what function, even what module exactly, is causing this segfault, but it must be either stats or special (as stats uses special). And both modules cannot be tested locally (special even doesn't have working unittests) as they import scipy and that makes finding/fixing bugs very time consuming and difficult. I also tried to remove the dependencies of these modules on scipy but gave up on short term, the dependencies were too deep and messy. On long term, I think these issues (the lack of independence and unittests) really ought to be fixed. So, unless someone will figure out quickly the source of this segfault, the Intel compiler remains unsupported. But I'll keep looking for the solution ... Until then, I'd suggest using g77 compiler for building scipy (see one of my previous mails how) and when we have fixed scipy for Intel compiler, you can later simply rebuild scipy. Sorry for inconvinience, Pearu

3 3

shape_base questions
by Pietro Berkes 18 Oct '02

18 Oct '02

Hi! I was taking a look to the 'shape_base.py' file, and I've got a couple of questions/suggestions: - 'apply_over_axes' won't work, since it cannot find SequenceType on line 15. SequenceType was defined in stats/rv2.py and stats/stats.py, where 'apply_over_axes' is used. I suggest to move the definition SequenceType = [types.TupleType, types.ListType, Numeric.ArrayType] together with a predicate function: def issequence(seq): return type(seq) in SequenceType in scipy_base/type_check.py and then to fix rv2.py, stats.py and shape_base.py. By the way: in rv2.py, SequenceType contains array.ArrayType and Numeric.ArrayType. Are they different? And if yes, why? - 'apply_over_axes' and 'expand_dims' accept negative axes, while other functions don't. Are there design guidelines regarding negative axes? The two functions correct negative indices in two different ways: 'apply_over_axes' : axis = len(shape) + axis 'expand_dims' : axis = len(shape) + axis + 1 I think the first way is the correct one in both cases, at least it would match the documentation of both functions. I could send a patch for these issues... what is the preferred way to do this? Should I use the SourceForge page (it appears to be void...)? Regards, Pietro.

2 1