Mailman 3 December 2005 - SciPy-Dev

newcore, icc compiled ATLAS problem
by Arnd Baecker Dec. 22, 2005

Dec. 22, 2005

Hi, I am again trying to install scipy core on an Itanium with ATLAS. FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/baecker/python2/lib/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.7.11\\""')] include_dirs = ['/usr/include'] The problem is that ATLAS is compiled with the intel compiler. Why is this so: According to http://math-atlas.sourceforge.net/errata.html#WhatComp on Itanium 2 icc 8.0 is MUCH (sic) faster than gcc 3.3! ====================================================================== ERROR: test_basic (scipy.base.matrix.test_matrix.test_algebra) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/baecker/python2/scipy_lintst_atlas/lib/python2.4/site-packages/scipy/base/tests/test_matrix.py", line 99, in test_basic from scipy.basic import linalg File "/home/baecker/python2//scipy_lintst_atlas/lib/python2.4/site-packages/scipy/basic/__init__.py", line 6, in ? import linalg File "/home/baecker/python2//scipy_lintst_atlas/lib/python2.4/site-packages/scipy/basic/linalg.py", line 9, in ? import lapack_lite ImportError: /home/baecker/python2/scipy_lintst_atlas/lib/python2.4/site-packages/scipy/basic/lapack_lite.so: undefined symbol: for_cpstr The library `libifcore.so` provides the needed routine nm /opt/intel/fc_90/lib/libifcore.so.6 | grep for_cpstr 00000000000d6900 T for_cpstr (+libcxa is needed by this one ...) So the solution is to get the following compile command /work/home/baecker/python2/bin/g77 -shared build/temp.linux-ia64-2.4/scipy/basic/lapack_lite/lapack_litemodule.o -L/home/baecker/python2/lib/atlas -llapack -lptf77blas -lptcblas -latlas -lg2c -L /opt/intel/fc_90/lib -lifcore -lcxa -o build/lib.linux-ia64-2.4/scipy/basic/lapack_lite.so I.e.: -L/opt/intel/fc_90/lib -lifcore -lcxa has to be added and then all tests of scipy core work fine. How can one achieve this? Best, Arnd

1 0

Re: [SciPy-dev] Scipy.org
by Joe Harrington Dec. 22, 2005

Dec. 22, 2005

While I agree with Travis on the state of the web site content (so much out of date that it all bears going over before posting on the new site), the random approach is how we got the mess we have now in the first place. It wasn't Plone that did it, it was us (or the lack of us). We are about (I hope, I dream, if I were religious I'd pray) to blossom into a popular site, with the majority of visitors not being developers and maybe not being more than budding scientists. The organization that developers or experienced users want will not meet the needs of a high-schooler who is using scipy to do a homework assignment for the first time on her school computer. Organic design works when everyone's goals are in line. Site visitors' goals are divergent, so letting the site "grow" in a pure-wiki way will make finding your way around the site like navigating a rose bush: beauty all around, but too painful to get to. We have some web design experts among us. Let's design a site structure that has room to grow, and slot content into it, rather than letting the site design evolve organically. It makes sense (to me) to have some "start here if you're a ..." pages that present an introduction to the appropriate audience and then index into the main site content. How to present these without slowing down the experienced user or developer is the question. It's not hard to think up successful strategies, but they may depend on a particular underlying organization, so we should do a little thinking before we start. I suggest a two-pronged approach: 1. Set up a staging area for the content itself. It should have a flat organization: items go in its top level for discussion and review before being moved into the main site. Items shouldn't stay here long. Doing this lets conversion start before we have a site structure. 2. While (eager!) content authors are moving their stuff over to the Moin staging area and updating for the new scipy, we all have a discussion of site organization, including input from web experts. Then, set up the structure and move over the content. Meanwhile, keep the current plone site up, with a note at the top saying what's going on (new Scipy, numeric and numarray unified, new site in development, discussion on how to organize it HERE, content conversion HERE, nothing new on this site, thanks for your indulgence). Set up the Moin site as new.scipy.org, and hang the staging area off it as new.scipy.org/staging. Be open about our process but don't stick it in newbies' faces until there's something there to stick. There will be a switchover time, likely before the new site is polished up, but it's not now. To begin with, I've made two lists. The first is a list of potential site user types. The second lists categories of content. Website Accessors New, inexperienced user New user experienced with similar software (IDL, Matlab) Existing user looking for news updates/downloads software main add-on docs place to report problem/track problem reports talk with other users seek advice give suggestion help others search for past coverage of issue resources on a topic links elsewhere topical page Developer all of the above talk with other developers active sources (SVN) web posting Potential user Media Donor Advocate Potential customer/employer Website Areas "What is" summary page demos on web demos you can run links Getting Started install intro docs intro demos community links Download packages docs News Community mailing lists events presentations user area past events Topical Developers bug reports SVN Media kit Donating Advocacy kits Store FAQ Docs --jh--

3 4

Re: [SciPy-dev] Issues "freezing" scipy using py2app
by Josh Marshall Dec. 22, 2005

Dec. 22, 2005

On Wed, 21 Dec 2005, Pearu Peterson wrote: >> I have an application that is using PyObjC, matplotlib and scipy to >> perform some image classification and feature extraction. I am now >> attempting to use py2app to bundle it up into a single OS X .app. >> >> ... > > Thanks for the patch. I have applied it to new pkgload feature. > scipy does > not use _import_tools.py anymore. As soon as this is available in SVN I will test it out. Not using _import_tools.py should make it easier to package the app. I will see how I go. >> There must be a simpler way, such as using find_module, but I >> couldn't get this to work since it requires the parent package to be >> imported before the subpackages. > > info.py files should be regarded as information files about > packages, they > are python modules only for convinience. And using imp.load_module is > necessary to prevent importing the whole package, at the import > time we > need only the information from the info.py file. I think it is a good idea to hold this information in python modules. If it is held in a data file, this becomes problematic with regards to the different ways of distributing scipy, eg install from source, frozen in a zip. This is similar to the problem matplotlib had with where to put their data files. Keep as is, works well. >> 2) The second issue was a problem involving get_config_vars being >> called from ppimport.py. distutils then requires the presence of the >> Python config file, which doesn't exist in the bundle. >> >> This was easily fixed by adding >> >> if sys.platform[:5]=='linux': >> so_ext = '.so' >> + elif sys.platform=='darwin': >> + so_ext = '.so' > > Hmm, what exception do you get when Python config file is not present? > ppimport._get_so_ext uses so_ext='.so' when ImportError appears. > I think the proper fix would be to catch proper exception. It was a problem related to trying to include the entirety of distutils in my packaged application. The exception came from distutils. I will try catching the exception and see if this is a valid fix. >> 3) Finally, I can't distribute my .app as-is, after being generated >> by py2app. py2app operates by all needed modules from the system >> site- >> packages being collected into a "site-packages.zip" within the >> application bundle. At runtime, some magic with import causes >> everything to work, and this is where scipy falls down. When trying >> to import the subpackages using the _import_utils.py magic, it cannot >> find them within the site-packages.zip. I can fix this by >> distributing the .app with the site-packages.zip expanded into a >> directory. However, I was wondering if anyone who knows the import >> magic better knows if we can fix this. > > I'd like to learn more about the magic that Python does with .zip > files. > Are there similar issues when using eggs? I will look into this after the Christmas break. It would be nice to be able to package scipy as an egg, and also create a proper Mac installer. Merry Christmas to all. Josh

1 0

Guide to SciPy: Core System
by Nils Wagner Dec. 21, 2005

Dec. 21, 2005

Hi Travis, When is your Guide to SciPy: Core System http://www.tramy.us/ available ? Nils

2 1

Reviewing scipy.test package
by Pearu Peterson Dec. 21, 2005

Dec. 21, 2005

Hi, We have scipy.test package that currently contains only two modules: testing.py and auto_test.py. testing.py is widely used in scipy test sites. auto_test.py is rather old and does not work without serious review. It is tempting to remove it from scipy core svn but it has some interesting ideas. This raises a general question: Shall we clean scipy up or should we keep old-interesting-but-not-working codes for future ideas? There is also a name conflict with scipy.test. When importing scipy, scipy.test module is overwritten with ScipyTest.test method. To resolve this conflict, either the name of the module or the method needs to be changed. I see two options: 1) rename ScipyTest.test -> ScipyTest.runtest or similar 2) rename scipy.test -> scipy.testing People are used to run scipy.test(), so I am inclined to applying the option (2). Any objections? Pearu

2 1

scipy-0.8.4 SRPM patch available
by Neal Becker Dec. 21, 2005

Dec. 21, 2005

I have patched scipy SRPM to build scipy-0.8.4 on Fedora4, x86_64. Also builds on i386. Please update the SPEC file. Here is the SPEC file I used: Note: I don't currently have atlas installed. If I did, I guess I would also need to add something for python setup.py to find it. %define name scipy_core %define version 0.8.4 %define release 1 %define python_sitearch %(%{__python} -c 'from distutils import sysconfig; print sysconfig.get_python_lib(1)') Summary: Core SciPy Name: %{name} Version: %{version} Release: %{release} Source0: %{name}-%{version}.tar.gz License: BSD Group: Development/Libraries BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot Prefix: %{_prefix} Vendor: SciPy Developers <scipy-dev(a)scipy.org> Url: http://numeric.scipy.org %description UNKNOWN %prep %setup %build env BLAS=/usr/lib64 LAPACK=/usr/lib64 CFLAGS="$RPM_OPT_FLAGS" python setup.py build #env CFLAGS="$RPM_OPT_FLAGS" python setup.py build %install python setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES %clean rm -rf $RPM_BUILD_ROOT %files -f INSTALLED_FILES #for some reason this was missed! %{python_sitearch}/scipy/lib/_dotblas.so %defattr(-,root,root)

1 0

Different number of tests
by Nils Wagner Dec. 21, 2005

Dec. 21, 2005

Hi all, I have installed the latest svn versions of core and scipy on two different machines. On the first box scipy.test(1,10) Ran 1372 tests in 8.086s FAILED (failures=1) <unittest.TextTestRunner object at 0x47fed68c> On the second box Ran 1230 tests in 7.358s FAILED (failures=1) <unittest.TextTestRunner object at 0xb2908c8c> What is the reason for the difference in the number of tests (1372 versus 1230) ? Nils

2 3

concatenate with None problem
by Robert Cimrman Dec. 21, 2005

Dec. 21, 2005

In [4]:nm.concatenate( (nm.array( [1,2,3] ), None) ) *** glibc detected *** free(): invalid pointer: 0xb61d6380 *** (SIGABRT) In [3]:nm.__scipy_version__ Out[3]:'0.4.3.1492' In [4]:nm.__core_version__ Out[4]:'0.8.6.1691' details: ======== *** glibc detected *** free(): invalid pointer: 0xb7a67380 *** Program received signal SIGABRT, Aborted. [Switching to Thread 16384 (LWP 11847)] 0xb7ba60b1 in kill () from /lib/libc.so.6 (gdb) bt #0 0xb7ba60b1 in kill () from /lib/libc.so.6 #1 0xb7d8c1e1 in pthread_kill () from /lib/libpthread.so.0 #2 0xb7d8c55b in raise () from /lib/libpthread.so.0 #3 0xb7ba5e44 in raise () from /lib/libc.so.6 #4 0xb7ba730d in abort () from /lib/libc.so.6 #5 0xb7bd85bc in __fsetlocking () from /lib/libc.so.6 #6 0xb7be2417 in mallopt () from /lib/libc.so.6 #7 0xb7be10df in mallopt () from /lib/libc.so.6 #8 0xb7bdfccf in free () from /lib/libc.so.6 #9 0xb7e28c1b in PyObject_Free (p=0xb7a67380) at obmalloc.c:798 #10 0xb7a4ddfc in arraydescr_dealloc () from /home/share/software/usr/lib/python2.4/site-packages/scipy/base/multiarray.so #11 0xb7a385e1 in array_dealloc () from /home/share/software/usr/lib/python2.4/site-packages/scipy/base/multiarray.so #12 0xb7a52041 in PyArray_Concatenate () from /home/share/software/usr/lib/python2.4/site-packages/scipy/base/multiarray.so #13 0xb7a59ab4 in array_concatenate () from /home/share/software/usr/lib/python2.4/site-packages/scipy/base/multiarray.so #14 0xb7e24220 in PyCFunction_Call (func=0xb7abba6c, arg=0xb5f33e4c, kw=0x0) at methodobject.c:77 #15 0xb7e66857 in call_function (pp_stack=0xbf81f628, oparg=1) at ceval.c:3558 #16 0xb7e639a6 in PyEval_EvalFrame (f=0x8100894) at ceval.c:2163

1 0

new.scipy.org/Wiki and external ReST documents
by Pearu Peterson Dec. 21, 2005

Dec. 21, 2005

Hi, I have added http://svn.scipy.org/svn/scipy_core/trunk/scipy/doc/DISTUTILS.txt to http://new.scipy.org/Wiki/Docs but it would be nice if ReST-ed DISTUTILS.txt would be processed by Wiki engine. Are there any such features possible in Wiki? If not, what would be a good practice to make ReST documents available to scipy Wiki? Thanks, Pearu

2 2

Re: [SciPy-dev] fftpack issues - not just OS X
by Tom Loredo Dec. 21, 2005

Dec. 21, 2005

> This sounds like issues with the new package loader scheme that is being > introduced. This needs to be fixed urgently. I don't know if this is related, but it seems undesirable to me: >>> from scipy.stats import norm Failed to import fftpack No module named fftpack Failed to import signal No module named fftpack [i.e., importing "norm" this way works as expected; but...] >>> from scipy.random import normal Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named random >>> import scipy.stats as stats >>> stats <module 'scipy.stats' from '/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/ stats/__init__.pyc'> >>> import scipy.random as random Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named random But random is certainly there: >>> from scipy import * >>> random <module 'scipy.basic.random' from '/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/ scipy/basic/random.pyc'> >>> random.normal <built-in method normal of mtrand.RandomState object at 0x3e0d0> >>> from scipy import random >>> -Tom ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/

1 0