Mailman 3 March 2002 - SciPy-Dev

preparing for release 0.2!
by eric 01 Apr '02

01 Apr '02

Hello all, Travis Oliphant and I had a talk today, and it sounds like SciPy 0.2 is about baked and ready to come out of the oven. The main things we wanted to add for this release were linalg and a cleaned up stats module. Both are pretty close to completion thanks to a ton of work by Pearu Peterson and Travis O. Seeing this is the case, I'd like to shoot for a 0.2 release candidate by Friday, April 5th. Here's a partial list of the things I know of before release: 1. Re-factor scipy_lite module (pending discussion). 2. Fix floating point NaN issues on multiple platforms (cephes module). 3. Have several stats people review the stats module 4. Add lu_factor and lu_solve (qr_factor and qr_solve, etc?) pairs to linalg. 5. Clean up weave documentation to match current implementation. 6. Update install instructions. 7. Test on multiple platforms. Feel free to add items -- this is meant as a starter list. Please don't put things like "document SciPy" or "complete set of unit tests." These aren't priorities for this release. If pydoc versions of the documentation get in, that's great. If they don't that is OK also. I'm most interested in getting something out there so that people have pre-built's for 2.2 (on Windows) and (reasonably) stable tarballs for Unix. I guess Pearu and Travis O. are the main guys to add other "todo" items. Also, we should coordinate a 2-4 hour period for one or two days next week on ICQ or IRC to work as a group on final clean up. Right now, I'd say Monday and Thursday are the best for me. Will that work for yall? Pearu, what time of day works best for you? I'm excited about getting this one out, because it is getting close to having a "functionally complete" core now. The next revision we can concentrate on docs and testing (everyone's favorite I know...). see ya, eric -- Eric Jones <eric at enthought.com> Enthought, Inc. [www.enthought.com and www.scipy.org] (512) 536-1057

5 8

Is the SciPy website down?
by rob 28 Mar '02

28 Mar '02

I was going to reinstall Weave and send you the verbose output, in the case where the compiler can't find stdc++. But your site is dead here. Rob. -- ----------------------------- The Numeric Python EM Project www.pythonemproject.com

2 1

Some thoughts on packaging SciPy
by Prabhu Ramachandran 28 Mar '02

28 Mar '02

hi, While I believe that its nice to have everything in one big package, I dont think that it fits the bill all the time. For instance I think weave, all the plot utilities, gui_thread and scipy_distutils and others do not belong in the scipy namespace. They are useful in scopes far beyond scipy. This is surely obvious to you all but I'd like to belabour it. scipy is of use to scientists and engineers but even we do not do scientific programming all the time. Many modules have use outside the scientific world. These should not be locked into scipy. OTOH, anything that is related to science/numerics should be part of scipy. It only makes sense. Consider plt. I'm sure its possible that a sys admin would like to use plt to plot stuff (maybe log files?). There is nothing scipy specific to wanting to plot data. Similarly with weave and maybe others. These are modules that should not be part of the scipy package. However, they should definitely be bundled as part of the scipy distribution. Hence the distinction between the scipy python package/namespace and the scipy distribution. I think they are different things and need to be noted. However, as I write this I realize that one viewpoint would be, "Why dont you just install the whole of scipy and just use what you want?" The problem with this approach is that its not possible to install scipy without other requirements. If someone wants just plt, they'd have to figure out the atlas related issues or any others that will arise in the future (and these will not be easy issues). I think there is only one way of dealing with this. We need a tool like cpan or ciphon. Or maybe not even that. Something like this might suffice. (1) Each module has a set of dependencies which it advertises in some file. (2) setup.py has a config file (or command line switches) that lets you select what packages you want installed. (3) setup.py also does not install anything that fails dependencies. For instance if someone wanted plt alone they select plt. plt ropes in gui_thread and requires wxPython. If wxPython is avlbl. plt and gui_thread are installed if not it is not installed. So the users can pick and choose what they want and install things with the least hassle. Once this is done it really does not matter if the modules are installed inside the scipy python package/namespace or outside it. The idea is somewhat like rpms and debs. There is one big source package from which smaller components are installable separately. I think this is definitely implementable but dont know how hard it should be to do. My apologies for not contributing any code that addresses these things. I thought that the least I can do is contribute ideas. prabhu

1 0

Re: [SciPy-dev] Some concerns on Scipy development
by eric 27 Mar '02

27 Mar '02

Hey Pearu, I've have thought about this a little lately also. There is a philosophical difference to packaging among the scientific developers. Some wish for small single purpose and stand alone packages that are installed one by one. Others wish for a single "standard library" of scientific tools that, once installed, is a one stop shop for a large number of scientific algorithms. There are benefits to both. However, I come squarely down in the second camp. A monolithic package is easier to install for end users, and it solves compatibility issues (such as SciPy changing the behavior of Numeric in some places). I believe the existence of such a package is required before there can be a mass conversion of engineers and scientist to Python as their tool of choice for daily tasks. This is the goal of SciPy. That said, the monolithic nature does pose some problems occasionally complicating development and following the CVS as Jochen has pointed out. Also, some may want to use some modules (weave, integrate, linalg) outside of scipy. This is useful in cases where you want to minimize install size or use something like py2exe (which currently doesn't work with SciPy). We should facilitate separating out packages when it is convenient, but not when it requires duplication of code or a lot of extra work. Perhaps re-organizing the architecture can make it convenient more often and come closer to making both camps happy. I agree that, when possible, it is nice to develop packages independent of SciPy -- that is how weave was developed. Later it was folded into SciPy, but it still runs separately. The new build structure (with separate setup_xxx.py files for each sub-package) implemented several months ago was developed, in part, to facilitate this sort of thing. Weave was easy in this respect because it doesn't need many numeric capabilities. So, I think this is a worthy goal for *some* of the modules (notably the ones people are discussing such as integrate, linalg, etc), with one caveat. These modules need access to some functions provided by scipy and will need to import at least one extra module. Scanning linalg, the needed functions are amax, amin, triu, etc. and a handful of functions subsumed from Numeric as well as some constants from scipy.limits. I consider it a bad idea to replicate these functions across multiple modules because of the maintenance issues associated with duplicate code. I don't want to go down that path. However, one thought is to make the idea of "levels" more explicit. We could define a package called "scipy_lite" or "scipy_level0" that would subsume Numeric and add the helper functions that are often used. It would not reference other scipy modules. This package would live in the scipy development tree, but would install as a separate package. So scipy_level0 would sit next to scipy in the site-packages directory. scipy_level0 would be easy to build without major dependencies -- much like Numeric. It would hold fastumath and maybe a few other extension modules, but it would be predominantly python code. The linalg, integrate, etc modules would import scipy_level0 instead of scipy. This way, people only have to port scipy_level0 instead of the whole of scipy if they want to use integrate in their package. I can't imagine much dissent about this approach by the people wanting single packages. If your willing to install Numeric and linalg on your own, then you should be willing to install the scipy_level0 package. Installing scipy_level0 outside the scipy package has some precedence since we've already done this with scipy_distutils and scipy_test. I'd rather leave it as a sub-directory of scipy, but pulling it out is necessary because of the way that python handles (or doesn't handle...) imports within packages -- that is if we want to make it easy to use sub-modules of scipy separately. So this is what the site-packages view of scipy would be: site-packages scipy_distutils scipy_test scipy_level0 subsumes and customizes Numeric handy.py misc.py scimath.py Matrix.py (?) fastumath.so (pyd) etc. scipy subsume scipy_base everything else In regards to higher level modules that use fft, svd, and other complex algorithms, they are just gonna have to import scipy. This requires some discussion before we make the change. It's also gonna require someone to step up and implement the change -- though it probably isn't a major effort. eric -- Eric Jones <eric at enthought.com> Enthought, Inc. [www.enthought.com and www.scipy.org] (512) 536-1057

4 6

More info on 100x100x100 FDTD's with Weave
by rob 27 Mar '02

27 Mar '02

The more info is that there is no more info that makes sense. Verbose=2 gave me nothing. When I switch from 80 cubed to 100 cubed region, as soon as the program hits the first Weave statement, my box starts paging with no CPU activity. It finally maxes out at 300 Meg swap and kills the app. No verbose debugging info to be found. Plain Numpy works fine with this sim at 300Meg+ memory. It does require a lot of swap on my HD, but there is plenty left to run Netscape, X11, etc. I have posted the program on my web site. Hit the "Whats New" button. Its called UFOblitz.py since I am doing a quasi-static analysis of those HV flying craft that were recently on Slashdot. If someone else can test this program and prove my results I will be happy. Otherwise this might be a FreeBSD problem. Just run the program normally, then run again after changing the "cells/meter" variable to 100. (Its in a 1M cubed sim space) Thanks, Rob. -- ----------------------------- The Numeric Python EM Project www.pythonemproject.com

1 0

Some concerns on Scipy development
by Pearu Peterson 27 Mar '02

27 Mar '02

Hi, I have some concerns about how different Scipy modules are dependent on each other by a global scipy namespace, though there is often little need for this dependency. Namely, most of the current modules import scipy and as result one can say that effectively all modules import each other. Such a policy is not good because of the following reasons: 1) It is generally a good style of programming if a module imports only thouse resources that it actually uses. This helps, for example, in localizing and fixing bugs in a particular module level, which will be much easier when doing it in a global scipy level. Testing routines should be particullary careful not importing irrelevant modules, escpecially, if these are also under testing. For a particular example, many modules import MLab or scipy just to access only the resources in Numeric. I would remove all such imports in favour of importing Numeric directly. 2) There has been wished (see numpy-discussion list) that scipy wouldn't be such a big monolitic environment. Instead, it should be possible to install Scipy subpackages as standalone packages if that is possible in principle. It has an advantage that other projects could use parts of Scipy codes without actually building the whole scipy. Or is this the Scipy policy to force installing/using either the whole scipy package or nothing from it? I think that then some experts will not use scipy at all (also I find it less stimulating to contribute to scipy). 3) The idea of different levels in SciPy is not working if the lowest level modules import everything. It would be less confusing to drop the idea of leveling from start -- I am not proposing to acctually go for it. Instead, as I see it, Scipy should be rather thin interface between an user and modules that it provides, with some additional (high level) convinience functions (that may import "everything"). Lower level modules should not depend each other unless there is a good reason for that. I wonder how other Scipy developers feel about these issues? Do you think that my concerns are somehow relevant or not? Is it acceptable trying gradually to turn lower level modules to be independent from each other if possible? Thanks, Pearu

4 6

Problems with CVS Scipy on Solaris
by Tim Lahey 26 Mar '02

26 Mar '02

Hi, I checked out the Scipy from CVS and had a number of difficulties in compiling. 1. Is there a way of using the Vendor LAPACK/BLAS instead of ATLAS? For the cases I'm concerned about, the Sun BLAS outperforms ATLAS. 2. I compiled ATLAS and managed to get it past the system_info check. But when I get to the X11 check, it says NOT FOUND then I get a traceback. 2a. Changing the directory list in the x11_info doesn't seem to work. 2b. I still couldn't get it to pick up the header and library files when I added the right directory. The tests were done on a Sun Blade 100 running Solaris 8. Cheers, Tim --- Tim Lahey PhD Student - Systems Design Engineering tjlahey(a)cgl.uwaterloo.ca

1 0

More experimenting continues
by rob 26 Mar '02

26 Mar '02

No, it turns out that an 80x80x80 sim uses the same memory with blitz() as with plain Numpy. I will try your debugging suggestion. Rob. -- ----------------------------- The Numeric Python EM Project www.pythonemproject.com

1 0

I'm doing some more experimenting
by rob 26 Mar '02

26 Mar '02

It turns out that what is happening is that blitz() is creating some overhead which makes my memory consumption higher. That is why a 100x100x100 wont work on my laptop with blitz, while is works with plain numpy. I dropped it down to 80x80x80, and now everything is Ok. I will try to get an exact amount for the extra memory overhead using top for 80x80x80. Right now top says 188Meg for the blitz version. I will have to make another file to use plain python. It is definitely true that the exact same blitz libraries work with the larger arrays, until I hit my swap limit. Rob. -- ----------------------------- The Numeric Python EM Project www.pythonemproject.com

1 0

Hi Eric
by Lytle, Robert TQO 26 Mar '02

26 Mar '02

The stdc++ thing doesn't bother me as much as the large array problem. I use blitz() three times for enclosing the field update equations in FDTD. Typically the compile uses up 200Meg ram and 200Meg swap, so its pretty slow. But afterwards it runs like a bat out of hell. But when I upped the matrix size it suddenly needed more memory for the compile, and crashed. Without blitz() this project consumes 400MB, since it has many other matrices for the UPML. Now here at work, I have a 1.8Ghz P4 with 768MB ram. I tried out blitz() with MingW and I get stack overflows. Unlike gcc, it can't handle such large compiles. Rob.

2 1