
SciPy's build process has been revamped. Thanks to Pearu for tons of help in this area on getting this all setup. Please grab the CVS and test on your given architecture so we can find out what has broken. Please report both success and failure. You'll need the latest snapshot of f2py (see below). Also, you'll need to set the following line in setup.py to point to your atlas libraries: scipy_distutils.atlas_info.library_path = ['home/ej/lib/atlas'] Then: python setup.py build python setup.py install python >>> import scipy >>> scipy.test() The main purpose of the reorg was (1) to modularize the build process so that it is easier to maintain, (2) make it easy for others to build packages that can be dropped into SciPy with minimum fuss, and (3) Allow for building and installing individual modules outside of SciPy without changing any install scripts. There are notes about how to do (2) at the bottom of this file. thanks, eric new dependencies: f2py2e (http://cens.ioc.ee/projects/f2py2e/) The latest snap shot is needed and is here: http://cens.ioc.ee/projects/f2py2e/rel-5.x/ This must be installed to build SciPy -- it has a setup.py script so it is easy. old dependencies: You still need these. atlas fftw changes: * scipy_distutils is a new package (installed by scipy, but as a separate package) that provides support for Fortran, f2py, and most of the other customizations that scipy uses for distutils. It is also used by Pearu's f2py package, and he's made a ton of contributions here. * setup.py is now very compact. Most setup work is delegated to sub-packages. * each sub-package has a setup_foo.py file in it that has a configuration() method. This returns a dictionary of key word arguments that can be passed to setup.py to build the package. A simple example can be found in fastumath/setup_fastumath.py. More complex examples are xplt/setup_xplt.py and linalg/setup_linalg.py. * scipy_distutils.atlas_info attempts to find your atlas library installation. It is pretty dumb right now. It has a variable called library_path that you can use to set the path to your atlas directory. Note that it takes a list, not a string as the argument: import scipy_distutils.atlas_info scipy_distutils.atlas_info.library_path = ['/home/ej/lib/atlas'] fftw_info.py serves the same purpose for fftw. Still need to add library_path to it. * scipy.compiler has been renamed to scipy.weave. It has been significantly enhanced in its capability. (still a few bugs here though.) * Numeric is no longer installed by the setup.py script. You'll have to install it separately (we'll continue to include it in the windows executables at point releases) * fastumath is installed as a separate module instead of being dropped in the Numeric directory. * scipy_test.py is now a package (do to distutils issues) and installed as a separate stand-alone package. * SciPy now builds with compilers other than g77 on Linux. Not much testing has been done yet with these: * Intel Fortran (by Pearu Peterson) * Absoft 5.0 (compiles, but still problems with upper/lowercase and underscore issues) ------------------ To include the package foo within SciPy, you'll need to place the foo directory within the SciPy directory, and create a setup_foo.py script within that directory: scipy/ foo/ foo1_source.c foo2_source.c __init__.py foo_script.py setup_foo.py setup_foo.py should have a single method in it called configuration() that looks like: #setup_foo.py import os from scipy_distutils.core import Extension from scipy_distutils.misc_util import get_path, default_config_dict def configuration(parent_package=''): # if we have a parent module, append a '.' to it # to simplify adding it to package names, etc. if parent_package: parent_package += '.' # find the local path of this module (/home/eric/scipy/foo) local_path = get_path(__name__) # grab a default configuration dictionary. We'll fill this # with this packages specifics. config = default_config_dict() # Set the name of this package. config['packages'].append(parent_package+'foo') # build the first extension module sources = ['foo1_source.c'] sources = [os.path.join(local_path,x) for x in sources] ext = Extension(parent_package+'foo.foo1',sources) config['ext_modules'].append(ext) # build the second extension module sources = ['foo2_source.c'] sources = [os.path.join(local_path,x) for x in sources] ext = Extension(parent_package+'foo.foo2',sources) config['ext_modules'].append(ext) # return the configuration information. return config Second, you have to add the name of the module 'foo' to one of the installation lists in setup.py. # setup.py (in scipy directory) . . . # standard modules standard_packages = ['io','linalg','special','signal','stats', 'interpolate','integrate','optimize', 'cluster','cow','ga','weave',] . . . That is about it. The build process should detect your package and install it as a sub-package of scipy. If you'd like to bundle your package with SciPy, but want to have it installed as a stand-alone package, put it in the following list of packages: # setup.py (in scipy directory) . . . # these packages aren't nested under scipy separate_packages = ['gui_thread','scipy_test','scipy_distutils', 'fastumath'] . . . The default_config_dict function looks like this: # scipy_distuils/misc_util.py . . . list_keys = ['packages', 'ext_modules', 'data_files', 'include_dirs', 'libraries', 'fortran_libraries', 'headers'] dict_keys = ['package_dir'] def default_config_dict(): d={} for key in list_keys: d[key] = [] for key in dict_keys: d[key] = {} return d . . . Let me know if the list of key words needs to be augmented to support something specail you are doing.

Hi,
To include the package foo within SciPy, you'll need to place the foo
<snip>
def configuration(parent_package=''): # if we have a parent module, append a '.' to it
<snip>
# return the configuration information. return config
To achive a better modularization of scipy, I would suggest the following addition to the setup_foo.py files: if __name__ == "__main__": from scipy_distutils.core import setup setup(**configuration()) so that when setup_foo.py build is run from foo/ directory, it must build and the resulting module must be usable without scipy. I think it makes it also easier to test submodules of scipy from their directories rather than editing the top level setup.py to disable other packages in order to just build the one module under testing. I realize that this might not be possible for all modules (due to scipy leveling scheme) but the lowest level modules as the most important ones should have more convinient testing facilities. What do you think? Regards, Pearu

Hey Pearu,
if __name__ == "__main__": from scipy_distutils.core import setup setup(**configuration())
<snip>
What do you think?
I think it is good. I'll add it. I have usually just added setup.py module to the packages that does exactly this. That is just because most people expect the "setup.py" to be the name of the setup script. Either (or both) are good ideas. eric

"eric" == eric <eric@scipy.org> writes:
eric> SciPy's build process has been revamped. Thanks to Pearu eric> for tons of help in this area on getting this all setup. eric> Please grab the CVS and test on your given architecture so eric> we can find out what has broken. Please report both success eric> and failure. eric> You'll need the latest snapshot of f2py (see below). Also, eric> you'll need to set the following line in setup.py to point eric> to your atlas libraries: eric> scipy_distutils.atlas_info.library_path = eric> ['home/ej/lib/atlas'] The new build seems to run fine and I didn't need to change the above flag. I also dont like this setting since you'll have trouble with CVS versioning when you change setup.py. The next time setup.py is changed in the CVS tree you risk the complication of a conflict. I'd suggest placing user configs in a separate file that is not in the CVS tree. prabhu

The new build seems to run fine and I didn't need to change the above flag. I also dont like this setting since you'll have trouble with CVS versioning when you change setup.py. The next time setup.py is changed in the CVS tree you risk the complication of a conflict. I'd suggest placing user configs in a separate file that is not in the CVS tree.
Good idea. Maybe the setup.cfg file is a good choice. I'm hoping that the atlas_info script gets more itelligent -- perhaps asking you for the path if it can't find it. It could then generate the file for you. If it didn't find the library, in the best scenarios, it would download it from the scipy website. This is probably a little ways out though. eric

"eric" == eric <eric@scipy.org> writes:
>> complication of a conflict. I'd suggest placing user configs >> in a separate file that is not in the CVS tree. eric> Good idea. Maybe the setup.cfg file is a good choice. eric> I'm hoping that the atlas_info script gets more itelligent eric> -- perhaps asking you for the path if it can't find it. It eric> could then generate the file for you. If it didn't find the That sounds like a good idea. eric> library, in the best scenarios, it would download it from eric> the scipy website. This is probably a little ways out eric> though. Maybe once ciphon (http://sourceforge.net/projects/pythonsiphon/) gets good enough we could use it. Also, I noticed on the weave documentation page that you were looking for a way to check if given an xterm if there is an X11 display available. The easiest way to do this is to check for the DISPLAY env variable. In [1]: import os In [2]: print os.environ['DISPLAY'] :0.0 On a remote telnet session this will cause a KeyError. $ python -c "import os; print os.environ.get('DISPLAY')" None Eric, weave.blitz is fantastic! Great job!! I haven't really checked out weave.inline yet but like blitz a lot. :) One thing I thought I'd mention. I'm sure you are aware of this but I think it worthy of mention. When you use blitz for an expression the computation is different from that when done by numeric. blitz will use the computed values immediately (Gauss-Siedel) while numeric will make a copy and do the computation (Gauss-Jordan). This means that sometimes the result of a computation when done by exec(expr) will be different from that of blitz(expr). Here is an example. # 4 point average.
expr = "u[1:-1, 1:-1] = (u[0:-2, 1:-1] + u[2:, 1:-1] + "\ ... "u[1:-1,0:-2] + u[1:-1, 2:])*0.25" u = zeros((5, 5), 'd'); u[0,:] = 100 exec (expr) u array([[ 100., 100., 100., 100., 100.], [ 0., 25., 25., 25., 0.], [ 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0.]])
u = zeros((5, 5), 'd'); u[0,:] = 100 weave.blitz (expr) u array([[ 100. , 100. , 100. , 100. , 100. ], [ 0. , 25. , 31.25 , 32.8125 , 0. ], [ 0. , 6.25 , 9.375 , 10.546875 , 0. ], [ 0. , 1.5625 , 2.734375 , 3.3203125, 0. ], [ 0. , 0. , 0. , 0. , 0. ]])
This is not a bad thing at all but worth knowing, IMHO. prabhu

Hey Prabhu,
eric> library, in the best scenarios, it would download it from eric> the scipy website. This is probably a little ways out eric> though.
Maybe once ciphon (http://sourceforge.net/projects/pythonsiphon/) gets good enough we could use it.
Yes, it might. Has anyone played with it at all? On the other hand, Python's ftp/http support is so good, that it only takes a few lines of code to handle the download. The architecture/CPU detection is more difficult, but not to bad. Still, if ciphon handles this, then know need to duplicate.
Also, I noticed on the weave documentation page that you were looking for a way to check if given an xterm if there is an X11 display available. The easiest way to do this is to check for the DISPLAY env variable.
In [1]: import os In [2]: print os.environ['DISPLAY'] :0.0
On a remote telnet session this will cause a KeyError.
$ python -c "import os; print os.environ.get('DISPLAY')" None
Good. Thanks. I'll try and work this into the code in the needed places.
Eric, weave.blitz is fantastic! Great job!! I haven't really checked out weave.inline yet but like blitz a lot. :)
Glad it is working for you.
One thing I thought I'd mention. I'm sure you are aware of this but I think it worthy of mention. When you use blitz for an expression the computation is different from that when done by numeric. blitz will use the computed values immediately (Gauss-Siedel) while numeric will make a copy and do the computation (Gauss-Jordan). This means that sometimes the result of a computation when done by exec(expr) will be different from that of blitz(expr). Here is an example.
This is true. I definitely need to mention this in the documentation somewhere. It can happen if values on the RHS of the equation are also on the LHS of the equation -- but only in certain circumstances like your example. Thanks for pointing this out. eric

On Thu, 10 Jan 2002, eric wrote:
In [1]: import os In [2]: print os.environ['DISPLAY'] :0.0
On a remote telnet session this will cause a KeyError.
$ python -c "import os; print os.environ.get('DISPLAY')" None
Good. Thanks. I'll try and work this into the code in the needed places.
However, there must be some better way for that as checking DISPLAY is not very reliable. For example, one can have DISPLAY variable not set also in X terminals: pearu@localhost:~$ python -c "import os;print repr(os.environ.get('DISPLAY'))" ':0.0' pearu@localhost:~$ unset DISPLAY pearu@localhost:~$ python -c "import os;print repr(os.environ.get('DISPLAY'))" None Regards, Pearu

"PP" == Pearu Peterson <pearu@cens.ioc.ee> writes:
PP> However, there must be some better way for that as checking PP> DISPLAY is not very reliable. For example, one can have PP> DISPLAY variable not set also in X terminals: I think that it is not too bad if DISPLAY is not set. (1) In such a case you cannot run an X application properly so you cant run the tests anyway. well, technically you could by passing -display but seldom do folks do it. $ unset DISPLAY $ display # an X11 app display: Unable to connect to X server () [Function not implemented]. (2) Even if you ssh to a remote host and enable X11 forwarding the DISPLAY variable is automatically set. (3) I dont know why one would want to unset DISPLAY. :) prabhu

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 10 Jan 2002 23:01:56 +0530 Prabhu Ramachandran wrote: Prabhu> (3) I dont know why one would want to unset DISPLAY. :) 'cause you don't want programs to spawn their X-interface over a modem line. Numerous (X)Emacs sessions from home come to mind... Greetings, Jochen - -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Liberté, Égalité, Fraternité GnuPG key: 44BCCD8E Sex, drugs and rock-n-roll -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Processed by Mailcrypt and GnuPG <http://www.gnupg.org/> iD8DBQE8Q3DCiJ/aUUS8zY4RAkkaAKCAwfHtXmC/GtvcPvVYuh42YvMlNQCgkVaJ 1mIzPux6CbAUfXWgyneOwXE= =5qvc -----END PGP SIGNATURE-----

"JK" == Jochen Küpper <jochen@jochen-kuepper.de> writes:
Prabhu> (3) I dont know why one would want to unset DISPLAY. :) JK> 'cause you don't want programs to spawn their X-interface over JK> a modem line. Numerous (X)Emacs sessions from home come to JK> mind... Yes, but would you want scipy to test the gui related things over that same modem line? My guess is no. Therefore using $DISPLAY to check if X is up or not and then decide to run the GUI tests seems to be a good idea. prabhu

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 15 Jan 2002 07:25:26 +0530 Prabhu Ramachandran wrote:
"JK" == Jochen Küpper <jochen@jochen-kuepper.de> writes:
Prabhu> (3) I dont know why one would want to unset DISPLAY. :) JK> 'cause you don't want programs to spawn their X-interface over JK> a modem line. Numerous (X)Emacs sessions from home come to JK> mind... Prabhu> Yes, but would you want scipy to test the gui related things over that Prabhu> same modem line? My guess is no. Oh, I actually fully agree, point taken! Don't know what I was thinking before. Greetings, Jochen - -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Liberté, Égalité, Fraternité GnuPG key: 44BCCD8E Sex, drugs and rock-n-roll -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Processed by Mailcrypt and GnuPG <http://www.gnupg.org/> iD8DBQE8Q6tqiJ/aUUS8zY4RAmgUAKCOtwpZG33T2DIYCHuGJ559vyKOEACguEV1 mcSPXChHKRU0cZh9k1SO2y8= =l9VV -----END PGP SIGNATURE-----
participants (4)
-
eric
-
Jochen Küpper
-
Pearu Peterson
-
Prabhu Ramachandran