Hi everyone,
I've spoken with Sam and Jeff and done some testing on my own, and we
now have the following pieces of information:
x86_64 machines at SLAC: clean compile
Triton: clean compile
Kraken: clean compile for me, Jeff had issues he's testing now
Ranger: clean compile
OS X: No tests yet, but I'd love it if someone would run this (my
laptop doesn't have enough space right now for scratch)
I've tarred up binary distributions for Triton, Kraken and x86_64
(RHEL4) and placed them here: http://raft.enzotools.org/files/ You
should be able to download these, untar them, and just run raft or
python2.6 or whatever. I have tested this, but only as the compiling
user.
If you compile RAFT on any other platforms, I'd love it if you could
tar it up with either the name of the supercomputing installation (as
above for Triton and Kraken) or the OS/Arch and put them somewhere I
can snag them, and I'll put them up on the website.
If you have any more feedback, that'd be awesome. I'm working right
now on understanding the SAGE notebook to see the best way of
distributing tutorial worksheets.
Thanks,
Matt
On Sat, Sep 5, 2009 at 11:51 PM, Matthew Turk
Hi everyone,
As some of you know, I've been working on a fork/port/rebranding of the FEMhub(dot org) framework. FEMhub itself is a fork of SAGE, which you can find at sagemath.org. Basically, SAGE and FEMhub are designed to be fully self-contained, web-based interfaces to finite element computation systems (FEMhub) and symbolic math systems (SAGE). Both are pretty amazing, and SAGE itself is a strong competitor with Maple and Mathematica.
With that out of the way, I'd like to take a moment to focus on a couple aspects of this. I saw this talk on FEMhub at SciPy 2009 by Ondrej Certik (more from Ondrej here: http://ondrejcertik.blogspot.com/2009/08/scipy-2009-conference.html ) where he described the development of FEMhub from a project called SPD (source python distribution.) These projects started as stripped down versions of SAGE, but now serve as generic Python installation mechanisms. The upside of this is that these projects can easily install complete python distributions, there is a large selection of existent packages available, and they even can (optionally) present web-interfaces to the complete python installation. I'll return to the idea of the web-interface in a moment, because at first I didn't really care about that but I have since come to. FEMhub actually has created and includes off-screen rendering mechanisms for VTK and OpenGL which work through the web notebook, which I think is pretty neat. However, you don't *have* to use it through the web -- all the python and everything you're familiar with works out of the box, and (best yet) it handles all the setting of library-loading and environment variables.
So here's what's up: I've taken FEMhub and rebranded it with a new name, RAFT. I've stripped out a bunch of the finite element codes and added in HDF5, h5py, and yt. In doing so, I believe I have completely replicated *and supplemented* the installation script, using a framework that has been externally developed and which continues to be developed externally. My installation scripts pale in comparison to the quality control that the SAGE project demands -- their system is much more robust than what we use, and it includes more packages. The RAFT project now benefits from this QA effort.
In the past, we've avoided including any code in yt that even *uses* scipy, simply because scipy is traditionally so difficult to install. FEMhub/SAGE/RAFT simply *work*, and they come with SciPy. Additionally, they come with the entire MayaVi distribution, which is not only able to render, but to render off-screen the need be. We get a lot of fun stuff with this, and it works on a ton of different systems without any hackery on the part of the user.
Now, if I can speak for a moment about the web notebook. This is something I still haven't figured out my feelings on, but I can tell you all this: it works seamlessly with yt with no modifications, which is more than I can say for other web-based interaction mechanisms I have tried. I was able to import yt, load a data file, add plots, and then whenever "save" is called they just show up in the web browser. Additionally, because it binds only to localhost and contains authentication mechanisms, this is accessible through ssh forwarding and should be considered feasible for running at supercomputer centers.
Okay. So. I think the modifications all work, and I've rebranded much of the Notebook. I have a couple more decisions to make on the back end about package distribution.
* Do we install VTK/MayaVi by default? This adds some compilation time. But, it also enables us to do 3D rendering more easily. I'm inclined to say no, because it can be automatically downloaded/installed by the raft command. * Do we completely replicate the SAGE spkg distribution mechanism on enzotools.org? This would enable us to have finer-grained control as well as to distribute personal packages, but we'd also have to stay up to date with upstream bug fixes *manually*. * What role should the notebook play? I was immediately and extremely opposed to this when I first saw it; but I have since warmed to it considerably, because of the ease-of-development, inline help, repeatable analysis, export/import capabilities and PDF output. It seems likely to me that this would be a very easy mechanism for new users, and we can distribute tutorial sheets *with* the RAFT, which users can step through on their own to learn both Python and yt. * Does anyone else want to help keep up with changes upstream, help with rebranding, and all-in-all test this out?
SO! On to the testing part.
You can test the RAFT 0.1 alpha distribution by snagging it here:
http://raft.enzotools.org/files/raft-0.1a.tar.bz2
You should ensure that no YT files are in your PATH or LD_LIBRARY_PATH -- we're doing this all fresh. Then,
$ tar xvfj raft-0.1a.tar.bz2 $ cd raft-0.1a $ make
and you should get an install, a little bit later. On triton it was pretty fast. You can now set your PATH to include /wherever/raft-0.1a/local/bin and when you run python2.6 you should get a yt installation. Now, you should be able to run anything you like. Running the command "raft" inside the raft-0.1a directory will give you a modified IPython. It might be fun to start that up, and type:
notebook(port=[something])
where [something] is some unused port. I like 7741. On your local machine, you can then type:
$ ssh -L 8000:localhost:7741 remote-host.org
and when you open up a web browser and point it to localhost:8000, you'll get your notebook interface. It's authenticated, so you'll need a u/pw combo, but the first time you start the notebook it'll prompt you. You can then add plots or whatever and anaylze any data accessible to you on that system. When a new PNG file is added (i.e., via the save() command on a PlotCollection) it will get included in the cell directly. Be sure to ctrl-c out when you're done. The "raft" command is pretty powerful, and you should also be able to install things with it.
Okay, that's a long enough email. I'm going to work on this a little bit more, but I'd *REALLY* appreciate it if you guys could just try installing and using it on a machine or two that you have access to. I intend for this to replace the installation script, but until it's definitely working, we can't really do that. :)
Thanks!
-Matt