On 16/11/2007, Rahul Garg <rahulgarg44@gmail.com> wrote:

It would be awesome if you guys could respond to some of the following questions : a) Can you guys tell me briefly about the kind of problems you are tackling with numpy and scipy? b) Have you ever felt that numpy/scipy was slow and had to switch to C/C++/Fortran? c) Do you use any form of parallel processing? Multicores? SMPs? Clusters? If yes how did u utilize them?

If you feel its not relevant to the list .. feel free to email me personally. I would be very interested in talking about these issues.

I think it would be interesting and on-topic to hear a few words from people to see what they do with numpy. a) I use python/numpy/scipy to work with astronomical observations of pulsars. This includes a range of tasks including: simple scripting to manage jobs on our computation cluster; minor calculations (like a better scientific calculator, though frink is sometimes better because it keeps track of units); gathering and plotting results; prototyping search algorithms and evaluating their statistical properties; providing tools for manipulating photon data; various other tasks. We also use the software package PRESTO for much of the heavy lifting; much of it is written in python. b) I have projects for which python is too slow, yes. Pulsar surveys are extremely compute-intensive (I estimated one that I'm involved with at two or three mega-core-hours), so any software that is going in the pipeline should have its programming-time/runtime tradeoff carefully examined. I wrote a search code in C, with bits in embedded assembler. All the driver code to shuffle files around and supply correct parameters is in python, though. PRESTO has a similar pattern, with more C code because it does more of the hard work. In most cases the communication between the heavy-lifting code and the python code is through the UNIX environment (the heavy-lifting code gets run as a separate executable) but PRESTO makes many functions available in python modules. On the other hand, I often write quick Monte Carlo simulations that run for a day or more, but since writing them takes about as long as running them, it's not worth writing them in a language that would run faster. c) Our problems tend to be embarrassingly parallel, so we tend not to use clever parallelism toolkits. For the survey I am working on, I wrote one (python) script to process a single beam on a single node (which takes about thirty hours), and another to keep the batch queue filled with an appropriate number of such jobs. I have thought about writing a more generic tool for managing this kind of job queue, but haven't invested the time yet. Anne

Hi Rahul,

a) Can you guys tell me briefly about the kind of problems you are tackling with numpy and scipy?

I'm a grad student "doing" computational biology. I primarily use the NumPy/SciPy/matplotlib triumvirate as a post processing tool to analyze what the heck happened after we run some learning algorithms we develop (or canned ones, like libsvm (for example)) to look for some sense in the results. I've been working w/ analyzing interaction networks/graphs, so I also use NetworkX[1] quite a bit as well (it's also a nice package w/ responsive authors). Many of the folks (in my lab, and collaborators) like to use MATLAB, so I've found scipy's io.loadmat invaluable for making this a bit more seamless. So, in general, for me (so far) numpy/scipy are generally used to integrate various datasets together and see if things "look kosher" (before runs and after runs).

b) Have you ever felt that numpy/scipy was slow and had to switch to C/C++/Fortran?

Yes, for things like boosting, svm, graph mining, etc ... but that's no real surprise since their iterative and need to run on large datasets. You should also note that there are python interfaces to these things out there as well, but I (thus far) haven't taken much of advantage of those and usually pipe out data into the expected text input formats and pull them back in when the algo is done.

c) Do you use any form of parallel processing? Multicores? SMPs? Clusters? If yes how did u utilize them?

I'd really like to (not just for Python), but I haven't. -steve [1] NetworkX: https://networkx.lanl.gov/wiki

Rahul Garg wrote:

It would be awesome if you guys could respond to some of the following questions : a) Can you guys tell me briefly about the kind of problems you are tackling with numpy and scipy?

I am using both numpy and scipy to solve PDEs in the context of finite element method (elasticity, porous media, ...).

b) Have you ever felt that numpy/scipy was slow and had to switch to C/C++/Fortran?

I use bits of C code wrapped by SWIG either to address real bottle-necks (FE assembling, element matrix computations) or due to the fact that I have lots of "legacy" C code that I reuse from my previous projects.

c) Do you use any form of parallel processing? Multicores? SMPs? Clusters? If yes how did u utilize them?

If I ever get myself to start, I will use petsc4py.

If you feel its not relevant to the list .. feel free to email me personally. I would be very interested in talking about these issues.

It is really interesting to see what others do. r.

a) Can you guys tell me briefly about the kind of problems you are tackling with numpy and scipy?

I'm using python with numpy,scipy, pytables and matplotlib for data analysis in the field of high energy particle physics. Most of the work is histograming millions of events, fitting functions to the distributions or applying cuts to yield optimized signal/background ratios. I often use the random number and optimization facilities for these purposes. Most of my colleagues use ROOT (root.cern.ch) which has also a python binding, however, I love the simplicity of numpy's ufuncs and indexing capabilities, which makes the code much denser and readable. Another part is developing toy simulation and reconstruction algorithms in python which later will be translated to C++ since the main software framework in our collaboration is written in C++. This includes log-likelihood track reconstruction algorithms based on the arrival times of photons measured with photomultipliers. Simulation involves particle generators and detector response simulations to test the reconstruction with known inputs.

b) Have you ever felt that numpy/scipy was slow and had to switch to C/C++/Fortran?

In particular for the simulation yes, depending on the level of detail of course. But only parts, eg. random number generation for certain distributions had to be coded in C/C++. Since the main software for my work is coded in C++, I often end up writing wrappers around parts of this software to extract the data I need for doing the analysis work in python.

c) Do you use any form of parallel processing? Multicores? SMPs? Clusters? If yes how did u utilize them?

We have a cluster at our lab which I use for my computations. This is not very difficult since the data can be split into several files and each can be treated in the same way. One just needs to pass a script over and over again to the cluster, this is done in a shell script or with the tools provided by the cluster scheduling system. Cheers! Bernhard

On Fri, Nov 16, 2007 at 07:50:07PM -0700, Rahul Garg wrote:

It would be awesome if you guys could respond to some of the following questions :

OK, I'll take a bit of time to do this.

a) Can you guys tell me briefly about the kind of problems you are tackling with numpy and scipy?

My day-to-day use is data processing of an atomic physics experiment (Bose Einstein condensation). The data created by the experiment is retrived from a bunch of instruments (scopes, cameras, and sensors) and stored in an HDF5 file for each run, along with the parameters of the run. This is done in Matlab for historical reasons, and MatLab is an absolute pain for this. I have implemented a similar system in Python for another experiment (I wrote an article sumerizing the lessons learnt and the patterns to follow, hell this was the fourth experiment control system I built, each time using a different platform www.gael-varoquaux.info/computers/agile_computer_control_of_an_experiment.pdf ). The data is first stored on a windows box that runs the experiment, then it is downloaded by our Linux server that keeps a hash table of all the experiments ran and some figures of merit. The data processing is done on the Linux box (quite often connected remotely using Nomachine's NX). The data is accessed in a database-like way through a custom Python module (1000 LOC). Recently I have switched to a object-relationnal-mapping-like loading of the HDF5-files (hierarchical data trees) through a database-like interface. It rocks, IMHO. We do simple fitting, blurring, filter, averaging, fft, ... on the data, and plot the results. Another of my uses is for simple toy models. Recently I have been elaborating a statistical model of an experiment, so I was running Monte Carlo simulations, but I have ran dynamical model integrations, and linear algebra calculations always revolving around the physics of cold atomic gases.

b) Have you ever felt that numpy/scipy was slow and had to switch to C/C++/Fortran?

Never. However I really like the fact that I can if I need to. When controling instruments in Python I often have to use their C SDK. In which case I use Pyrex or Ctypes.

c) Do you use any form of parallel processing? Multicores? SMPs? Clusters? If yes how did u utilize them?

No. Most often I am limited by IO. I have the feeling that unlike most people on the list, my main work is not on a computer, but on an experiment. Gaël