[Numpy-discussion] numpy : your experiences?

[Gael Varoquaux]

On Fri, Nov 16, 2007 at 07:50:07PM -0700, Rahul Garg wrote:
> It would be awesome if you guys could respond to some of the following
> questions :

OK, I'll take a bit of time to do this.

> a) Can you guys tell me briefly about the kind of problems you are
> tackling with numpy and scipy?

My day-to-day use is data processing of an atomic physics experiment
(Bose Einstein condensation). The data created by the experiment is
retrived from a bunch of instruments (scopes, cameras, and sensors) and
stored in an HDF5 file for each run, along with the parameters of the
run. This is done in Matlab for historical reasons, and MatLab is an
absolute pain for this. I have implemented a similar system in Python for
another experiment (I wrote an article sumerizing the lessons learnt and
the patterns to follow, hell this was the fourth experiment control
system I built, each time using a different platform
www.gael-varoquaux.info/computers/agile_computer_control_of_an_experiment.pdf
). The data is first stored on a windows box that runs the experiment,
then it is downloaded by our Linux server that keeps a hash table of all
the experiments ran and some figures of merit.

The data processing is done on the Linux box (quite often connected
remotely using Nomachine's NX). The data is accessed in a database-like
way through a custom Python module (1000 LOC). Recently I have switched
to a object-relationnal-mapping-like loading of the HDF5-files
(hierarchical data trees) through a database-like interface. It rocks,
IMHO.

We do simple fitting, blurring, filter, averaging, fft, ... on the data,
and plot the results.

Another of my uses is for simple toy models. Recently I have been
elaborating a statistical model of an experiment, so I was running Monte
Carlo simulations, but I have ran dynamical model integrations, and
linear algebra calculations always revolving around the physics of
cold atomic gases.

> b) Have you ever felt that numpy/scipy was slow and had to switch to
> C/C++/Fortran?

Never. However I really like the fact that I can if I need to. When
controling instruments in Python I often have to use their C SDK. In
which case I use Pyrex or Ctypes.

> c) Do you use any form of parallel processing? Multicores? SMPs?
> Clusters? If yes how did u utilize them?

No. Most often I am limited by IO.

I have the feeling that unlike most people on the list, my main work is
not on a computer, but on an experiment.

Gaël