[pypy-dev] Talk in the Supercomputing Day, Madrid

Guillem Borrell i Nogueras guillem at torroja.dmt.upm.es
Tue Jan 13 16:17:53 CET 2009


Hi!

El Tuesday 13 January 2009 13:31:07 holger krekel escribió:

>
> Could you give examples on the concrete challenges you face?
>

Multicore architectures are killing supercomputing because two kinds of 
parallelism are needed at the same time. A couple of years ago the computer 
architecture departments started suggesting multiple OpenMP threads in every MPI 
processes. This required a huge amount of work and never worked as expected 
(maybe compilers are to blame?).

Now the trend seems to be using MPI for processes and a superscalar compiler 
that uses all the cores and SPU for just a few kind of operations. This is not 
more useful than using vectorized versions of Blas, levels 1 and 2.

Some years ago I started experimenting with python and MPI. I coded a very 
limited MPI bindings collection for Python and launched python interpreters as 
MPI processes. Instantly thought that a JIT-accelerated, multicore-aware 
interpreter and a fast MPI-like message passing library would lower the 
complexity of using parallel computers. That would not give the highest 
performance but one has to be realistic. There's no need to distribute a VM, 
message passing is not that hard.

Maybe Java would be more suitable but I am contrary to Java from an almost 
religious point of view. I just hate it.

That's what I mean with narrowing the software gap. I've think that pypy has all 
the blocks to build this kind of interpeter, but maybe I am wrong. This is 
because I cant' really understand what pypy is ;) (maybe that's the reason why I 
find it so interesting).

>
> The PyPy project is mostly concerned with reducing the complexity
> of writing efficient dynamic language interpreters.
> PyPy's JIT and its flexible backend implementation
> should help with bridging the software gap IIUC.
> It probably makes sense to stress and elaborate
> on this point and to discuss examples in the talk.
>
> Managing the complexity of software deployment and integration
> is not PyPy's primary concern but it provides some interesting
> building blocks.  For example, i am interested in using using
> PyPy's sandboxing and virtualization capabilities to build an
> open "cloud at home" network: people would run a bare-bones
> executable, configure & control CPU, RAM, Network, Disk
> resources and access rights. On this network e.g. open source
> developers or scientists could launch programs and
> computations or start os-level virtual machines where
> non-python code runs.
>
> > In addition, most of postprocessing tools are written in matlab, an
> > interpreted language.  Running not-so-high performance tasks in a
> > workstation efficiently is sometimes as important as running a simulation
> > in a 12000 node supercomputer. It yould be nice if someone would remind
> > the audience that matlab is not the only suitable (or best) tool for that
> > job.
>
> Right, I guess that something like Numpy combined with PyPy's JIT
> would could make a killer combination.  That's work, though.

I think that you'll have to start explaining what a JIT is. See the note on the 
audience below. I've been using Python+Numpy+Scipy+... as a complete replacement 
for matlab for the last two years. I suggested all of my colleagues to do the 
same but they all believe that python is a toy language (!). I think that you 
could give a more authorized opinion on this.

> > I hope that I managed to explain the topics I'm interested in.
>
> sure, let us know what you think makes most sense for
> the audience - who will be the audience btw?

The talks will be in the school of aeronautics. Most of people in the audience 
will be engineers, mathematicians and physicists users of supercomputing without 
any academic background on Computer Science. 

Thany you so much!
-- 
guillem

Guillem Borrell i Nogueras
Laboratorio de Mecánica de Fluidos Computacional
Escuela Técnica Superior de Ingenieros Aeronáuticos
guillem at torroja.dmt.upm.es
Web: http://torroja.dmt.upm.es/guillem/blog



More information about the Pypy-dev mailing list