From Perl to Python: restructuring a HPC workflow

neurino lelli.luca at googlemail.com
Wed Mar 27 12:29:50 CET 2013


In the need for restructuring our daily workflow, i think it might be a 
good idea to ask the Python community and hopefully initiate a thread 
about pros and cons.

We are a small group of people (approx. 10), working separetely on 
their own projects (each employee manages approx. 2-3 projects). We 
deal with high loads of data everyday.

While the processing is accomplished with fortran and C programs mainly 
on three systems (one cluster, two standalone IBM HPCs,  8852 and p770, 
all managed by a grid-Engine), networking, pre/postprocessing, jobs 
queue administration and numerical analysis have been accomplished with 
Perl.

This workflow has been flawless now for at least 15 years. New 
generations of employees have been given Perl scripts and they 
developed the tools further.

If i think at the actual situation of Perl, i can't see a shiny time 
ahead. Perl 6 is far to be a reliable solution, the CPAN archive is 
slowing down. My idea is to persuade my colleagues to move toward 
Python-based solutions. But our concerns are that, in 3-4 years from 
now, the tools we are going to develop must be still scalable, 
mantainable, portable and of high-performance.

We don't have any solid in-house know-how on Python. We just have to 
start everything from scracth. Where do you see advantages and 
drawbacks in switching from Perl to Python, given the work picture 
above?

Thanks in advance for any opinions you might have.




More information about the Python-list mailing list