From Perl to Python: restructuring a HPC workflow
lelli.luca at googlemail.com
Wed Mar 27 12:29:50 CET 2013
In the need for restructuring our daily workflow, i think it might be a
good idea to ask the Python community and hopefully initiate a thread
about pros and cons.
We are a small group of people (approx. 10), working separetely on
their own projects (each employee manages approx. 2-3 projects). We
deal with high loads of data everyday.
While the processing is accomplished with fortran and C programs mainly
on three systems (one cluster, two standalone IBM HPCs, 8852 and p770,
all managed by a grid-Engine), networking, pre/postprocessing, jobs
queue administration and numerical analysis have been accomplished with
This workflow has been flawless now for at least 15 years. New
generations of employees have been given Perl scripts and they
developed the tools further.
If i think at the actual situation of Perl, i can't see a shiny time
ahead. Perl 6 is far to be a reliable solution, the CPAN archive is
slowing down. My idea is to persuade my colleagues to move toward
Python-based solutions. But our concerns are that, in 3-4 years from
now, the tools we are going to develop must be still scalable,
mantainable, portable and of high-performance.
We don't have any solid in-house know-how on Python. We just have to
start everything from scracth. Where do you see advantages and
drawbacks in switching from Perl to Python, given the work picture
Thanks in advance for any opinions you might have.
More information about the Python-list