Hi We're gathering success stories for the website. Anyone feel like providing name, company and some data how pypy worked for them (we accept info how it didn't work on bugs.pypy.org all the time :) Cheers, fijal
I have a fully-functional wireless network simulation tool written in pypy+swig. Is that nice? Have couple papers to refer to as well. If you want I could write a small abstract on how it was so great to use pypy (which, in fact, was really great)? Alex. On Friday 07 October 2011 20:17:01 Maciej Fijalkowski wrote:
Well... that took a bit long-ish because I was preparing a small conference we are having here in Tampere. Anyway, as promised. You are free to do whatever you like with the stuff (cut/remove stuff if it is too big) as long as there is my name somewhere nearby. PYPY + SWIG as a basis for simulation platform System-level simulations of communications networks are very resource-hungry, mostly due to complex statistical relations that have to be uncovered. For example, it is not enough to observe just one or two packets in the network to conclude if it is stable or not, the actual observation should last a lot longer then any feedback process in the network. If the buffer sizes are large, this can mean hundreds of thousands of packets. Wireless networks add one extra problem - the channel model is typically an $N^2$ complexity algorithm, and it has to be run for each and every symbol transmitted over the radio interface. Unfortunately, most of the existing simulation platforms for wireless networks do not allow for multiple sources to coexist and cooperate, and therefore are unsuitable for my research. A new simulation environment was to be devised. Originally, my simulator was implemented as a time-advance model with Matlab. Unfortunately, complex logic of the network in question made it so slow, that it was practically useless. The second implementation in C was scalable enough for a single wireless cell, but it was clearly unsuitable for implementation of complex scheduling algorithms. Generally, scheduling algorithms operate with such terms as sets, maps and dynamic priority queues. The relay selection algorithms, which are the prime topic, are also operating with sets and mappings. Unfortunately, such high- level data structures do not fit well into the paradigm of C/C++. Therefore, it was decided to split the model into two parts: the link-level model implemented with C and the control level implemented with Python. Obviously, the bulk of calculations was to be performed at the link-level, and the complex control logic was to be implemented with Python. Unfortunately, the performance of Python interpreter soon appeared to be a bottleneck, but redesigning the model again did not seem like a right thing to do. And here, following the Zen of Python, the one obvious solution popped out - PyPy. PyPy is essentially a JIT-compiler for Python, and it allows to run Python programs with speeds close to those approachable with pure C++. At the same time it retains the flexibility of the scripting language. As for the C part, the SWIG wrapper generator allows to maintain the linking between the python top-level and binary backend in a clean, efficient and cross-platform manner. It also allows for a nice trick - the wrappers can be also generated for Octave, which allows to verify the channel model parts against their analytical models in Matlab-like environment. Unfortunately, SWIG can not generate interfaces for Matlab yet, but in most cases I have actually found Octave better and faster for small prototyping and testing projects. Currently, the PyPy 1.6 runs the model about 200\% faster then standard CPython 2.7.2. Also it provides the unique opportunities in configuration, as most of the model scenario can be constructed with object constructors and for loops, rather then some freaky config files or infinite bulks of XML. Overall, I am very statisfied with PyPy experience, and I plan on trying it in even more demanding environment - as an experimental engine for a P2P protocol tracker. Best regards, Alexander Pyattaev, Researcher, Department of Communications Engineering, Tampere University of Technology, Finland On Friday 21 October 2011 16:36:10 Bea During wrote:
I have a fully-functional wireless network simulation tool written in pypy+swig. Is that nice? Have couple papers to refer to as well. If you want I could write a small abstract on how it was so great to use pypy (which, in fact, was really great)? Alex. On Friday 07 October 2011 20:17:01 Maciej Fijalkowski wrote:
Well... that took a bit long-ish because I was preparing a small conference we are having here in Tampere. Anyway, as promised. You are free to do whatever you like with the stuff (cut/remove stuff if it is too big) as long as there is my name somewhere nearby. PYPY + SWIG as a basis for simulation platform System-level simulations of communications networks are very resource-hungry, mostly due to complex statistical relations that have to be uncovered. For example, it is not enough to observe just one or two packets in the network to conclude if it is stable or not, the actual observation should last a lot longer then any feedback process in the network. If the buffer sizes are large, this can mean hundreds of thousands of packets. Wireless networks add one extra problem - the channel model is typically an $N^2$ complexity algorithm, and it has to be run for each and every symbol transmitted over the radio interface. Unfortunately, most of the existing simulation platforms for wireless networks do not allow for multiple sources to coexist and cooperate, and therefore are unsuitable for my research. A new simulation environment was to be devised. Originally, my simulator was implemented as a time-advance model with Matlab. Unfortunately, complex logic of the network in question made it so slow, that it was practically useless. The second implementation in C was scalable enough for a single wireless cell, but it was clearly unsuitable for implementation of complex scheduling algorithms. Generally, scheduling algorithms operate with such terms as sets, maps and dynamic priority queues. The relay selection algorithms, which are the prime topic, are also operating with sets and mappings. Unfortunately, such high- level data structures do not fit well into the paradigm of C/C++. Therefore, it was decided to split the model into two parts: the link-level model implemented with C and the control level implemented with Python. Obviously, the bulk of calculations was to be performed at the link-level, and the complex control logic was to be implemented with Python. Unfortunately, the performance of Python interpreter soon appeared to be a bottleneck, but redesigning the model again did not seem like a right thing to do. And here, following the Zen of Python, the one obvious solution popped out - PyPy. PyPy is essentially a JIT-compiler for Python, and it allows to run Python programs with speeds close to those approachable with pure C++. At the same time it retains the flexibility of the scripting language. As for the C part, the SWIG wrapper generator allows to maintain the linking between the python top-level and binary backend in a clean, efficient and cross-platform manner. It also allows for a nice trick - the wrappers can be also generated for Octave, which allows to verify the channel model parts against their analytical models in Matlab-like environment. Unfortunately, SWIG can not generate interfaces for Matlab yet, but in most cases I have actually found Octave better and faster for small prototyping and testing projects. Currently, the PyPy 1.6 runs the model about 200\% faster then standard CPython 2.7.2. Also it provides the unique opportunities in configuration, as most of the model scenario can be constructed with object constructors and for loops, rather then some freaky config files or infinite bulks of XML. Overall, I am very statisfied with PyPy experience, and I plan on trying it in even more demanding environment - as an experimental engine for a P2P protocol tracker. Best regards, Alexander Pyattaev, Researcher, Department of Communications Engineering, Tampere University of Technology, Finland On Friday 21 October 2011 16:36:10 Bea During wrote:
participants (4)
-
Alex Pyattaev -
Alexander Pyattaev -
Bea During -
Maciej Fijalkowski