Sorry if this is a dumb question - I'm very new to PyPy, but also very interested in using it in several applications. I currently use PyTables and NumPy in a particular application, and I was playing around with the idea of migrating it to PyPy. I don't really need the querying or compression capabilities of PyTables, and most processing of the data is done in pure Python code, so I was hoping to see a speed-up with PyPy. I ran some tests yesterday on storing data on disk in flat binary files, and PyPy was significantly slower than CPython. I haven't done any comparisons with PyTables yet. Note that I'm running CPython 2.7.2 and PyPy 1.5 on a rather old Windows XP machine. I generated a 1 million element array using the built-in array module, wrote it to disk, and then read it back in. See http://pastie.org/2342676 for the code. Each operation was slower with PyPy than with CPython. * Array creation - CPython: 0.16s - PyPy: 0.47s * Writing file - CPython: 0.05s - PyPy: 0.11s * Reading file: CPython: 0.02s - PyPy: 0.08s This method won't quite work for me in any case - I need to store 64 bit integers, and the built-in array module doesn't support them. To get around that, I modified the pure-python array.py that comes in the pypy\lib_pypy directory. I added a "q" to the end of the line "TYPECODES = ..." which represents a 64 bit signed integer within the struct module. I saved that modified file as array2.py and imported it in place of the built-in array. See http://pastie.org/2342721 for the code. That allowed me to use a 64 bit integer, but the array creation step was again much slower on PyPy than it was on CPython. The disk accessing steps were more similar, and are probably at about the limit of the hard disk anyway, but creating the array takes much longer under PyPy. * Array creation - CPython: 0.31s - PyPy: 1.42s * Writing file - CPython: 0.16s - PyPy: 0.25s * Reading file: CPython: 0.83s - PyPy: 0.13s Any ideas on what could be causing this speed difference? Am I doing anything egregiously stupid in my code? Any ideas on better methods for efficiently storing and retrieving binary data from disk under PyPy? Thanks in advance for your help. Sincerely, Josh Ayers
On Tue, Aug 9, 2011 at 5:07 AM, Josh Ayers <josh.ayers@gmail.com> wrote:
I generated a 1 million element array using the built-in array module, wrote it to disk, and then read it back in. See http://pastie.org/2342676 for the code.
Each operation was slower with PyPy than with CPython.
* Array creation - CPython: 0.16s - PyPy: 0.47s
Whats taking time here is to iterate over the range-list and unwrapping all the integers. If all you want is to allocate an array it's significantly faster (both on pypy and on cpython) to do: a = array.array(outputDataType,[0]) * dataSize
* Writing file - CPython: 0.05s - PyPy: 0.11s * Reading file: CPython: 0.02s - PyPy: 0.08s
The builtin array module uses space.call_method(w_f, 'write') and space.call_method(w_f, 'read') to implement fromfile and tofile. For fromfile that means copying the data atleast once, and maybe that's whats going on with tofile too. I dont know how hard it would be to add some fast path for common cases that reads/writes data directly into the array buffer?
This method won't quite work for me in any case - I need to store 64 bit integers, and the built-in array module doesn't support them. To get around that, I modified the pure-python array.py that comes in the pypy\lib_pypy directory. I added a "q" to the end of the line "TYPECODES = ..." which represents a 64 bit signed integer within the struct module. I saved that modified file as array2.py and imported it in place of the built-in array. See http://pastie.org/2342721 for the code.
That allowed me to use a 64 bit integer, but the array creation step was again much slower on PyPy than it was on CPython. The disk accessing steps were more similar, and are probably at about the limit of the hard disk anyway, but creating the array takes much longer under PyPy.
Why do you think this is limited by the harddisk? I would imagine this approach to be slower than using the builtin module. Did you try this approach with a datatype supported by the built in module to compare the performance of the two approaches? -- Håkan Ardö
participants (2)
-
Hakan Ardo -
Josh Ayers