[issue865] Struct.pack and unpack 3x and 6x slower than CPython
New submission from Josh Ayers <josh.ayers@gmail.com>: Using struct.pack and struct.unpack is significantly slower on PyPy 1.6 than on CPython 2.7.2 when packing a large list. This is under Windows 7. A summary of the timing results is shown below and the code used is attached. The attached code uses 4 byte integers. Floats show similar results. 4 MB 40 MB 80 MB pack unpack pack unpack pack unpack CPython 0.13 0.03 1.30 0.29 2.60 0.60 PyPy 0.43 0.20 3.90 1.77 8.03 3.59 Factor 3.3x 6.7x 3.0x 6.1x 3.1x 6.0x The benchmark code packs a large list, writes it to a file, then reads the file back in and unpacks it. I didn't include it in the table above, but the reading portion was also slower in PyPy. However, it wasn't as large of a difference as the struct packing and unpacking portion. ---------- messages: 3105 nosy: jayers2003, pypy-issue priority: feature status: unread title: Struct.pack and unpack 3x and 6x slower than CPython ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue865> ________________________________________
Justin Peel <peelpy@gmail.com> added the comment: Yes, struct.pack and struct.unpack are known to be slower, especially with ints. Alex Gaynor's unroll-if-alt branch, which I think will be merged soon, should help out a great deal with this. Reading and writing to files are also known to be slow because of the extra memory allocation and copying currently needed in order to make sure that the buffers used for the actual reading and writing of the files are not moved during copying by the garbage collector. Some ideas have been discussed to try to remedy this, but no one has actually worked to fix it yet to my knowledge. ---------- nosy: +justinpeel status: unread -> chatting ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue865> ________________________________________
Josh Ayers <josh.ayers@gmail.com> added the comment: Thanks for the update. Glad to hear it's a known issue with at least a partial solution in work. ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue865> ________________________________________
Alex Gaynor <alex.gaynor@gmail.com> added the comment: The attachment you referenced seems to have gotten lost, any chance you can upload it so I can take a look? ---------- nosy: +agaynor ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue865> ________________________________________
Josh Ayers <josh.ayers@gmail.com> added the comment: File attached. ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue865> ________________________________________
Carl Friedrich Bolz <cfbolz@gmx.de> added the comment: This is mostly a property of the insanely long format strings for pack and unpack. For human-sized format strings (e.g. struct.pack("iii", 1, 2, 3)) PyPy is much faster than CPython, both for pack and unpack. Therefore I am closing the issue. If somebody is of the opinion that struct format strings are in practice commonly thousands of characters long, please reopen it. ---------- nosy: +cfbolz status: chatting -> resolved ________________________________________ PyPy bug tracker <tracker@bugs.pypy.org> <https://bugs.pypy.org/issue865> ________________________________________
participants (4)
-
Alex Gaynor -
Carl Friedrich Bolz
-
Josh Ayers
-
Justin Peel