Re: [Python-Dev] Fast Implementation for ZIP decryption
Hello list, resurrecting a rather old thread from here: http://mail.python.org/pipermail/python-dev/2009-August/091450.html I've actually come across a use case where faster zipfile decryption would be very helpful to me (though one could certainly argue that it's a frivolous case). I'm writing a map editor for a game whose graphics datafiles are stored inside an encrypted zipfile. I've received the game developer's blessing for my editor to be able to read inside the zipfile, so legality isn't an issue here, but the pure-Python decryption currently available in the 'zipfile' module really is dog-slow. When I have my app load directly from already-uncompressed files on the HD, it's rare for a map to take more than 1 second to load; when the app has to use the pure-Python decryption routines to load all the graphics images, it can take over a minute (and this is on a 3GHz Core 2 Duo). Bundling the necessary graphics with my app isn't an option, nor is asking users to crack the zipfile encryption themselves to provide an uncompressed directory to read from. So anyway, I'd at least love to have a peek at the C code mentioned in the thread; if nothing more, perhaps I could bundle it into something that could be installed optionally by someone who wants it. Does anyone still have this code available somewhere? Thanks a bunch, CJ -- WOW: Flemmy | "Happiness isn't good enough for me! I pez@apocalyptech.com | demand euphoria!" 24.24.2.3171 | - Calvin
CJ Kucera wrote:
Hello list, resurrecting a rather old thread from here:
http://mail.python.org/pipermail/python-dev/2009-August/091450.html
... and one final update from me, mostly just so Google and the like will pick it up. I did actually end up packaging up something I called "czipfile," which is just the stock Python Lib/zipfile.py optimized in Cython, which provides very fast zipfile decryption. It's available here: http://pypi.python.org/pypi/czipfile So, anyone with a need for fast zipfile decryption can now head over that way. FWIW, while playing around with things, I found that you can get some pretty nontrivial performance improvements in pure Python by unrolling the decryption loop. For instance, line 608 in Lib/zipfile.py used to be: newdata = ''.join(map(self.decrypter, newdata)) Substituting the following results in some significant speed improvements (though still quite a lot slower than the C-based extension) - key0 = self.decrypter.key0 key1 = self.decrypter.key1 key2 = self.decrypter.key2 crctable = self.decrypter.crctable datalist = [] for c in newdata: k = key2 | 2 cord = ord(c) ^ (((k * (k^1)) >> 8) & 255) datalist.append(chr(cord)) key0 = ((key0 >> 8) & 0xffffff) ^ crctable[(key0 ^ cord) & 0xff] key1 = (key1 + (key0 & 255)) & 4294967295 key1 = (key1 * 134775813 + 1) & 4294967295 key2 = ((key2 >> 8) & 0xffffff) ^ crctable[(key2 ^ ((key1 >> 24) & 255)) & 0xff] self.decrypter.key0 = key0 self.decrypter.key1 = key1 self.decrypter.key2 = key2 newdata = ''.join(datalist) Anyone looking to speed up decryption who didn't want to absolutely depend on the user having czipfile installed might want to consider bundling their own modified version of Lib/zipfile.py with the above changes. Okay, that's all. Enjoy! -CJ -- WOW: Flemmy | "The ships hung in the sky in much the same pez@apocalyptech.com | way that bricks don't." - Douglas Adams, 24.24.2.3171 | _The Hitchhiker's Guide To The Galaxy_
participants (1)
-
CJ Kucera