[issue10900] bz2 module fails to uncompress large files

Charles-Francois Natali report at bugs.python.org
Tue Mar 1 12:27:57 CET 2011


Charles-Francois Natali <neologix at free.fr> added the comment:

@Eric.Wolf

Could you try with this:

            # Read in anohter chunk of the file
            # NOTE: It's possible that an XML tag will be greater than buffsize
            #       This will break in that situation
-            newb = self.fp.read(self.bufpos)
+            newb = self.fp.read(self.buffsize)

Also, could you provide the output of
strace -emmap2,sbrk,brk python <script>

I could be completely wrong, but both in your case and in wrobell's case, there's a lot of _PyBytes_Resize going on, and given how PyObject_Realloc is implemented, this could lead to heavy heap fragmentation.

----------
nosy: +neologix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10900>
_______________________________________


More information about the Python-bugs-list mailing list