[ python-Bugs-1622896 ] Exception when compressing certain data with bz2

[SourceForge.net]

Bugs item #1622896, was opened at 2006-12-27 15:26
Message generated for change (Comment added) made by gsasha
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1622896&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Extension Modules
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Gontmakher (gsasha)
Assigned to: Nobody/Anonymous (nobody)
Summary: Exception when compressing certain data with bz2

Initial Comment:
Looks like an out-of-bounds array access... might be a security problem.

The attached file includes a script which, when executed, tries to pack the two given directories with a bz2 compressor. On my machine (stock 32 bit Ubuntu Edgy), the program fails with the following exception: 
--------------------
Traceback (most recent call last):
  File "test.py", line 13, in ?
    block = compressor.compress(open("compress-0067/"+file, "rb").read())
ValueError: the bz2 library has received wrong parameters
--------------------

The problem occurs under either python2.4 or 2.5 (I don't have other versions to test with).

Sorry, the file is large... I tried to reduce the example to smaller number of files etc., but no such luck. The file is too large to be submitted here as an attachment, so I have uploaded it to http://www.cs.technion.ac.il/~gsasha/testcase.tar.bz2



----------------------------------------------------------------------

>Comment By: Alex Gontmakher (gsasha)
Date: 2006-12-28 11:17

Message:
Logged In: YES 
user_id=47707
Originator: YES

I saw that the effect occurs only when one compressor finishes and is
destroyed, and then the second one gives the "wrong parameters". Usually,
for such an effect to happen, the first compressor must have some permanent
effect on the system - and since it was destroyed before the second one was
created, it was only natural to assume that the reason was an out of bounds
access.

My approach is that such a bug should be suspected as an out-of-bounds
access until shown othervise, especially in a language like Python where it
is natural to assume that such problems shouldn't occur and that it is safe
to use security-wise. Of course I'll be happy to learn that it's just a
simple bug.

On a related note, will the fix be backported to all the relevant Python
versions? If you understand why the bug happens (I don't, sorry, have no
idea of internals of Python), you might be able to generate a compact test
case that would capture the erroneous behavior.

Best regards,
Alex


----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-12-28 10:27

Message:
Logged In: YES 
user_id=33168
Originator: NO

I don't understand why you think this is an array out of bounds.  I ran
your test case under valgrind and it reported no problems.

I can reproduce the problem.  I have attached a patch that fixes the
problem for me.  I am not certain it's the correct fix however.  The unit
tests pass with this modification.
File Added: bz2.diff

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1622896&group_id=5470