Problem with tarfile module to open *.tar.gz files - unreliable ?

[m_ahlenius]

On Aug 20, 9:10 am, Dave Angel <da... at ieee.org> wrote:
> m_ahlenius wrote:
> > On Aug 20, 6:57 am, m_ahlenius <ahleni... at gmail.com> wrote:
>
> >> On Aug 20, 5:34 am, Dave Angel <da... at ieee.org> wrote:
>
> >>> m_ahlenius wrote:
>
> >>>> Hi,
>
> >>>> I am relatively new to doing serious work in python.  I am using it to
> >>>> access a large number of log files.  Some of the logs get corrupted
> >>>> and I need to detect that when processing them.  This code seems to
> >>>> work for quite a few of the logs (all same structure)  It also
> >>>> correctly identifies some corrupt logs but then it identifies others
> >>>> as being corrupt when they are not.
>
> >>>> example error msg from below code:
>
> >>>> Could not open the log file: '/disk/7-29-04-02-01.console.log.tar.gz'
> >>>> Exception: CRC check\
> >>>>  failed 0x8967e931 !=x4e5f1036L
>
> >>>> When I manually examine the supposed corrupt log file and use
> >>>> "tar -xzvof /disk/7-29-04-02-01.console.log.tar.gz "  on it, it opens
> >>>> just fine.
>
> >>>> Is there anything wrong with how I am using this module?  (extra code
> >>>> removed for clarity)
>
> >>>>  if tarfile.is_tarfile( file ):
> >>>>         try:
> >>>>             xf =arfile.open( file, "r:gz" )
> >>>>             for locFile in xf:
> >>>>                 logfile =f.extractfile( locFile )
> >>>>                 validFileFlag =rue
> >>>>                 # iterate through each log file, grab the first and
> >>>> the last lines
> >>>>                 lines =ter( logfile )
> >>>>                 firstLine =ines.next()
> >>>>                 for nextLine in lines:
> >>>>                     ....
> >>>>                         continue
>
> >>>>                 logfile.close()
> >>>>                  ...
> >>>>             xf.close()
> >>>>         except Exception, e:
> >>>>             validFileFlag =alse
> >>>>             msg =\nCould not open the log file: " + repr(file) + "
> >>>> Exception: " + str(e) + "\n"
> >>>>  else:
> >>>>         validFileFlag =alse
> >>>>         lTime =xtractFileNameTime( file )
> >>>>         msg =>>>>>>> Warning " + file + " is NOT a valid tar archive
> >>>> \n"
> >>>>         print msg
>
> >>> I haven't used tarfile, but this feels like a problem with the Win/Unix
> >>> line endings.  I'm going to assume you're running on Windows, which
> >>> could trigger the problem I'm going to describe.
>
> >>> You use 'file' to hold something, but don't show us what.  In fact, it's
> >>> a lousy name, since it's already a Python builtin.  But if it's holding  
> >>> fileobj, that you've separately opened, then you need to change that
> >>> open to use mode 'rb'
>
> >>> The problem, if I've guessed right, is that occasionally you'll
> >>> accidentally encounter a 0d0a sequence in the middle of the (binary)
> >>> compressed data.  If you're on Windows, and use the default 'r' mode,
> >>> it'll be changed into a 0a byte.  Thus corrupting the checksum, and
> >>> eventually the contents.
>
> >>> DaveA
>
> >> Hi,
>
> >> thanks for the comments - I'll change the variable name.
>
> >> I am running this on linux so don't think its a Windows issue.  So if
> >> that's the case
> >> is the 0d0a still an issue?
>
> >> 'mark
>
> > Oh and what's stored currently in
> > The file var us just the unopened pathname to the
> > Target file I want to open
>
> No, on Linux, there should be no such problem.  And I have to assume
> that if you pass the filename as a string, the library would use 'rb'
> anyway.  It's just if you pass a fileobj,  AND are on Windows.
>
> Sorry I wasted your time, but nobody else had answered, and I hoped it
> might help.
>
> DaveA

Hi Dave

thanks for responding - you were not wasting my time but helping me to
be aware of other potential issues.

Appreciate it much.

Its just weird that it works for most files and even finds corrupt
ones, but some of the ones it marks as corrupt seem to be OK.

thanks

'mark