Parsing a potentially corrupted file
Chris Angelico
rosuav at gmail.com
Wed Dec 14 07:57:03 EST 2016
On Wed, Dec 14, 2016 at 10:43 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> This is a messy format to parse, but it's manageable. However, there's a catch. Because the logging software involved is broken, I can occasionally get a log record prematurely terminated with a new record starting mid-stream. So something like the following:
>
> [2016-11-30T20:04:08.000+00:00] [Component] [le[2016-11-30T20:04:08.000+00:00] [Component] [level] [] [] [id] Description of the issue goes here
>
> I'm struggling to find a "clean" way to parse this. I've managed a clumsy approach, by splitting the file contents on the pattern [ddd-dd-ddTdd:dd:dd.ddd+dd:dd] (the timestamp - I've never seen a case where this gets truncated) and then treating each entry as a record and parsing it individually. But the resulting code isn't exactly maintainable, and I'm looking for something cleaner.
>
Is the "[Component]" section something you could verify? (That is - is
there a known list of components?) If so, I would include that as a
secondary check. Ditto anything else you can check (I'm guessing the
[level] is one of a small set of values too.) The logic would be
something like this:
Read line from file.
Verify line as a potential record:
Assert that line begins with timestamp.
Verify as many fields as possible (component, level, etc)
Search line for additional timestamp.
If additional timestamp found:
Recurse. If verification fails, assume we didn't really have a
corrupted line.
(Process partial line? Or discard?)
If "[[" in line:
Until line is "]]":
Read line from file, append to description
If timestamp found:
Recurse. If verification succeeds, break out of loop.
Unfortunately it's still not really clean; but that's the nature of
working with messy data. Coping with ambiguity is *hard*.
ChrisA
More information about the Python-list
mailing list