Hi everyone,
So I have to validate vary large xml files (gigabytes large) against xsd
schema files. I have been asked to produce line and column numbers where
the errors occurred in the file.
I'm using:
Python 3.5.2
lxml 3.7.3
libxml2 2.9.3+dfsg1-1ubuntu0.2
Ubuntu 16.04
The following code does a good job of getting a list of all the issues with
the xml file that I can nicely present to the user (for brevity I left out
a bunch of try/catch statements). This works great because it also gives
the line and column numbers in the xml file where the issues occurred:
schema_root = etree.XML(xsd_file.read())
schema = etree.XMLSchema(schema_root)
parser = etree.XMLParser(schema=schema)
parsed_xml = etree.XML(xml_file.read())
if not schema.validate(parsed_xml):
errors_list = schema.errors_log
Unfortunately, this results in reading the entire file in memory.
Alternatively, I can do the following:
schema_root = etree.XML(xsd_file.read())
schema = etree.XMLSchema(schema_root)
for event, elem in etree.iterparse(xml_file, schema=schema):
#clear stuff from memory
elem.clear()
parent = elem.getparent()
while elem.getprevious() is not None:
del elem.getparent()[0]
If the xml fails to validate against the xsd file then the iterator
returned by etree.iterparse's next method raises an etree.XMLSyntaxError
for the first xsd issue it finds.
This does a great job of keeping the memory usage constant over even very
large files, but it only tells me if the xsd file is valid or not. It does
not give me a list of issues like the previously mentioned method does.
I tried just catching the XMLSyntaxError by surrounding only the iterator's
next method with a try/catch and attempting to move on with the file even
if the error was raised, but that just resulted in an infinite loop where
the iterator's next method continuously raised the same error for the same
element and stopped progressing through the file.
It's very tempting to think that there must be a way to get the best of
both worlds here, but I've scoured the documentation, tried a few things,
as well as briefly dug into the code a bit but I haven't had any luck yet.
I'm not even sure doing this would be possible using iterparse.
Any advice in this area would be greatly appreciated.
Regards,
Phil Zerull