Hello, I've run up against a problem and I've tried to create a "minimal" example to demonstrate. What I'm trying to achieve is to incrementally parse xml files but still use xpath expressions to extract "records" from the xml. A user would provide an xml file, a "record xpath" and a set of "column xpaths". My plan was to read the xml file incrementally by only reading "a" first child node of the root, extract records from that node, discard the node and continue with the next "first child". This works fine for most of the xml files we're importing, except when they get really big. Of course, the whole point of this exercise is to be able to parse *big* files without running out of memory. The problem I'm seeing is that *some* nodes aren't loaded correctly (they are "empty"), but I can't figure out why. python 3.6.5 lxml 4.3.0.0 libxml 2.9.9 libxslt 1.1.32 If I run the following script: ------8<----------------------------------------------------- from lxml import etree as ET def create_big_xml(file_path, node_count=1000): with ET.xmlfile(file_path, encoding='utf-8') as fh: fh.write_declaration() with fh.element('root'): for i in range(node_count): c0 = ET.Element('c0') c1 = ET.SubElement(c0, 'c1') c2 = ET.SubElement(c1, 'c2') c2.text = str(i) fh.write(c0) def iter_xml_records(file_path, record_xpath, column_xpaths): doc = ET.iterparse(file_path, events=('start', 'end')) try: _, root = next(doc) except StopIteration: return start_tag = None for event, element in doc: if event == 'start' and start_tag is None: start_tag = element.tag if event == 'end' and element.tag == start_tag: for record_node in record_xpath(root): yield [xp(record_node) for xp in column_xpaths] start_tag = None root.clear() fp = 'test-incr.xml' errors = 0 node_count = 100 while not errors: node_count *= 10 create_big_xml(fp, node_count) rec_xp = ET.XPath('./c0') col_xps = [ ET.XPath('./c1/c2/text()'), ] for i, record in enumerate(iter_xml_records(fp, rec_xp, col_xps)): if record != [[str(i)]]: errors += 1 print(node_count, "nodes:", errors, "errors") ------8<----------------------------------------------------- The output on my machine is:

...

...

python bug_lxml.py 1000 nodes: 0 errors 10000 nodes: 5 errors

If I comment out the line "fh.write_declaration()", it takes longer to produce errors:

...

...

python bug_lxml.py 1000 nodes: 0 errors 10000 nodes: 0 errors 100000 nodes: 0 errors 1000000 nodes: 411 errors

If your're still with me, thanks for taking the time! Is this a bug in lxml, or am I doing something that I shouldn't? Regards, Wietse