Hello everyone,

I've been cracking my head about this performance issue I'm having and I could use some help.

At my work we have to parse extremely large XML files - 20GB and even larger. The basic algorithm is as follows:

with open(file, "rb") as reader:

context = etree.iterparse(reader, events=('start', 'end'))

for ev, el in context:

(processing)

el.clear()

In Python 2.7, the processing time for a 20GB XML file is approximately 40 minutes.

In Python 3.13, it's 7 hours, more than ten times from Python 2.

We went through a fine-toothed comb to find the reason why (there were minimal changes in the porting process), and out of desperation I commented out the el.clear() line, and apparently that is the reason - without it, performance in Python 3 matches with 2.

Unfortunately when we tested this in a less well-endowed server, the program crashed due to running out of memory (it worked fine with Python 2).

I tried substituting el.clear() with del el instead but it did not work - apparently there were still references somewhere, so the garbage collector didn't fire.

Questions:

1. What is the difference between the Python 2 and Python 3's implementation of clear()?

2. Is there a way to solve this issue of performance penalty? I tried fast_iter, clearing the root element, re-assigning element to None, nothing works.

Any help would be greatly appreciated.

Regards,