Performance issues when using element.clear() in Python 3.x
Hello everyone, I've been cracking my head about this performance issue I'm having and I could use some help. At my work we have to parse extremely large XML files - 20GB and even larger. The basic algorithm is as follows: with open(file, "rb") as reader: context = etree.iterparse(reader, events=('start', 'end')) for ev, el in context: (processing) el.clear() In Python 2.7, the processing time for a 20GB XML file is approximately 40 minutes. In Python 3.13, it's 7 hours, more than ten times from Python 2. We went through a fine-toothed comb to find the reason why (there were minimal changes in the porting process), and out of desperation I commented out the el.clear() line, and apparently that is the reason - without it, performance in Python 3 matches with 2. Unfortunately when we tested this in a less well-endowed server, the program crashed due to running out of memory (it worked fine with Python 2). I tried substituting el.clear() with del el instead but it did not work - apparently there were still references somewhere, so the garbage collector didn't fire. Questions: 1. What is the difference between the Python 2 and Python 3's implementation of clear()? 2. Is there a way to solve this issue of performance penalty? I tried fast_iter, clearing the root element, re-assigning element to None, nothing works. Any help would be greatly appreciated. Regards,
Hi, Noorulamry Daud schrieb am 13.02.25 um 12:28:
I've been cracking my head about this performance issue I'm having and I could use some help.
At my work we have to parse extremely large XML files - 20GB and even larger. The basic algorithm is as follows:
with open(file, "rb") as reader: context = etree.iterparse(reader, events=('start', 'end')) for ev, el in context: (processing) el.clear()
I guess this is not a look-alike example but just meant as a hint, right? Clearing the elements on both start and end events seems useless, clearing them on start is probably outright dangerous, etc. You would at least want to pass the "keep_tail=True" option and clear them only at the end. https://lxml.de/parsing.html#modifying-the-tree https://lxml.de/parsing.html#incremental-event-parsing
In Python 2.7, the processing time for a 20GB XML file is approximately 40 minutes.
In Python 3.13, it's 7 hours, more than ten times from Python 2.
Are you using the same versions of lxml (and libxml2) in both? There shouldn't be a difference in behaviour, except for the obvious language differences (bytes/unicode). Does the memory consumption stay constant over time or does it continuously grow as it parses? Have you run a memory profiler on your code? Or a (statistical) line profiler to see where the time is spent? Stefan
On 13 Feb 2025, at 15:18, Stefan Behnel via lxml - The Python XML Toolkit wrote:
Are you using the same versions of lxml (and libxml2) in both?
There shouldn't be a difference in behaviour, except for the obvious language differences (bytes/unicode).
Based on the parsing code we use in Openpyxl, I'd agree with this. NB., we discovered that, for pure parsing, ie. you just want to get at the data, the standard library's etree module is often significantly faster, but YMMV.
Does the memory consumption stay constant over time or does it continuously grow as it parses?
Have you run a memory profiler on your code? Or a (statistical) line profiler to see where the time is spent
Excellent suggestions: memory_profiler and pympler are useful tools for this. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 Düsseldorf D- 40489 Tel: +49-203-3925-0390 Mobile: +49-178-782-6226
Hi everyone, Thank you for your replies.
I guess this is not a look-alike example but just meant as a hint, right?
Yes. My work is very protective about the source code, so I am only allowed to sketch out the rough approximation. The start events are involved during some of the processing, and as you mentioned, element.clear() was not invoked with it.
Are you using the same versions of lxml (and libxml2) in both?
No, and that's what makes it so frustrating. I cannot tell management that using the latest version of Python and lxml actually causes a significant performance penalty. By rights using the latest versions should be at least as good as, if not better, than the older version.
Does the memory consumption stay constant over time or does it continuously grow as it parses?
Have you run a memory profiler on your code? Or a (statistical) line
It grows larger until it eventually crashed. My colleague expanded the page file and managed to delay said crash, but it happened eventually. profiler to see where the time is spent I used Python's cprofile to find the bottlenecks, but unfortunately the results weren't making sense. It identified which functions were taking the most time, but when I did a line-by-line analysis the times didn't add up. Since commenting out the element.clear() lines did bring the result close the Python 2.7's performance, the rest of the team decided that this is where the issue is.
the standard library's etree module is often significantly faster,
I have not considered that angle since what I can find on Google indicated that lxml is the fastest; but I'll give this a try. On Fri, 14 Feb 2025 at 00:21, Charlie Clark < charlie.clark@clark-consulting.eu> wrote:
On 13 Feb 2025, at 15:18, Stefan Behnel via lxml - The Python XML Toolkit wrote:
Are you using the same versions of lxml (and libxml2) in both?
There shouldn't be a difference in behaviour, except for the obvious language differences (bytes/unicode).
Based on the parsing code we use in Openpyxl, I'd agree with this. NB., we discovered that, for pure parsing, ie. you just want to get at the data, the standard library's etree module is often significantly faster, but YMMV.
Does the memory consumption stay constant over time or does it continuously grow as it parses?
Have you run a memory profiler on your code? Or a (statistical) line profiler to see where the time is spent
Excellent suggestions: memory_profiler and pympler are useful tools for this.
Charlie
-- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 Düsseldorf D- 40489 Tel: +49-203-3925-0390 Mobile: +49-178-782-6226 _______________________________________________ lxml - The Python XML Toolkit mailing list -- lxml@python.org To unsubscribe send an email to lxml-leave@python.org https://mail.python.org/mailman3/lists/lxml.python.org/ Member address: noorulamry.daud@gmail.com
Hi, Noorulamry Daud schrieb am 14.02.25 um 09:56:
Are you using the same versions of lxml (and libxml2) in both?
No, and that's what makes it so frustrating. I cannot tell management that using the latest version of Python and lxml actually causes a significant performance penalty. By rights using the latest versions should be at least as good as, if not better, than the older version.
It should be. This seems more of a memory problem.
Does the memory consumption stay constant over time or does it continuously grow as it parses?
It grows larger until it eventually crashed. My colleague expanded the page file and managed to delay said crash, but it happened eventually.
Then you're not cleaning up enough of the XML tree. Some of it remains in memory after processing it, and thus leads to swapping and long waiting times. Try to find out how the tree looks after a few iterations. You're collecting "start" events, so grab the first returned element (that's the root element) and print its tostring() after each ".clean()" call. That should show you what data you're missing in the cleanup.
Have you run a memory profiler on your code? Or a (statistical) line profiler to see where the time is spent
I used Python's cprofile to find the bottlenecks, but unfortunately the results weren't making sense. It identified which functions were taking the most time, but when I did a line-by-line analysis the times didn't add up.
That's not unusual. Line profiling takes additional time *per line*, so the results are often different from simple *per function* timings. Statistical profilers are much better than cProfile here since they add less overhead.
Since commenting out the element.clear() lines did bring the result close the Python 2.7's performance, the rest of the team decided that this is where the issue is.
Sort-of, but probably for other reasons.
the standard library's etree module is often significantly faster,
I have not considered that angle since what I can find on Google indicated that lxml is the fastest; but I'll give this a try.
I can second that. It uses a different parser and in-memory model, so the performance is different – better for some things, worse for others. Try it to see where your code ends up. Note that the feature set is also very different, though. lxml adds lots of functionality that "xml.etree" cannot provide. Stefan
On 14 Feb 2025, at 11:12, Stefan Behnel via lxml - The Python XML Toolkit wrote:
Then you're not cleaning up enough of the XML tree. Some of it remains in memory after processing it, and thus leads to swapping and long waiting times.
It's definitely a memory issue. You can write some code to catch memory use quickly. This is something we wrote for openpyxl while we trying to "contain" memory use: ```python import os import openpyxl from memory_profiler import memory_usage def test_memory_use(): """Naive test that assumes memory use will never be more than 120 % of that for first 50 rows""" folder = os.path.split(__file__)[0] src = os.path.join(folder, "files", "very_large.xlsx") wb = openpyxl.load_workbook(src, read_only=True) ws = wb.active initial_use = None for n, line in enumerate(ws.iter_rows(values_only=True)): if n % 50 == 0: use = memory_usage(proc=-1, interval=1)[0] if initial_use is None: initial_use = use assert use/initial_use < 1.2 print(n, use) if __name__ == '__main__': test_memory_use() ``` You should be able to adapt this for your parser and it'll tell you soon enough how far in you get before your memory use balloons. If memory serves I had one problem where I was clearing in the wrong place, which meant that other elements were sticking around. Thanks to Stefan for helping me sort it. I think your code made be too aggressive. It might help to look at the Openpyxl worksheet parser which has to handle what happens if you do additional processing within nodes. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 Düsseldorf D- 40489 Tel: +49-203-3925-0390 Mobile: +49-178-782-6226
Hi everyone, Despite the brief respite, the issue my team is having with the element.clear() persists. I honestly have no idea why lxml 2.2.3 can do it instantly while the latest version took ages. I do wonder about something though; I used sys.getsizeof to see the size of the elements before and after clear, but to my surprise the size remained constant at 56 bytes. In that case what are we clearing? On Fri, 14 Feb 2025, 21:08 Charlie Clark, <charlie.clark@clark-consulting.eu> wrote:
On 14 Feb 2025, at 11:12, Stefan Behnel via lxml - The Python XML Toolkit wrote:
Then you're not cleaning up enough of the XML tree. Some of it remains in memory after processing it, and thus leads to swapping and long waiting times.
It's definitely a memory issue. You can write some code to catch memory use quickly. This is something we wrote for openpyxl while we trying to "contain" memory use:
import osimport openpyxl from memory_profiler import memory_usage
def test_memory_use(): """Naive test that assumes memory use will never be more than 120 % of that for first 50 rows""" folder = os.path.split(__file__)[0] src = os.path.join(folder, "files", "very_large.xlsx") wb = openpyxl.load_workbook(src, read_only=True) ws = wb.active
initial_use = None
for n, line in enumerate(ws.iter_rows(values_only=True)): if n % 50 == 0: use = memory_usage(proc=-1, interval=1)[0] if initial_use is None: initial_use = use assert use/initial_use < 1.2 print(n, use) if __name__ == '__main__': test_memory_use()
You should be able to adapt this for your parser and it'll tell you soon enough how far in you get before your memory use balloons. If memory serves I had one problem where I was clearing in the wrong place, which meant that other elements were sticking around. Thanks to Stefan for helping me sort it. I think your code made be too aggressive. It might help to look at the Openpyxl worksheet parser which has to handle what happens if you do additional processing within nodes.
Charlie
-- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 <https://www.google.com/maps/search/Sengelsweg+34+%0D%0AD%C3%BCsseldorf+%0D%0AD-+40489?entry=gmail&source=g> Düsseldorf <https://www.google.com/maps/search/Sengelsweg+34+%0D%0AD%C3%BCsseldorf+%0D%0AD-+40489?entry=gmail&source=g> D- 40489 <https://www.google.com/maps/search/Sengelsweg+34+%0D%0AD%C3%BCsseldorf+%0D%0AD-+40489?entry=gmail&source=g> Tel: +49-203-3925-0390 Mobile: +49-178-782-6226 _______________________________________________ lxml - The Python XML Toolkit mailing list -- lxml@python.org To unsubscribe send an email to lxml-leave@python.org https://mail.python.org/mailman3/lists/lxml.python.org/ Member address: noorulamry.daud@gmail.com
You're clearing the subelements, the attributes, the text, and the tail https://github.com/lxml/lxml/blob/0eb4f0029497957e58a9f15280b3529bdb18d117/s... By default sys.getsizeof only measures the "intrinsic" size of an object, it does not traverse pointers unless the object specifically overrode `__sizeof__` to expose this information. lxml does not, it always returns 56. That's trivially testable, just create an empty element, check its size, it's 56, then add a bunch of children, check the root element's size, still 56. An lxml element would be 56 bytes because (w/ GIL and GC) a Python object has a baseline of 4*8 = 32 bytes (class pointer, refcount, prev and next gc pointers) and to that it adds 3*8 = 24 bytes for pointers to the internal _Document, to the libxml2 xmlNode, and to the cached tag string (in clark's notation). On 3/03/25 07:41, Noorulamry Daud via lxml - The Python XML Toolkit wrote:
Hi everyone,
Despite the brief respite, the issue my team is having with the element.clear() persists. I honestly have no idea why lxml 2.2.3 can do it instantly while the latest version took ages.
I do wonder about something though; I used sys.getsizeof to see the size of the elements before and after clear, but to my surprise the size remained constant at 56 bytes. In that case what are we clearing?
On Fri, 14 Feb 2025, 21:08 Charlie Clark, <charlie.clark@clark-consulting.eu <mailto:charlie.clark@clark-consulting.eu>> wrote:
__
On 14 Feb 2025, at 11:12, Stefan Behnel via lxml - The Python XML Toolkit wrote:
Then you're not cleaning up enough of the XML tree. Some of it remains in memory after processing it, and thus leads to swapping and long waiting times.
It's definitely a memory issue. You can write some code to catch memory use quickly. This is something we wrote for openpyxl while we trying to "contain" memory use:
|import os import openpyxl from memory_profiler import memory_usage def test_memory_use(): """Naive test that assumes memory use will never be more than 120 % of that for first 50 rows""" folder = os.path.split(__file__)[0] src = os.path.join(folder, "files", "very_large.xlsx") wb = openpyxl.load_workbook(src, read_only=True) ws = wb.active initial_use = None for n, line in enumerate(ws.iter_rows(values_only=True)): if n % 50 == 0: use = memory_usage(proc=-1, interval=1)[0] if initial_use is None: initial_use = use assert use/initial_use < 1.2 print(n, use) if __name__ == '__main__': test_memory_use() |
You should be able to adapt this for your parser and it'll tell you soon enough how far in you get before your memory use balloons. If memory serves I had one problem where I was clearing in the wrong place, which meant that other elements were sticking around. Thanks to Stefan for helping me sort it. I think your code made be too aggressive. It might help to look at the Openpyxl worksheet parser which has to handle what happens if you do additional processing within nodes.
Charlie
-- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 <https://www.google.com/maps/search/Sengelsweg+34+%0D%0AD%C3%BCsseldorf+%0D%0AD-+40489?entry=gmail&source=g> Düsseldorf <https://www.google.com/maps/search/Sengelsweg+34+%0D%0AD%C3%BCsseldorf+%0D%0AD-+40489?entry=gmail&source=g> D- 40489 <https://www.google.com/maps/search/Sengelsweg+34+%0D%0AD%C3%BCsseldorf+%0D%0AD-+40489?entry=gmail&source=g> Tel: +49-203-3925-0390 Mobile: +49-178-782-6226
_______________________________________________ lxml - The Python XML Toolkit mailing list -- lxml@python.org <mailto:lxml@python.org> To unsubscribe send an email to lxml-leave@python.org <mailto:lxml-leave@python.org> https://mail.python.org/mailman3/lists/lxml.python.org/ <https://mail.python.org/mailman3/lists/lxml.python.org/> Member address: noorulamry.daud@gmail.com <mailto:noorulamry.daud@gmail.com>
_______________________________________________ lxml - The Python XML Toolkit mailing list -- lxml@python.org To unsubscribe send an email to lxml-leave@python.org https://mail.python.org/mailman3/lists/lxml.python.org/ Member address: xmo@odoo.com
I see. Then what I'm wondering is, for the "clearing the elements" process, is there a difference in what Python 2.7 did that is different than Python 3.8+? Because we have the same code but vastly different execution times when we run it through the different versions. And we narrowed it down to the clear() method, so what is the secret of Py2.7's performance? Out of desperation my manager is suggesting that we use Python's native xml processing instead of lxml but I feel like I've lost the battle if I do that. On Mon, 3 Mar 2025, 16:04 Xavier Morel via lxml - The Python XML Toolkit, < lxml@python.org> wrote:
You're clearing the subelements, the attributes, the text, and the tail
https://github.com/lxml/lxml/blob/0eb4f0029497957e58a9f15280b3529bdb18d117/s...
By default sys.getsizeof only measures the "intrinsic" size of an object, it does not traverse pointers unless the object specifically overrode `__sizeof__` to expose this information.
lxml does not, it always returns 56. That's trivially testable, just create an empty element, check its size, it's 56, then add a bunch of children, check the root element's size, still 56.
An lxml element would be 56 bytes because (w/ GIL and GC) a Python object has a baseline of 4*8 = 32 bytes (class pointer, refcount, prev and next gc pointers) and to that it adds 3*8 = 24 bytes for pointers to the internal _Document, to the libxml2 xmlNode, and to the cached tag string (in clark's notation).
On 3/03/25 07:41, Noorulamry Daud via lxml - The Python XML Toolkit wrote:
Hi everyone,
Despite the brief respite, the issue my team is having with the element.clear() persists. I honestly have no idea why lxml 2.2.3 can do it instantly while the latest version took ages.
I do wonder about something though; I used sys.getsizeof to see the size of the elements before and after clear, but to my surprise the size remained constant at 56 bytes. In that case what are we clearing?
On Fri, 14 Feb 2025, 21:08 Charlie Clark, <charlie.clark@clark-consulting.eu <mailto:charlie.clark@clark-consulting.eu>> wrote:
__
On 14 Feb 2025, at 11:12, Stefan Behnel via lxml - The Python XML Toolkit wrote:
Then you're not cleaning up enough of the XML tree. Some of it remains in memory after processing it, and thus leads to swapping and long waiting times.
It's definitely a memory issue. You can write some code to catch memory use quickly. This is something we wrote for openpyxl while we trying to "contain" memory use:
|import os import openpyxl from memory_profiler import memory_usage def test_memory_use(): """Naive test that assumes memory use will never be more than 120 % of that for first 50 rows""" folder = os.path.split(__file__)[0] src = os.path.join(folder, "files", "very_large.xlsx") wb = openpyxl.load_workbook(src, read_only=True) ws = wb.active initial_use = None for n, line in enumerate(ws.iter_rows(values_only=True)): if n % 50 == 0: use = memory_usage(proc=-1, interval=1)[0] if initial_use is None: initial_use = use assert use/initial_use < 1.2 print(n, use) if __name__ == '__main__': test_memory_use() |
You should be able to adapt this for your parser and it'll tell you soon enough how far in you get before your memory use balloons. If memory serves I had one problem where I was clearing in the wrong place, which meant that other elements were sticking around. Thanks to Stefan for helping me sort it. I think your code made be too aggressive. It might help to look at the Openpyxl worksheet parser which has to handle what happens if you do additional processing within nodes.
Charlie
-- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 < https://www.google.com/maps/search/Sengelsweg+34+%0D%0AD%C3%BCsseldorf+%0D%0AD-+40489?entry=gmail&source=g
Düsseldorf < https://www.google.com/maps/search/Sengelsweg+34+%0D%0AD%C3%BCsseldorf+%0D%0AD-+40489?entry=gmail&source=g
Tel: +49-203-3925-0390 Mobile: +49-178-782-6226
_______________________________________________ lxml - The Python XML Toolkit mailing list -- lxml@python.org <mailto:lxml@python.org> To unsubscribe send an email to lxml-leave@python.org <mailto:lxml-leave@python.org> https://mail.python.org/mailman3/lists/lxml.python.org/ <https://mail.python.org/mailman3/lists/lxml.python.org/> Member address: noorulamry.daud@gmail.com <mailto:noorulamry.daud@gmail.com>
_______________________________________________ lxml - The Python XML Toolkit mailing list -- lxml@python.org To unsubscribe send an email to lxml-leave@python.org https://mail.python.org/mailman3/lists/lxml.python.org/ Member address: xmo@odoo.com
lxml - The Python XML Toolkit mailing list -- lxml@python.org To unsubscribe send an email to lxml-leave@python.org https://mail.python.org/mailman3/lists/lxml.python.org/ Member address: noorulamry.daud@gmail.com
On 4 Mar 2025, at 7:16, Noorulamry Daud via lxml - The Python XML Toolkit wrote:
I see.
Then what I'm wondering is, for the "clearing the elements" process, is there a difference in what Python 2.7 did that is different than Python 3.8+? Because we have the same code but vastly different execution times when we run it through the different versions. And we narrowed it down to the clear() method, so what is the secret of Py2.7's performance?
I think this is going round in circles. You haven't provided the real code you're using and seem to have ignored many of the suggestions that we have made. Many of us have worked with both Python 2 and Python 3 and I have not had to change the code working with XML as a result, nor have I seen any significant performance differences. In fact, the most recent Python versions (starting with Python 3.7) are faster. This is from 2019. 
Out of desperation my manager is suggesting that we use Python's native xml processing instead of lxml but I feel like I've lost the battle if I do that.
I've already said that this should be an option and could, indeed, be faster. Why don't you write some simple benchmarks? Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 Düsseldorf D- 40489 Tel: +49-203-3925-0390 Mobile: +49-178-782-6226
participants (4)
-
Charlie Clark -
Noorulamry Daud -
Stefan Behnel -
Xavier Morel