On 6 Nov 2013, at 19:35, Stefan Behnel <stefan_ml@behnel.de> wrote:
Ed Singleton, 06.11.2013 19:49:
On 6 Nov 2013, at 18:25, Stefan Behnel wrote: Would this be likely to cause a memory leak in non-threaded code?
I don't think so. Single-threaded code should always use the same (global) dictionary, so it can't leak memory more than once. The leak here seemed to be a couple of bytes per thread, that's so tiny that you wouldn't even notice it with only one thread.
No, I'm looking at an app that uses up 1GB of memory over 20,000 iterations, with about 100 xpaths per iteration, so it would need to be something bigger than that.
I've been investigating a memory leak in an application that makes very heavy use of lxml, and I would be delighted if this was the cause.
As you've seen, the best way to get it fixed is to invest the time to strip it down to just a couple of operations. That's work, sure, but someone has to invest it.
You are, off course, right. Breaking it into smaller and smaller chucks seems the way to go, until I can find out where it is.
You should also make sure that you are using the latest libxml2/libxslt.
I hadn't thought of that. Good idea.
And you could run your program through valgrind, which can detect memory that doesn't get freed. The Makefile in lxml's sources has an example command line.
Thanks. I'll try this out. Ed