Hello, I'd like to suggest a trival feature for lxml. libxml2 and libxslt support extended memory debugging when both libraries are compiled with the the '--with-mem-debug' flag [1]. This is useful to debug how many memory is actually used by the XML and XSTL code. In order to show the amount of used memory and memory blocks, the two functions xmlMemBlocks() [2] and xmlMemUsed() [3] needs to be wrapped. Both functions are available in non-debug builds and return 0 in that case. It might be worth to wrap the other debug helpers like xmlMemDisplay(FILE *) [4], too. Regards, Christian Example patch: diff -ru lxml-2.3.4.orig/src/lxml/lxml.etree.pyx lxml-2.3.4/src/lxml/lxml.etree.pyx --- lxml-2.3.4.orig/src/lxml/lxml.etree.pyx 2012-03-26 12:28:58.000000000 +0200 +++ lxml-2.3.4/src/lxml/lxml.etree.pyx 2012-05-03 18:33:00.702130662 +0200 @@ -2956,6 +2956,11 @@ except _TargetParserResult, result_container: return result_container.result +def memoryUsed(): + return tree.xmlMemUsed() + +def memoryBlocks(): + return tree.xmlMemBlocks() ################################################################################ # Include submodules diff -ru lxml-2.3.4.orig/src/lxml/tree.pxd lxml-2.3.4/src/lxml/tree.pxd --- lxml-2.3.4.orig/src/lxml/tree.pxd 2011-09-25 18:58:05.000000000 +0200 +++ lxml-2.3.4/src/lxml/tree.pxd 2012-05-03 17:44:50.323798041 +0200 @@ -322,6 +322,7 @@ cdef extern from "libxml/xmlmemory.h": cdef void* xmlMalloc(size_t size) nogil cdef int xmlMemBlocks() nogil + cdef int xmlMemUsed() nogil cdef extern from "etree_defs.h": cdef bint _isElement(xmlNode* node) nogil [1] http://xmlsoft.org/html/libxml-xmlmemory.html#DEBUG_MEMORY [2] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemBlocks [3] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemUsed [4] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemDisplay
Christian Heimes, 03.05.2012 18:37:
I'd like to suggest a trival feature for lxml. libxml2 and libxslt support extended memory debugging when both libraries are compiled with the the '--with-mem-debug' flag [1]. This is useful to debug how many memory is actually used by the XML and XSTL code.
Agreed.
In order to show the amount of used memory and memory blocks, the two functions xmlMemBlocks() [2] and xmlMemUsed() [3] needs to be wrapped. Both functions are available in non-debug builds and return 0 in that case.
It might be worth to wrap the other debug helpers like xmlMemDisplay(FILE *) [4], too.
I can see this being useful. However, I'd also like to keep the API impact low for the majority of users who will never use this. Maybe we can add only two functions, one that returns the current memory info as a tuple (or named tuple in Python versions that support it), and one that allows dumping the entire memory info straight into a file (given by a file path string from Python space for now). Do you think that would work? Stefan
Christian Heimes, 03.05.2012 18:37:
I'd like to suggest a trival feature for lxml. libxml2 and libxslt support extended memory debugging when both libraries are compiled with the the '--with-mem-debug' flag [1]. This is useful to debug how many memory is actually used by the XML and XSTL code.
In order to show the amount of used memory and memory blocks, the two functions xmlMemBlocks() [2] and xmlMemUsed() [3] needs to be wrapped. Both functions are available in non-debug builds and return 0 in that case.
It might be worth to wrap the other debug helpers like xmlMemDisplay(FILE *) [4], too.
[1] http://xmlsoft.org/html/libxml-xmlmemory.html#DEBUG_MEMORY [2] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemBlocks [3] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemUsed [4] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemDisplay
Here's a proposal for an API: https://github.com/lxml/lxml/commit/878b6be76e921a6b2ffc480f5dc173a4d2e6b2ac Please give it a try to see if it suits your needs. Patches are welcome. Stefan
participants (2)
-
Christian Heimes -
Stefan Behnel