Mailman 3 - lxml - The Python XML Toolkit

[lxml-dev] Memory management redux
by Martijn Faassen Dec. 11, 2004

Dec. 11, 2004

Hi there, Inspired by discussions with Vic and through browsing the vlibxml2 code, I've implemented bit of memory management functionality which, after a lot of manual debugging, seems to be doing the right thing. So far... A new addition to the lxml trunk is nodereg, and associated testing stuff (noderegtest.pyx and test_nodereg.py). nodereg is a system for registering Python-level node proxies, plus some base classes for the document and node objects in a typical libxml2 tree wrapper. … [View More]

1 1

[lxml-dev] reorganisation confusion
by Martijn Faassen Dec. 7, 2004

Dec. 7, 2004

Hi there, Tonight I attempted a package reorganisation myself, integrating vlibxml2 into lxml from the other direction, leaving most of lxml's build scripts and testing framework intact. My attempts are here: http://codespeak.net/svn/lxml/branch/lxml-reorg/ The build story works ('make' does the trick). The tests are picked up by the test runner. I noticed that on reorg-20041120 'make' tries to install in site-packages directly, which is something that doesn't happen on my branch. I … [View More]

2 2

[lxml-dev] some comments about the file shuffling
by Martijn Faassen Nov. 26, 2004

Nov. 26, 2004

Hi there, Just took a glance at the reshuffling branch. I'm glad you're doing this work. Some comments about it though: it's now lxml.etree.etree and lxml.etree.dom in the old situation, and intended, it was just lxml.etree and lxml.dom i.e. there's no need for this separate etree package, we can just put the modules in the lxml container package. dom has nothing to do with etree; it's a completely different way to layer over libxml2. Where did the testrunner test.py go? I see you … [View More]

2 2

[lxml-dev] Re: etree compile problem
by Martijn Faassen Nov. 25, 2004

Nov. 25, 2004

Victor Ng wrote: > Does etree in the trunk line compile for you? Yup, it seems to work just fine: > make python2.3 setup.py build_ext -i running build_ext building 'lxml.etree' extension creating build creating build/temp.linux-i686-2.3 creating build/temp.linux-i686-2.3/src creating build/temp.linux-i686-2.3/src/lxml gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/usr/include/python2.3 -c src/lxml/etree.c -o build/temp.… [View More]

2 4

[lxml-dev] etree doesn't build for me
by Victor Ng Nov. 23, 2004

Nov. 23, 2004

Hi all, I've been reorganizing the lxml/etree and vlibxml2 stuff and I'm basically done wedging vlibxml2 into the lxml.vlibxml2 package - but I can't get etree to build, much less run any unittests. Does this work for anyone else? The trunk line doesn't seem to build for me. vic --- "Consequences, Schmonsequences, as long as I'm rich." -- Looney Tunes, Ali Baba Bunny (1957, Chuck Jones)

1 0

[lxml-dev] vlibxml2 and lxml
by Martijn Faassen Nov. 19, 2004

Nov. 19, 2004

Hi there, Back from my honeymoon I see that Victor has released vlibxml2. Cool! Thanks Victor! I hope that we can still converge it all into a single package -- vlibxml2 aims to do a fairly low-level straightforward mapping for parts of the libxml2 API, as I understand it, while lxml aims to raises the level of the API. The approaches don't need to exclude each other however and pooling them together could help both. Victor, I'm curious about the reasons why you didn't use the existing lxml … [View More]

4 17

[lxml-dev] victree
by Martijn Faassen Nov. 12, 2004

Nov. 12, 2004

Hi there, I now found the 'victree' module (as opposed to the empty package) in vlibxml2. Quite interesting. The Python wrapper approach certainly makes it a lot less large than the Pyrex strategy, though I have the hope this will be made easier with vlibxml2 as a base too. etree.pyx would have the benefit of being faster, too, as many of the operations are happening at C speed. What are your plans for victree? Regards, Martijn

1 0

[lxml-dev] memory management strategies
by Victor Ng Oct. 19, 2004

Oct. 19, 2004

Hmm... I've got too many email addresses and the lxml mailing list held up my last email. Lets try that again. vic -- I've released the first embryonic version of the libxml2 wrapper that I've been working on. I'm not feeling very witty so I called it vlibxml2. Short blurb: http://www.crankycoder.com/archives/2004/10/vlibxml2_first.html The tarball: http://www.crankycoder.com/archives/vlibxml2-0.0.1.tar.gz There's a bunch of stuff that still needs work, but in general, I think the … [View More]code is not so embarrassing that I have to hide it anymore. I've tried to maintain API compatibility with libxml2 - with the exception that I do garbage collection for you. I chose not to use ElementTree as the base line, mainly because I don't want to have to think too much as I add in more features from libxml2's C API. Adding abstractions like ElementTree over top of a properly garbage collected libxml2 should be easy anyway - I'm more concerned with having a complete XML featureset. === How to use it: Easy - it's basically a really really stripped down libxml2. import vlibxml2 node = vlibxml2.newNode('foo') node.setProp('bar', '123') batz= vlibxml2.newNode('batz') node.addChild( batz ) assert node.children()[0] == batz assert batz.parent() == node assert node.prop('bar') == '123' assert node.xpathEval('//batz')[0] == batz That's pretty much everything for now. === Stuff that will get fixed in the immediate future (in order): - get a proper memory leak detector into vlibxml2. I'm planning on porting the libxml2 memory leak detector over. I'm currently just asserting that in DEBUG mode, the garbage collector never free's the same pointer twice. This could potentially give you a false positive failure in the case that you free a node, allocate a new node and free the new node, but the new node has the same address as the old node. Anyway - I really want to fix it as it's the only way I'll really feel comfortable with the memory management. - currently all xmlNode objects belong to some xmlDoc object. Creating a new xmlNode object implicitly creates a root xmlDoc object. This is not the same behavior as libxml2, so I should probably not do the same thing. Plus it's a double malloc/free which is never a really good thing. - add namespace support. I want to be able to generate and parse SOAP, XML-RPC and RSS/Atom. That means I need to suport xmlNamespace stuff. - add xmlReader support. I actually have a real need to parse XML files of ~100MB. I'd like to be able to handle close to 500MB of data if possible. It should at least be possible to do something like that - and that means having an alternate to the DOM way of doing things. Getting this stuff to work may be finicky as it probably involves playing with the global interpreter lock as libxml2 will need to make callbacks to Pyrex code. Luckily - there's been some example code posted to the Pyrex mailing list recently so I'm not too intimidated by this. ==The sort've bad news== This release is being released under the GPL. I'm not-so-slowly turning into a free software nerd from an open source nerd. This may make some people mad - I'm open to changing the license, but for now I'm most comfortable with the GPL. If I hear a good argument, I'd switch to the PSF Python license, but for now - it's GPL. Source access: I've decided to keep all the code on my own SVN repository - I need to sort out some security issues before I start expose the repository to other people. I've already got Trac installed (http://trac.edgewall.com) for bug tracking and wiki documentation. Again - it's just a matter of setting up security and splicing up projects in Trac, but it's coming. vic --- Don't be humble ... you're not that great. -- Golda Meir On 15-Oct-04, at 10:09 AM, Fred Drake wrote: > On Fri, 15 Oct 2004 04:14:07 +0200, Martijn Faassen > <faassen(a)infrae.com> wrote: >> If you want checkin rights for lxml, check with Philipp (see mail to >> Victor). > > Got those already, just haven't had any time to work on them. > > > -Fred > > -- > Fred L. Drake, Jr. <fdrake at gmail.com> > Zope Corporation > _______________________________________________ > lxml-dev mailing list > lxml-dev(a)codespeak.net > http://codespeak.net/mailman/listinfo/lxml-dev > [View Less]

1 0

[lxml-dev] memory management strategies
by Martijn Faassen Oct. 15, 2004

Oct. 15, 2004

Hi everybody, I'm trying to figure out strategies to manage memory automatically in lxml in the context of libxml2 trees. Some of my incomplete thinking and description of the problem has been encapsulated in the following document: http://codespeak.net/svn/lxml/trunk/doc/memorymanagement.txt It's quite possible that I'm making things more complicated than necessary, as Vic said. I'm quite curious to hear Vic's and other people's thinking about this. The basic concept is that libxml2 … [View More]

4 13

[lxml-dev] wrapping libxml2 with ctypes
by Thomas Heller Oct. 11, 2004

Oct. 11, 2004

I'm interested in wrapping (parts of) libxml2 with ctypes. Not because I have a use case for it right now, but because I need a larger project to drive ctypes to the limit, and to get more experience with it. As I understand it, there are already Python bindings using pyrex to parts of libxml - it would also be very interesting to compare the speed of the pyrex bindings to ctypes bindings. So I'm interested in answers to these questions: Are the bindings stable? Is the python api documented?… [View More]

2 3