Mailman 3 June 2005 - lxml - The Python XML Toolkit

[lxml-dev] Type returned by xpath
by Florian Wagner 01 Jul '05

01 Jul '05

Hi, I want to be able to do something like this with lxml for attr in doc.xpath('//somenode@someattr'): attr.text = 'sometext' But since the xpath method returns strings for the attributes this is not possible. Is there some way to make xpath return a list of node objects instead? Greetings Florian

2 1

[lxml-dev] Feature request / patch: Copy a node
by Florian Wagner 01 Jul '05

01 Jul '05

I do have to make copies of nodes in an application I'm writing, so I've tried making a patch for lxml to allow me to do that. The patch is attached, but since I don't have experience with Pyrex and libxml2 there could well be something wrong with it. It seems to work, though. Also I don't know if creating a new document / _ElementTree for the new nodes is the right thing to do, but I suppose it should be... Regards Florian Index: src/lxml/etree.pyx =================================================================== --- src/lxml/etree.pyx (revision 12556) +++ src/lxml/etree.pyx (working copy) @@ -370,7 +370,21 @@ c_node = c_node.next else: raise ValueError, "Matching element could not be found" + + def copy(self, recursive = True): + # 1 = copy recursive; 2 = don't copy recursive + recursive = int(not recursive) + 1 + + cdef xmlNode* c_node + cdef xmlDoc* c_doc + c_doc = theParser.newDoc() + etree = _elementTreeFactory(c_doc) + c_node = tree.xmlDocCopyNode(self._c_node, c_doc, recursive) + tree.xmlDocSetRootElement(c_doc, c_node) + + return _elementFactory(etree, c_node) + # PROPERTIES property tag: def __get__(self):

2 3

[lxml-dev] More timing and profiling
by Olivier Grisel 01 Jul '05

01 Jul '05

Hi list, I updated my script to use the 'profile' module instead of 'hotshot' since 'profile' displays calls to C functions and builtins. Before, actually profiling, I wanted to check that cElementTree was actually faster on the local branching operation by performing it three times (only on the first 100 revisions of BZR instead of more than 700 at the time of writing): ogrisel@localhost:~/Developments/lxml-testcase/grisel/profiling $ python profile_bzr.py -a timeit -x lxml Timing against /tmp/bzr.dev-lxml/bzrlib/__init__.pyc branching took 2.34565591812 ogrisel@localhost:~/Developments/lxml-testcase/grisel/profiling $ python profile_bzr.py -a timeit -x cetree Timing against /tmp/bzr.dev/bzrlib/__init__.pyc branching took 1.86508107185 So cElementTree is faster on the local branching operation. Now here is the profile of the same operation in both cases: ogrisel@localhost:~/Developments/lxml-testcase/grisel/profiling $ python profile_bzr.py -a profile -x lxml Profiling against /tmp/bzr.dev-lxml/bzrlib/__init__.pyc Added 230 texts. Added 100 inventories. Added 100 revisions. Thu Jun 23 23:42:22 2005 /tmp/bzr_data.profile 385004 function calls (374095 primitive calls) in 5.700 CPU seconds Ordered by: internal time, call count List reduced from 429 to 20 due to restriction <20> ncalls tottime percall cumtime percall filename:lineno(function) 64844 0.860 0.000 0.860 0.000 :0(get) 430 0.520 0.001 0.520 0.001 :0(compress) 8992 0.430 0.000 1.310 0.000 /tmp/bzr.dev-lxml/bzrlib/inventory.py:189(from_element) 21611 0.390 0.000 0.600 0.000 /usr/lib/python2.4/posixpath.py:56(join) 17627/9095 0.230 0.000 0.470 0.000 /tmp/bzr.dev-lxml/bzrlib/inventory.py:317(iter_entries) 9009 0.190 0.000 0.770 0.000 /tmp/bzr.dev-lxml/bzrlib/store.py:132(__contains__) 10165 0.170 0.000 0.490 0.000 /tmp/bzr.dev-lxml/bzrlib/store.py:69(_path) 204 0.160 0.001 0.160 0.001 :0(parse) 18878 0.160 0.000 0.160 0.000 :0(access) 22295 0.130 0.000 0.130 0.000 :0(startswith) 1 0.130 0.130 4.730 4.730 /tmp/bzr.dev-lxml/bzrlib/branch.py:727(update_revisions) 3601/1238 0.120 0.000 0.540 0.000 /tmp/bzr.dev-lxml/bzrlib/changeset.py:682(get_new_path) 9977 0.110 0.000 0.230 0.000 /usr/lib/python2.4/posixpath.py:74(split) 430 0.110 0.000 0.110 0.000 :0(flush) 103 0.110 0.001 1.540 0.015 /tmp/bzr.dev-lxml/bzrlib/inventory.py:487(from_element) 1218 0.110 0.000 0.110 0.000 :0(decompress) 9090 0.100 0.000 0.120 0.000 /tmp/bzr.dev-lxml/bzrlib/inventory.py:408(add) 3184 0.090 0.000 0.090 0.000 :0(seek) 1836 0.080 0.000 0.480 0.000 /usr/lib/python2.4/gzip.py:242(_read) 22160 0.080 0.000 0.080 0.000 :0(endswith) ogrisel@localhost:~/Developments/lxml-testcase/grisel/profiling $ python profile_bzr.py -a profile -x cetree Profiling against /tmp/bzr.dev/bzrlib/__init__.pyc Added 230 texts. Added 100 inventories. Added 100 revisions. Thu Jun 23 23:42:44 2005 /tmp/bzr_data.profile 404062 function calls (393056 primitive calls) in 5.120 CPU seconds Ordered by: internal time, call count List reduced from 431 to 20 due to restriction <20> ncalls tottime percall cumtime percall filename:lineno(function) 430 0.560 0.001 0.560 0.001 :0(compress) 21611 0.460 0.000 0.650 0.000 /usr/lib/python2.4/posixpath.py:56(join) 8992 0.350 0.000 0.670 0.000 /tmp/bzr.dev/bzrlib/inventory.py:189(from_element) 64642 0.310 0.000 0.310 0.000 :0(get) 1 0.250 0.250 3.990 3.990 /tmp/bzr.dev/bzrlib/branch.py:727(update_revisions) 9977 0.220 0.000 0.310 0.000 /usr/lib/python2.4/posixpath.py:74(split) 18878 0.200 0.000 0.200 0.000 :0(access) 17627/9095 0.170 0.000 0.480 0.000 /tmp/bzr.dev/bzrlib/inventory.py:317(iter_entries) 204 0.160 0.001 0.310 0.002 :0(_parse) 9009 0.140 0.000 0.650 0.000 /tmp/bzr.dev/bzrlib/store.py:132(__contains__) 3601/1238 0.130 0.000 0.680 0.001 /tmp/bzr.dev/bzrlib/changeset.py:682(get_new_path) 22295 0.120 0.000 0.120 0.000 :0(startswith) 10165 0.110 0.000 0.380 0.000 /tmp/bzr.dev/bzrlib/store.py:69(_path) 9090 0.100 0.000 0.120 0.000 /tmp/bzr.dev/bzrlib/inventory.py:408(add) 2639 0.090 0.000 0.450 0.000 /usr/lib/python2.4/gzip.py:242(_read) 430 0.080 0.000 0.080 0.000 :0(flush) 15767 0.080 0.000 0.080 0.000 :0(rstrip) 103 0.080 0.001 0.880 0.009 /tmp/bzr.dev/bzrlib/inventory.py:487(from_element) 1617 0.080 0.000 0.080 0.000 :0(decompress) 22160 0.070 0.000 0.070 0.000 :0(endswith) The main difference is the :0(get) time that is much higher in the lxml case. C functions are tagged :0(function_name) and agreggated by names. This is annoying since for instance, lxml/_elementpath.py:167(_compile) calls C function named 'get' whose stats get agreggated whith the stats of all of other 'get' C functions of the python standard library. One has to use the print_callers method of the pstats.Stats object to list all the different contribitions: :0(get) /tmp/bzr.dev-lxml/bzrlib/changeset.py:809(longest_to_shortest)(194) 0.000 /tmp/bzr.dev-lxml/bzrlib/changeset.py:829(rename_to_temp_delete)(98) 0.000 /tmp/bzr.dev-lxml/bzrlib/changeset.py:870(rename_to_new_create)(98) 0.100 /tmp/bzr.dev-lxml/bzrlib/changeset.py:1133(get_inventory_change)(98) 0.070 /tmp/bzr.dev-lxml/bzrlib/changeset.py:1366(make_basic_entry)(198) 0.160 /tmp/bzr.dev-lxml/bzrlib/changeset.py:1516(get_path)(100) 0.010 /tmp/bzr.dev-lxml/bzrlib/inventory.py:189(from_element)(62944) 1.430 /tmp/bzr.dev-lxml/bzrlib/revision.py:159(unpack_revision)(808) 0.040 /usr/lib/python2.4/encodings/__init__.py:69(search_function)(3) 0.000 /usr/lib/python2.4/site-packages/lxml/_elementpath.py:167(_compile)(202) 0.010 /usr/lib/python2.4/sre.py:213(_compile)(100) 0.000 /usr/lib/python2.4/sre_parse.py:225(_class_escape)(1) 0.000 This :0(get) time is the main difference between the two profiles. The parsing times :0(_parse) and :0 are(parse) are the same. So to summarize, this study tells us that to improve lxml over cElementTree on this particular case, one should first focus on optimizing the following piece of code (or the related code that uses it in lxml) in _elementpath.py: """ _cache = {} ## # (Internal) Compile path. def _compile(path): p = _cache.get(path) if p is not None: return p p = Path(path) if len(_cache) >= 100: _cache.clear() _cache[path] = p return p """ Best, -- Olivier

2 4

[lxml-dev] libxml2-2.2.19/libxslt-1.1.14 with lxml on Mac OS X
by Jan-Wijbrand Kolman 28 Jun '05

28 Jun '05

Hi, In my attempt to install lxml on my Powerbook, I started out using the latest libxml2 and libxslt versions - 2.2.19 and 1.1.14 respectively. Compiling lxml did seem to work, however, the tests wouldn't run and crash with segfaults. Since it was confirmed that lxml did actually compile and run using these versions of libxml2 and libxslt on Linux, I initially thought it was me doing something wrong, either in installing lxml prerequisites or lxml itself. After wasting some more hours trying :) it finally occured to me to actually try the libxml2 and libxslt versions as advertised in the lxml installation instructions: 2.2.16 and 1.1.12. Compiling then worked just fine and indeed the lxml tests run without problems too! I guess, for lxml to work with latest libxml2 and libxslt on OS X some work needs to be done? Some more details: I'm using OS X 10.3.9, tried with both Python2.3 from Fink 'unstable' and a 'manually' installed Python2.3. In both cases I 'manually' installed libxml2 and libxslt (since Fink's libxml2/libxslt do not include the Python bindings). If I can be somehow of any help in debugging this, or if I need to provide more details, I'm glad to! kind regards, jw -- Jan-Wijbrand Kolman jw(a)infrae.com

1 0

[lxml-dev] [PATCH] cElementTree compat for bazaaz-ng
by Olivier Grisel 22 Jun '05

22 Jun '05

Hi list, I have identified 3 incompatibility issues that prevented lxml.etree to be used in place of Fredrik's cElementTree (or ElementTree) module in for the bazaar-ng SCM system. I attach a unittest test suite (test_grisel.py) that highlights those issues. It is meant to be added to the unittests area of the testcase area on codespeak: http://codespeak.net/svn/lxml/testcase/unittests/ However, it is runs perfectly as standalone script (you just need lxml and optionally cElementTree in your PYTHONPATH). Change the 'import' lines to feel the difference :) And here is the good news: the second attached file is a patch against the current trunk (rev 13608) to make lxml compatible enough to make those tests pass and bzr's test suite as well :) However, I used the basic self._c_doc replacement and I have no clue whether or not this will cause memory leaks (I don't know pyrex nor libxml2 at all). And now the benchmarks: % rsync -av --delete bazaar-ng.org::bazaar-ng/bzr/bzr.dev /tmp/ % cp -r /tmp/bzr.dev/ /tmp/bzr.dev-lxml/ % perl -p -i -e 's/from cElementTree/from lxml.etree/g' /tmp/bzr.dev-lxml/bzrlib/*.py # bzr log with cElementTree % cd /tmp/bzr.dev/ % time python ./bzr log > /dev/null real 0m0.524s user 0m0.481s sys 0m0.037s # bzr log with lxml.etree % cd /tmp/bzr.dev-lxml/ % time python ./bzr log > /dev/null real 0m0.394s user 0m0.338s sys 0m0.049s I did repeat the measure 10 times each, and the results where almost always the same. So lxml is a bit faster on this case. Regards, -- Olivier

2 6

[lxml-dev] StringIO vs cStringIO
by mss＠mawhrin.net 18 Jun '05

18 Jun '05

Hi Now that the issue with gcc 4.0 is resolved :) I have another question. I propose the following patch to support cStringIO whlie providing the document to parse. For me it gave some performance improvement. Regards -- Misha

1 0

[lxml-dev] lxml on gcc 4.0
by Martijn Faassen 17 Jun '05

17 Jun '05

Hi there, I did some hacking against Pyrex to see whether I could make it compile lxml with gcc 4.0. Unguided by any actual knowledge, I created a patch which goes on top of the Red Hat patch, which then is applied to Pyrex 0.9.3. I've attached the two patches. Apply the pyrex-0.9.2.1-gcc4.patch first (it does apply against 0.9.3 too), and then my gcc-4.0-hack.patch. I'd be happy to hear reports on whether this works for others! Regards, Martijn

4 13

[lxml-dev] tostring(None) Segfaults with 0.7
by dharana 16 Jun '05

16 Jun '05

# python Python 2.4.1 (#1, Jun 2 2005, 01:24:02) [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import lxml.etree >>> xmldoc = lxml.etree.fromstring('<section></section>') >>> omlnode = xmldoc.find('oml') >>> type(omlnode) <type 'NoneType'> >>> lxml.etree.tostring(omlnode) Violación de segmento -- dharana

2 3

[lxml-dev] lxml version?
by dharana 16 Jun '05

16 Jun '05

Is there any way of checking the current version of lxml? If it's not possible could it be possible to add it lxml._version? It would make it easier to write check scripts. Regards, -- dharana

2 1

[lxml-dev] lxml 0.7 released!
by Martijn Faassen 15 Jun '05

15 Jun '05

Hey everybody, I've just released lxml 0.7. This is a release with quite a bit of new features and bug fixes, including XPath expression parameters, XInclude support, XMLSchema validation support, more namespace prefix support, better encoding support, and more. As usual, it can be found here: http://codespeak.net/lxml/ Regards, Martijn

3 7