Mailman 3 February 2014 - lxml - The Python XML Toolkit

Re: [lxml] copy or deepcopy to detect changes?
by Frank Millman 20 Feb '14

20 Feb '14

----- Original Message ----- From: "Stefan Behnel" <stefan_ml(a)behnel.de> To: <lxml(a)lxml.de> Sent: Thursday, February 20, 2014 9:52 AM Subject: Re: [lxml] copy or deepcopy to detect changes? > Frank Millman, 20.02.2014 08:23: >> I need to make a copy of an etree Element before calling a function, so >> that I can compare it on return to see if it has changed. >> >> Normal assignment does not work, as the two names refer to the same >> object, >> so even if changes are made they still compare equal. >> >> I tried copy.copy() and copy.deepcopy(), and they both seem to work. >> However, I ran a simple benchmark to compare the speed, and deepcopy() >> runs >> much faster than copy(), contrary to my expectation. > > In lxml, copy() internally calls deepcopy(). That's an implementation > detail, though, and is due to the split between the C level tree and the > Python level view on it. The right thing to do is to write what you > intend, > not what happens to be faster in this case. > > The speed difference you get might just be the internal call overhead. > Deep > copying is *very* fast, even though I guess you only benchmarked it with a > fairly small tree anyway. > Thanks for the reply, Stefan. Actually I have just realised that everything I said above was rubbish - sorry about that. My benchmark was mistakenly copying the string version of the element, not the element itself. Ahem, I am still wrong - 'copying' a long string is quicker than 'deepcopying' it. What I was actually doing was copying a byte-encoded string. A simple test shows that python (3.3.2 in my case) is blindingly fast to 'deepcopy' a byte-encoded string, and comparatively slow to 'copy' it. Now that I am copying the actual etree Element, the speeds are fairly similar, but deepcopy is slightly slower, which is what I would expect. Sorry for the noise. Frank

1 0

copy or deepcopy to detect changes?
by Frank Millman 19 Feb '14

19 Feb '14

Hi all I need to make a copy of an etree Element before calling a function, so that I can compare it on return to see if it has changed. Normal assignment does not work, as the two names refer to the same object, so even if changes are made they still compare equal. I tried copy.copy() and copy.deepcopy(), and they both seem to work. However, I ran a simple benchmark to compare the speed, and deepcopy() runs much faster than copy(), contrary to my expectation. So two questions - 1. Is this expected? 2. Is it safe to use deepcopy for my purposes? Thanks Frank Millman

2 1

white space problem
by Martin Mueller 19 Feb '14

19 Feb '14

I have a TEI text in which word or punctuation tokens are wrapped in <w> or <pc> tags and white space is wrapped in <c> tags. Now I want to reduce that text to a version in which the contents of <w>, <pc>, and <c> elements become the text nodes of higher structural elements like <l> or <p>. I can do this with simple command etree.strip_tags(dom, tei + 'w', tei + 'pc', tei+'c') This leaves a lot of newlines and white space in the serialized output. If I want to make the XML text more human readable, I can do this by first serializing the text with text = etree.tostring(dom, encoding = 'unicode') and then using half a dozen or more regular expressions to clean it up. Such as whitespace = re.compile(r'\s+') #delete redundant blank space and newlines punctuation = re.compile(r'(\w)\s+([,;:.?!])') #remove space before punctuation and a sequence of commands like text = whitespace.sub(' ', text) text = punctuation.sub(r'\1\2', text) This works and is even reasonably fast, but it strikes me as very kludgy, and it becomes very awkward if you want to add newlines before or after certain elements, depending on where they are in the hierarchy. Is there a way of tackling this problem within lxml? My assumption is there isn't and white space between elements is a case of "that's not my department, says Wernher von Braun." Martin Mueller Professor emeritus of English and Classics Northwestern University

2 1

Re: [lxml] white space problem
by Holger Joukl 18 Feb '14

18 Feb '14

Hi, > Von: Martin Mueller <martinmueller(a)northwestern.edu> > An: "lxml(a)lxml.de" <lxml(a)lxml.de>, > > I have a TEI text in which word or punctuation tokens are wrapped in <w> > or <pc> tags and white space is wrapped in <c> tags. Now I want to reduce > that text to a version in which the contents of <w>, <pc>, and <c> > elements become the text nodes of higher structural elements like <l> or > <p>. I can do this with simple command > > etree.strip_tags(dom, tei + 'w', tei + 'pc', tei+'c') > > This leaves a lot of newlines and white space in the serialized output. If > I want to make the XML text more human readable, I can do this by first > serializing the text with > > text = etree.tostring(dom, encoding = 'unicode') > > [...] > This works and is even reasonably fast, but it strikes me as very kludgy, > and it becomes very awkward if you want to add newlines before or after > certain elements, depending on where they are in the hierarchy. Is there a > way of tackling this problem within lxml? My assumption is there isn't and > white space between elements is a case of "that's not my department, says > Wernher von Braun." Have you looked into (re)parsing with a parser that removes whitespace from elements that are not data-only, either before or after you do strip_tags(), like (http://lxml.de/tutorial.html#parsing-from-strings-and-files): >>> parser = etree.XMLParser(remove_blank_text=True) And maybe using tostring(..., pretty_print=True) for a (hierarchically) formatted final text representation? You might also consider iterating over the tree and "normalizing" whitespace in elements' .text attributes by s.th. like " ".join(elem.text.split()): >>> " ".join("fooo, \n bar moo ".split()) 'fooo, bar moo' >>> I must admit to not having any experience with TEI and not understanding your intention too well. Could you give a small sample of your source document and how you want it to look like after the transformation? Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart

3 2

Core dump in etree
by Frank Astier 13 Feb '14

13 Feb '14

Hi - I am using lxml etree 2 ways to parse a file. The first way works, the second core dumps. But the second way seems to be allowed by the doc. Any idea? First way - no problem: >>> from lxml import etree >>> etree.parse(open("test.src")) <lxml.etree._ElementTree object at 0xd41a70> Second way - core dump: >>> from lxml import etree >>> etree.parse("test.src") Segmentation fault (core dumped) Test.src xml file (from w3c site): "<note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>" I'm in python 2.6.6 on linux 2.6.32-131.4.1.el6.x86_64 and my version of lxml is 2.3.4. Thans for any tip, Frank

2 1

how to validate this xml
by Jachym Cepicky 13 Feb '14

13 Feb '14

Hi, I would like to validate this XML: http://geo102.fsv.cvut.cz:8080/geoserver/sf/ows?service=WFS&version=1.0.0&r… I tried: 1. Download the xml (bugsites.gml) 2. Downloaded the schema http://geo102.fsv.cvut.cz:8080/geoserver/schemas/wfs/1.0.0/WFS-basic.xsd (schemax.xml) 3. written script from lxml import etree xmlschema_doc = etree.parse(open('schema.xsd')) xmlschema = etree.XMLSchema(xmlschema_doc) doc = etree.parse(open('bugsites.gml')) print xmlschema.validate(doc) It ends up with False response :-( Thanks Jachym -- Jachym Cepicky URL: http://les-ejk.cz e-mail: jachym.cepicky at gmail com PGP: http://les-ejk.cz/pgp/JachymCepicky.pgp @jachymc

2 1

lxml 3.3.1 released
by Stefan Behnel 12 Feb '14

12 Feb '14

Hi all, I'm happy to announce the release of lxml 3.3.1. This is a bug-fix-only release for the stable lxml 3.3 series. It also fixes build problems on Windows, so we can hope for Windows binary builds becoming available soon. The documentation is here: http://lxml.de/ Download: http://lxml.de/files/lxml-3.3.1.tgz Signature: http://lxml.de/files/lxml-3.3.1.tgz.asc Changelog: http://lxml.de/3.3/changes-3.3.1.html Github: https://github.com/lxml/lxml/commit/3f8cba77030383ad52e45263d416e903ad2bea4d This release was built using Cython 0.20.1. If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.3.1 (2014-02-12) ================== Bugs fixed ---------- * LP#1014290: HTML documents parsed with ``parser.feed()`` failed to find elements during tag iteration. * LP#1273709: Building in PyPy failed due to missing support for ``PyUnicode_Compare()`` and ``PyByteArray_*()`` in PyPy's C-API. * LP#1274413: Compilation in MSVC failed due to missing "stdint.h" standard header file. * LP#1274118: iterparse() failed to parse BOM prefixed files.

1 0

suppress namespace prefix in printouts
by Martin Mueller 04 Feb '14

04 Feb '14

If you use lxml to extract elements from an XML text associated with a schema, you must give the program the element name with its namespace, and it will print out the element with its name space. Thus a TEI element <w xml:id="someID">Mary</w> will be written out to the outputfile as <w xmlns="http://www.tei-c.org/ns/1.0" xml:id="someID">Mary</w> Is there a way of suppressing the name space so that the output looks like the input, i.e. <w xml:id="someID">Mary</w> Martin Mueller Professor emeritus of English and Classics Northwestern University

3 8

lxml - The Python XML Toolkit February 2014