Mailman 3 April 2011 - lxml - The Python XML Toolkit

Re: [lxml] Parsing HTML files with HTML entities
by Stefan Behnel April 20, 2011

April 20, 2011

Henrik, 20.04.2011 16:05: > Hello list, > > I've searched around but can't find an answer on this. > > The problem is that if I parse some HTML which have certain characters converted to HTML enties i.eö they are stripped away. > > I.e<h1>Björn</h1> becomes<h1>Bjrn</h1> > > I'm using lxml 2.3 on Mac OS X 10.6 > > The parser is setup up like this; > > parser = html.XHTMLParser(recover=True, ns_clean=True, remove_blank_text=True, resolve_entities=False) Using "recover=True" with an XML parser has a clear smell to me. Fix your data instead. Regarding XHTML entity references, consider reading the documentation: http://lxml.de/api/lxml.html.XHTMLParser-class.html Stefan

1 0

lxml.html.html5parser causes lxml.etree._utf8 to raise TypeError
by Simon Sapin April 20, 2011

April 20, 2011

Hi, I get an exception when I try to use lxml.html.html5parser, with any source. Here is a test case: from lxml.html import html5parser html5parser.fromstring('') And the traceback I get: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/lxml/html/html5parser.py", line 137, in fromstring guess_charset=guess_charset) File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/lxml/html/html5parser.py", line 54, in document_fromstring return parser.parse(html, useChardet=guess_charset).getroot() File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/html5lib/html5parser.py", line 211, in parse parseMeta=parseMeta, useChardet=useChardet) File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/html5lib/html5parser.py", line 111, in _parse self.mainLoop() File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/html5lib/html5parser.py", line 194, in mainLoop self.phase.processEOF() File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/html5lib/html5parser.py", line 589, in processEOF self.parser.phase.processEOF() File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/html5lib/html5parser.py", line 600, in processEOF self.insertHtmlElement() File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/html5lib/html5parser.py", line 595, in insertHtmlElement self.tree.insertRoot(impliedTagToken("html", "StartTag")) File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/lxml/html/_html5builder.py", line 91, in insertRoot root_element = self.elementClass(name) File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/html5lib/treebuilders/etree.py", line 31, in __init__ namespace)) File "/home/ssapin/virtualenvs/weasy/lib/python2.7/site-packages/lxml/html/__init__.py", line 1563, in Element v = html_parser.makeelement(*args, **kw) File "parser.pxi", line 878, in lxml.etree._BaseParser.makeelement (src/lxml/lxml.etree.c:74798) File "apihelpers.pxi", line 120, in lxml.etree._makeElement (src/lxml/lxml.etree.c:11828) File "apihelpers.pxi", line 1462, in lxml.etree._getNsTag (src/lxml/lxml.etree.c:23247) File "apihelpers.pxi", line 1480, in lxml.etree.__getNsTag (src/lxml/lxml.etree.c:23373) File "apihelpers.pxi", line 1364, in lxml.etree._utf8 (src/lxml/lxml.etree.c:22190) TypeError: Argument must be bytes or unicode, got 'dict' The versions are: Python : sys.version_info(major=2, minor=7, micro=1, releaselevel='final', serial=0) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) html5lib : 0.90 The test case is so simple that I fail to see how I could have gotten it wrong. Any idea of what is going on? Best regards, -- Simon Sapin

1 0

Parsing HTML files with HTML entities
by Henrik April 20, 2011

April 20, 2011

Hello list, I've searched around but can't find an answer on this. The problem is that if I parse some HTML which have certain characters converted to HTML enties i.e ö they are stripped away. I.e <h1>Björn</h1> becomes <h1>Bjrn</h1> I'm using lxml 2.3 on Mac OS X 10.6 The parser is setup up like this; parser = html.XHTMLParser(recover=True, ns_clean=True, remove_blank_text=True, resolve_entities=False) //Henrik

2 1

Value error at 'import etree'
by Pelle Svensson April 20, 2011

April 20, 2011

Hi, I can't import lxml att all. test.py source only one line from lxml import etree []$ python3 test.py Traceback (most recent call last): File "test.py", line 2, in <module> from lxml import etree File "xmlschema.pxi", line 22, in init lxml.etree (src/lxml/lxml.etree.c:148019) File "xpath.pxi", line 396, in lxml.etree.XPath.__init__ (src/lxml/lxml.etree.c:107688) File "xpath.pxi", line 117, in lxml.etree._XPathEvaluatorBase.__init__ (src/lxml/lxml.etree.c:104888) File "xpath.pxi", line 55, in lxml.etree._XPathContext.__init__ (src/lxml/lxml.etree.c:104232) File "extensions.pxi", line 83, in lxml.etree._BaseContext.__init__ (src/lxml/lxml.etree.c:96555) File "extensions.pxi", line 120, in lxml.etree._BaseContext._to_utf (src/lxml/lxml.etree.c:97031) File "apihelpers.pxi", line 1295, in lxml.etree._utf8 (src/lxml/lxml.etree.c:21267) ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes Linux Fedora 14 Python 3.1.2 python-lxml.x86_64 2.2.8-1.fc14 In http://code.google.com/p/html5lib/issues/detail?id=139 they indicate that the encodings table is bogus but I can't figure out what to do!

2 2

Fwd: ANN: whole day lxml training in Leipzig/Germany on Mai 20, 2011
by Stefan Behnel April 20, 2011

April 20, 2011

Hi, just a little reminder. There are still places left for this event. Stefan -------- Original-Message -------- Hi all, I'm giving a one day introductory level XML/lxml training at the Python Academy in Leipzig, Germany, titled "High Performance XML with Python", on Mai 19, 2011. Course language is English. You can find the course page and registration links here: http://www.python-academy.com/courses/python_course_xml.html It is part of a one week Python Power Course with medium to advanced level Python courses (to be booked separately or in a row): http://www.python-academy.com/courses/python_power_course.html Please forward this announcement to interested parties. Thanks! Stefan ## Target Audience The course targets medium level to experienced Python programmers who want to generate and/or process XML (and, to some extend, HTML) content efficiently. A basic understanding of XML is helpful but not required. ## Motivation The objective of this course if to get an understanding of important XML technologies, and to learn how to use the available tools by example. ## Content Initially, the course will build up a common understanding of XML (specifically the XML Infoset) and some of its applications. The main theme then deals with efficient processing of XML (and a bit of HTML) in Python. The presented tool set includes the ElementTree library that comes with Python since version 2.5, and the freely available lxml library that combines a compatible Python API with a large set of additional XML features. 1) Introduction to XML * XML and the XML Infoset * XML Namespaces * Dealing with XML formats 2) Fast XML processing * Parsing and serializing XML files * Extracting information from XML documents (tree navigation, XPath, CSS selectors) * Processing and transforming XML documents in main memory * Generating XML documents * Stream processing of large XML files that do not fit into main memory 3) Advanced topics * Creating proprietary XML formats * Validating XML formats with schema languages (e.g. RelaxNG, Schematron) * Binding XML documents to Python objects (lxml.objectify) * Creating application specific XML APIs with lxml * Introduction to stylesheet transformations (XSLT processing) Note that the advanced topics are subject to time constraints. A choice will be made based on the interest of the participants. _______________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de http://mailman-mail5.webfaction.com/listinfo/lxml

1 0

xslt fn:replace function
by Pasithee Jupiter April 19, 2011

April 19, 2011

Hi All Does lxml supports xslt functions ( http://www.w3schools.com/xpath/xpath_functions.asp) I have the following xsl file <xsl:template match="PropertyIsLike"> <xsl:variable name="literal" select="//Literal"/> <xsl:variable name="wildcard" select="//Literal[@wildCard]"/> <Statement><xsl:value-of select="//ValueReference" /> LIKE '<xsl:value-of select="fn:replace($literal, $wildcard, '%')" />'</Statement> </xsl:template> and try to replace some characters in the //Literal tag of the xml file xml file looks like the following <Filter><PropertyIsLike wildCard="*" singleChar="?" escapeChar="!"><ValueReference>highway</ValueReference><Literal>bus?st*</Literal></PropertyIsLike></Filter> Any hint, ideas, ... welcome. Thans a lot in advance

3 2

Xml Special Characters Escaping issue
by Aditi Barua April 15, 2011

April 15, 2011

Hello, I am working with a localization tool Wordforge Editor. I am trying to keep the tags inside a text of a xliff file without changing the xml special characters. For example, here is a source string and a target string of a xliff file. <source xml:lang="en-us"> *Texts before inline tag<bpt id="1">texts inside inline tag</bpt> Texts after inline tag*</source> <target"> *Texts before inline tag<bpt id="1">texts inside inline tag</bpt> Texts after inline tag*</target> The source and target string can be shown in the localization tool and user translates the English target to another language. Here in the target, the text *"<bpt id="1">texts inside inline text</bpt>"*has to be kept as it is. It can not be changed or distorted. But when I save the text in the xliff file, it can not keep the original tag. Rather it escapes the special character of xml tag "<" and ">" to *"<"* and *">" * respectively. As a result, the string is saved in the file as <target>Texts before inline tag *<bpt id="1"> texts inside inline tag </bpt>* Texts after inline tag <target>* *It supposed to be saved in the file like this : * *<target"> *Texts before inline tag<bpt id="1">texts inside inline tag</bpt> Texts after inline tag*</target> * *Can you please tell me how can I prevent xml special characters*("<" and ">")* from being replaced with character entities (*"<"* and *">"*) while saving in the file. Is there any function or technique in lxml library to handle this? *-- * Thank you, Aditi.

2 1

Memory usage with iterparse()
by Roger Hoover April 12, 2011

April 12, 2011

Hi, I'm trying to use iterparse to parse a large XML file without bringing it all into memory at once. However, I notice that the position of the element in the XML file makes a huge different how much memory gets used. In this case, I'm parsing a 41MB xml file with 1 "<venue>" xml element near the top of the file and others at the bottom of the file. When parsing the "<venue>" element near the top of the file, memory usage is low (~6K) but when we get down to the bottom of the file it jumps to ~150K. Any ideas how to avoid this? [rhoover@localhost]$ ps -o "rss,pmem,vsz" -p 8665 RSS %MEM VSZ 5876 0.5 11520 [rhoover@localhost]$ ps -o "rss,pmem,vsz" -p 8665 RSS %MEM VSZ 149744 14.4 155688 import time from lxml import etree def do_it(elem): time.sleep(10) def fast_iter(context, func): for event, elem in context: func(elem) elem.clear() del context context = etree.iterparse("vtop.xml", events=['end'], tag='venue') fast_iter(context, do_it)

3 9

Skipping import of schema ... , since this namespace was already imported ...
by Martin Paces April 11, 2011

April 11, 2011

Hello, I am using following python code to load an XML Schema to be used for later XML valiadtion: #------------ import urllib from lxml import etree schema_url = "http://schemas.opengis.net/wcs/1.1/wcsGetCapabilities.xsd" schema_base = "http://schemas.opengis.net/wcs/1.1/" schema = etree.XMLSchema( etree.XML( urllib.urlopen( schema_url ).read() , base_url = schema_base ) ) #------------ The etree.XMLSchema fails with following error: File "xmlschema.pxi", line 103, in lxml.etree.XMLSchema.__init__ (src/lxml/lxml.etree.c:116069) lxml.etree.XMLSchemaParseError: Element '{http://www.w3.org/2001/XMLSchema}import': Skipping import of schema located at 'http://www.w3.org/2001/xml.xsd' for the namespace 'http://www.w3.org/XML/1998/namespace', since this namespace was already imported with the schema located at 'http://schemas.opengis.net/gml/3.1.1/smil/xml-mod.xsd'., line 20 The issue is obvious two included independent schema documents include the same schema, but I see no reason why the XMLSchema constructor should fail by throwing an exception, N.B., when the exception says it is skipping the action. It should continue to work rather than to give it up. So far I managed to test this issue on several Linux distributions with the same result: Ubuntu 10.4 lxml 2.2.4 Debian Lenny lxml 2.1.1 ScinetificLin.6 lxml 2.2.3 I plan to download and check the latest 2.3 version and possibly to try an alternative XML Schema validator. But maybe I could get a faster answer from someone of you saving me some time. Also, I am not sure whether this issue should be reported as a bug, but I guess so. Thank you. Martin

2 4

AssertionError: ElementTree not initialized, missing root
by Baþar ÖZDEMÝR April 9, 2011

April 9, 2011

Hi, Why the crash? I am thinking about that it must be related with the web site which i am parsing. Please point me in the right direction. Regards. (lxml-2.3 on Python 2.5.2) The code: #-*- coding: utf-8 -*- #!/usr/bin/env python import lxml.html root = "http://www.forumla.net" r = lxml.html.parse(root) title = r.find(".//title").text print title The traceback File "ll.py", line 8, in <module> title = r.find(".//title").text File "lxml.etree.pyx", line 1936, in lxml.etree._ElementTree.find (src/lxml/lxml.etree.c:44987) File "lxml.etree.pyx", line 1698, in lxml.etree._ElementTree._assertHasRoot (src/lxml/lxml.etree.c:42865) AssertionError: ElementTree not initialized, missing root

2 1