Mailman 3 May 2008 - lxml - The Python XML Toolkit

[lxml-dev] install lxml 2.0.5 on Mac OS X Leopard - why is it so hard?
by Kumar McMillan 16 May '08

16 May '08

I know this has been discussed over and over but I'm writing to see if anyone has made a breakthrough yet. The problem of course is that Leopard's builtin libxml2 and libxslt are too old for lxml 2.0. You have to build libxml2 either from source or use a port. There is currently a problem with the libxml2 port, but the workaround is going fine for me: http://trac.macports.org/ticket/15230 (I know because postgres built just fine and I have some tests exerising psycopg2 as well) So after updating my libxml2 to 2.6.31 and libxslt to 1.1.23 and updating my $PATH so that the new xml2-config and xslt-config can be found, I can build lxml *without errors* but I see these warnings: $ sudo easy_install lxml-2.0.5.tgz Processing lxml-2.0.5.tgz Running lxml-2.0.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3azY8e/lxml-2.0.5/egg-dist-tmp-t80esG Building lxml version 2.0.5. NOTE: Trying to build without Cython, pre-generated 'src/lxml/etree.c' needs to be available. Using build configuration of libxslt 1.1.23 ld: warning in /opt/local/lib/libxslt.dylib, file is not of required architecture ld: warning in /opt/local/lib/libexslt.dylib, file is not of required architecture ld: warning in /opt/local/lib/libxml2.dylib, file is not of required architecture [... and more like this ...] ... Finished processing dependencies for lxml==2.0.5 What doesn't make sense is these files seem fine to me: $ file -L /opt/local/lib/libxslt.dylib /opt/local/lib/libxslt.dylib: Mach-O dynamically linked shared library i386 $ file -L /opt/local/lib/libexslt.dylib /opt/local/lib/libexslt.dylib: Mach-O dynamically linked shared library i386 I was having similar trouble like this on Tiger and I had test cases in my own test suite that would consistently segfault. On Leopard, those same test cases were *not* segfaulting but now I have some different test cases that are consistently segfaulting. The segfault looks like this in the crash log: Exception Type: EXC_BAD_ACCESS (SIGBUS) Exception Codes: KERN_PROTECTION_FAILURE at 0x0000000000000008 Crashed Thread: 0 Thread 0 Crashed: 0 libxml2.2.dylib 0x90d39419 xmlDictLookup + 360 1 libxml2.2.dylib 0x025626e4 xmlXPathCompExprAdd + 212 2 libxml2.2.dylib 0x025709c6 xmlXPathCompPathExpr + 3910 [etc....] Setting my dyld path (like suggested in the docs, export DYLD_LIBRARY_PATH=/opt/local/lib:/usr/lib) *does* make my test cases run without segfault so I'm assuming what's happening is lxml is using the older dylibs at runtime. This is a really lame way to fix the problem! Specifically, my svn binaries do not like this dylib setting, producing errors like: $ svn ls dyld: lazy symbol binding failed: Symbol not found: _iconv_open Referenced from: /usr/lib/libaprutil-1.0.dylib Expected in: /opt/local/lib/libiconv.2.dylib [etc] (This is slightly odd since I included /usr/lib but whatever.) *sigh* Next, I tried doing a static build of lxml by setting STATIC_LIBRARY_DIRS = ['/opt/local/lib'] in setup.py and running: python setup.py bdist_egg --static --with-xml2-config=/opt/local/bin/xml2-config --with-xslt-config=/opt/local/bin/xslt-config I had to fiddle with gcc to get this to build but otherwise it built fine and installed ok but I did not see any difference. Still consistent segfaults that are fixed by setting the dyld path. Now I'm out of ideas. Does anyone have another suggestion? Until then I have a stupid bash file that I have to source anytime I want to work on lxml. -Kumar

3 15

Re: [lxml-dev] html entities and lxml.html.ElementSoup
by roger patterson 15 May '08

15 May '08

Hi Viksit, What you typed was correct, except you have to note that lxml.html.soupparser.convert_tree(soup) returns a *list* of root elements, so you can't just do a lxml.etree.tostring() on the list. Depending on your HTML, choosing the first element will probably work. I have moved to the trunk now, so am working well with the new lxml.html.soupparser. But if you're stuck on that branch, then that work-around worked for me. Hope it works for you! cheers -Roger 2008/5/14 Viksit Gaur <viksit(a)aya.yale.edu>: > Hi there, > >>Roger Patterson wrote: >>> I'm getting an interesting situation. When using the very cool >>> ElementSoup add-on to lxml.html with certain source-html files that >>> already encode entities (eg. £), using the ElementSoup.parse() >>> messes up the entities. > > I'm running into the same problem. > >>It looks like it's not the parse(), but rather the serialisation. What >> >happens >>is that the entity references end up in the /text/ content, which is >> >clearly >>wrong as it leads to re-escaping of the references on the way out. > >>> What I'm currently doing to solve this is first parsing it with >>> BeautifulSoup(html, convertEntities="html"), then calling >>> ElementSoup.convert_tree(soup). This work-around works fine, but I >>> thought I'd bring it to your attention. > > Did you mean something of the sort, > > soup = BeautifulSoup(doc, convertEntities="html") > root = lxml.html.soupparser.convert_tree(soup) > > Because I get an error of the form: > > File "lxml.etree.pyx", line 2491, in lxml.etree.tostring > (src/lxml/lxml.etree.c:21792) > TypeError: Type 'list' cannot be serialized. > > > >>ElementSoup should do that for you. I fixed it on the trunk. > >>Stefan > > Unfortunately, I can't switch to lxml trunk. Would it be possible for you to > point me to the code change in lxml so I can patch it myself? > > Thanks and Cheers, > Viksit >

1 0

[lxml-dev] Help with an error message
by Konstantin Ryabitsev 14 May '08

14 May '08

Hi, everyone: I'm having trouble with the following case. One of my automatic import scripts takes data from one source and submits it to another as an XML feed. Recently, it started failing because one of the entries contains a null. The testcase is such: from lxml.etree import Element sourcestr = 'Contains a null: \x00' unistr = unicode(sourcestr, 'utf-8') elt = Element('foo').text = unistr Running it will cause the following error: Traceback (most recent call last): File "foo.py", line 6, in <module> elt = Element('foo').text = unistr File "etree.pyx", line 741, in etree._Element.text.__set__ File "apihelpers.pxi", line 344, in etree._setNodeText File "apihelpers.pxi", line 648, in etree._utf8 AssertionError: All strings must be XML compatible, either Unicode or ASCII Can someone suggest the best way to deal with this? Kind regards, -- Konstantin Ryabitsev Montréal, Québec

4 5

Re: [lxml-dev] install lxml 2.0.5 on Mac OS X Leopard - why is it so hard?
by Stefan Behnel 13 May '08

13 May '08

Hi, Mike Meyer wrote: > Apple's official position is that static linking of > applications is unsupported. They don't provide static versions of any > of the system libraries. > > Likewise, macports doesn't provide static libraries for the libraries > it installs, and the docs don't hint at anyway to get it to do so. Great! Now that would have been too easy anyway, wouldn't it? :-/ Thanks for the infos. Now, anyone for a plan B? Stefan

3 4

[lxml-dev] Looking for general insight.
by Kenneth Miller 12 May '08

12 May '08

All, Any opinions on my problem would be greatly appreciated. I've got a large pre-defined XML schema, tons of data types etc. I want to be able to create python objects from the schemas and traffic these objects in and out of some sort of a database. Could I perhaps create these objects using lxml and extend lxml to use zope persistence? Regards, Kenneth Miller

2 1

[lxml-dev] Getting info from an XML file that has invalid character data in it (and how to specify recover option)
by Ben 09 May '08

09 May '08

Hello I'm writing some code to check whether our daily backups worked. Backup Exec stores its results in XML files. Sometimes bad characters - or maybe it is binary data - ends up in these XML files and then lxml chokes: C:\>python sb-lxml.py Traceback (most recent call last): File "sb-lxml.py", line 5, in <module> Xml = etree.parse(XmlFileName) File "lxml.etree.pyx", line 2520, in lxml.etree.parse (src/lxml/lxml.etree.c:22062) File "parser.pxi", line 1309, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:53088) File "parser.pxi", line 1338, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:53337) File "parser.pxi", line 1248, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:52584) File "parser.pxi", line 828, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:50115) File "parser.pxi", line 452, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:47023) File "parser.pxi", line 536, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:47861) File "parser.pxi", line 478, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:47285) lxml.etree.XMLSyntaxError: PCDATA invalid Char value 11, line 132, column 95 The offending line looks like this (not sure if the bad characters will make it through the email): </error><error>Directory not found. Can not backup directory \Data\\l Strategy - Progress Rep.doc\\\??ā?\\VIC-ve\TT\miscellaneous and its subdirectories. Example code to demonstrate how I use it (with lxml-2.0.5 and Python 2.5.2): ################################## Xml = etree.parse(XmlFileName) print Xml.findtext(".//end_time") print Xml.findtext(".//engine_completion_status") ############################## The code works fine unless there are invalid characters in, and I am happy for any suggestion, because the bit I'm interested in is always near the end of the xml file, and there should be a way to get it reliably regardless of the gunk elsewhere in the file (or that's what I hope) Also, I've tried the 'recover' parser option, but I'm doing something wrong, because I get this: C:\>python sb-lxml.py Traceback (most recent call last): File "sb-lxml.py", line 9, in <module> print Xml.findtext(".//end_time") File "lxml.etree.pyx", line 1656, in lxml.etree._ElementTree.findtext (src/lxml/lxml.etree.c:15354) File "lxml.etree.pyx", line 1489, in lxml.etree._ElementTree._assertHasRoot (src/lxml/lxml.etree.c:14116) AssertionError: ElementTree not initialized, missing root The code I tried for the 'recover' parser option: XmlFileName = r'c:/BEX03194.xml' parser = etree.XMLParser(recover=True) Xml = etree.parse(StringIO(XmlFileName), parser) print Xml.findtext(".//end_time") print Xml.findtext(".//engine_completion_status") I guess I'm just specifying the option wrong, but can't see how I should be doing it. Any suggestion, including how to circumvent/work around the problem is most welcome. ReplyReply AllForwardTrash ____________________________________________________________ FREE 3D MARINE AQUARIUM SCREENSAVER - Watch dolphins, sharks & orcas on your desktop! Check it out at http://www.inbox.com/marineaquarium

2 2

[lxml-dev] Building lxml 2.0.5 on RHEL/CentOS 4
by Jeffrey Ollie 09 May '08

09 May '08

Has anyone built lxml 2.0.5 on RHEL 4 or CentOS 4? When I submit it to the Fedora/EPEL buildsystem I get the following error: libxml/schematron.h: No such file or directory I don't have direct access to a RHEL/CentOS 4 box so I can't do much more debugging until I do get one set up. libxml2 is at version 2.6.16 in RHEL/CentOS 4. The full build log is here: http://buildsys.fedoraproject.org/logs/fedora-4-epel/38964-python-lxml-2.0.… Jeff

2 2

[lxml-dev] generative building of xml?
by kris 09 May '08

09 May '08

I am generating, processing and eventually serializing several XML streams. I was wondering if this was possible to do with lxml? Here's the setup. I've got several databases generating XML content (which can be quite large), I really want to be able to process the database record progressively generating XML and sending out on its own stream. An aggregator/filter (elsewhere) will read the streams and parse them processing similar members and generate a new stream based on the combined streams. DB1 DB2 DB3 Core database XML XML XML XML genaration WS WS WS delivery over a stream using generator | | | +------+-----+ AGG Parse and match incoming streams (iterparse) XML WS send resulting merge as XML using generator. So the questions: 1.. Does anybody have a recipe to build a recursive generator using Element? 2. Given the above generator, is there any such thing as a generator version etree.tostring? -- Kristian Kvilekval kris(a)cs.ucsb.edu http://www.cs.ucsb.edu/~kris w:805-636-1599 h:504-9756

2 3

Re: [lxml-dev] Querying valid children of an element?
by Brad Smith 08 May '08

08 May '08

To clarify about what I'm doing. The goal is to have a shorthand language (not entirely tag-based) that is easier for subject matter experts to learn than docbook, which can then be converted into full docbook once they've written a first draft. So, to illustrate one aspect of it, instead of writing <itemizedlist> <listitem><para><command>foomaster</command> example...</para> <itemizedlist> <listitem><screen>$ foomaster [OPTIONS]</screen></listitem> </itemizedlist> </listitem> </itemizedlist> They can write * <command>foomaster</command> example... ** <screen>$ foomaster [OPTIONS]</screen> As you can see, the translation process consists of not just converting asterisks into the appropriate combination of itemizedlists and listitems, but also protecting cdata within paras where necessary. In the first one, the interpreter sees that <command> isn't allowed inside <listitem>, which is its cue to try inserting a <para>. <screen> is allowed within <listitem>, so it does not insert a para. Making that determination is what I'm trying to find the best approach for. Currently I use a function like this: def validateAppend(parent,child): parent.append(child) if not dtd.validate(parent): dbg("Appending %s to %s failed DTD validation" % (child.tag,parent.tag)) del(parent[-1]) return False return True This works but, like I said, is not terribly efficient, so I just wanted to see if there was another method for making the determination. --Brad On Thu, May 8, 2008 at 11:44 AM, Mike Meyer <mwm(a)mired.org> wrote: > On Thu, 08 May 2008 09:33:17 +0200 Stefan Behnel <stefan_ml(a)behnel.de> wrote: > >> > I am writing a tool that translates xml tags mixed with a wiki-like >> > shorthand into full xml. It would be helpful to be able to >> > sanity-check the mix of explicit tags and implicit tags I'm deriving >> > from the shorthand by querying our DTD along the lines: "Is element >> > foo legal within element bar" Same for CDATA. >> > >> > Is this possible using lxml? If not, is it possible using anything >> > else? >> >> You could define your grammar in a way that is easily usable for you in your >> program and then generate a DTD from that. > > Are you really using DTDs, and not using that as a catchall for the > various Schema languages? > > If so, then you might consider switching to a modern schema > language. RelaxNG lets you write regular expressions for CDATA, which > ought to work with wiki-like "tags", and I wouldn't be surprised to > find that Schematron is turing complete. > > <mike > -- ~ Second Shift: An original, serialized audio adventure ~ http://www.secondshiftpodcast.com

1 0

[lxml-dev] Querying valid children of an element?
by Brad Smith 08 May '08

08 May '08

Hello, I just discovered lxml and am pretty excited about it. There is one thing I'm having trouble figuring out how to do, though, if it's even possible: I am writing a tool that translates xml tags mixed with a wiki-like shorthand into full xml. It would be helpful to be able to sanity-check the mix of explicit tags and implicit tags I'm deriving from the shorthand by querying our DTD along the lines: "Is element foo legal within element bar" Same for CDATA. Is this possible using lxml? If not, is it possible using anything else? The best I've been able to come up with so far is to assemble a tree of dummy nodes in the proposed order and then validate it, but this seems wasteful. Thanks in advance for any help offered, --Brad -- ~ Second Shift: An original, serialized audio adventure ~ http://www.secondshiftpodcast.com

2 1