Mailman 3 August 2008 - lxml - The Python XML Toolkit

[lxml-dev] Odd looking exception
by Sidnei da Silva 09 Aug '08

09 Aug '08

Hi there, I've just got an exception while upgrading to lxml 2.1.1. Still trying to find out if it's some backwards-incompatible change. But what triggered my attention is that the exception seems to have tried to display the line number from etree.c but failed: File "enfold\lxml\transform.pyo", line 429, in transform File "xslt.pxi", line 399, in lxml.etree.XSLT.__init__ (src/lxml/lxml.etree.c:%u) XSLTParseError: Cannot parse stylesheet This is on Windows FWIW. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

2 1

[lxml-dev] Converting from XML to HTML parsed trees.
by John Krukoff 08 Aug '08

08 Aug '08

I have some XML data already parsed into an lxml ElementTree. Is there any easy way to reparse that using lxml.html, or is the only way to do it to serialize the XML to a string and reparse using one of the lxml.html parsers? -- John Krukoff <jkrukoff(a)ltgc.com> Land Title Guarantee Company

2 1

Re: [lxml-dev] [Bug 255800] parser.feed(t) + parser.feed('\n') != parser.feed(t+'\n')
by Stefan Behnel 08 Aug '08

08 Aug '08

(forwarding this to the list) Vladimir Vcelak wrote: > Dne Thu, 07 Aug 2008 19:13:04 +0200 (CEST) Stefan Behnel napsal: >> Vladimir Vcelak wrote: >>> >>> t = '<a>x</a>' >>> >>> parser = etree.XMLParser(target = EchoTarget()) >>> >>> parser.feed(t) ; parser.feed('\n') >>> start a {} >>> data u'x' >>> end a >>> >>> >>> parser = etree.XMLParser(target = EchoTarget()) >>> >>> parser.feed(t + '\n') >>> (print none!!) >> >> The parser is free to start/stop/continue parsing at any time. As long >> as the callback sequence in both cases is the same after >> close(), I don't see the bug here. > > Thank you for your answer, but I can't depend on calling close(). My idea is use XML for dialogue between Server and Client: > > SERVER: > from lxml import etree > > class MyParser: > def start(): > level += 1 > def end(): > level -= 1 > if level == 0: > end_command = True > > socket = create_TCP_server(port) > while True: > s = socket.accept() > end_command = False > parser = etree.XMLParser(target = MyTarget()) > while not end_command: > parser.feed(s.readline()) Have you actually tried calling close() at this point? It might even work, and it will raise an exception if the command wasn't well-formed. BTW, I didn't check, but creating the parser once outside the loop and making sure you call close() after each iteration might actually be enough. > answer = execute(command) > s.write(answer) > > > > CLIENT: > socket = connect_to(server, port) > socket.write("<cmd1> ..........</cmd1>\n") > answer = socket.read() > socket.write("<cmd2> ..........</cmd2>\n") > answer = socket.read() > .... > > or CLIENT for debug: > telnet server port > <cmd1> .... > ... > </cmd1> > answer > <cmd2> .... > ... > </cmd2> > answer

1 0

[lxml-dev] Some HTML target processing issues
by Max Ivanov 08 Aug '08

08 Aug '08

Hi all! I've attached small piece of code. lxml target parsing has some problems from my point of view. 1) I use lxml.html.HTMLParser which should handle unknown HTML tags since it uses lxml.html.HtmlElementClassLookup which contain this code in its lookup function: "if node_type == 'element': return self._element_classes.get(name.lower(), HtmlElement)". If I understand it right, then even unknown tags should be handled properly. But I still get error at the end of the code: lxml.etree.XMLSyntaxError: Tag noindex invalid, line 266, column 17 I don't understand why, hope someone give me a good advice :) 2) Even if the whole process fails, etree.fromstring continue to call target methods (start,end,comment etc...) even after invelid tag <noindex> is appeared. It's ok, it's some sort of fault tolerance. But why it do not call target.close() at the end? Instead of that it raises exception. If document processing continues even after error, then call target.close() too! Maybe i'ts better to pass all accured errors to close function, so target could decide what to do. 3) lxml should stop processing when target raises exception. Nowdays it's just ignored and all continue.

2 1

[lxml-dev] combining target parser class with DTD validation
by Gary V. Vaughan 07 Aug '08

07 Aug '08

Hi, I have an xml.sax based parser that works like this: .-----. ,-------------------. file.xml -> | gpp | -> preprocessed.xml -> |saxexts.make_parser| --. `-----' `-------------------' | ,-------------------. | custom class heirarchy <- |sax DocumentHandler| <-' `-------------------' I'm in the process of converting all of this from xml.sax to lxml.etree. The DocumentHandler function is very complex, but well debugged, so I'd really like to convert it to work with an lxml.XMLParser target keyword, which is straight forward enough (changing startElement to start, characters to data etc) to avoid churning the handler function and the custom class heirarchy it builds. So for so good... Also, performance is important, so I'm passing the the output of gpp (general pre-processor) to the parser with a feed function as gpp is running on file.xml. The main driver behind moving the project to lxml is to perform DTD validation on 'preprocessed.xml', but setting a target keyword in etree.XMLParser turns off DTD validation at parse time :( Also, with dtd_validation=True, even xml documents with no DOCTYPE declaration throw an exception. Since I have a zillion files that I'd like to migrate gradually, while interoperating with other users that haven't installed the lxml based parser yet, I only want to do validation when there is a DOCTYPE declaration. Older files don't have it, so I'd like to ignore the missing DTD reference on those until they are upgraded. My question is: what is the cleanest/fastest way to combine (i) passing input to the parser with a feed function (ii) reusing most of the sax DocumentHandler with a target class (iii) performing DTD validation on the fly during parsing (iv) but skipping validation if there is no DOCTYPE declaration I've ended up implementing an lxml based validating parser to replace the above like this: # input file fh = open (gpp_input, 'r') # For backwards compatibility, skip dtd validation when there # is no DOCTYPE declaration: match = re.compile ('^<!DOCTYPE ', re.M).search (fh.read (1024), 1) fh.seek(0) # prepare XML parser to read data parser = etree.XMLParser (dtd_validation=(match != None)) gpp_r, my_w = os.pipe () my_r, gpp_w = os.pipe () gpp = os.fork () if gpp == 0: ... # set up pipes to gpp stdin and stdout ... while fds: ... # collect gpp stdout with select ... parser.feed (gpp_output) # get the etree tree = parser.close () # walk the etree and fire synthetic sax events xmlh = old_sax_DocumentHandler () context = etree.iterwalk (tree, events=("start", "end")) for action, element in context: if action == 'start': xmlh.startElement (element.tag, element.attrib) if element.text and hasattr (xmlh, 'characters'): xmlh.characters (element.text) elif action == 'end': xmlh.endElement (element.tag) if element.tail and hasattr (xmlh, 'characters'): xmlh.characters (element.tail) It works well enough, but it feels kludgy to manually peek into each xml file and look for a DOCTYPE at the start; and since I have to walk the tree once while building it and again when calling the handler function, I'm sure it is slower than it could be. Advice gratefully received! Cheers, Gary -- Gary V. Vaughan (gary(a)thewrittenword.com) -- Gary V. Vaughan (gary(a)thewrittenword.com)

2 1

[lxml-dev] FAIL: test_parser_target_entity
by Cesar Ortiz 06 Aug '08

06 Aug '08

Hi all, I am new to lxml, but I have been using libxml2 for a while. In my system I´ve got libxml2 2.6.26 and libxslt 1.1.17, and I tried to install lxml. I could not install with easy_install (because of the installation I have got I think) so I tried to install from source, and it worked. After that I run the tests and one failed: [phe1246@pandora lxml]$ make test python setup.py build_ext -i Building lxml version 2.2.alpha1-56897. Building with Cython 0.9.8. Using build configuration of libxslt 1.1.17 Building against libxml2/libxslt in the following directory: /home/phe1246/install//lib running build_ext building 'lxml.etree' extension gcc -pthread -shared build/temp.linux-i686-2.4/src/lxml/lxml.etree.o -L/home/phe1246/install//lib -lxslt -lexslt -lxml2 -lz -lm -o src/lxml/etree.so building 'lxml.objectify' extension gcc -pthread -shared build/temp.linux-i686-2.4/src/lxml/lxml.objectify.o -L/home/phe1246/install//lib -lxslt -lexslt -lxml2 -lz -lm -o src/lxml/objectify.so python test.py -p -v TESTED VERSION: 2.2.alpha1-56897 Python: (2, 4, 3, 'final', 0) lxml.etree: (2, 2, -199, 56897) libxml used: (2, 6, 26) libxml compiled: (2, 6, 26) libxslt used: (1, 1, 17) libxslt compiled: (1, 1, 17) 970/970 (100.0%): Doctest: xpathxslt.txt ====================================================================== FAIL: test_parser_target_entity (lxml.tests.test_elementtree.ETreeTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/unittest.py", line 260, in run testMethod() File "/home/phe1246/software/lxml/src/lxml/tests/test_elementtree.py", line 3417, in test_parser_target_entity events) File "/usr/lib/python2.4/unittest.py", line 333, in failUnlessEqual raise self.failureException, \ AssertionError: ['start-root', 'start-sub', 'end-sub', 'start-sub', 'data-this is an entity', 'end-sub', 'start-sub', 'end-sub', 'end-root'] != ['start-root', 'start-sub', 'end-sub', 'start-sub', u'data-this is an entityan entity', 'end-sub', 'start-sub', 'end-sub', 'end-root'] ---------------------------------------------------------------------- Ran 970 tests in 33.731s FAILED (failures=1) make: *** [test_inplace] Error 1 Should I worry about it? I didn´t see anything related in the mailing-list repository. Thanks in advance. NB: By the way, is it possible to get a .tar.gz version from any place?

3 3

Re: [lxml-dev] Transform parameter variables
by Stefan Behnel 06 Aug '08

06 Aug '08

Hi, it's good practice to a) reply to the list, b) avoid top-posting and c) read what people post. Niels Bjerre wrote: > Thank You for your response > > But I have problem with the_dict: > These 2 statements don't give the same result. > > 1. newdoc = transform(places.myplaces, area="'3751'") > 2. newdoc = transform(places.myplaces, {'area':'\"\'3751\'\"'}) > > 1. is passed to xslt: <xsl:param name="area" select="'ost'"/> > 2. is ignored > > I have tried with > {'area':'\'3751\''} and others The last line will work, but as I wrote before: > 2008/7/28 Stefan Behnel <stefan_ml(a)behnel.de> >> You can pass more than one kayword parameter, as everywhere in Python. >> If you want to pass them from a dictionary, do the usual >> >> result = transform(xml_tree, **the_dict) >> >> trick. Note the two stars before "the_dict". This is standard Python syntax for expanding a mapping into keyword arguments. Stefan

3 3

[lxml-dev] Failure with custom resolvers and relative xsl:import
by Sidnei da Silva 06 Aug '08

06 Aug '08

Hi there, I've got a reproducible failing test when using custom resolvers and relative xsl:import. Basically, if I use resolve_string() or let the default resolver do it's work, everything works fine. If I use resolve_file though, the *next* uri to be resolved will have a relative (to the previous uri resolved) filename, and then there's not enough information available to compute the full URI. This happens with lxml 2.1.1. I am pretty sure it didn't happen with lxml 1.3.x series (which is what I was using before). I'm attaching the problematic test and related files. For reference, this is the problematic code: ... def resolve(self, uri, id, ctx): print uri # return None # works # return self.resolve_string(open(uri, 'r').read(), ctx) # works return self.resolve_file(open(uri, 'r'), ctx) # fails ... -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

1 0

[lxml-dev] Passing UTF-8 bytestrings to lxml
by John J Lee 04 Aug '08

04 Aug '08

Hi Apologies in advance if this is the wrong list -- I'm suggesting a change to lxml, so I guess this is the right place... I'm working on some existing code that makes use of both unicode objects and UTF-8 encoded bytestring objects (both of which sometimes contain non-ASCII characters). I'm making changes to the code to ensure that it supports the unicode character set. Unfortunately, it's not practical to change all of the code to use unicode objects (partly because there's a lot of code, and partly because fixing that would probably entail fixing PyGTK to return unicode objects instead of UTF-8 encoded bytestrings). So, the plan is to live with both unicode and UTF-8 encoded bytestrings, and to ensure Python's default encoding is always set to UTF-8. I'm sure the wisdom that approach could be debated (!), but I hope that somebody will be kind enough to answer the following question anyway :-) Looking at the code, it seems that changing function _utf8 in apihelpers.pxi to accept UTF-8 encoded bytestrings (see patch below) would be sufficient to make lxml accept UTF-8 encoded bytestrings. Indeed, that seems to work. 1. Will what I'm doing subtly break lxml in some way if I make use of this patched lxml in my own code? 2. Should lxml be changed in this way? If it's considered important to avoid accidentally passing non-ASCII bytestrings to lxml, would it be acceptable to add a global switch to enable accepting UTF-8 encoded bytestrings? Thanks for any help (This patch is against lxml 1.3.6, but this function in SVN trunk is very similar) --- apihelpers.pxi.orig 2008-08-04 12:52:34.000000000 +0100 +++ apihelpers.pxi 2008-08-04 12:41:57.000000000 +0100 @@ -640,13 +640,12 @@ cdef object _utf8(object s): if python.PyString_Check(s): - assert not isutf8py(s), \ - "All strings must be XML compatible, either Unicode or ASCII" + assert isutf8py(s) != -1, \ + "All strings must either unicode objects or UTF-8" elif python.PyUnicode_Check(s): - # FIXME: we should test these strings, too ... s = python.PyUnicode_AsUTF8String(s) assert isutf8py(s) != -1, \ - "All strings must be XML compatible, either Unicode or ASCII" + "All strings must be either unicode objects or UTF-8" else: raise TypeError, "Argument must be string or unicode." return s John

2 3