Mailman 3 July 2008 - lxml - The Python XML Toolkit

[lxml-dev] Proposal: Automatic unique ID generation for each tag or persistend user data for Element
by Ivan Begtin 18 Jul '08

18 Jul '08

Hi, I am working on several HTML processing algorithms for so named "structured data extraction". Earlier I've used BeautifulSoup and ElementTree/Xpath for these tasks, but now looking for something more advanced. For some of processing procedures I need to calculate lot's of metadata about each tag whereever it's located on page or whatever size it is. Calculation is quite simple task - much harder to map calculated values back to specified nodes. For example, I could generate MD5 hash value for each 'tag.text' and to keep it as dictionary somethere (DB / Python dict / e.t.c) but later I need to find back tag by specified hash and this is not so simple. Same situation is during processing saved HTML file. For example If I've found important block of tags during initiatial HTML parse, I want to find this block later on saved copy of this HTML page. Yes, sure, I can generate original xpath query for this tag to find it later but it's resource sonsuming operation and surely could be simplified I've tried one way to achive this task by creation of specific Element class with metadata populated on _init(), so I don't loose connection between metadata and node by itselft. The problem is that lxml generates Elements dynamically so when I created custom xpath method to obtain metadata performance degrated significantly since Elements were created once again. I propose one of two changes or both of them: 1. Automatically generate ID unique for tag in this etree instance which will be same for different instances of the Element. To be able to find unique id of the tag and to be able to get node by this unique id For example it could be as getNodeByID(id) method of etree and each Element will have "id" property 2. Persistent user data for Element Another solution would be it lxml could support "data transfer" between dynamic instances of Element of same tag. For example it could be setUserData(key, value) and getUserData(key) method for Element and property 'userdata' to access it. So if I set my data for this Element once, I will be able to access it in any following Element intsance of this tag. If I missed something and it could be done even using current lxml API, please, let me know. Best Regards, Ivan Begtin

5 4

Re: [lxml-dev] Proposal: Automatic unique ID generation for each tag or persistend user data for Element
by Stefan Behnel 18 Jul '08

18 Jul '08

Hi, please keep the list involved and avoid top-posting. Ivan Begtin wrote: > 2008/7/18 Stefan Behnel: >> Dr R. Sanderson wrote: >>> tree = node.getroottree() >>> elems = node.xpath('//p') >>> info = {} >>> for e in elems: >>> eid = abs(hash(tree.getpath(e))) >>> # processing here... >>> >>> Then later I can check if the data came from the same element or not by >>> comparing the eids. >> That's a nice way of doing it. If you additionally want to make the ID >> stick >> with the Element to avoid recalculation or to make it survive a tree split, >> you can store the hash value as xml:id of the node, as in >> >> e.set("{http://www.w3.org/XML/1998/namespace}id", >> md5sum(tree.getpath(e))) >> >> Just remember to remove all xml:id attributes and run cleanup_namespaces() >> in lxml 2.1 before you serialise it to plain HTML. > > It solved about 99% of problem but still we have html comments which have > attributes. Rarely, but sometimes they are needed too. Could you explain what an html comment with an attribute is? > And one more question: Is it possible to add ID without changing any > original data ? Or to print tag via etree.tostring(...) without data which I > add manually? Repeating myself: >> Just remember to remove all xml:id attributes and run cleanup_namespaces() >> in lxml 2.1 before you serialise it to plain HTML. Here's the code: for el in root.iter(): try: del el.attrib["{http://www.w3.org/XML/1998/namespace}id"] except KeyError: pass etree.cleanup_namespaces(root) print etree.tostring(root, method="html") Stefan

2 1

[lxml-dev] Proper syntax to insert elements with namespaces?
by Mike MacCana 18 Jul '08

18 Jul '08

Hello all, I'm reading some OpenDocument files with lxml, and inserting various elements. A snippet, which works fine, is below. Yay LXML! ## Parse our content as 'document' document = etree.parse(contentfile) print etree.tostring(document.getroot(), pretty_print=True), ## The body of the document, where we'll add our text elements body = document.xpath('/office:document-content/office:body/office:text', namespaces={'office':'urn:oasis:names:tc:opendocument:xmlns:office:1.0','text':'urn:oasis:names:tc:opendocument:xmlns:text:1.0'}) ## Insert some text as element one print "adding element:" body[0].insert(1, etree.Element("text",name="Standard")) ## Identify the element by it's xpath and change its contents. newelement = document.xpath('/office:document-content/office:body/office:text/text', namespaces={'office':'urn:oasis:names:tc:opendocument:xmlns:office:1.0','text':'urn:oasis:names:tc:opendocument:xmlns:text:1.0'}) newelement[0].text="new element text" 1. Is there a better way to refer to, and set the text of, the newly created element? 2. How do I insert an element with a namespace? The code above creates: <text name="Standard">new element text</text> But I'd prefer: <text:p text:style-name="Standard">old element</text:p> Cheers, Mike ________________________________________________ Mike MacCana Technical Specialist Australia Linux and Virtualisation Services IBM Global Services Level 14, 60 City Rd Southgate Vic 3000 Phone: +61-3-8656-2138 Fax: +61-3-8656-2423 Email: mmaccana(a)au1.ibm.com

2 4

[lxml-dev] Controlling attributes in lxml.html.clean
by Bruno Barberi Gnecco 18 Jul '08

18 Jul '08

Hi, Is there someway to control tag attributes in lxml.html.clean? Specifically I'm trying to get rid of any 'id' attributes. It seems that the only control available is the safe_attrs_only flag, which only allows defs.safe_attrs to be used. May I suggest an API somewhat like this: http://amisphere.com/contrib/python-html-filter/ for the next releases? I'd be happy to collaborate to implement it. Thanks a lot, Bruno

2 1

[lxml-dev] How to specify library files location when building lxml?
by lists＠h2co3.com 17 Jul '08

17 Jul '08

Hi, I've been struggling to get lxml built on Solaris 2.9 *without* root. The problem seems to be that when I go to build lxml with easy_install it is looking in /usr/lib (where very old versions exist) but I want it to use the libxml and libxslt that are in /usr/local/lib. Is there a way to specify that? I've checked all the options and distutils.cfg but can't figure out how. So far I have: - built Python 2.5.1 from source - setup virtual-env - run easy_install with --prefix successfully to install modules The output I see is: bash-2.05$ easy_install --prefix=~/alt lxml Searching for lxml Reading http://pypi.python.org/simple/lxml/ Reading http://codespeak.net/lxml Best match: lxml 2.1 Downloading http://codespeak.net/lxml/lxml-2.1.tgz Processing lxml-2.1.tgz Running lxml-2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-1-bIHk/lxml-2.1/egg-dist-tmp-Elegr4 Building lxml version 2.1. NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' needs to be available. Using build configuration of libxslt 1.0.19 Building against libxml2/libxslt in the following directory: /usr/lib src/lxml/lxml.etree.c:108:28: libxml/chvalid.h: No such file or directory .etc Many thanks, Daniel Daniel Seltzer dseltzer(a)h2co3.com 646-322-5657

2 1

[lxml-dev] Proposal: Better html5lib Support
by Armin Ronacher 16 Jul '08

16 Jul '08

Hi, I'm lately working a lot with html5lib which has a tree builder that can generate an lxml tree which is awesome :-) There are however a few inconveniences in the html5lib lxml support. Mostly because the html5lib API is quite complex to use and I've seen that there is a beautiful soup parser support in html5lib, so why not move the html5lib tree builder into an lxml.html.html5 module or so that provides the same API as the html (that is `fragment_fromstring`, `document_fromstring`, etc.) html5lib is currently the most advanced HTML parsing module for Python I know about and it is able to deal with most HTML the same way popular browsers do. There is another small problem with html5lib and lxml interoperability that is the HTML5 doctype ("<!DOCTYPE HTML>") that lxml naturally cannot handle. I know that lxml is an XML library after all, but maybe support for this special doctype could be added. Regards, Armin

3 14

[lxml-dev] problem with e-factory input text field
by Dominique.Holzwarth＠ch.delarue.com 14 Jul '08

14 Jul '08

Hi everyone I'm trying to generate a HTML page with the E-Factory (from lxml.html import builder as E) but I'm having some problem with checkboxes... The output I'd like to get is the following: <input type="checkbox" name="something" value="1">label text</input> I'm creating the input field with: E.INPUT("label text", name="something", type="checkbox", value="1") but when i do a lxml.html.tostring(...) of my whole page object the "label" text" will be missing and the tag will look like that: <input type="checkbox" name="something" value="1"> <-- Note: no closing tag and no text-node When i do a lxml.etree.tostring of just that tag the result is perfect, but that method will cause troubles with tags like <script type="text/javascript" src="..."> so I think I can't use that on the whole page... Does anyone got an idea how to create the above state output? Greetings Dominique Dominique Holzwarth | De La Rue International Limited Software Engineer dominique.holzwarth(a)ch.delarue.com<mailto:dominique.holzwarth@ch.delarue.com> | tel: +41 (0) 31 997 56 13 | fax: +41 (0) 997 56 80 Berne Branch, Morgenstrasse 131, 3018 Berne, Switzerland | www.delarue.com<http://www.delarue.com/> ***************************************************************************** This e-mail and any files attached are strictly confidential, may be legally privileged and are intended solely for the addressee. If you are not the intended recipient please notify the sender immediately by return email and then delete the e-mail and any attachments immediately. The views and or opinions expressed in this e-mail are not necessarily the views of De La Rue plc or any of its subsidiaries and the De La Rue Group of companies, their directors, officers and employees make no representation about and accept no liability for its accuracy or completeness. You should ensure that you have adequate virus protection as the De La Rue Group of companies do not accept liability for any viruses. De La Rue plc Registered No.3834125, De La Rue Holdings plc Registered No 58025 and De La Rue International Limited Registered No 720284 are all registered in England with their registered office at: De La Rue House, Jays Close, Viables, Hampshire RG22 4BS *****************************************************************************

2 1

Re: [lxml-dev] questions about xslt with xml, where to get help?
by Stefan Behnel 10 Jul '08

10 Jul '08

Hi, please keep the list on CC. Dominique.Holzwarth(a)ch.delarue.com wrote: > Ok, that seems to work now :-) Thanks a lot! I indeed was wondering what > this weird syntax a="'A'" means in the doc. Is it correct, that if I want > to pass a complete DOM-node as xsl-param I can just write something like > myParam="node-tag"? That doesn't pass the node, it just passes the reference (i.e. the tag name), which will be resolved from the XML source document that you are transforming, in the current transformation context (i.e. from within the node you are operating on). I think your best bet to pass tree content is through XSLT extension elements or XPath extension functions. Stefan

1 0

[lxml-dev] questions about xslt with xml, where to get help?
by Dominique.Holzwarth＠ch.delarue.com 10 Jul '08

10 Jul '08

Hi everyone I'm not completely sure whether this mailing list only intended to further develop lxml or as well for _users_ of lxml... but I haven't found any other mailing list / forum :-( I'm having problems with getting xsl transformation run. When i do: xslt_doc = etree.parse(stylesheet) transform = etree.XSLT(xslt_doc) result = transform(DOMdocument, **{statustext='some text', mode='navigation'}) the result will just be empty even tho in the documentation it is stated the paramters should be passed as keyword parameters... greetings Dominique Dominique Holzwarth | De La Rue International Limited Software Engineer dominique.holzwarth(a)ch.delarue.com<mailto:dominique.holzwarth@ch.delarue.com> | tel: +41 (0) 31 997 56 13 | fax: +41 (0) 997 56 80 Berne Branch, Morgenstrasse 131, 3018 Berne, Switzerland | www.delarue.com<http://www.delarue.com/> ***************************************************************************** This e-mail and any files attached are strictly confidential, may be legally privileged and are intended solely for the addressee. If you are not the intended recipient please notify the sender immediately by return email and then delete the e-mail and any attachments immediately. The views and or opinions expressed in this e-mail are not necessarily the views of De La Rue plc or any of its subsidiaries and the De La Rue Group of companies, their directors, officers and employees make no representation about and accept no liability for its accuracy or completeness. You should ensure that you have adequate virus protection as the De La Rue Group of companies do not accept liability for any viruses. De La Rue plc Registered No.3834125, De La Rue Holdings plc Registered No 58025 and De La Rue International Limited Registered No 720284 are all registered in England with their registered office at: De La Rue House, Jays Close, Viables, Hampshire RG22 4BS *****************************************************************************

3 4

[lxml-dev] lxml 2.1 released
by Stefan Behnel 10 Jul '08

10 Jul '08

Hi, lxml 2.1 finally made it to PyPI! This is a major new release that follows the 2.0 series with a couple of cleanups and tons of new features. The complete changelog follows below. This is also the first version that officially supports Python 3, as released in 3.0beta1. Have fun, Stefan 2.1 (2008-07-09) ================ Features added -------------- * Smart strings can be switched off in XPath (``smart_string`` keyword option). * ``lxml.html.rewrite_links()`` strips links to work around documents with whitespace in URL attributes. * Pickling ``ElementTree`` objects in lxml.objectify. * Major overhaul of ``tools/xpathgrep.py`` script. * Pickling ``ElementTree`` objects in lxml.objectify. * Support for parsing from file-like objects that return unicode strings. * New function ``etree.cleanup_namespaces(el)`` that removes unused namespace declarations from a (sub)tree (experimental). * XSLT results support the buffer protocol in Python 3. * Polymorphic functions in ``lxml.html`` that accept either a tree or a parsable string will return either a UTF-8 encoded byte string, a unicode string or a tree, based on the type of the input. Previously, the result was always a byte string or a tree. * Support for Python 2.6 and 3.0 beta. * File name handling now uses a heuristic to convert between byte strings (usually filenames) and unicode strings (usually URLs). * Parsing from a plain file object frees the GIL under Python 2.x. * Running ``iterparse()`` on a plain file (or filename) frees the GIL on reading under Python 2.x. * Conversion functions ``html_to_xhtml()`` and ``xhtml_to_html()`` in lxml.html (experimental). * Most features in lxml.html work for XHTML namespaced tag names (experimental). * All parse functions in lxml.html take a ``parser`` keyword argument. * lxml.html has a new parser class ``XHTMLParser`` and a module attribute ``xhtml_parser`` that provide XML parsers that are pre-configured for the lxml.html package. * Error logging in Schematron (requires libxml2 2.6.32 or later). * Parser option ``strip_cdata`` for normalising or keeping CDATA sections. Defaults to ``True`` as before, thus replacing CDATA sections by their text content. * ``CDATA()`` factory to wrap string content as CDATA section. * New event types 'comment' and 'pi' in ``iterparse()``. * ``XSLTAccessControl`` instances have a property ``options`` that returns a dict of access configuration options. * Constant instances ``DENY_ALL`` and ``DENY_WRITE`` on ``XSLTAccessControl`` class. * Extension elements for XSLT (experimental!) * ``Element.base`` property returns the xml:base or HTML base URL of an Element. * ``docinfo.URL`` property is writable. Bugs fixed ---------- * Custom resolvers were not used for XMLSchema includes/imports and XInclude processing. * CSS selector parser dropped remaining expression after a function with parameters. * Descending dot-separated classes in CSS selectors were not resolved correctly. * ``ElementTree.parse()`` didn't handle target parser result. * Potential threading problem in XInclude. * Crash in Element class lookup classes when the __init__() method of the super class is not called from Python subclasses. * A number of problems related to unicode/byte string conversion of filenames and error messages were fixed. * Building on MacOS-X now passes the "flat_namespace" option to the C compiler, which reportedly prevents build quirks and crashes on this platform. * Windows build was broken. * Rare crash when serialising to a file object with certain encodings. * Incorrect evaluation of ``el.find("tag[child]")``. * Moving a subtree from a document created in one thread into a document of another thread could crash when the rest of the source document is deleted while the subtree is still in use. * Passing an nsmap when creating an Element will no longer strip redundantly defined namespace URIs. This prevented the definition of more than one prefix for a namespace on the same Element. * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31. * lxml.etree accepted non well-formed namespace prefix names. * Hanging thread in conjunction with GTK threading. * Crash bug in iterparse when moving elements into other documents. * HTML elements' ``.cssselect()`` method was broken. * ``ElementTree.find*()`` didn't accept QName objects. * Default encoding for plain text serialisation was different from that of XML serialisation (UTF-8 instead of ASCII). Other changes ------------- * ``objectify.enableRecursiveStr()`` was removed, use ``objectify.enable_recursive_str()`` instead * Speed-up when running XSLTs on documents from other threads * Non-ASCII characters in attribute values are no longer escaped on serialisation. * Passing non-ASCII byte strings or invalid unicode strings as .tag, namespaces, etc. will result in a ValueError instead of an AssertionError (just like the tag well-formedness check). * Up to several times faster attribute access (i.e. tree traversal) in lxml.objectify. * lxml should now build without problems on MacOS-X. * If the default namespace is redundantly defined with a prefix on the same Element, the prefix will now be preferred for subelements and attributes. This allows users to work around a problem in libxml2 where attributes from the default namespace could serialise without a prefix even when they appear on an Element with a different namespace (i.e. they would end up in the wrong namespace). * Major cleanup in internal ``moveNodeToDocument()`` function, which takes care of namespace cleanup when moving elements between different namespace contexts. * New Elements created through the ``makeelement()`` method of an HTML parser or through lxml.html now end up in a new HTML document (doctype HTML 4.01 Transitional) instead of a generic XML document. This mostly impacts the serialisation and the availability of a DTD context. * Minor API speed-ups. * The benchmark suite now uses tail text in the trees, which makes the absolute numbers incomparable to previous results. * Generating the HTML documentation now requires Pygments_, which is used to enable syntax highlighting for the doctest examples. .. _Pygments: http://pygments.org/ Most long-time deprecated functions and methods were removed: - ``etree.clearErrorLog()``, use ``etree.clear_error_log()`` - ``etree.useGlobalPythonLog()``, use ``etree.use_global_python_log()`` - ``etree.ElementClassLookup.setFallback()``, use ``etree.ElementClassLookup.set_fallback()`` - ``etree.getDefaultParser()``, use ``etree.get_default_parser()`` - ``etree.setDefaultParser()``, use ``etree.set_default_parser()`` - ``etree.setElementClassLookup()``, use ``etree.set_element_class_lookup()`` Note that ``parser.setElementClassLookup()`` has not been removed yet, although ``parser.set_element_class_lookup()`` should be used instead. - ``xpath_evaluator.registerNamespace()``, use ``xpath_evaluator.register_namespace()`` - ``xpath_evaluator.registerNamespaces()``, use ``xpath_evaluator.register_namespaces()`` - ``objectify.setPytypeAttributeTag``, use ``objectify.set_pytype_attribute_tag`` - ``objectify.setDefaultParser()``, use ``objectify.set_default_parser()``

2 3