Mailman 3 June 2010 - lxml - The Python XML Toolkit

[lxml-dev] objectify.DataElement() fixed to use stringify
by jholg＠gmx.de 25 Jun '10

25 Jun '10

Hi, I recently noticed that DataElement() doesn't make use of PyType.stringify. Patch committed to trunk, with test: Committed revision 75584. 0 pytaf@adevp02 .../lxml $ svn diff -rCOMMITTED Index: src/lxml/tests/test_objectify.py =================================================================== --- src/lxml/tests/test_objectify.py (revision 75482) +++ src/lxml/tests/test_objectify.py (working copy) @@ -318,6 +318,13 @@ arg = objectify.DataElement(3.1415) self.assertRaises(ValueError, objectify.DataElement, arg, _xsi="xsd:int") + + def test_data_element_element_arg(self): + arg = objectify.Element('arg') + value = objectify.DataElement(arg) + self.assert_(isinstance(value, objectify.ObjectifiedElement)) + for attr in arg.attrib: + self.assertEquals(value.get(attr), arg.get(attr)) def test_root(self): root = self.Element("test") @@ -1968,6 +1975,14 @@ self.assertEquals(r.date.pyval, parse_date(stringify_date(time))) self.assertEquals(r.date.text, stringify_date(time)) + date = objectify.DataElement(time) + + self.assert_(isinstance(date, DatetimeElement)) + self.assert_(isinstance(date.pyval, datetime)) + + self.assertEquals(date.pyval, parse_date(stringify_date(time))) + self.assertEquals(date.text, stringify_date(time)) + def test_object_path(self): root = self.XML(xml_str) path = objectify.ObjectPath( "root.c1.c2" ) Index: src/lxml/lxml.objectify.pyx =================================================================== --- src/lxml/lxml.objectify.pyx (revision 75482) +++ src/lxml/lxml.objectify.pyx (working copy) @@ -1973,6 +1973,9 @@ if dict_result is not NULL: _pytype = (<PyType>dict_result).name + if _pytype is None: + _pytype = _pytypename(_value) + if _value is None and _pytype != u"str": _pytype = _pytype or u"NoneType" strval = None @@ -1984,11 +1987,12 @@ else: strval = u"false" else: - strval = unicode(_value) + stringify = unicode + dict_result = python.PyDict_GetItem(_PYTYPE_DICT, _pytype) + if dict_result is not NULL: + stringify = (<PyType>dict_result).stringify + strval = stringify(_value) - if _pytype is None: - _pytype = _pytypename(_value) - if _pytype is not None: if _pytype == u"NoneType" or _pytype == u"none": strval = None Btw.: Before I noticed the existing stringify test in test_objectify I thought I'd add my custom pure-python DecimalElement class to have something to test. I didn't need to in the end but I wondered: What (and where?) about adding this very simple DecimalElement class to lxml? Maybe have an extra subdir for objectify element classes that are not enabled per default but can be imported & registered? I could also provide a DatetimeElement implementation, though this currently depends on dateutil - but I don't think this should be a bigproblem, it simply wouldn't work on machines without dateutil. Holger -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

1 0

[lxml-dev] German 1-day course on XML/ET/lxml in Leipzig, Germany - September 3rd, 2010
by Stefan Behnel 25 Jun '10

25 Jun '10

Hi everyone, [English version] I will be giving a beginners course on XML, ElementTree and lxml in September. It will be held in German (although sufficient interest may convince me to give it in English as well). It's called "High-Performance XML with Python" and will take place at the Python Academy in Leipzig/Germany on September 3rd, 2010. http://www.python-academy.de/Kurse/kurs_xml_python.html http://www.python-academy.com/courses/prices.html If you are interested, please contact me and/or Mike Müller of the PyA. http://www.python-academy.com/contact.html [German version] Am 3. September 2010 gebe ich einen eintägigen deutschsprachigen Einsteigerkurs zu XML, ElementTree und lxml. Der Kurstitel ist "High-Performance XML mit Python", Veranstaltungsort ist die Python-Akademie in Leipzig. http://www.python-academy.de/Kurse/kurs_xml_python.html http://www.python-academy.de/Kurse/preise.html Bei Intresse bitte an mich und/oder Mike Müller von der PyA wenden. http://www.python-academy.de/kontakt.html Have fun, Stefan

1 0

[lxml-dev] broken link for 2.3alpha release on PyPI "simple" index
by Leonardo Rochael Almeida 23 Jun '10

23 Jun '10

Hi, The "simple" index for lxml on PyPI [1] has a "2.3alpha1 download_url" link that results in 404 [2] This is breaking "easy_install" based systems (ex. buildout) that point use "simple" PyPI index [3] [1] http://pypi.python.org/simple/lxml/ [2] http://pypi.python.org/packages/source/l/lxml/lxml-2.3alpha1.tar.gz [3] http://pypi.python.org/simple/ Work-arounds include: - using the regular PyPI as index - explicitly requiring version 2.2.6 of lxml (ex. pinning the version in buildout) Cheers, Leo

3 2

[lxml-dev] Xml to flat file
by spuzhava＠purdue.edu 22 Jun '10

22 Jun '10

Hello, I was looking for a way to convert an xml fragment (_ElementTree object) to a flat file (comma separated perhaps). Though I understand this can be done quite easily by iterating, I just wanted to check if there was already something built in the library for this purpose. I tried going through the API (http://codespeak.net/lxml/api/index.html) but wasn't able to lay my hands on something that did this. I would be grateful if someone could let me know if they have come across a function that does this in lxml. Thanks, Shankar.

2 1

[lxml-dev] Wrapping one Tag around existing text
by Kyle Hanson 22 Jun '10

22 Jun '10

Hello, I was trying to find all the text elements in an html doc whose parents are divs and make it so their parents were p=elements. e.g. turn this: <div><h1>Title</h1>text<h2>stuff</h2></div> into this: <div><h1>Title</h1><p>text</p><h2>stuff</h2></div> so far I have: for text in (x for x in parsed.xpath('//div/text()') if len(x.strip())): -----p = builder.P(''+text) #I have to put ""+ because it doesnt recognize text is not a string type and str() chokes on string conversion because of unicode. I just dont know how to insert the created p element into the div while deleting the text node. I tried text.getparent().insert(text.getparent().index(text), p) but it says that the argument is supposed to be an _Element not an _ElementStringResult Also it says that the _ElementStringResult "text" 's parent is h1, and not div. Why is this? In order for me to get the div container (the true parent of 'text') I had to do text.getparent().getparent() Thanks

2 2

[lxml-dev] XPath vs lxml methods
by Lashkara Singh 21 Jun '10

21 Jun '10

Hi everyone I am new to Python and etree.lxml in general, so please answer accordingly! I am parsing a file with Python and lxml, and after reading up stuff, I have a confusion as to where to choose between XPath and lxml, given a choice. For example, I will take the following code to illustrate what I mean: To get the text of all tags with name *price*, I can use this: for ele in Tree.getiterator('price'): print ele.text > Or, I can do this using XPath: price_list = Tree.xpath('//price/text()') > The question is, given both of them return the same output and mean the same thing, which one should I use? XPath or the methods provided by the lxml API? Please note that I am not looking to optimize my code, though that is one part, what I am more concerned about is which is good, from a code point of view. Thanks for your time. L Singh

2 1

[lxml-dev] lxml 2.3alpha1 released
by Stefan Behnel 21 Jun '10

21 Jun '10

Hi all, I'm happy to announce the first alpha release of lxml 2.3. This is *not* a production-ready release, and there will be further fixes and changes before the final 2.3 release. Therefore, it is not the default version on PyPI but only available via "easy_install lxml==2.3alpha1" and from here: http://pypi.python.org/pypi/lxml/2.3alpha1 http://codespeak.net/lxml/dev/ The main reasons for this release are: There hasn't been a release for too long; there were too many changes in the meantime, including various bug fixes; I'd like to make it easier for users to benefit from those improvements and to report bugs and unexpected changes that get in their way. Note that not all reported bugs in the tracker that I'd like to fix for 2.3 are closed with this alpha, but I hope that a couple of others will make it before the final release. If you think that your Very Important Bug hasn't received enough interest so far, consider adding a comment to the bug tracker. ("fix this bug!" may not be enough ;) Major new features in this release include better compatibility with ElementTree 1.3 (which will be in the Python 3.2 standard library), ISO-Schematron support and various usability fixes for XSLT and XPath. The (IMHO) most important bug fix is the hardening of the API against potential crashes due to badly initialised wrapper objects. It is the clear intention for the future development to make lxml safer to use and hard to crash. I'm also planning to offer commercial (paid) support for lxml's further development, so if you really need a new feature or can't wait for a bug to get fixed, you may consider paying me to do it for you. Important notice: This does *not* mean that lxml is going non-free in any way, or that its OpenSource development is endangered. Lxml will continue to ship its complete source code under the liberal BSD license, and any (generally interesting) paid changes will become part of the mainline distribution. It just means that there will be a way for users to financially support my work on the project and to invest resources into targeted work that they can't (or don't want to) do themselves. This release was built using revision "c71560698d0d" of the cython-closures development branch (pre-0.13). A final Cython 0.13 is expected for early July, which will be the base for the final lxml 2.3 release. ... and I'm pretty sure this is the longest release announcement I've ever written. ;) Have fun, Stefan Here's the complete post-2.2.6 changelog for this release: 2.3alpha1 (2010-06-19) ====================== Features added -------------- * Keyword argument ``namespaces`` in ``lxml.cssselect.CSSSelector()`` to pass a prefix-to-namespace mapping for the selector. * New function ``lxml.etree.register_namespace(prefix, uri)`` that globally registers a namespace prefix for a namespace that newly created Elements in that namespace will use automatically. Follows ElementTree 1.3. * Support 'unicode' string name as encoding parameter in ``tostring()``, following ElementTree 1.3. * Support 'c14n' serialisation method in ``ElementTree.write()`` and ``tostring()``, following ElementTree 1.3. * The ElementPath expression syntax (``el.find*()``) was extended to match the upcoming ElementTree 1.3 that will ship in the standard library of Python 3.2/2.7. This includes extended support for predicates as well as namespace prefixes (as known from XPath). * During regular XPath evaluation, various ESXLT functions are available within their namespace when using libxslt 1.1.26 or later. * Support passing a readily configured logger instance into ``PyErrorLog``, instead of a logger name. * On serialisation, the new ``doctype`` parameter can be used to override the DOCTYPE (internal subset) of the document. * New parameter ``output_parent`` to ``XSLTExtension.apply_templates()`` to append the resulting content directly to an output element. * ``XSLTExtension.process_children()`` to process the content of the XSLT extension element itself. * ISO-Schematron support based on the de-facto Schematron reference 'skeleton implementation'. * XSLT objects now take XPath object as ``__call__`` stylesheet parameters. * Enable path caching in ElementPath (``el.find*()``) to avoid parsing overhead. * Setting the value of a namespaced attribute always uses a prefixed namespace instead of the default namespace even if both declare the same namespace URI. This avoids serialisation problems when an attribute from a default namespace is set on an element from a different namespace. * XSLT extension elements: support for XSLT context nodes other than elements: document root, comments, processing instructions. * Support for strings (in addition to Elements) in node-sets returned by extension functions. * Forms that lack an ``action`` attribute default to the base URL of the document on submit. * XPath attribute result strings have an ``attrname`` property. * Namespace URIs get validated against RFC 3986 at the API level (required by the XML namespace specification). * Target parsers show their target object in the ``.target`` property (compatible with ElementTree). Bugs fixed ---------- * API is hardened against invalid proxy instances to prevent crashes due to incorrectly instantiated Element instances. * Prevent crash when instantiating ``CommentBase`` and friends. * Export ElementTree compatible XML parser class as ``XMLTreeBuilder``, as it is called in ET 1.2. * ObjectifiedDataElements in lxml.objectify were not hashable. They now use the hash value of the underlying Python value (string, number, etc.) to which they compare equal. * Parsing broken fragments in lxml.html could fail if the fragment contained an orphaned closing '</div>' tag. * Using XSLT extension elements around the root of the output document crashed. * ``lxml.cssselect`` did not distinguish between ``x[attr="val"]`` and ``x [attr="val"]`` (with a space). The latter now matches the attribute independent of the element. * Rewriting multiple links inside of HTML text content could end up replacing unrelated content as replacements could impact the reported position of subsequent matches. Modifications are now simplified by letting the ``iterlinks()`` generator in ``lxml.html`` return links in reversed order if they appear inside the same text node. Thus, replacements and link-internal modifications no longer change the position of links reported afterwards. * The ``.value`` attribute of ``textarea`` elements in lxml.html did not represent the complete raw value (including child tags etc.). It now serialises the complete content on read and replaces the complete content by a string on write. * Target parser didn't call ``.close()`` on the target object if parsing failed. Now it is guaranteed that ``.close()`` will be called after parsing, regardless of the outcome. Other changes ------------- * Official support for Python 3.1.2 and later. * Static MS Windows builds can now download their dependencies themselves. * ``Element.attrib`` no longer uses a cyclic reference back to its Element object. It therefore no longer requires the garbage collector to clean up. * Static builds include libiconv, in addition to libxml2 and libxslt.

2 3

[lxml-dev] Using Regex to search CSSselectors?
by Kyle Hanson 19 Jun '10

19 Jun '10

I am switching my code from BeautifulSoup to LXML for HTML parsing however there is one area of my code that I was hoping someone could help on. Beautfiul soup allows re compiled patterns as arguments to search for nodes. I was hoping that LXML would allow for something similar. Basically I need to find all tags with a class name that matches the regex expression r'post_?\d+' and another one that matches the tag's id to the regex r'(user|profile)_\d+' Thanks

2 1

[lxml-dev] another bug
by Sergio Monteiro Basto 19 Jun '10

19 Jun '10

Hi, I have many sites on test. a parking domain, import lxml.html hparser = lxml.html.HTMLParser(encoding='utf-8', remove_comments=True) content="""<frameset> <frame src="main.php" name="srcpg" id="srcpg" frameborder="0" scrolling="Auto" marginwidth="" marginheight="0"> </frameset>""" etree_document = lxml.html.fromstring(content, parser=hparser) TypeError Traceback (most recent call last) /home/sergio/<ipython console> in <module>() /usr/lib/python2.6/site-packages/lxml/html/__init__.pyc in fromstring(html, base_url, parser, **kw) 634 other_head.drop_tree() 635 return doc --> 636 if (len(body) == 1 and (not body.text or not body.text.strip()) 637 and (not body[-1].tail or not body[-1].tail.strip())): 638 # The body has just one element, so it was probably a single TypeError: object of type 'NoneType' has no len() thanks, -- Sérgio M. B.

2 1

[lxml-dev] comment processing (bug?)
by kristian kvilekval 19 Jun '10

19 Jun '10

I am having a problem with embedded comments not being ignored by the parser. >From the examples on iterparse and iterwalk http://codespeak.net/lxml/parsing.html I tried the examples and it seems to work as advertised (comments ignored unless 'comment' is in events). However, I changed the input slightly to use an embedded comment: commented_xml = ''' <root> <element key='value'>text</element>  </root>''' xml = etree.XML (commented_xml) context = etree.iterwalk (xml, events = ('start','end')) for action, elem in context: print("%s: -%s-" % (action, elem.tag)) start: -root- start: -element- end: -element- start: -<built-in function Comment>- end: -<built-in function Comment>- end: -root- Since events does not contain comments, I wasn't expecting the comment and seem to unable to filter it out. Any pointers appreciated.. Kris

2 1