Mailman 3 September 2012 - lxml - The Python XML Toolkit

Installation / PyPI issues: 3.0(alpha|beta)
by Tim Diggins Sept. 28, 2012

Sept. 28, 2012

Hi Stefan and co - I've installed lxml 3.0alpha 2 using pip last week (simply: pip install lxml) However, working in a fresh virtualenv I tried again, and It looks like lxml 3.0 alpha 2 has been removed from pypi and lxml 3.0 beta isn't yet on pypi. When I try to install using pip install pip install git+github.com/lxml/lxml@26358d2218a84ad8ee89cce64828f907489c43af I get some compilation warnings and errors, which indicate that either: * src/lxml/lxml.etree.c should be present (is that because I'm using the git version rather than a pypi version?) or * I should install Cython Is that right? But will 3.0beta be out on pypi in a few days? (in which case I'll just wait for that) thanks Tim Tim Diggins http://red56.co.uk

2 1

root.docinfo.externalDTD is None
by Thomas Larsen Wessel Sept. 28, 2012

Sept. 28, 2012

I have an xml document with a doctype pointing to a dtd in the same folder: <!DOCTYPE point SYSTEM "point.dtd"> <point> <address>abc</address> </point> I parse it, and check out the dtd's root = etree.parse(open('point_dtd.xml')) intdtd = root.docinfo.internalDTD # lxml.etree.DTD object extdtd = root.docinfo.externalDTD # None I was expecting externalDTD to return a DTD for the point.dtd schema, and the internalDTD to be None. Furthermore intdtd is "empty", for instance intdtd.elements returns an empty list, while point.dtd does in fact define elements! Shouldn't externalDTD point to dtd for the point.dtd?

2 3

Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 27, 2012

Sept. 27, 2012

Hi Stefan,, I tested by checking sys.path and through strace but couldn't get much out of it. However, a clean install has worked :) Special thanks to you and Simon :) Regards Pankaj Singh From: Pankaj Singh Sent: 26-09-2012 22:58 To: lxml(a)lxml.de Subject: RE: [lxml] PyUnicodeUCS2_DecodeLatin1 error Hi Stefan, Thanks for the inputs. I'll test again with a clean install. Will post the results. Regards Pankaj Singh From: Stefan Behnel Sent: 26-09-2012 21:59 To: lxml(a)lxml.de Subject: Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error Simon Sapin, 26.09.2012 17:18: > Le 26/09/2012 17:10, Pankaj Singh a écrit : >> Is there an option wherein I can specify unicode style while building >> lxml? No. > I *think* it should just pick it up from the current python at > compile-time Yes, that's fully automatic. There is no way it can get that wrong. > Just to be clear: you’re installing/compiling lxml from a source tarball > with a USC4 python, and when importing it in the same python get > "undefined symbol: PyUnicodeUCS2_DecodeLatin1". Is this correct? That can't be the case. I assume there's something wrong with either the import path (PYTHONPATH) or the local installation, potentially both. For diagnosis, run strace -e open,stat python -c 'import lxml.etree' | fgrep etree That will show from where it tries to import the lxml.etree module. In any case, a clean install should fix this. Stefan _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

lxml 3.0 beta1 released
by Stefan Behnel Sept. 26, 2012

Sept. 26, 2012

Hi everyone, I'm proud to announce the first beta release of lxml 3.0. This release fixes a serious memory leak in the last alpha release, so please upgrade if you currently use a pre-release of 3.0. The 3.0 release series features initial support for PyPy and a new tree API for DTDs (by Walter Dörwald), as well as some major cleanups in the code base. The lxml.cssselect package was moved into a separate PyPI project nicely extended and maintained by Simon Sapin. Originally, this release was targeted as lxml 2.4, but some major improvements as well as a couple of potentially incompatible changes make "3.0" appear more appropriate. The final release of 3.0 is scheduled for early October, so now is a good time to report any remaining bugs and to send in your bug fix pull requests for approval. The documentation is here: http://lxml.de/3.0/ Download: http://lxml.de/files/lxml-3.0beta1.tgz Signature: http://lxml.de/files/lxml-3.0beta1.tgz.asc GitHub: https://github.com/lxml/lxml/commit/26358d2218a84ad8ee89cce64828f907489c43af This release was built using Cython 0.17.1. Development on lxml's sources now requires this version. If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.0beta1 (2012-09-26) ===================== Features added -------------- * Python level access to (optional) libxml2 memory debugging features to simplify debugging of memory leaks etc. Bugs fixed ---------- * Fix a memory leak in XPath by switching to Cython 0.17.1. * Some tests were adapted to work with PyPy. Other changes ------------- * The code was adapted to work with the upcoming libxml2 2.9.0 release. 3.0alpha2 (2012-08-23) ====================== Features added -------------- * The .iter() method of elements now accepts tag arguments like "{*}name" to search for elements with a given local name in any namespace. With this addition, all combinations of wildcards now work as expected: "{ns}name", "{}name", "{*}name", "{ns}*", "{}*" and "{*}*". Note that "name" is equivalent to "{}name", but "*" is "{*}*". The same change applies to the .getiterator(), .itersiblings(), .iterancestors(), .iterdescendants(), .iterchildren() and .itertext() methods, the strip_attributes(), strip_elements() and strip_tags() functions as well as the iterparse() function. * C14N allows specifying the inclusive prefixes to be promoted to top-level during exclusive serialisation. Bugs fixed ---------- * Passing long Unicode strings into the feed() parser interface failed to read the entire string. Other changes ------------- 3.0alpha1 (2012-07-31) ====================== Features added -------------- * Initial support for building in PyPy (through cpyext). * DTD objects gained an API that allows read access to their declarations. * xpathgrep.py gained support for parsing line-by-line (e.g. from grep output) and for surrounding the output with a new root tag. * E-factory in lxml.builder accepts subtypes of known data types (such as string subtypes) when building elements around them. * Tree iteration and iterparse() with a selective tag argument supports passing a set of tags. Tree nodes will be returned by the iterators if they match any of the tags. Bugs fixed ---------- * The .find*() methods in lxml.objectify no longer use XPath internally, which makes them faster in many cases (especially when short circuiting after a single or couple of elements) and fixes some behavioural differences compared to lxml.etree. Note that this means that they no longer support arbitrary XPath expressions but only the subset that the ElementPath language supports. The previous implementation was also redundant with the normal XPath support, which can be used as a replacement. * el.find('*') could accidentally return a comment or processing instruction that happened to be in the wrong spot. (Same for the other .find*() methods.) * The error logging is less intrusive and avoids a global setup where possible. * Fixed undefined names in html5lib parser. * xpathgrep.py did not work in Python 3. * Element.attrib.update() did not accept an attrib of another Element as parameter. * For subtypes of ElementBase that make the .text or .tail properties immutable (as in objectify, for example), inserting text when creating Elements through the E-Factory feature of the class constructor would fail with an exception, stating that the text cannot be modified. Other changes -------------- * The code base was overhauled to properly use 'const' where the API of libxml2 and libxslt requests it. This also has an impact on the public C-API of lxml itself, as defined in etreepublic.pxd, as well as the provided declarations in the lxml/includes/ directory. Code that uses these declarations may have to be adapted. On the plus side, this fixes several C compiler warnings, also for user code, thus making it easier to spot real problems again. * The functionality of "lxml.cssselect" was moved into a separate PyPI package called "cssselect". To continue using it, you must install that package separately. The "lxml.cssselect" module is still available and provides the same interface, provided the "cssselect" package can be imported at runtime. * Element attributes passed in as an attrib dict or as keyword arguments are now sorted by (namespaced) name before being created to make their order predictable for serialisation and iteration. Note that adding or deleting attributes afterwards does not take that order into account, i.e. setting a new attribute appends it after the existing ones. * Several classes that are for internal use only were removed from the lxml.etree module dict: _InputDocument, _ResolverRegistry, _ResolverContext, _BaseContext, _ExsltRegExp, _IterparseContext, _TempStore, _ExceptionContext, __ContentOnlyElement, _AttribIterator, _NamespaceRegistry, _ClassNamespaceRegistry, _FunctionNamespaceRegistry, _XPathFunctionNamespaceRegistry, _ParserDictionaryContext, _FileReaderContext, _ParserContext, _PythonSaxParserTarget, _TargetParserContext, _ReadOnlyProxy, _ReadOnlyPIProxy, _ReadOnlyEntityProxy, _ReadOnlyElementProxy, _OpaqueNodeWrapper, _OpaqueDocumentWrapper, _ModifyContentOnlyProxy, _ModifyContentOnlyPIProxy, _ModifyContentOnlyEntityProxy, _AppendOnlyElementProxy, _SaxParserContext, _FilelikeWriter, _ParserSchemaValidationContext, _XPathContext, _XSLTResolverContext, _XSLTContext, _XSLTQuotedStringParam * Several internal classes can no longer be inherited from: _InputDocument, _ResolverRegistry, _ExsltRegExp, _ElementUnicodeResult, _IterparseContext, _TempStore, _AttribIterator, _ClassNamespaceRegistry, _XPathFunctionNamespaceRegistry, _ParserDictionaryContext, _FileReaderContext, _PythonSaxParserTarget, _TargetParserContext, _ReadOnlyPIProxy, _ReadOnlyEntityProxy, _OpaqueDocumentWrapper, _ModifyContentOnlyPIProxy, _ModifyContentOnlyEntityProxy, _AppendOnlyElementProxy, _FilelikeWriter, _ParserSchemaValidationContext, _XPathContext, _XSLTResolverContext, _XSLTContext, _XSLTQuotedStringParam, _XSLTResultTree, _XSLTProcessingInstruction

1 0

Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 26, 2012

Sept. 26, 2012

Hi Stefan, Thanks for the inputs. I'll test again with a clean install. Will post the results. Regards Pankaj Singh From: Stefan Behnel Sent: 26-09-2012 21:59 To: lxml(a)lxml.de Subject: Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error Simon Sapin, 26.09.2012 17:18: > Le 26/09/2012 17:10, Pankaj Singh a écrit : >> Is there an option wherein I can specify unicode style while building >> lxml? No. > I *think* it should just pick it up from the current python at > compile-time Yes, that's fully automatic. There is no way it can get that wrong. > Just to be clear: you’re installing/compiling lxml from a source tarball > with a USC4 python, and when importing it in the same python get > "undefined symbol: PyUnicodeUCS2_DecodeLatin1". Is this correct? That can't be the case. I assume there's something wrong with either the import path (PYTHONPATH) or the local installation, potentially both. For diagnosis, run strace -e open,stat python -c 'import lxml.etree' | fgrep etree That will show from where it tries to import the lxml.etree module. In any case, a clean install should fix this. Stefan _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 26, 2012

Sept. 26, 2012

Hi Simon, Thanks for the quick reply. Lxml was built from the tar using the same python. For safety sake i'll test again after removing all other python versions from the system. Is there an option wherein I can specify unicode style while building lxml? Thanks Pankaj Singh From: Simon Sapin Sent: 26-09-2012 20:24 To: lxml(a)lxml.de Subject: Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error Le 26/09/2012 16:39, Pankaj Singh a écrit : > *ImportError: > /usr/local/Python/lib/python2.6/site-packages/lxml/etree.so: undefined > symbol: PyUnicodeUCS2_DecodeLatin1* lxml can support either UCS4 or UCS2 as the internal Unicode representation, but the switch is made at compile-time. Make sure that the Python that compiled etree.so and the Python that uses it are the same. Same version (at least minor as in 2.7.x, maybe micro as in 2.7.3), same architecture, and in this case same Unicode settings (UCS4 vs UCS2.) The easiest way to do this is to re-install lxml from the source tarball. Do you still have this issue? -- Simon Sapin _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

3 2

Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 26, 2012

Sept. 26, 2012

Yes, that is correct. Pankaj Singh From: Simon Sapin Sent: 26-09-2012 20:48 To: lxml(a)lxml.de Subject: Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error Le 26/09/2012 17:10, Pankaj Singh a écrit : > Is there an option wherein I can specify unicode style while building > lxml? I *think* it should just pick it up from the current python at compile-time, but there might be something buggy. Just to be clear: you’re installing/compiling lxml from a source tarball with a USC4 python, and when importing it in the same python get "undefined symbol: PyUnicodeUCS2_DecodeLatin1". Is this correct? -- Simon Sapin _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 26, 2012

Sept. 26, 2012

Hi, Im getting the following error while importing etree *ImportError: /usr/local/Python/lib/python2.6/site-packages/lxml/etree.so: undefined symbol: PyUnicodeUCS2_DecodeLatin1* Environment specs are as follows OS : Red Hat 4.1.2-14 Python : 2.6.4 lxml : 2.3.5 *I needed to install Python with flag --enable-unicode=ucs4* It looks as if lxml is not supporting unicode ucs4 through which python was installed. What are my options if I want to use lxml with above mentioned unicode based python installation? My apologies if this question has been asked before but I ain't quite sure as to how I can search the previous posts without going through posts from every month one by one in the archives :( Thanks and Regards Pankaj Singh

2 1

Installing with pip seems to always choose version 3.0.alpha2
by Tal Einat Sept. 25, 2012

Sept. 25, 2012

pip install lxml==2.3 Running setup.py egg_info for package lxml Building lxml version 3.0.alpha2. ... This is very annoying, I can't seem to get pip to install any version of lxml other than 3.0.alpha2! I don't know if it's an issue with pip or with lxml, but please fix this... Regards, - Tal Einat

3 6

Strange memory leakage problem on Ubuntu
by zzjjzzgggg Sept. 25, 2012

Sept. 25, 2012

Hello, I am new to lxml. I installed lxml on Ubuntu 12.04 via easy_install as suggested easy_install --allow-hosts=lxml.de,*.python.org lxml The version of libxml2 is Version: 2.7.8.dfsg-5.1ubuntu4.1 I use python 3.2. When I use lxml to process some HTML fragments, I find serious memory leakage issues. The test code and test data are attached. When I remove this line ps=html.xpath("//p[@node-type='feed_list_content']") , There isn't memory leakage problem. What's the problem please? Many thanks. I can provide more info if you need. I post a thread in stackoverflow. Somebody suggests it is a bug. http://stackoverflow.com/questions/12383537/lxml-memeory-leakage-with-pytho… Best regards zzjjzzgggg

2 2