Mailman 3 August 2016 - lxml - The Python XML Toolkit

Import lxml module cause PyImport_ImportModule failed
by 刘云枫 18 Aug '16

18 Aug '16

I wrote python test code which uses the module lxml. I want to call foo in c++. It will fail in step PyImport_ImportModule if I add the “from lxml import html” but works well when I remove it. Directly run Test.py in cmd is ok. My OS: windows 10 Python: python 2.7 x86 lxml: lxml-3.6.0.win32-py2.7.exe Test.py import os import sys import requests from lxml import html #it will cause failed def foo(): host = "http://www.baidu.com" s = requests.session() res = s.get(host) return res c++ code: Py_Initialize(); PyRun_SimpleString("import sys"); PyRun_SimpleString("sys.path.insert(0,'./')"); PyObject* pModule = PyImport_ImportModule("Test"); // here failed if (pModule == NULL || PyErr_Occurred()) { PyErr_Print(); } PyObject* pDict = PyModule_GetDict(pModule); if (pDict == NULL || PyErr_Occurred()) { PyErr_Print(); } PyObject *pFunHi = PyDict_GetItemString(pDict, "foo"); PyObject *ret = PyObject_CallFunction(pFunHi,NULL); Py_DECREF(pFunHi); Py_Finalize(); The error msg is : Traceback (most recent call last): File "C:\Users\Administrator\Desktop\test\Test\Debug\Test.py", line 4, in <module> from lxml import html File "E:\python27\lib\site-packages\lxml\html\__init__.py", line 54, in <module> from .. import etree ImportError: DLL load failed: Unable to find the specified module。 how to use lxml module correct?

2 1

XMLSyntaxError: value is not an element of the set
by iiijjjiii＠gmail.com 17 Aug '16

17 Aug '16

I downgraded libxml2 to 2.9.3 and the problem goes away. Thanks, Jim

1 0

XMLSyntaxError: value is not an element of the set
by Jim Karsten 17 Aug '16

17 Aug '16

I am getting an unusual exception from lxml.objectify. Traceback (most recent call last): File "/root/tmp/lxml_test.py", line 56, in <module> main() File "/root/tmp/lxml_test.py", line 52, in main root = objectify.fromstring(xml, parser) File "src/lxml/lxml.objectify.pyx", line 1801, in lxml.objectify.fromstring (src/lxml/lxml.objectify.c:25171) File "src/lxml/lxml.etree.pyx", line 3213, in lxml.etree.fromstring (src/lxml/lxml.etree.c:77697) File "src/lxml/parser.pxi", line 1819, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:116494) File "src/lxml/parser.pxi", line 1707, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:115144) File "src/lxml/parser.pxi", line 1079, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:109543) File "src/lxml/parser.pxi", line 573, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:103404) File "src/lxml/parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:105058) File "src/lxml/parser.pxi", line 613, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:103967) lxml.etree.XMLSyntaxError: Element 'link', attribute 'rel': [facet 'enumeration'] The value '' is not an element of the set {'self', 'group'}. Demo code: #!/usr/bin/env python2 # -*- coding: utf-8 -*- """ lxml_test.py Script for testing lxml. """ from cStringIO import StringIO from lxml import etree, objectify def main(): """Main processing.""" xsd = """<?xml version="1.0"?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="shipment-info"> <xsd:complexType> <xsd:all> <xsd:element ref="links"/> </xsd:all> </xsd:complexType> </xsd:element> <xsd:element name="links"> <xsd:complexType> <xsd:sequence> <xsd:element ref="link" maxOccurs="unbounded"/> </xsd:sequence> </xsd:complexType> </xsd:element> <xsd:element name="link" type="LinkType"/> <xsd:complexType name="LinkType"> <xsd:attribute name="rel" type="RelType" use="required"/> <xsd:attribute name="index" type="xsd:nonNegativeInteger" use="optional"/> </xsd:complexType> <xsd:simpleType name="RelType"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="self"/> <xsd:enumeration value="group"/> </xsd:restriction> </xsd:simpleType> </xsd:schema> """ xml = """<?xml version="1.0" encoding="UTF-8"?> <shipment-info><links><link rel="self" index="1"></link></links></shipment-info>""" fileobject = StringIO(xsd) schema = etree.XMLSchema(file=fileobject) parser = objectify.makeparser(schema=schema) root = objectify.fromstring(xml, parser) if __name__ == '__main__': main() The xml and xsd validate. xmlschema_doc = etree.parse(fileobject) xmlschema = etree.XMLSchema(xmlschema_doc) xml_f = StringIO(xml) doc = etree.parse(xml_f) valid = xmlschema.validate(doc) print 'valid: {var}'.format(var=valid) # prints: valid: True So I suspect the problem arises from lxml converting to python objects. Versions: Python 2.7.12 lxml-3.6.0 libxml2.so.2.9.4 libxslt.so.1.1.29 No exception is raised on my old server. Only when I upgraded was the exception raised. The old server runs: Python 2.7.2 lxml-2.3.4 libxml2.so.2.7.8 libxslt.so.1.1.26 Any suggestions? Thanks, Jim

2 1

lxml.html.fragments_fromstring whitespace issue
by Fabio Sangiovanni 13 Aug '16

13 Aug '16

Hi, with no_leading_text=False (which is the default), lxml.html.fragments_fromstring should strip leading whitespace from the first item of the returned list, if it's a string (from the docs: "The first item in the list may be a string (though leading whitespace is removed)"). This doesn't seem to happen: >>> from lxml.html import fragments_fromstring >>> html = ' prefix <div>TEXT</div>TAIL' >>> frags = fragments_fromstring(html) >>> frags [' prefix ', <Element div at 0x7f0f3fce51d8>] Python : sys.version_info(major=3, minor=5, micro=1, releaselevel='final', serial=0) lxml.etree : (3, 6, 0, 0) libxml used : (2, 9, 4) libxml compiled : (2, 9, 3) libxslt used : (1, 1, 29) libxslt compiled : (1, 1, 28) If I may express a personal opinion: I'd appreciate if the behaviour could remain the current one (this would also leave the API untouched). In this case, only the docs need to be updated. Thanks, Fabio ps. I've opened an issue on launchpad about this - https://bugs.launchpad.net/lxml/+bug/1603631

2 1

Internal error in xsltKeyFunction
by Mikhail Edoshin 13 Aug '16

13 Aug '16

Hello, I'm getting the following error when I try to transform XML with XSLT. lxml.etree.XSLTApplyError: Internal error in xsltKeyFunction(): Could not get the document info of a context doc. Could I get some information about that error? Which context document does it mean? For example, if I've created a document on the fly or maybe isolated a part of a document with ElementTree and then tried to transform it with XSLT that had keys, could the error be caused by the fact that these objects are not "real" documents? Or it's some other context document? My task context is that I'm trying to dynamically modify an XML tree and transform various parts of the tree (as opposed to the whole document). It works fine until I try to use keys. I'm totally willing to rewrite it but I cannot quite understand what might solve the problem. That is my goal is to avoid saving results to disk and just keep transforming. For example: 1) Transform 'foo.xml' into a tree with root 'transformedFoo'. Keep the result in memory. 2) Transform 'bar.xml' into a tree with root 'transformedBar'. Keep the result in memory. 3) Combine these two elements into something like: <baz> <transformedFoo>...</transformedFoo> <transformedBar>...</transformedBar> </baz> 4) Transform the resulting <baz> element and finally save the result to disk. Kind regards, -- Mikhail Edoshin mikhail.edoshin(a)gmail.com Skype: mikhail_edoshin (GMT+3)

2 1

Fwd: Parse cyrillic names and values attributes
by Thomas Morgan 13 Aug '16

13 Aug '16

Hi, everyone! It posible in future make parse cyrillic names and values attributes (I understand that it sounds awful)? Beautiful Soup 4 in this case works normally. File 'file.html' is attached to the letter. For example: Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> from lxml import etree >>> >>> print("%-20s: %s" % ('Python', sys.version_info)) Python : sys.version_info(major=3, minor=5, micro=1, releaselevel='final', serial=0) >>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION)) lxml.etree : (3, 6, 0, 0) >>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION)) libxml used : (2, 9, 3) >>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION)) libxml compiled : (2, 9, 3) >>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION)) libxslt used : (1, 1, 28) >>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_ VERSION)) libxslt compiled : (1, 1, 28) >>> from lxml.html import fromstring, tostring >>> html_str = "" >>> with open('file.html', encoding="utf-8") as f: ... html_str = f.read() ... >>> html1 = fromstring(html_str.encode("utf-8")) >>> div = html1.cssselect("div")[0] >>> div.items() [('class', 'test')] >>> div.set("новый_ключ", "новое_значение") >>> div.items() [('class', 'test'), ('новый_ключ', 'новое_значение')] >>> >>> html2 = fromstring(tostring(html1, encoding='utf-8')) >>> div2 = html2.cssselect('div')[0] >>> div2.items() [('class', 'test')] >>> import bs4 >>> html3 = bs4.BeautifulSoup(html_str, 'html.parser') >>> div3 = html3.find_all('div')[0] >>> div3.attrs {'ключ': 'значение', 'class': ['test']} >>> div3["новый_ключ"] = "новое_значение" >>> div3.attrs {'новый_ключ': 'новое_значение', 'ключ': 'значение', 'class': ['test']}

2 1

Re: [lxml] Prevent lxml from adding a default doctype
by Charlie Clark 13 Aug '16

13 Aug '16

Am .08.2016, 11:21 Uhr, schrieb Shadab Zafar <dufferzafar0(a)gmail.com>: Please keep this on list, even if the default setting is reply to author. > We are using lxml in mitmproxy to display the data, but don't want to > add new fields to it. Yes, but you want lxml to produce valid HTML, don't you? > lbxml2 supposedly has an option to not do this, but I can't figure out > how to activate it in lxml. Seems to me that all you have to do is *not* pass in docinfo.doctype into lxml.etree.tostring. Easy enough to write a conditional for this. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Kronenstr. 27a Düsseldorf D- 40217 Tel: +49-211-600-3657 Mobile: +49-178-782-6226

3 5

Prevent lxml from adding a default doctype
by Shadab Zafar 12 Aug '16

12 Aug '16

lxml seems to add a default doctype when one is missing in the html document. See this demo code: import lxml.etree import lxml.html def beautify(html): parser = lxml.etree.HTMLParser( strip_cdata=True, remove_blank_text=True ) d = lxml.html.fromstring(html, parser=parser) docinfo = d.getroottree().docinfo return lxml.etree.tostring( d, pretty_print=True, doctype=docinfo.doctype, encoding='utf8' ) with_doctype = """ <!DOCTYPE html> <html> <head> <title>With Doctype</title> </head> </html> """ # This passes! assert "DOCTYPE" in beautify(with_doctype) no_doctype = """<html> <head> <title>No Doctype</title> </head> </html>""" # This fails! assert "DOCTYPE" not in beautify(no_doctype) # because the returned html contains this line # <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> # which was not present in the source before --- How can I tell lxml to not do this? This issue was originally raised here: https://github.com/mitmproxy/mitmproxy/issues/845 Quoting a comment on reddit [1] as it might be helpful: > lxml is based on libxml2, which does this by default unless you pass the option `HTML_PARSE_NODEFDTD`, I believe. See code. [2] > I don't know if you can tell lxml to pass that option though.. libxml has python bindings that you could perhaps use directly but they seem really hairy. > EDIT: did some more digging and that option does appear in the lxml soure [3]. That option does exactly what you want but I'm not sure how to activate it yet, if it's even possible. [1] - https://www.reddit.com/r/Python/comments/42rb90/comparing_execution_time_of… [2] - https://github.com/GNOME/libxml2/blob/master/HTMLparser.c#L4777 [3] - https://github.com/lxml/lxml/blob/master/src/lxml/includes/htmlparser.pxd#L…

2 1