Mailman 3 October 2006 - lxml - The Python XML Toolkit

[lxml-dev] Can't build on Windows
by Dan Posluns 01 Nov '06

01 Nov '06

I'm not normally a Windows user so I'm at my wit's end here trying to get lxml to install. When I try easy_install I get the following: C:\Python24\Scripts>easy_install lxml Searching for lxml Reading http://www.python.org/pypi/lxml/ Reading http://codespeak.net/lxml Reading http://www.python.org/pypi/lxml/1.1.2 Best match: lxml 1.1.2 Downloading http://codespeak.net/lxml/lxml-1.1.2.tgz Processing lxml-1.1.2.tgz Running lxml-1.1.2\setup.py -q bdist_egg --dist-dir c:\docume~1\dposluns\locals~ 1\temp\easy_install-vc3ufv\lxml-1.1.2\egg-dist-tmp-vxzlya Building lxml version 1.1.2 warning: no files found matching 'etree.c' under directory 'src\lxml' warning: no files found matching 'objectify.c' under directory 'src\lxml' warning: no files found matching 'etree.h' under directory 'src\lxml' warning: no files found matching 'etree_defs.h' under directory 'src\lxml' warning: no files found matching 'pubkey.asc' under directory 'doc' warning: no previously-included files found matching 'doc\pyrex.txt' warning: no previously-included files found matching 'src\lxml\etree.pxi' cl : Command line warning D4025 : overriding '/W3' with '/w' cl : Command line warning D4029 : optimization is not available in the standard edition compiler etree.c c:\Documents and Settings\dposluns\Local Settings\Temp\easy_install-vc3ufv\lxml- 1.1.2\src\lxml\etree_defs.h(20) : fatal error C1083: Cannot open include file: ' libxml/xmlversion.h': No such file or directory error: Setup script exited with error: command '"C:\Program Files\Microsoft Visu al Studio .NET 2003\Vc7\bin\cl.exe"' failed with exit status 2 When I try David Sankel's technique to build the libraries statically, I manage to get as far as adding wsock32 (which took me long enough to figure out how to do - again, not a Windows programmer) before the build process borked out on me with the same errors. I was using the Windows distributions of libxml, libxslt, iconv and zlib from zlatkovic.com. Can anyone help me figure this one out? Thanks, Dan. -- Dan Posluns, B. Eng. & Scty. (Software Engineering and Society) dan(a)danposluns.com - ICQ: 35758902 http://www.danposluns.com "The great thing about being the only species on the planet that makes a distinction between right and wrong is that we get to make the rules up for ourselves as we go." - Douglas Adams, Last Chance to See

2 1

[lxml-dev] Test failures on Windows
by Sidnei da Silva 31 Oct '06

31 Oct '06

I think I've mentioned the other day the fact some failures are seen when running tests for trunk on Windows. Here's the log: http://tinyurl.com/yhd4hc If you want to follow the status of the builds, you can use this page. You can also force a build from there by clicking on the builder name (same line as 'changes' header): http://tinyurl.com/yykhz7 -- Sidnei da Silva http://www.enfoldsystems.com

3 20

[lxml-dev] lxml 1.1.2 released
by Stefan Behnel 30 Oct '06

30 Oct '06

Hi everyone, lxml 1.1.2 finally made it to the cheeseshop. http://cheeseshop.python.org/pypi/lxml This is mainly a bugfix release for the stable 1.1 series, the changelog is below. As there were a number of important fixes, updating is recommended. Eggs for x86-64 are uploaded already, and I'd love to see more eggs thrown in that direction. Have fun, Stefan 1.1.2 (2006-10-30) Features added * Data elements in objectify support repr(), which is now used by dump() * Source distribution now ships with a patched Pyrex * New C-API function makeElement() to create new elements with text, tail, attributes and namespaces * Reuse original parser flags for XInclude * Simplified support for handling XSLT processing instructions Bugs fixed * Parser resources were not freed before the next parser run * Open files and XML strings returned by Python resolvers were not closed/freed * Crash in the IDDict returned by XMLDTDID * Copying Comments and ProcessingInstructions failed * Memory leak for external URLs in _XSLTProcessingInstruction.parseXSL() * Memory leak when garbage collecting tailed root elements * HTML script/style content was not propagated to .text * Show text xincluded between text nodes correctly in .text and .tail * 'integer * objectify.StringElement' operation was not supported

1 0

[lxml-dev] lxml now ships with Pyrex included
by Stefan Behnel 27 Oct '06

27 Oct '06

Hi, as lxml will most likely continue to depend on a non-release version of pyrex for quite a while, I decided to add the patched Pyrex version to the source distribution. It is part of lxml's SVN checkout, I only add the Pyrex directory to the project root in my release script. The setup.py script simply prepends that directory to the Python path so that the modified Pyrex is used automatically. This has a couple of consequences for users: * Distributors that want to apply patches to the lxml sources no longer have to check out Pyrex themselves and can shrink their build scripts to a simple run of setup.py. * Users can now modify and build lxml distributions without globally installing a patched Pyrex. They can also run "make clean" if something went wrong, without breaking the build afterwards, even if they have an unpatched Pyrex installed in their site-packages. I hope this helps in making lxml easier to build. Note that SVN users still have to install our Pyrex version, but I think that's acceptable. Now that this works, we could also remove the .c files to shrink the source distribution. Any objections? Stefan

2 2

[lxml-dev] NameError on docloader.pxi
by Sidnei da Silva 27 Oct '06

27 Oct '06

Hi there, I had spotted a NameError in docloader.pxi, but fixed it locally then forgot to send the patch. So here it is. It did bite me on the back when I tried to compile lxml-trunk on another box :) Traceback: File "xslt.pxi", line 593, in etree._XSLTProcessingInstruction.parseXSL File "parser.pxi", line 889, in etree._parseDocument File "parser.pxi", line 893, in etree._parseDocumentFromURL File "parser.pxi", line 810, in etree._parseDocFromFile File "parser.pxi", line 522, in etree._BaseParser._parseDocFromFile File "parser.pxi", line 591, in etree._handleParseResult File "etree.pyx", line 201, in etree._ExceptionContext._raise_if_stored File "parser.pxi", line 285, in etree._parser_resolve_from_python File "docloader.pxi", line 84, in etree._ResolverRegistry.resolve File "lxmlfilter.pyc", line 50, in resolve File "docloader.pxi", line 47, in etree.Resolver.resolve_file NameError: _ParserInput -- Sidnei da Silva http://www.enfoldsystems.com

2 1

[lxml-dev] [lxml][objectify] optimization questions
by Holger Joukl 26 Oct '06

26 Oct '06

Hi, sorry for the inconvenience, I now put this into a new thread. And I'd have gotten back to that sooner but have been ill. >Then: what you observe are most likely GC 'issues'. The thing is: if the >element already exists as Python object, it is reused, which is much faster >then creating a new one. So in the cases where your code runs faster, you can >assume that the object survived a larger portion of your code without being >re-instantiated. I probably have some misunderstandings how the reuse of elements works. When I "visit" a node, like: >>> from lxml import etree >>> from lxml import objectify >>> parser = etree.XMLParser(remove_blank_text=True) >>> lookup = etree.ElementNamespaceClassLookup(objectify.ObjectifyElementClassLookup()) >>> parser.setElementClassLookup(lookup) >>> objectify.setDefaultParser(parser) >>> objectify.enableRecursiveStr() >>> root = objectify.Element('root') >>> root.i = 17 >>> root.i <Element i at 1e94b0> >>> the Python Element object for "i" is being created. Will that Python Element be garbage-collected afterwards, if I do not explicitly delete "i" from the xml tree? I thought this element survived in the element proxy. >Especially recursive printing instantiates the entire tree, so if the objects >are not deleted directly afterwards, this has a performance effect on code >that runs afterwards. I see, but why would "manual access" of the nodes not have the same effect: Runs slow: ========== python2.4 -m timeit -v -s""" from lxml import etree from lxml import objectify parser = etree.XMLParser(remove_blank_text=True) lookup = etree.ElementNamespaceClassLookup(objectify.ObjectifyElementClassLookup()) parser.setElementClassLookup(lookup) objectify.setDefaultParser(parser) objectify.enableRecursiveStr() root = objectify.Element('root') root.i = 17 root.f = 238.3343 root.s = 'what' root.d = '2006-03-03' print root.i print root.f print root.s print root.d """ "n = root.i; n = root.f; n = root.s; n = root.d" 17 238.3343 what 2006-03-03 10 loops -> 0.0102 secs 17 238.3343 what 2006-03-03 100 loops -> 0.101 secs 17 238.3343 what 2006-03-03 1000 loops -> 1.02 secs 17 238.3343 what 2006-03-03 17 238.3343 what 2006-03-03 17 238.3343 what 2006-03-03 raw times: 1.03 1.02 1.02 1000 loops, best of 3: 1.02 msec per loop Runs fast: ========== python2.4 -m timeit -v -s""" from lxml import etree from lxml import objectify parser = etree.XMLParser(remove_blank_text=True) lookup = etree.ElementNamespaceClassLookup(objectify.ObjectifyElementClassLookup()) parser.setElementClassLookup(lookup) objectify.setDefaultParser(parser) objectify.enableRecursiveStr() root = objectify.Element('root') root.i = 17 root.f = 238.3343 root.s = 'what' root.d = '2006-03-03' print root """ "n = root.i; n = root.f; n = root.s; n = root.d" root = None [ObjectifiedElement] i = 17 [IntElement] f = 238.33430000000001 [FloatElement] s = 'what' [StringElement] d = '2006-03-03' [StringElement] 10 loops -> 0.00109 secs root = None [ObjectifiedElement] i = 17 [IntElement] f = 238.33430000000001 [FloatElement] s = 'what' [StringElement] d = '2006-03-03' [StringElement] 100 loops -> 0.00928 secs root = None [ObjectifiedElement] i = 17 [IntElement] f = 238.33430000000001 [FloatElement] s = 'what' [StringElement] d = '2006-03-03' [StringElement] 1000 loops -> 0.0897 secs root = None [ObjectifiedElement] i = 17 [IntElement] f = 238.33430000000001 [FloatElement] s = 'what' [StringElement] d = '2006-03-03' [StringElement] 10000 loops -> 0.905 secs root = None [ObjectifiedElement] i = 17 [IntElement] f = 238.33430000000001 [FloatElement] s = 'what' [StringElement] d = '2006-03-03' [StringElement] root = None [ObjectifiedElement] i = 17 [IntElement] f = 238.33430000000001 [FloatElement] s = 'what' [StringElement] d = '2006-03-03' [StringElement] root = None [ObjectifiedElement] i = 17 [IntElement] f = 238.33430000000001 [FloatElement] s = 'what' [StringElement] d = '2006-03-03' [StringElement] raw times: 0.893 0.911 0.911 10000 loops, best of 3: 89.3 usec per loop Recursively outputting root before accessing its child elements really speeds things up, even though I accessed all elements in the slow example, too. Why is this? I'm clueless. Holger Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, verständigen Sie bitte den Absender sofort und löschen Sie die E-Mail sodann. Das unerlaubte Kopieren sowie die unbefugte Übermittlung sind nicht gestattet. Die Sicherheit von Übermittlungen per E-Mail kann nicht garantiert werden. Falls Sie eine Bestätigung wünschen, fordern Sie bitte den Inhalt der E-Mail als Hardcopy an. The contents of this e-mail are confidential. If you are not the named addressee or if this transmission has been addressed to you in error, please notify the sender immediately and then delete this e-mail. Any unauthorized copying and transmission is forbidden. E-Mail transmission cannot be guaranteed to be secure. If verification is required, please request a hard copy version.

2 8

[lxml-dev] [lxml][objectify] optimization of recursive object dumping
by Holger Joukl 26 Oct '06

26 Oct '06

Hi, I've posted this before but messed with the threads, so here it is again: (Note: patch line numbers might differ, this was based on 1.1 branch of 2 weeks ago, but I could of course update this and send a new patch) First some background: I'm experimenting with a custom objectified datetime class based on Python's datetime that employs the dateutil.parser module to detect if some element value is in a valid datetime format, i.e. the parse function from dateutil.parser is used to implement the type_check for the PyType type registry. 1) Invoking this parse method is quite expensive, so I want this to happen rarely. As I am using "recursive element dumping" as default I found that for every __str__ call .pyval of the ObjectifiedDataElements in a tree is accessed, which in turn triggers parsing for my custom datetime class. As I don't really see a way to avoid this I propose the introduction of an additional property "_pyval_repr" that can be overridden in subclasses, which makes it possible to simply return element.text, if getting .pyval is expensive. S.th. like: *** ORIG/lxml-1.1/src/lxml/objectify.pyx Wed Sep 27 09:18:30 2006 --- src/lxml/objectify.pyx Wed Oct 4 11:00:09 2006 *************** *** 484,489 **** --- 484,493 ---- def __get__(self): return textOf(self._c_node) + property _pyval_repr: + def __get__(self): + return self.pyval + def __str__(self): return textOf(self._c_node) or '' *************** *** 931,938 **** cdef object _dump(_Element element, int indent): indentstr = " " * indent ! if hasattr(element, "pyval"): ! value = element.pyval else: value = textOf(element._c_node) if value and not value.strip(): --- 935,942 ---- cdef object _dump(_Element element, int indent): indentstr = " " * indent ! if hasattr(element, "_pyval_repr"): ! value = element._pyval_repr else: value = textOf(element._c_node) if value and not value.strip(): This can substantially speed up things for complicated type_check routines (in my usecase :) Holger Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, verständigen Sie bitte den Absender sofort und löschen Sie die E-Mail sodann. Das unerlaubte Kopieren sowie die unbefugte Übermittlung sind nicht gestattet. Die Sicherheit von Übermittlungen per E-Mail kann nicht garantiert werden. Falls Sie eine Bestätigung wünschen, fordern Sie bitte den Inhalt der E-Mail als Hardcopy an. The contents of this e-mail are confidential. If you are not the named addressee or if this transmission has been addressed to you in error, please notify the sender immediately and then delete this e-mail. Any unauthorized copying and transmission is forbidden. E-Mail transmission cannot be guaranteed to be secure. If verification is required, please request a hard copy version.

2 3

[lxml-dev] Resolvers and open files
by Sidnei da Silva 25 Oct '06

25 Oct '06

I have a long-running process that uses a custom resolver to resolve a simple filename to a file relative to a pre-configured directory. It looks like this: class RelativeUrlResolver(etree.Resolver): def __init__(self, prefix): self.prefix = prefix def resolve(self, url, id, context): print "Resolving URL '%s'" % url if not url.startswith('http'): url = self.prefix + urllib.quote_plus(url) ssf = urllib.urlopen(url) if ssf is None: raise ValueError, 'could not resolve url: %r' % url return self.resolve_file(ssf, context) I'm creating the parser like this: parser = etree.XMLParser() parser.resolvers.add(RelativeUrlResolver(BASE)) (Where BASE = 'file:///path/to/some/dir') Now, the issue that's biting me is that it looks like the file is kept open after the processing has finished. The parser is re-created every time ATM, and goes 'out of scope' right after doing the transformation, so I would expect it all to be garbage collected, and the file to be closed. Do I need to do anything special to get this file to be closed? Thanks. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

2 2

[lxml-dev] Failure to Compile on Windows
by Sidnei da Silva 25 Oct '06

25 Oct '06

I've got pretty far with setting up the dependencies and all but after that the compilation failed with this error. Anyone have a clue? Looks like it's missing some .h that 'nano http' depends on. C:\Arquivos de programas\Microsoft Visual C++ Toolkit 2003\bin\link.exe /DLL /no logo /INCREMENTAL:NO /LIBPATH:c:\Python24\libs /LIBPATH:c:\Python24\PCBuild /EXPORT:initobjectify build\temp.win32-2.4\Release\src/lxml/objectify.obj /OUT:build\lib.win32-2.4\lxml\objectify.pyd /IMPLIB:build\temp.win32-2.4\Release\src/lxml\objectify.lib ..\libxml2-2.6.26.win32\lib\libxml2_a.lib ..\libxslt-1.1.17.win32\lib\libxslt_a.lib ..\libxslt-1.1.17.win32\lib\libexslt_a.lib ..\zlib-1.2.3.win32\lib\zlib.lib ..\iconv-1.9.2.win32\lib\iconv_a.lib Creating library build\temp.win32-2.4\Release\src/lxml\objectify.lib and object build\temp.win32-2.4\Release\src/lxml\objectify.exp libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSAGetLastError@0 referenced in function _socket_errno libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSACleanup@0 referenced in function _xmlNanoHTTPCleanup libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__WSACleanup@0 libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__closesocket@4 referenced in function _xmlNanoHTTPFreeCtxt libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__closesocket@4 referenced in function _xmlNanoFTPNewCtxt libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__select@20 referenced in function _xmlNanoHTTPSend libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__select@20 libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__send@16 referenced in function _xmlNanoHTTPSend libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__send@16 libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__recv@16 referenced in function _xmlNanoHTTPRecv libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__recv@16 libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__getsockopt@20 referenced in function _xmlNanoHTTPConnectAttempt libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol ___WSAFDIsSet@8 referenced in function _xmlNanoHTTPConnectAttempt libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__connect@12 referenced in function _xmlNanoHTTPConnectAttempt libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__connect@12 libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__ioctlsocket@12 referenced in function _xmlNanoHTTPConnectAttempt libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__socket@12 referenced in function _xmlNanoHTTPConnectAttempt libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__socket@12 libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__htons@4 referenced in function _xmlNanoHTTPConnectHost libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__htons@4 libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__gethostbyname@4 referenced in function _xmlNanoHTTPConnectHost libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__gethostbyname@4 libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSAStartup@8 referenced in function _xmlNanoHTTPInit libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__WSAStartup@8 libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__listen@8 referenced in function _xmlNanoFTPGetConnection libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__bind@12 referenced in function _xmlNanoFTPGetConnection libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__getsockname@12 referenced in function _xmlNanoFTPGetConnection build\lib.win32-2.4\lxml\objectify.pyd : fatal error LNK1120: 17 unresolved externals error: command '"C:\Arquivos de programas\Microsoft Visual C++ Toolkit 2003\bin\link.exe"' failed with exit status 1120 -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

3 6

[lxml-dev] doctype declarations in XML output
by Raymond Wiker 20 Oct '06

20 Oct '06

Hi. I've just started using Python 2.5 and lxml.etree, and am very happy so far - with this combination, I have an easy-to-use XML solution that includes good XPath and XSLT support. One question: is it possible to add doctype definitions to the output? I have not been able to find anything about this in the documentation, and I cannot see any references to xmlCreateIntSubset or xmlNewDtd in the source code. These should be trivial to add, as far as I can see.

2 1