Mailman 3 March 2008 - lxml - The Python XML Toolkit

Re: [lxml-dev] Python XML Validator
by Stefan Behnel 12 Mar '08

12 Mar '08

Hi, moving this here from python-dev (where it started for whatever reason...) Mike Meyer wrote: > On Tue, 11 Mar 2008 18:01:29 +0100 Stefan Behnel wrote: >> BTW, we had MacOS builds a while ago, so I wouldn't mind having someone >> volunteer to contribute builds on a regular basis (static builds preferred). > > For which Python build? python.org? Activatestate? Leopard? Macports? > Fink? pkgsrc? Any idea if a single build will work for all of them? I have no idea. At the very least, different Python major versions will pose a problem. And I guess the builds provided by package distributions like fink and macports will also require newer dependencies on other ends, or be built with newer compilers... >>> The second time for OS-X, I used an older version of lxml (1.3.6), and >>> just did "setup.py install". Worked like a charm. That's not hard. >> Interesting. 1.3.6 should also require libxml2 2.6.20 (although maybe less >> strictly than 2.0). > > I just grabbed it and tried parsing thing with it; I didn't try the > advanced features that I depend on lxml for (rng validation and lots > of xpath), or what the OP was looking for (validation). Running the > test.py suite turns has one failure: > > File "/Users/mwm/lxml-1.3.6/src/lxml/tests/../../../doc/parsing.txt", line 369, in parsing.txt > Failed example: > etree.tounicode(root) > Expected: > u'<test> \uf8d1 + \uf8d2 </test>' > Got: > u'<test>  +  </test>' If that's the only problem, then 1.3.x works 'acceptably' with 2.6.16 - except that newer versions are much better in parsing HTML and validating with XML schema (amongst other things). Note that the test suite tends to avoid testing features that only depend on libxml2, and especially stuff that has changed between library versions. It's a test suite for lxml, not for libxml2. However, 2.0 will not work that easily. Things like parse-time schema validation and schematron support do not work on versions below 2.6.20 (or actually 2.6.21, but we disable schematron on 2.6.20). We might be able to work around some more stuff by spreading some #ifdef's and #defines, but so far, I find it perfectly acceptable if 2.0 requires newer dependencies for new features. People who care about reliability will not use libraries as old as 2.6.16 anyway. The list of fixed bug only gets longer with newer versions. >>>>> Which means you wind up having to >>>>> build those yourself if you want a recent version of lxml, even if >>>>> you're using a system that includes lxml in it's package system. >>>> If you want a clean system, e.g. for production use, buildout has proven to be >>>> a good idea. And we also provide pretty good instructions on our web page on >>>> how to install lxml on MacOS-X and what to take care of. >>> Yes, but the proposal was to include it in the Python standard >>> library. Software that doesn't work on popular target platforms >>> without updating a standard system library isn't really suitable for >>> that. >> Hmm, coming somewhat back on-topic: how does Python currently handle its >> dependencies under MacOS-X? SQLite, for example? Does it use system libraries >> only, or are there libraries it ships with? (The MacOS distro is much bigger, >> but that might be due to the universal build - although that suggests that >> MacOS-X users do not care about disk space or download size anyway) > > For most of them, it checks for the existence of the libraries and > header files for those packages, and then builds the wrapper libraries > if it finds their requirements. Look through the 2.5.2 setup.py for > how sqlite3 is handled (it's a bit much to include here). Funny, looking for the sqlite setup was actually a good idea. It does all sorts of things to figure out a good one to use, specifically on MacOS-X. There even appears to be some trickery to take the first library it finds, static or dynamic, instead of continuing to look for a dynlib. I wouldn't mind adding a similar setup to lxml's setupinfo.py. Maybe someone can get a hand on this? It would be great to have an automatic static build on MacOS, so that people could just run setup.py and be sure it uses the expected libs the next time they use it. Is there a standard directory prefix where macport & Co. install libraries and related stuff like xslt-config? Stefan

2 1

[lxml-dev] .base and docinfo.URL
by Ian Bicking 11 Mar '08

11 Mar '08

Does .base inherit from docinfo.URL? It doesn't seem like it does. I tried changing .base_url to just return self.base, but if I do: >>> from lxml.html import parse >>> doc = parse('http://python.org').getroot() >>> print doc.base None >>> doc.getroottree().docinfo.URL 'http://python.org'

2 2

[lxml-dev] Some benchmarks
by Ian Bicking 11 Mar '08

11 Mar '08

For the curious, I've attached some benchmarks. These are preliminary, I'm putting together the numbers for my HTML talk at PyCon. One thing that I'd like to test is the memory use for documents. To do this I'm parsing about 4.5Mb of documents and keeping them in memory, and looking at the VSZ/RSS sizes reported by ps before and after. I don't think this is the right/best way to do this. For instance, transient memory use by some parsers makes Python grab a bunch of memory, but it might be free after parsing, and usable for other things. Also, I don't know if VSZ/RSS is valid at all. I get the impression it isn't that valid. And the increases I'm seeing for lxml don't seem to be sufficient; at least the process should grow by 4.5Mb, right? lxml can't be that much more efficient than the serialized form of these files. Another clear indication that we're measuring transient stuff is that when using the BeautifulSoup or html5 parser with an lxml document the memory increases substantially. So any ideas on how to test memory would be much appreciated. (Maybe I could look at ps, and then start creating Python objects until the memory use increases, so that I know I've used up any extra allocated memory?) I've also attached the script, though you'll need to grab your own HTML files. html_lxml is broken; I patched it locally to work (http://code.google.com/p/html5lib/issues/detail?id=65). Ian Parsing 355 files, 4524Kb (ripped from python.org) lxml = lxml.html bs = BeautifulSoup html5_cet = html5 parser with cElementTree model html5_et = html5 parser with ElementTree model html5_lxml = html5 parser with lxml.html model html5_minidom = html5 parser with minidom model html5_simple = html5 parser with internal simple_tree model lxml_bs = BeautifulSoup parser with lxml model htmlparser = HTMLParser, with no parser actions, document string is its own model python tester.py --no-gc lxml : 0.5156 sec ( 100% of lxml) bs : 10.3816 sec (2013% of lxml) html5_cet : 29.5829 sec (5737% of lxml) html5_et : 30.2433 sec (5865% of lxml) html5_lxml : 31.7533 sec (6158% of lxml) html5_minidom : 34.2963 sec (6651% of lxml) html5_simple : 28.7421 sec (5574% of lxml) lxml_bs : 12.2269 sec (2371% of lxml) htmlparser : 3.0968 sec ( 600% of lxml) python tester.py --no-gc --serialize lxml : 0.2704 sec ( 100% of lxml) bs : 1.8265 sec ( 675% of lxml) html5_cet : 1.5960 sec ( 590% of lxml) html5_et : 1.7677 sec ( 653% of lxml) html5_lxml : 0.2755 sec ( 101% of lxml) html5_minidom : 3.4696 sec (1283% of lxml) html5_simple : 1.4929 sec ( 552% of lxml) lxml_bs : 0.2834 sec ( 104% of lxml) VSZ/RSS increase: lxml: 1168 / 120 bs: 82508 / 82176 html5_cet: 54620 / 54756 html5_et: 64688 / 64960 html5_lxml: 49076 / 49124 html5_minidom: 194304 / 192928 html5_simple: 98608 / 98004 lxml_bs: 104920 / 104852 htmlparser: 5412 / 4456 Note: htmlparser keeps all the strings of the documents in memory.

3 5

Re: [lxml-dev] lxml for Python 2.3
by Stefan Behnel 10 Mar '08

10 Mar '08

Hi, Valeriy Pogrebitskiy wrote: > Does this mean that easy_install will, or will not build lxml? > And yes - I am looking for Windows version. If you have the necessary tools installed on your machine (which seems to be VC6 for Python 2.3), you should be able to build lxml. You might also be successful with MinGW, but you'll have to work that out. May I ask what the reason is why you are tied to Py2.3? Stefan

1 0

Re: [lxml-dev] lxml for Python 2.3
by Stefan Behnel 10 Mar '08

10 Mar '08

Hi, Valeriy Pogrebitskiy wrote: > This communication may contain privileged or confidential information. If > you are not the intended recipient, you are hereby notified that you have > received this message in error and that any review, dissemination, > distribution or copying of this message is strictly prohibited. Note that your message is archived, which is clearly stated on the subscription page. Stefan

1 0

[lxml-dev] should lxml.html have the same API as lxml.etree?
by Ian Bicking 10 Mar '08

10 Mar '08

I've noticed in html5lib that they do (in html5lib.treebuilder.etree_lxml): try: import lxml.html as etree except ImportError: from lxml import etree with the expectation that the two work the same way. They don't work the same, specifically there's no etree.Comment. Is it a reasonable expectation that they act the same? (I think they haven't tested the code much with lxml 2, so basically they haven't exercised the first case... though looking at the code some I'm not sure it works with the second case either) Ian

3 4

Re: [lxml-dev] lxml for Python 2.3
by Stefan Behnel 09 Mar '08

09 Mar '08

Hi, VALERIY POGREBITSKIY wrote: > I am looking for a lxml distribution for Python 2.3, and found no such > distribution (it exeists for Python 2.4 and 2.5 only). I also found some of > your comments that you found a way to compile 'lxml' for Python 2.3. > > Do you have (or do you know where I can find) such distribution (install) - > for Python 2.3? > > I would greatly appreciate your help. Since lxml compiles and runs nicely under Python 2.3 (use easy_install, as usual), I assume that what you actually want is a pre-built binary for MS Windows, right? Maybe someone on the list has a Windows build for Py2.3, or can get one ready? Stefan

2 1

[lxml-dev] XHTML handling in lxml.html
by Stefan Behnel 04 Mar '08

04 Mar '08

Ian Bicking wrote: > translating HTML to XHTML is kind of an outstanding issue for lxml.html, > and it seems reasonable to me that XHTML could be parsed into the same > classes as HTML. The only real caveat there is that XHTML uses different > (namespaced) tag names. I agree that there is more we could do. For example, we could add "xhtml" as a serialisation method and do stuff internally to add a namespace declaration to the serialised "<html>" (iff there isn't a namespace declared already). I'm not sure if it would be an error if the tree contains non-HTML elements, I guess we could just leave that to the user. > If you remove the tag names, then the classes and > the lookup applies just fine. (Presumably the lookup could be changed to > support XHTML fairly easily.) I would say so, yes. There would also be issues with the XPath expressions in things like html.clean, I assume. It would definitely be a good thing if the whole machinery could handle namespace-free HTML and namespaced XHTML equally well. Stefan

2 1

[lxml-dev] error installing lxml
by Elena Soutyrina 04 Mar '08

04 Mar '08

I am having trouble to install lxml. I already installed libxml2 (2.6.30) and libxslt, Cython (0.9.6.12) easy_install lxml gives me error Building with Cython 0.9.6.12. Building lxml version 2.0.2. warning: no previously-included files found matching 'doc/pyrex.txt' src/lxml/lxml.etree.c:1536: error: syntax error before 'xmlSchemaSAXPlugStruct' src/lxml/lxml.etree.c:1536: error: syntax error before 'xmlSchemaSAXPlugStruct' What am I missing? Best regards, Elena Soutyrina Application Engineer | Scope Seven 310 220 3939 x430 2201 Park Place, Suite 100 | El Segundo, CA 90245 www.scopeseven.com

2 1

[lxml-dev] Using EXSLT extensions on Windows with standard lxml binaries
by Alexander Kozlovsky 03 Mar '08

03 Mar '08

Hello! I'm trying to use EXSLT extension functions on Windows with standard lxml binary distribution (lxml-1.3.6.win32-py2.4.exe) I'm trying to do the next, but it is not work as expected: <?xml version="1.0"?> <xsl:stylesheet version = '1.0' xmlns:xsl='http://www.w3.org/1999/XSL/Transform' xmlns:str="http://exslt.org/strings" extension-element-prefixes="str"> <xsl:template match="body"> <body> <h1 class="{str:replace('abc', 'b', 'x')}">test</h1> </body> </xsl:template> </xsl:stylesheet> Is it possible to use EXSLT extensions on Windows without rebuilding of standard binaries? Thanks in advance -- Best regards, Alexander mailto:alexander.kozlovsky@gmail.com

3 12