Mailman 3 April 2008 - lxml - The Python XML Toolkit

Re: [lxml-dev] CDATA and lxml
by Stefan Behnel 11 Apr '08

11 Apr '08

Silfheed wrote: > So first off I know that CDATA is generally hated and just shouldn't > be done, but I'm simply required to parse it and spit it back out. > Parsing is pretty easy with lxml, but it's the spitting back out > that's giving me issues. The fact that lxml strips all the CDATA > stuff off isnt really a big issue either, so long as I can create > CDATA blocks later with <>&'s showing up instead of <>& . > I've scoured through the lxml docs, but probably not hard enough, so > anyone know the page I'm looking for or have a quick how to? There's nothing in the docs because lxml doesn't allow you to create CDATA sections. You're not the first one asking that, but so far, no one really had a take on this. It's not as trivial as it sounds. Removing the CDATA sections in the parser is just for fun. It simplifies the internal tree traversal and text aggregation, so this would be affected if we allowed CDATA content in addition to normal text content. It's not that hard, it's just that it hasn't been done so far. Stefan

1 2

[lxml-dev] Help getting lxml to work reliably on MacOS-X
by Stefan Behnel 09 Apr '08

09 Apr '08

Hi, it looks like many Mac users have problems with lxml on their platform. This usually involves installing up-to-date dependencies (libxml2/libxslt) in addition to the system libraries. I would like to get these problems resolved. To get a start on this, we must collect some information. We had a few reports, but I need to know in more detail what people did, what they tried, and to what avail. So here is a list of questions for Mac users. Please help us by answering them. Some instructions follow at the end. When building lxml, please move any installed Cython versions out of the way and run the build on the unpacked lxml-2.0.tar.gz release sources. It must say "trying to build without Cython" at the beginning. Please provide the following information: - what package management system (fink/macports) do you use? - are you using the stock Python or one that is installed separately? - what library versions are you using of libxml2, libxslt, zlib, libiconv? - which library versions are preinstalled on your platform? (I do not know how to find that out, can anyone provide instructions here?) - what library versions does lxml.etree find? (see below) Just in case there are people who actually have a working installation, - has anyone successfully built lxml statically against libxml2/libxslt? * does it work reliably? (see "Testing" below) * did you build with the --without-threads option? * does it work with *and* without that option? - has anyone managed to get lxml working reliably (see "Testing" below) with a dynamic build? * did you set DYLD_LIBRARY_PATH? * is DYLD_LIBRARY_PATH required for you or does it work without? * is there anything special you did to make it work? * if there are crashes, is the install unusable or are there things you can still do reliably? If lxml crashes for you, - does it work if you set DYLD_LIBRARY_PATH at runtime? (dynamic builds only) - does it work when building with the --without-threads option? - does it crash when running the normal tests? - if the tests pass fine, does it crash with the "ot-test" script below? Every reply is appreciated. You can reply in private e-mail, if you prefer, although it might be helpful to others to see your answer. Thanks in advance, Stefan Here are some instructions: * Checking library versions Please report the library versions that lxml.etree thinks it is using: http://codespeak.net/lxml/FAQ.html#i-think-i-have-found-a-bug-in-lxml-what-… * Setting DYLD_LIBRARY_PATH When you run tests or an application with lxml, you can pass the environment variable DYLD_LIBRARY_PATH to your program. It needs to be set to the directory (or directories) where lxml's library dependencies are installed, i.e. libxml2 and libxslt. http://codespeak.net/lxml/build.html#providing-newer-library-versions-on-ma… * Testing When I say "works reliably", I mean, without crashes. The first thing to verify that is to run "make test". If it already crashes here, there is no need to try the script below. Please report anything you can find out about the crash in this case. A debugger run and/or a stack trace might give us some useful hints here. If the normal tests pass, please try another test. Here is an XML document and a script that fires off a bunch of threads to parse the document and run multiple XSLTs over it. The archive is about 1MB. http://codespeak.net/lxml/ot-test.zip It's kind of a worst-case scenario that I use to find problems, so if you have a working installation of lxml 2.0, please run it to see if it crashes for you. Parsing may fail (usually for threads 7-9), that's fine, but it must not crash with your otherwise working setup. It normally starts up 16 threads, which should require a couple of hundred MB of memory. If you find that it runs out of memory, you can try running less threads instead by passing the number to the script at the command line.

7 17

[lxml-dev] sednaobject - Pythonic interface to Sedna XML Database
by Jim Washington 09 Apr '08

09 Apr '08

I've put out a new alpha (0.10alpha2) of zif.sedna, a Python adapter to Sedna, a multi-user XML database, at the Python Cheese Shop. The new alpha has a start at objectifying XML from the Sedna database in a manner kind of like sqlobject does for SQL. The aim is to make easy, pythonic CRUD (create, read, update, delete) operations for the XML store. Push data in, get data back out. sednaobject, now included in zif.sedna, provides three classes. First is SednaXQuery, which gives you list-like semantics for the results of any arbitrary XQuery or XPath expression. You init it with a Sedna cursor, a query statement, and an optional parser method. Then, you can iterate, or obtain items by index or slice. If you do not provide a parser, you get items as unicode strings. Second is SednaContainer, which behaves like a SednaXQuery, except it is read-write. Like SednaXQuery, you init it with a cursor, a query statement, and an optional parser method. The query statement must refer to exactly one element in the database. This is the container, and you can obtain and replace items in the container by index. Slicing works for retrieval, and append, remove, insert, and del work as per the elementtree API. Third is SednaObjectifiedElement, which also operates on a single element in the database. SednaObjectifiedElement is a thin wrapper around lxml.objectify. Alter the item with the objectify API, and save(). Thanks, lxml team, for making this really easy! Since, in XML, an element is an element is an element, you can use the second and third sednaobject classes on any element in the database. Which you would use in a situation depends on the aspect you are interested in at the moment. I see Sedna as an attractive middle ground between SQL databases and object databases like ZODB. Data size is practically unlimited. You can alter a small portion of a data set transactionally, in a multi-user environment, without a full rewrite of the data. Like SQL databases, it uses a query language to obtain and format just the data you want, from anywhere in the database. XQuery has nice built-in functionality for counting, filtering, reordering, doing math, etc., on items. Like ZODB, you can store and retrieve items of arbitrary complexity without too much fuss. A Sedna database can have multiple XML documents and multiple 'collections' of (similar) documents that can be queried together or separately. The Sedna team just released version 3.0 of the Sedna server, which has improved speed and reliability. 3.0 now runs on Mac OSX, in addition to x86 Linux and Win2K/XP. zif.sedna with sednaobject version 0.10 is alpha, so interfaces can and probably will change. The included doctests all pass using a Sedna 2.x server. I have not included the new features of 3.0 (e.g., faster, read-only queries) yet. Testing with a 3.0 server results in a single harmless failure. Speed? I'm getting 60-70 single-query transactions per second through Pylons on a 2Ghz Opteron. Transaction speed of course depends on how many queries are in the transaction and what the queries do. zif.sedna: http://pypi.python.org/pypi/zif.sedna/ Sedna: http://modis.ispras.ru/sedna/ I am currently the sole developer for zif.sedna. Feedback is welcome. - Jim Washington

1 0

Re: [lxml-dev] etree.parse hangs with a lot of parallel requests
by Stefan Behnel 09 Apr '08

09 Apr '08

>> It seems you only want to parse DTDs locally from disc, so setting >> "no_network=True" (which is the default in lxml 2.0) should prevent any >> accidental remote access. > > Eventually it turned out that I'm working fine without DTD. So, > setting no_network = True and load_dtd = False really solved the problem. Hmm, do you really need to turn off DTD loading or is disabling network access enough? I wouldn't expect loading the DTD from the disk cache to take that much time (although, if you can live without it and time is really critical, then it's obviously better to safe that bit of time also). > The parsing time is almost insignificant now. Shameless plug: http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/ > If you're interested, here it goes: > http://beta.rambler.ru/srch?query=python+lxml&searchtype=web > This is a search engine frontend, xml\xslt based, everything is > carried via lxml. Nice. That's a meta search engine, right? Will that site stay online? I.e. does it make sense to set a link from our "who uses lxml" FAQ entry? http://codespeak.net/lxml/FAQ.html#who-uses-lxml Stefan

1 0

[lxml-dev] [lxml] adding __float__, __int__ etc. to objectify.StringElement
by Holger Joukl 09 Apr '08

09 Apr '08

Hi, I suggest adding __float__, __int__ etc. methods to lxml.objectify StringElement, to enable things like >>> float(objectify.DataElement("234")) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: float() argument must be a string or a number >>> These will just try to invoke the very same operation on the underlying pyval. Maybe there are other classes where such methods would be helpful (BoolElement comes to mind, but I'll have to look). Any objections? I can add this plus some tests otherwise. Holger -- Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger

3 5

[lxml-dev] etree.parse hangs with a lot of parallel requests
by Dmitri Fedoruk 09 Apr '08

09 Apr '08

Hi all, I'm using lxml-2.0_1 now (I have not upgraded since to most recent versions as I have not noticed any features relevant to me), libxml2-2.6.30 , libxslt-1.1.22, FreeBSD 6.2 and 7.0 , the application runs within mod_python / apache 2.2.8 . My situation is pretty straightforward: fetch xml as plain text via http, parse it and get etree object, than apply xslt and get resulting html. The code is the following: self.xmlParser = etree.XMLParser(no_network = False, resolve_entities = False, load_dtd = True ) I use load_dtd=True as sometimes I encounter html entities in my input data. They are included in my dtd in this way: <!ENTITY % HTMLlat1 SYSTEM "xhtml-lat1.ent"> %HTMLlat1; <!ENTITY % HTMLsymbol SYSTEM "xhtml-symbol.ent"> %HTMLsymbol; <!ENTITY % HTMLspecial SYSTEM "xhtml-special.ent"> %HTMLspecial; Then eventually it comes up to ... xmlres = etree.parse( StringIO.StringIO( reply['data'] ), self.xmlParser ) And here I have serious problems. Parsing time is usually up to 100 ms (even this is critical time for me). But sometimes I have 3, 5 and even 60 seconds (!) of parsing. This situation happens under a heavy load (~20 simultaneous parsings/transformations per sec). So, I have several questions: 1) What am I doing wrong? 2) Is there any way to limit the runtime of the etree.parse? Is there any way to kill a thread maybe? I can not afford to wait even 150 ms, to say nothing about 1 second and more. Any help would be appreciated! Cheers, Dmitri

3 2

[lxml-dev] ElementTree.find does not accept QName objects.
by John Krukoff 06 Apr '08

06 Apr '08

Since I was the one that complained about the find method on Elements not accepting QNames, it's probably not surprising that I expected them to work with the ElementTree find method as well. Instead an unsliceable error is thrown, due to the value being expected to be a string: >>> from lxml import etree >>> x = etree.XML( '<a><b/></a>' ) >>> e = etree.ElementTree( x ) >>> e.find( 'b' ) <Element b at b7f4d914> >>> e.find( etree.QName( 'b' ) ) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "etree.pyx", line 508, in etree._ElementTree.find TypeError: 'etree.QName' object is unsliceable Though using e.getroot( ).find is working fine as a workaround for me. -- John Krukoff <jkrukoff(a)ltgc.com> Land Title Guarantee Company

2 4