Mailman 3 May 2014 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel April 10, 2023

April 10, 2023

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon April 9, 2023

April 9, 2023

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede Jan. 30, 2023

Jan. 30, 2023

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel Jan. 11, 2023

Jan. 11, 2023

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva Aug. 30, 2022

Aug. 30, 2022

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

generating html with incremental writer
by Burak Arslan Sept. 25, 2014

Sept. 25, 2014

Hi, Is there a way to pass method='html' to etree.xmlfile()? Or any other way to serialize incrementally to html? Best, Burak

2 9

Re: [lxml] Lxml aborts with an odd error message
by Stefan Behnel May 28, 2014

May 28, 2014

Martin Mueller, 26.05.2014 16:18: > On 5/26/14, 8:02, "Stefan Behnel" wrote: >> Martin Mueller, 25.05.2014 22:49: >>> Using the option "huge_tree=True" on the parser works, but has >>> performance issues that grow progressively worse. >>> >>> The default setting for parser seems to be about 1.2 million ID. Up to >>> that point it processes a play with ~20,000 words and associated IDs at >>> the rate of 1 a second. By the time the program has worked through its >>> 9.7 >>> million words, the checking of IDs takes 8 seconds per play: performance >>> degrades by almost an order of magnitude. If progress were linear, the >>> program should take about 500 seconds. In fact, it took 2400 seconds. >> >> Yes, I noticed that, too. I'm currently coding up a new parser option, >> working title "collect_ids=False", that would allow you to disable the ID >> hash table building. When I do that, performance jumps from ~52 seconds >> per >> million IDs to 2 seconds on my side for an extreme test case. >> >> However, it's not ready for a release yet. I get test failures in other >> areas with this change, so it needs a bit more work. I can publish a 3.4 >> alpha when it's ready, might take a couple of days, though. > > That will be a great feature! I look forward to it, and I'll be happy > to help test it. Here's an implementation: https://github.com/lxml/lxml/commit/35316b052af48921657813bb68563fe4a301d1b8 I attached a little test program that I used for benchmarking. It stress tests the XML ID handling by parsing lots of elements with different IDs and discarding them right after parsing. The new implementation performs 5x better with the normal parser and about 50x better with the new collect_ids=False option. Given how rare the usage of the XML ID hash table should be in real code, this makes me wonder if the option should not be switched off by default, however backwards incompatible that is. Can you test it from the latest github version? BTW, the lxml homepage has a Paypal link to allow for sponsorship of lxml's development, just in case this wasn't generally known. :) Stefan

1 0

Lxml aborts with an odd error message
by Martin Mueller May 26, 2014

May 26, 2014

I have run into an odd problem with the current version of lxml running on Python3.4 on a six-year old Mac Pro laptop with 8GB of memory. I want to loop through ~500 TEI encoded plays, where each word token has an xml:id, like this: <w lemma="act" n="1-b-0140" ana="#vvn" reg="acted" xml:id="A07064-000200">acted</w> where the ID is composed from a text id (A07064) and a wordcounter The basic program goes like plays = os.walk(sourcePlayDirectory) for directory in plays: for item in directory[2]: filename = directory[0] + '/' + item tree = etree.parse(filename, parser) for element in tree.iter(tei + 'w'): {some conditions} This works, but after not quite a minute--about 50 plays and probably about looking at a million <w> elements, the program stops and produces an error message like this: tree = etree.parse(filename, parser) File "lxml.etree.pyx", line 3239, in lxml.etree.parse (src/lxml/lxml.etree.c:69970) File "parser.pxi", line 1749, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:102081) File "parser.pxi", line 1775, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:102345) File "parser.pxi", line 1679, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:101380) File "parser.pxi", line 1110, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:96832) File "parser.pxi", line 582, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:91290) File "parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:92476) File "parser.pxi", line 622, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:91772) lxml.etree.XMLSyntaxError: ID A07897-308750 already defined, line 59347, column 40 Now there is nothing wrong with ID A07897-308750. It hasn't been used previously, and if you start the program again with, say, the play previous to the one that raised the exception, it will sail right through text A07897 and continue for not quite a minute but produce the same error message with a different ID. So it's not the ID. I don't know what is happening, but I suspect that something inside lxml or python hits a limit, and when that limit is hit, you get the syntax error message with the ID at the point at which the program "had enough" whatever it had enough of.. The Activity monitor on my Mac shows nothing extraordinary: Python memory use at the point of failure is about 150MB, but I have run similar programs on an older Mac with an earlier version of lxml without any trouble. Martin Mueller Professor emeritus of English and Classics Northwestern University

3 7

Re: [lxml] Lxml aborts with an odd error message
by Martin Mueller May 25, 2014

May 25, 2014

Ps to my earlier email. Is there a way in which I can make lxml "forget" everything about a file as soon as it is done with it. I have a dim sense that iterparse has something to do with that, but I don't know how to tell the program, " when you're done with file A, wipe out everything in memory before moving on to file B." Some command of that kind should do the trick, because the program has no trouble processing individually any of the files it complains about when they are encountered in the aggregate. Martin Mueller Professor emeritus of English and Classics Northwestern University Yes, I am sure that the IDs are unique across the corpus. I don't think that the same file is encountered twice. If that were the case, I would expect the error to occur at the first ID of the file that is encountered twice. But it happens somewhere in the middle, so we would posit that the program happily encounters some number of IDs twice, but suddenly complains. On 5/24/14, 19:59, "Ivan Pozdeev" <vano(a)mail.mipt.ru> wrote: >1) You run everything through one parser, so it appears to accumulate IDs >it has seen so far. >Are you sure your IDs are unique across all files? > >2) Are there any chances it encounters the same file twice >(hardlinks/symlinks/restarts from a dirty state)? > > >> I have run into an odd problem with the current version of lxml running >>on >> Python3.4 on a six-year old Mac Pro laptop with 8GB of memory. > >> I want to loop through ~500 TEI encoded plays, where each word token has >> an xml:id, like this: > >> <w lemma="act" n="1-b-0140" ana="#vvn" reg="acted" >> xml:id="A07064-000200">acted</w> > >> where the ID is composed from a text id (A07064) and a wordcounter > > >> The basic program goes like > >> plays = os.walk(sourcePlayDirectory) > >> for directory in plays: >> for item in directory[2]: >> filename = directory[0] + '/' + item > >> tree = etree.parse(filename, parser) >> for element in tree.iter(tei + 'w'): >> {some conditions} > >> This works, but after not quite a minute--about 50 plays and probably >> about looking at a million <w> elements, the program stops and produces >>an >> error message like this: > >> tree = etree.parse(filename, parser) >> File "lxml.etree.pyx", line 3239, in lxml.etree.parse >> (src/lxml/lxml.etree.c:69970) >> File "parser.pxi", line 1749, in lxml.etree._parseDocument >> (src/lxml/lxml.etree.c:102081) >> File "parser.pxi", line 1775, in lxml.etree._parseDocumentFromURL >> (src/lxml/lxml.etree.c:102345) >> File "parser.pxi", line 1679, in lxml.etree._parseDocFromFile >> (src/lxml/lxml.etree.c:101380) >> File "parser.pxi", line 1110, in >> lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:96832) >> File "parser.pxi", line 582, in >> lxml.etree._ParserContext._handleParseResultDoc >> (src/lxml/lxml.etree.c:91290) >> File "parser.pxi", line 683, in lxml.etree._handleParseResult >> (src/lxml/lxml.etree.c:92476) >> File "parser.pxi", line 622, in lxml.etree._raiseParseError >> (src/lxml/lxml.etree.c:91772) >> lxml.etree.XMLSyntaxError: ID A07897-308750 already defined, line 59347, >> column 40 > >> Now there is nothing wrong with ID A07897-308750. It hasn't been used >> previously, and if you start the program again with, say, the play >> previous to the one that raised the exception, it will sail right >>through >> text A07897 and continue for not quite a minute but produce the same >> error message with a different ID. > >> So it's not the ID. I don't know what is happening, but I suspect that >> something inside lxml or python hits a limit, and when that limit is >>hit, >> you get the syntax error message with the ID at the point at which the >> program "had enough" whatever it had enough of.. > >> The Activity monitor on my Mac shows nothing extraordinary: Python >>memory >> use at the point of failure is about 150MB, but I have run similar >> programs on an older Mac with an earlier version of lxml without any >> trouble. > > > > >> Martin Mueller >> Professor emeritus of English and Classics >> Northwestern University > > >> _________________________________________________________________ >> Mailing list for the lxml Python XML toolkit - http://lxml.de/ >> lxml(a)lxml.de >> https://mailman-mail5.webfaction.com/listinfo/lxml > > > >-- >Best regards, > Ivan mailto:vano@mail.mipt.ru > >_________________________________________________________________ >Mailing list for the lxml Python XML toolkit - http://lxml.de/ >lxml(a)lxml.de >https://mailman-mail5.webfaction.com/listinfo/lxml

3 3

Re: [lxml] Load entity definitions from DTD
by Frederik Elwert May 22, 2014

May 22, 2014

1 0