Mailman 3 September 2012 - lxml - The Python XML Toolkit

lxml and Unicode
by Martin Mueller Sept. 14, 2012

Sept. 14, 2012

I ma new to lxml and a little wobbly on Unicode in Python 2.7. Here is my problem: 1. I take an SGML text with character entities and convert them to UTF-8 characters 2. I extract note elements from the text with the converted characters. This works if I use two separate Python scripts. The first goes like this: # This Python file uses the following encoding: utf-8 import entityrules filein = open('/users/martin/dropbox/eebo-SGMLfiles/200504/A10675.sgm', 'r') fileout = open('/users/martin/dropbox/lxml/A10675.xml', 'w') text = filein.read() text = entityrules.replaceCharacterEntities(text) print >> fileout, text It takes the SGML file as an input. applies the "replaceCharacterEntities" and writes that output to a new file. The second script goes like this: #encoding = 'utf-8' from lxml import etree filein ='/users/martin/dropbox/lxml/a10675.xml' fileout = open('/users/martin/dropbox/lxml/notesA10675.xml', 'w') doc = etree.parse(filein) for note in doc.getiterator('note'): print >>fileout, etree.tostring(note, with_tail = False, encoding = 'utf-8') Its input is the file generated by the first script. It converts the input file into an element tree and then iterates through the tree, extracting a particular element . Now it should be possible to combine those two scripts in a single script as follows: # This Python file uses the following encoding: utf-8 import entityrules from lxml import etree #converting the SGML character entities to UTF-8 characcters filein = open('/users/martin/dropbox/eebo-SGMLfiles/200504/A10675.sgm', 'r') fileout = open('/users/martin/dropbox/lxml/notesA10675.xml', 'w') text = filein.read() text = entityrules.replaceCharacterEntities(text) print "I finished the first part of this script" #creating an element tree and extracting instances of a particular element doc = etree.parse(text) for note in doc.getiterator('note'): print >>fileout, etree.tostring(note, with_tail = False, encoding = 'utf-8') This script produces the following output: Martin-Muellers-Mac-Pro:lxml martin$ python getNotes.py I finished the first part of this script Traceback (most recent call last): File "getNotes.py", line 14, in <module> doc = etree.parse(text) File "lxml.etree.pyx", line 2954, in lxml.etree.parse (src/lxml/lxml.etree.c:56220) File "parser.pxi", line 1533, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:82303) File "parser.pxi", line 1562, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:82596) File "parser.pxi", line 1462, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:81635) File "parser.pxi", line 1002, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:78544) File "parser.pxi", line 569, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:74488) File "parser.pxi", line 650, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:75379) File "parser.pxi", line 584, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:74650) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1126: ordinal not in range(128) The script does the first part of the job correctly, as is apparent from the first line. And the "text" variable that is passed to the etree function should in principle be identical with the content of the file that is passed to my script from a file. But it isn't, and somewhere along the road Python seems to have forgotten that we 're dealing with Unicode texts. What am I doing wrong? Or is there a bug? I am using the latest Enthought version of Python, which comes bundled with lxml Martin Mueller Professor of English and Classics Northwestern University

2 1

Re: [lxml] Building lxml with mac os x mountain lion fails.
by Mark Grandi Sept. 12, 2012

Sept. 12, 2012

My mistake, I should of clarified that this was for python 3. Easy_install works great but it doesn't run under python3... From: Brett Sent: 9/12/2012 12:11 AM To: lxml(a)lxml.de Subject: Re: [lxml] Building lxml with mac os x mountain lion fails. mark grandi <markgrandi <at> gmail.com> writes: > > Well, I seem to of figured it out. It seems that libxml2 by default tries to > include lzma, but since I recently installed the xz libraries, its probably > having some conflict with the static vs not static version of liblzma, so i > edited the buildlibxml.py script to pass in a '--without-lzma' option, and > that worked. My entire compile process is listed below. The setup script > REALLY needs to be updated to work with recent versions of mac os x... > > Corvidae:lxml-2.3.5 markgrandi$ echo $CFLAGS > -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform > /Developer/SDKs/MacOSX10.8.sdk/ > > Corvidae:lxml-2.3.5 markgrandi$ echo $LDFLAGS > -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform > /Developer/SDKs/MacOSX10.8.sdk/usr/lib > ...snip... > add '--without-lzma' to that list, so it becomes > > # build libxml2 > libxml2_configure_cmd = configure_cmd + [ > '--without-python', > '--with-iconv=%s' % prefix, > '--without-lzma'] > ...snip... I tried the following and it seems to work fine: lilliput:lxml-2.3.5 brett$ STATIC_DEPS=true sudo easy_install "lxml==2.3.5" lilliput:lxml-2.3.5 brett$ sudo python setup.py install lilliput:lxml-2.3.5 brett$ python >>> from lxml import etree >>> x = etree.fromstring('''<note> <to> Tove </to> <from> Jani </from> <heading> Reminder </heading> <body> Don't forget me this weekend! </body> </note>'''.encode("utf-8")) >>> x <Element note at 0x103b0cfa0> >>> etree.tostring(x) "<note>\n <to>\n Tove\n </to>\n <from>\n Jani\n </from>\n <heading>\n Reminder\n </heading>\n <body>\n Don't forget me this weekend!\n </body>\n </note>" _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

3 2

Building lxml with mac os x mountain lion fails.
by Mark Grandi Sept. 12, 2012

Sept. 12, 2012

Hello, I'm trying to build lxml on mac os x 10.8 and the build fails. I've also tried to change the compiler from gcc to clang, and it still fails. Output: http://paste.ubuntu.com/1172867/ (this is the output when CC=clang) any ideas? =/ ~mark

4 5

Strange memory leakage problem on Ubuntu
by zzjjzzgggg Sept. 12, 2012

Sept. 12, 2012

Hello, I am new to lxml. I installed lxml on Ubuntu 12.04 via easy_install as suggested easy_install --allow-hosts=lxml.de,*.python.org lxml The version of libxml2 is Version: 2.7.8.dfsg-5.1ubuntu4.1 I use python 3.2. When I use lxml to process some HTML fragments, I find serious memory leakage issues. The test code and test data are attached. When I remove this line ps=html.xpath("//p[@node-type='feed_list_content']") , There isn't memory leakage problem. What's the problem please? Many thanks. I can provide more info if you need. Best regards zzjjzzgggg

1 0

Strange memory leakage problem on Ubuntu
by zzjjzzgggg Sept. 12, 2012

Sept. 12, 2012

Hello, I am new to lxml. I installed lxml on Ubuntu 12.04 via easy_install as suggested easy_install --allow-hosts=lxml.de,*.python.org lxml The version of libxml2 is Version: 2.7.8.dfsg-5.1ubuntu4.1 I use python 3.2. When I use lxml to process some HTML fragments, I find serious memory leakage issues. The test code and test data are attached. When I remove this line ps=html.xpath("//p[@node-type='feed_list_content']") , There isn't memory leakage problem. What's the problem please? Many thanks. I can provide more info if you need. Best regards zzjjzzgggg

1 0

the lxml.etree bug
by descusr Sept. 8, 2012

Sept. 8, 2012

Hello: I try to parse the page: http://www.liqucn.com. My py-code: #coding=utf-8 from lxml import etree,tostring import requests resp = requests.get('http://www.liqucn.com') html = resp.text page = etree.HTML(html) print tostring(page) #------------------------------------- I found many missing html-code where affter '<span class="app_ico"><a href="http://www.liqucn.com/yx/17773.shtml" target="_blank" lz_src="http://images.liqucn.com/mini/60x60/h005/h79/img201208220736340_60x60.png " '.I am doubt this result very confused.You can you open this url, see the source code, and then run my code.Thank you!

1 0

Sphinx Documentation
by Peter Hoffmann Sept. 2, 2012

Sept. 2, 2012

Hi all, I have started to port[0] the documentation to sphinx. As I have read[1] on the mailling list that's a task where help would be appreciated. Because the documentation is already in rst format, an initial build with sphinx was easy and the result[2] looks quite promising. The next steps will be: * remove markup specific to the current build system and replace it with sphinx idioms * fix pdf generation * use sphinx for api documentation (with sphinx autodoc where possible) * theme/layout If you have additional ideas or things I should pay attention to please let me know. Peter [0]: https://github.com/hoffmann/lxml [1]: http://thread.gmane.org/gmane.comp.python.lxml.devel/5821/focus=5822 [2]: http://vps.peter-hoffmann.com/lxml-sphinx/

2 1