Mailman 3 September 2010 - lxml - The Python XML Toolkit

[lxml-dev] Attribute Whitelist in lxml.html.clean
by Richard Zurad 11 Oct '10

11 Oct '10

Greetings, I'm currently working on a project that requires an attribute whitelist for the clean_html method in lxml.html.clean (more specifically, the __call__ method of the Cleaner object). I did some digging and found a post to this mailing list from about two years ago that asked the same thing: http://article.gmane.org/gmane.comp.python.lxml.devel/3875/ While I agree that it would be nice to link an attribute whitelist to a list of attributes that are only valid for a given tag, a solution like that seems a bit overkill for what my project requires as we are only cleaning subtrees of the HTML DOM where we know exactly what we're expecting and can narrow things down to a very small whitelist of tags and attributes. Also, to only restrict attributes to standards-compliant attributes per tag puts a bit of a damper on some web development trends where developers define custom attributes for the sake of their own requirements (adding an in-line editor, etc) I've implemented a simplistic fix that adds the ability to whitelist attributes. The addition is a simple five lines of code to the __call__ method of the Cleaner object in lxml.html.clean. It simply checks if the keyword argument whitelist_args (a list of valid args for any tag) was passed in to __init__. if it was, then on line 257 of clean.py, instead of using defs.safe_attrs, we use the supplied whitelist (although if both safe_attrs_only and whitelist_args is passed to the constructor, throw a ValueError, as it doesn't make sense to supply both a whitelist and to specify to use the feedparser attribute whitelist) I'd like to submit a patch to lxml with this code, however there is a design decision that must be made. In the __call__ method where we do javascript sanitization if the contructor was called with javascript=True (or the default behavior), there is code that simply skips javascript sanitization on attributes if the safe_attrs_only is True since the feedparser whitelist does not include event attributes. This leads to a question of what to do if the object is instantiated with, for example, an attribute whitelist that includes 'onchange'? Would it be safe to assume that if an event attribute is passed in as part of the whitelist, we want to allow javascript on that attribute and that attribute only? Or should we raise an error because it doesn't make sense to pass in an event attribute in the whitelist along with javascript=True? Or should we just silently delete the attribute, ignoring the whitelist? For the project I'm working on, this situation won't arise. However, since I'd like to submit a patch to lxml for potential inclusion of this feature in a future release, I'd like to make sure that the concept behind it is in line with the rest of the lxml mantra. Thanks, -Rich -- http://www.greyboxware.com

2 1

Re: [lxml-dev] lxml-dev Digest, Vol 72, Issue 4
by Dimitrios Pritsos 05 Oct '10

05 Oct '10

On 20/09/10 14:46, lxml-dev-request(a)codespeak.net wrote: > Send lxml-dev mailing list submissions to > lxml-dev(a)codespeak.net > > To subscribe or unsubscribe via the World Wide Web, visit > http://codespeak.net/mailman/listinfo/lxml-dev > or, via email, send a message with subject or body 'help' to > lxml-dev-request(a)codespeak.net > > You can reach the person managing the list at > lxml-dev-owner(a)codespeak.net > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of lxml-dev digest..." > > > Today's Topics: > > 1. Re: lxml 2.3 beta 1 released (Pascal) > 2. Finding the media-type and method of an XSLT (Laurence Rowe) > 3. lxml can't work with html5lib (flya flya) > 4. Re: Access to ElementTree for XML schema (Dave Kuhlman) > 5. lxml.html.submit_form and unicode values (Eugene Van den Bulke) > 6. Re: lxml.html.submit_form and unicode values (Vojt?ch Rylko) > 7. Re: lxml.html.submit_form and unicode values > (Eugene Van den Bulke) > 8. Re: lxml.html.submit_form and unicode values > (Eugene Van den Bulke) > 9. Problem with utf-8 and etree (Vetoshkin Nikita) > > 10. Concurrent Programming and lxml but recipe for Newbies like > me - any solution? (Dimitrios Pritsos) > Ok I have a correction to make in 10. I posted earlier: In fact the Threading Setup is working Fine STILL the MultiProcessing Set up is NOT. One problem that occured is that Queue.Queue() with in a Manager() object has some kind of problem to trasfere Trees, While Queue.Queue() works fine with Threads. Any Idea how the MultiProcessing setup could work ?

2 1

[lxml-dev] Performance gets bad when parsing xml with namespaces
by Max Kisselew 01 Oct '10

01 Oct '10

Hello, recently I discovered a problem with lxml/LibXML2. I guess it's likely that the problem comes from Libxml2. I hope despite that you can help me. I'm working on a university project where I use Python and lxml for xml parsing and processing. First there were no namespace definitions in the xml files we used but recently the format has slightly changed and some namespace definitions were added. Here the xml format as it was in the beginning: <?xml version="1.0" encoding="UTF-8"?> <D-Spin> <MetaData> <source>IMS, Uni Stuttgart</source> </MetaData> <TextCorpus lang="de"> <text>European Medicines Agency EMEA/H/C/471 [...] Wegen</text> <tokens> <token ID="t1">European</token> <token ID="t2">Medicines</token> [...] <token ID="t145906">Wegen</token> </tokens> <sentences> [...] <sentence ID="s5921" tokenIDs="t145906"/> </sentences> </TextCorpus> </D-Spin> And here the xml with the recently added namespace definitions: <?xml version="1.0" encoding="UTF-8"?> <D-Spin xmlns="http://www.dspin.de/data" version="0.4"> <MetaData xmlns="http://www.dspin.de/data/metadata"> <source>IMS, Uni Stuttgart</source> </MetaData> <TextCorpus xmlns="http://www.dspin.de/data/textcorpus" lang="de"> <text>European Medicines Agency EMEA/H/C/471 [...] Wegen</text> <tokens> <token ID="t1">European</token> <token ID="t2">Medicines</token> [...] <token ID="t145906">Wegen</token> </tokens> <sentences> [...] <sentence ID="s5921" tokenIDs="t145906"/> </sentences> </TextCorpus> </D-Spin> I wanted to extract all the content from the <token> elements. In the xml file without the namespace definitions that takes just a moment (less that 30 seconds). But when I tried to perform the same on the new file with namespaces, it took much longer, more that 30 minutes (!). The xml file was about 7 MB. Since the same problem occurs when one tries to parse the xml file with the LibXML2 binding for Perl, I guess the problem comes from LibXML2 itself. It is also strange that the performance problem seems to grow with the amount of the <token> tags to be parsed. So the first 10 000 tags only need about a second. But when we parse the first 20 000 tags, it takes 21 seconds! Do you have any idea about the cause of this problem and how it could be solved? Thanks Max

2 2

[lxml-dev] Fwd: Problem with utf-8 and etree
by Vetoshkin Nikita 30 Sep '10

30 Sep '10

Thanks for your reply, Stefan. You're right, I'm using Python2.6. Doesn't lxml asumes UTF-8 encoding or is there any way to tell about encoding? I'm solving the problem: Firebird DBMS via SQLAlchemy returns Unicode string, then I convert them to bytestrings in UTF-8 for csv module, after that I want to send that string as a text element and convert it back to Unicode, and that serialize to UTF-8 and actually send. That no good. I digged a bit and found that lxml is checking if characters fit UTF-8, but I couldn't find the wrong one in my string. 2010/9/30 Stefan Behnel <stefan_ml(a)behnel.de>: > Vetoshkin Nikita, 17.09.2010 08:48: >> >> Not sure if it's a bug, so here I am. >> Test case: >> >> from lxml import etree >> e = etree.Element("Test") >> l = open("1.txt").read() >> e.text = l #<-- ValueError: All strings must be XML compatible... > > I assume you are using Python 2? In that case, you get back a byte string > when reading the file. Since lxml.etree can't know which encoding the byte > sequence in that string uses, it rejects the string. You must decode it to > Unicode first. > > Stefan >

2 1

[lxml-dev] PCDATA question
by Mag Gam 30 Sep '10

30 Sep '10

I am trying to parse a XML file I keep getting this error: lxml.etree.XMLSyntaxError: PCDATA invalid Char value 27, line 29053, column 2111 By looking at this character, it does look wierd. There is an escape sequence of some sort in my file. Is there a way to include this or convert this bad character into something useful?

2 5

[lxml-dev] Problem with utf-8 and etree
by Vetoshkin Nikita 30 Sep '10

30 Sep '10

Not sure if it's a bug, so here I am. Test case: from lxml import etree e = etree.Element("Test") l = open("1.txt").read() e.text = l # <-- ValueError: All strings must be XML compatible... etree.fromstring("""<?xml version="1.0" encoding="utf-8" standalone="no" ?>\r\n<Test>""" + l + "</Test>") # <-- works 1.txt file is attached. My setup: lxml.etree: (2, 3, -99, 0) libxml used: (2, 7, 7) libxml compiled: (2, 7, 7) libxslt used: (1, 1, 26) libxslt compiled: (1, 1, 26)

2 1

[lxml-dev] Error with etree.XML(mystr) what i cant identify
by Ovnicraft 29 Sep '10

29 Sep '10

Hi folks, trying with etree.XML(mystr) i get this error http://openerp.pastebin.com/rPF9fKYZ my str is http://openerp.pastebin.com/CbU4y04Y, i cant identify the problem. Waiting for your help. Regards, -- Cristian Salamea @ovnicraft

2 3

[lxml-dev] exploring a relaxng schema
by Tim Arnold 29 Sep '10

29 Sep '10

hi, I'd like to generate a list of legal child elements given a parent element, using the DocBook 5 relaxng schema. Is it possible to do this with lxml? something like this is what I'd like to eventually get/write: somefunc('varname','inlineequation')# returns False somefunc('varname','inlinemediaobject')# returns True thanks, --Tim

2 2

[lxml-dev] automatic windows eggs for lxml
by Adam GROSZER 27 Sep '10

27 Sep '10

Hello, With the help of the Zope Foundation I build a bot that does windows binary eggs of some zope packages. Using that infrastructure it should be rather easy to do all sorts of windows binary eggs for lxml too. Are you interested in that? -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: If you realize that you aren't as wise today as you thought you were yesterday, you're wiser today. - Michigan Presbyterian Church

3 6

[lxml-dev] lxml-2.2.8-py2.5-win32.egg is missing from pypi
by Adam GROSZER 23 Sep '10

23 Sep '10

Hello, lxml-2.2.8-py2.5-win32.egg is missing from pypi. lxml 2.2.8 is a dependency for the upcoming ZTK 1.0 release. Would someone please do the egg? -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: The truest test of independent judgment is being able to dislike someone who admires us, and to admire someone who dislikes us. - Sydney J. Harris

1 0