[lxml-dev] HTML character code interpretation
I am using lxml to process some xhtml files. The files have html character codes embedded in them. For instance: & #39;s rather than a '. When I parse the files, edit them, and then write them back out, I want my edits to be the only changes in the output files, but lxml is replacing the character codes with the actual characters they are supposed to represent as well. So if I have: It& #39;s an example. <-- Space inserted to help readability. It is writing out: It's an example. I've tried setting resolve_entities to false, ala: tree = etree.parse(input, etree.XMLParser(resolve_entities=False)) But this seems to have no effect. There a way to tell lxml to ignore these/leave them as is? Thanks. -s
Hi, Spencer Crissman wrote:
I am using lxml to process some xhtml files. The files have html character codes embedded in them. For instance: & #39;s rather than a '. When I parse the files, edit them, and then write them back out, I want my edits to be the only changes in the output files, but lxml is replacing the character codes with the actual characters they are supposed to represent as well.
Just to clarify, does that mean you want the character references and entities exactly as they were before, or do you want certain characters escaped, or ...
So if I have: It& #39;s an example. <-- Space inserted to help readability.
It is writing out: It's an example.
I've tried setting resolve_entities to false, ala: tree = etree.parse(input, etree.XMLParser(resolve_entities=False))
The best place to find that out is the libxml2 parser source code, but IIRC this option only relates to entities defined in DTDs, not to character references.
There a way to tell lxml to ignore these/leave them as is?
No. Maybe you could give some more background on what you try to achieve. There may still be ways to do what you want. Stefan
Sorry, I tried to be as concise as possible, here is some more info as to what I am trying to get. We use the files as templates for our website. They have some markers in them to insert data that get processed when the files get served up. I need to add some new tags to a number of these pages, enough of them that I don't wish to do so by hand. I was hoping to use lxml to read in the files, add the markers, and write out the files. The output would need to be exactly like the input except for the tags and/or attributes that I specifically add to the element tree. Most of this works, except for a few things: 1) The whitespace gets mangled a bit. I lose some newlines, and a couple get added. This doesn't matter much, and I could live with it. 2) lxml is adding a meta attribute to the output's header section. This also doesn't matter so much. 3) All the HTML character codes are getting replaced by the actual characters. 4) We have some custom tags that are self closing, and when they get written out, they are getting written as open and close tag pairs rather than a self-closing element. So if anyone has a suggestion on how to address issues 3 and/or 4, either with lxml or a different parser/x(ht)ml lib, I would appreciate any pointers. I was hoping to avoid a full blown parser, but if I can't get a little closer with something pre-built, I may have to go that way. Thanks for the response, Spencer On Tue, Jul 29, 2008 at 1:52 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Hi,
Spencer Crissman wrote:
I am using lxml to process some xhtml files. The files have html character codes embedded in them. For instance: & #39;s rather than a '. When I parse the files, edit them, and then write them back out, I want my edits to be the only changes in the output files, but lxml is replacing the character codes with the actual characters they are supposed to represent as well.
Just to clarify, does that mean you want the character references and entities exactly as they were before, or do you want certain characters escaped, or ...
So if I have: It& #39;s an example. <-- Space inserted to help readability.
It is writing out: It's an example.
I've tried setting resolve_entities to false, ala: tree = etree.parse(input, etree.XMLParser(resolve_entities=False))
The best place to find that out is the libxml2 parser source code, but IIRC this option only relates to entities defined in DTDs, not to character references.
There a way to tell lxml to ignore these/leave them as is?
No. Maybe you could give some more background on what you try to achieve. There may still be ways to do what you want.
Stefan
Hi, Spencer Crissman wrote:
We use the files as templates for our website. They have some markers in them to insert data that get processed when the files get served up. I need to add some new tags to a number of these pages, enough of them that I don't wish to do so by hand. I was hoping to use lxml to read in the files, add the markers, and write out the files. The output would need to be exactly like the input except for the tags and/or attributes that I specifically add to the element tree.
Here I have to assume that your templates are HTML based (as opposed to XML or XHTML) and that you use the HTML parser to parse them. The HTML parser has a certain knowledge about the HTML structure, and can therefore work different from a normal XML parser. Also, the HTML parser parses with error recovery enabled by default (recover=True), so it will try to fix up the structure if it finds any problems.
Most of this works, except for a few things: 1) The whitespace gets mangled a bit. I lose some newlines, and a couple get added. This doesn't matter much, and I could live with it.
Could you provide the code that you use for parsing and serialising?
2) lxml is adding a meta attribute to the output's header section. This also doesn't matter so much.
AFAIR, the HTML serialiser does that, not the parser. It's easy to remove from the serialised string. We do that somewhere near the end (?) of lxml/html/__init__.py.
3) All the HTML character codes are getting replaced by the actual characters.
I don't think there is a way to prevent that. I'm having a hard time to understand why the character references are so important to you? The example you showed was escaping a plain ASCII character, no browser on earth should have a problem with that. If it's a fixed set of character references that must be escaped (for whatever reason), you can serialise the tree into a unicode string (encoding=unicode), replace them with the respective charref sequence and then encode the string into the target encoding.
4) We have some custom tags that are self closing, and when they get written out, they are getting written as open and close tag pairs rather than a self-closing element.
Hmm, yes, that's a problem. You can't extend the serialiser with additional knowledge about special tags. The only special serialiser that libxml2 supports is the one for HTML. But since the self-closing tags are necessarily empty, you can always treat the serialised string and replace all "></specialtag>" substrings by "/>" or ">" before you write it back out. Stefan
participants (2)
-
Spencer Crissman -
Stefan Behnel