[David Priest]
I'm feeling that I've perhaps pissed in someone's cornflakes, but I'm going to respond anyways. If I've offended, I apologize profusely: no offense was intended.
No offense perceived or taken. We're just debating the technical issues; nothing personal. Sorry if it seemed too blunt, but email tends to look that way. I don't have the time to be super-friendly or the inclination to sprinkle smilies throughout. When discussing issues via email, one needs a thick skin. Assume that the writer is smiling continuously, trying to help, which I was and am. [David Goodger]
XML character entities (&xNNNN;) are unknown to reStructuredText and are the wrong way to do it. Docutils is correct in substituting "&" in HTML output for every "&" in the input file (how could it tell the difference between *using* an XML character entity and just *talking* about one?).
The charent pattern can be detected easily enough, and the "&" encoding skipped for those entities. If you want to talk about an entity directly, literalizing it would do the trick. In all other cases the ampersand can be safely encoded.
By literalizing do you mean ``inline literals`` or literal blocks? That's not always acceptable. I might want to say The '&' entity is used by HTML and XML to represent the '&' character. I shouldn't have to use inline literals here. Docutils uses Unicode internally, and I don't see a need for it to grow a character entity subsystem. So far, you're the only one who has asked for one, and that's not convincing enough. I suspect that you may be asking Docutils to cover a deficiency in your toolset, or there's a misunderstanding. Please answer these questions from my last message to help clear this up: Why are you unable to insert the actual, encoded characters into the text? What *are* you able to insert? What encoding are your files using? What platform (OS, editor, etc.)? It could be that you *can* insert real characters but don't know it.
But the substitution table using "proper" characters is good enough, although I'm not entirely sure that all backends output will be able to deal with two-byte Unicode.
HTML can handle UTF-8. XML uses Unicode internally and assumes UTF-8 or UTF-16 unless told otherwise. As for back-ends, that's a Writer issue. If output format X can't handle Unicode, then the X format's Writer needs to encode those characters or signal an error. TeX can't handle the &#xNNNN; form. Here's an alternative for you. If you want to use &whatever; XML character entities in your source, just put a simple filter into your tool chain that converts those entities into UTF-8. Something like:: charents2utf8 input.txt | docutils/tools/html.py > output.html There must be filters like that "out there". If not, it wouldn't be hard to write one, I think. A codec would do just as well, but Python doesn't come with such a codec (more's the pity). I just realized the front-ends don't have support for explicit stdin/stdout with "-" arguments. I'll add that soon. [re: interpreted text roles]
There's absolutely no need for any regex substitution here. That's exactly what the Writers are for. The ":gui:`File`" input may become "<gui>File</gui>" in the internal document tree. The HTML Writer would write it with bold (or, better yet, with <span class="gui">, made bold by the stylesheet). The DocBook Writer would write it with <guilabel>.
The problem with implementing it as a Writer is that Writers don't travel with the source text files. If I send you a ReST file with :gui: roles in it, what's your DocUtils installation going to do with it?
The set of roles built into Docutils itself will grow. If the growth proves to be unlimited or unmanageable, there will have to be an alternative. If those roles are not handled by the default Docutils set, then they can be local to your installation. They wouldn't be portable, true; there's only so far a "standard" can go and we can't please everybody 100%. There's also this alternative:
There has been some discussion about parameterizing the interpreted text system somehow, to avoid proliferation of element types (gui, keypress, etc.). No decision or action yet.
See the Doc-SIG thread, "master plan for interpreted text?" from last month.
If "parameterizing the interpreted text system" means that simple role substitutions -- the kind that can be handled by regex -- can be placed within the source text files, great! It makes the source text more portable.
But I doubt it will take the form of "regex substitutions". That's just too low-level, IMHO.
If the role can't be handled by a regex, then of course it's going to require a Writer. (Although... if one could embed a Python script... but, no. That's verging on silly.)
Have you read PEP 256 & 258 yet? Please do. They explain the Docutils architecture and the purpose of the components (Writer, Reader, Parser, etc.).
And if this email has been scrunched into a blob, I apologize.
Try leaving a space on each blank line. I.e., [return] [space] [return]. I've done that just above. -- David Goodger http://starship.python.net/~goodger Programmer/sysadmin for hire: http://starship.python.net/~goodger/cv