Actually, I did install pip with Python3.3 in the site-packages directory of Python3.3, and it shows up as pip-1.2.1-py3.3.egg. But there doesn't seem to be a way of calling it. 'pip' will do the 2.7 routine, and pip3.3 or pip-3.3 are not recognized as commands. I checked the http://www.pip-installer.org/en/latest/usage.html#pip-install for any information about using different versions of pip with different versions of Python. But there is no such information. I installed pip by running 'python3.3. setup.py install' in the downloaded pip directory. That worked in the sense that it put various pip files in the appropriate site packages directory Martin Mueller Professor of English and Classics Northwestern University On 2/13/13 1:07 PM, "Stefan Behnel" <stefan_ml@behnel.de> wrote:

Martin Mueller, 13.02.2013 19:19:

...

I have been trying to install the latest version of lxml on OS Lion. I've had varying success. It worked for Python2.7, but didn't work for Python3.3.

I used the installation instructions given on the lxml website:

STATIC_DEPS=true sudo pip install lxml

I got an error message about a missing "llvm-gcc-4.2", but there was good advice on how to handle this at

http://waqasshabbir.tumblr.com/post/19073648382/llvm-gcc-4-2-exe-error-on -m ac-osx-lion-when-building

My goal was to install lxml in Python3.3, but the command with a specified target installed lxml in Python2.7. I learned how to use the target flag and how to install lxml in a particular version of Python with this command:

STATIC_DEPS=true sudo pip install lxml --target

/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-pack ag es

That's the wrong way round. You are asking a Py2.7 pip to install its package into the site-packages directory of you Py3.3 installation. Rather, you want to run pip with the Py3.3 "python" executable so that it can build a proper Py3.3 package for you. When you install pip under Py3.3, it usually creates a "pip-3.3" that you can just run (and there should be a "pip-2.7" that's identical to your current "pip").

Stefan

_________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml@lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

I am baffled by the behaviour of lxml 3.1.0 when you feed it a perfectly ordinary TEI file that begins with the standard xml declaration: <?xml version="1.0" encoding="UTF-8"?> It produces the error message "Unicode strings with encoding declaration are not supported." If you take out the line, things work. How does that make sense? The xml declaration is a standard part of an XML file. If you run xqueries with oXygen on the file, the processor does not complain. This seems to be an lxml rather than a Python feature: if you do something with the file that doesn't involve lxml, it doesn't complain about the declaration,and it handles Unicode characters just fine. Martin Mueller Professor of English and Classics Northwestern University On 2/13/13 1:07 PM, "Stefan Behnel" <stefan_ml@behnel.de> wrote:

Martin Mueller, 13.02.2013 19:19:

...

I have been trying to install the latest version of lxml on OS Lion. I've had varying success. It worked for Python2.7, but didn't work for Python3.3.

I used the installation instructions given on the lxml website:

STATIC_DEPS=true sudo pip install lxml

I got an error message about a missing "llvm-gcc-4.2", but there was good advice on how to handle this at

http://waqasshabbir.tumblr.com/post/19073648382/llvm-gcc-4-2-exe-error-on -m ac-osx-lion-when-building

My goal was to install lxml in Python3.3, but the command with a specified target installed lxml in Python2.7. I learned how to use the target flag and how to install lxml in a particular version of Python with this command:

STATIC_DEPS=true sudo pip install lxml --target

/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-pack ag es

That's the wrong way round. You are asking a Py2.7 pip to install its package into the site-packages directory of you Py3.3 installation. Rather, you want to run pip with the Py3.3 "python" executable so that it can build a proper Py3.3 package for you. When you install pip under Py3.3, it usually creates a "pip-3.3" that you can just run (and there should be a "pip-2.7" that's identical to your current "pip").

Stefan

_________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml@lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

Thank you. That works, but there are other ways in which lxml 3 behaves quite differently from 2, and in ways that don't seem to be documented in the tutorial. For instance, its default prefaces every line with a b', which I take it stands for binary, and it transforms utf-8 characters into character entities. Also, when I worked with 2.7 and a pre 3 version of lxml, printed output behaved in a predictable way. But in lxml 3 the etree.tostring output makes a mess of white space and prints newline characters in the form "\n" I work a lot with Python3.3 because I don't have to worry about UTF-8: the scripts handle text files as if they were ASCII, and I don't have to worry about the encoding. The same is true of Xquery. But lxml seems to be very fussy about character issues, and oddly enough it seems even fussier in Python 3 than in Python 2. Is there a set of simple rules about what to do or not to do when you routinely work with Utf-8 and multilingual documents? Sorry to be so difficult about this. I thought lxml was going to be easier than xquery. And the ability to combine XML parsing with other operations in one program is certainly an advantage. But the management of character problems seems to be a real headache, and it's a lot harder than ordinary operations in Python. Martin Mueller Professor of English and Classics Northwestern University On 2/16/13 2:28 PM, "Stefan Behnel" <stefan_ml@behnel.de> wrote:

Martin Mueller, 16.02.2013 21:24:

...

I am baffled by the behaviour of lxml 3.1.0 when you feed it a perfectly ordinary TEI file that begins with the standard xml declaration: <?xml version="1.0" encoding="UTF-8"?>

It produces the error message "Unicode strings with encoding declaration are not supported."

If you take out the line, things work. How does that make sense? The xml declaration is a standard part of an XML file. If you run xqueries with oXygen on the file, the processor does not complain.

This seems to be an lxml rather than a Python feature: if you do something with the file that doesn't involve lxml, it doesn't complain about the declaration,and it handles Unicode characters just fine.

It looks like you're feeding it Unicode data instead of a byte stream. If you're really reading it from a file, make sure you open it in binary mode, or just pass the filename in to lxml's parse() function.

Stefan

_________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml@lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

Tres Seaver, 17.02.2013 04:41:

On 02/16/2013 04:31 PM, Martin Mueller wrote:

...

there are other ways in which lxml 3 behaves quite differently from 2, and in ways that don't seem to be documented in the tutorial. For instance, its default prefaces every line with a b', which I take it stands for binary

"bytes" actually, which is a builtin type in Python.

...

and it transforms utf-8 characters into character entities.

There is no such thing as "utf-8 characters". "UTF-8" is an encoding, i.e. a mapping from Unicode text (i.e. characters) to a byte sequence (i.e. bytes). And there's no difference in lxml between Py2 and Py3 in its behaviour regarding character escaping. By default, it does it in both, because the default encoding is ASCII with character references for characters that ASCII cannot represent. If you want plain, unescaped, UTF-8 encoded output, pass "utf-8" into the "encoding" option.

...

Also, when I worked with 2.7 and a pre 3 version of lxml, printed output behaved in a predictable way. But in lxml 3 the etree.tostring output makes a mess of white space and prints newline characters in the form "\n"

That has nothing to do with lxml. It's how repr() works for bytes objects. And it's actually quite predictable and deterministic.

...

I work a lot with Python3.3 because I don't have to worry about UTF-8: the scripts handle text files as if they were ASCII, and I don't have to worry about the encoding. The same is true of Xquery. But lxml seems to be very fussy about character issues, and oddly enough it seems even fussier in Python 3 than in Python 2.

...

Is there a set of simple rules about what to do or not to do when you routinely work with Utf-8 and multilingual documents?

The simplest rule is: always encode your data (preferably to UTF-8) before passing it into the lxml parsing mechanism.

Hmm, actually, it's the other way round: "do not decode your XML data yourself". That's a part of the work of an XML parser. Just pass it in unchanged, it'll always do the right thing, by specification. XML is explicitly defined as a stream of bytes. It's not Unicode text. Take a look at the spec, it's all about byte sequences and how to map them to text and structure. The error that lxml emits is explicitly there to keep people from doing the wrong thing. The ability to parse XML from Unicode text input is only there to simplify parsing fragments from code, not to do anything major with it. Specifically not to parse whole documents and (that's where the error comes from) specifically not whole documents that lie about their encoding. Basically, what lxml and libxml2 have to do when they see Unicode text input is to 1) decode the underlying system specific byte buffer to Unicode 2) encode that to UTF-8 3) parse the result as XML. That's a bit inefficient. Even worse, if the input comes from an actual file that is opened in text mode, the process becomes 1) Python reads bytes from the file 2) Python decodes the data to a unicode object 3) lxml and libxml2 decode the underlying buffer to Unicode 4) encode it to UTF-8 5) parse the result as XML. For straight bytes input from a UTF-8 encoded file, it's more like 1) read bytes from the file 2) parse them as XML This is very efficient, especially when you pass the input file by file name instead of letting Python open it for you (but the latter is ok, too). Stefan

Martin Mueller, 19.02.2013 06:38:

I'm still not quite out of the woods, and perhaps I should stay in the woods of Python2.7

"woods" seems like the right word. Once you've seen the light, it's hard to go back there. :)

If parse an XML file with lxml under Python2.7, iterate through all its elements and use the command print etree.tostring(element, encoding = 'utf-8), the output of the print operation is identical with the input, as it should be.

If I do the same thing with Python3.3 and add no encoding information, the output of the print operation is not identical with the input. The lines are prefaced by b' and Unicode characters show up as character entities (ſ)

As I said, that's unrelated to Py2/3. If you leave out the encoding option in Py2, you'll get the same form of escaping there. The only thing that changes is the repr() of bytes strings in Py3, which prefixes them with the 'b' prefix. But you'll only see that when you *print* them. It doesn't change your data, it just makes it clearer what you have in your hands.

If I now use the command print etree.tostring(element, encoding = 'utf-8), the output is also not identical, but the utf-8 characters show up differently (the long 's' (ſ) shows up as '\xc5\xbf'. and new line characters show up as \n

The output is escaped by repr(). You get the same thing in Py2 when you *explicitly* call repr() on the object.

What do I do to decode the bytes version so that I can parse a file with lxml and then print it out unchanged?

If all you want to do is to print it, pass encoding="unicode". That doesn't really make sense semantically, because "unicode" is not an encoding, but it makes lxml serialise the result to a Python unicode string, which prints properly in both Py2 and Py3.

That's a basic operation for me: before you introduce changes, you want to make sure that you can put a file through a series of operations without change. The section on "serializing to Unicode strings" has a sentence that says "However, if you want to save the result to a file or pass it over the network, you should use write() or tostring() with a byte encoding (typically UTF-8) to serialize the XML." If that means something other than adding "encoding = "utf-8" , I don't know what it would be.

Different use cases. Printing data to make it readable to humans is something entirely different from serialising it for machine processing.

If I simply open an XML file in 'rb' mode and then cycle through its lines, giving a print command like

print(line.decode('utf-8')

the output transforms the bytes input into ordinary utf-8.

In Py2, you mean. That's a bug that was fixed in Py3. What happens is that the line first gets decoded (!) into Unicode behind your back, using a default encoding (which may change, meaning that the operation may fail arbitrarily) and only then it does what you asked it for, namely it encodes the result into a UTF-8 byte sequence. The implicit decoding is a step that leads to rather unpredictable and hard to understand behaviour in real-life, especially because the codec it uses is system specific. Users can change it and thus break your code on their installation. These problems eventually lead to the decision to separate byte strings from Unicode strings in Py3.

There must surely be something that does the same for the parsed XML tree, but what is it?

See my comment on the two use cases above.

The scripts I write mix XML parsing with other ordinary string operations.

Could you give an example? It's not all that common to do that.

This is unproblematic in 2.7

It's actually *very* problematic in Py2. You just didn't notice yet because you were lucky, and most likely also because you were not dealing with Unicode data outside of the ASCII range. Those who do were almost certainly bitten more than once by the infamous Py2 auto-decoding bug. It's also a bit problematic in Py3, but for a different reason. Byte strings no longer support the complete string API there, which means that some operations that used to work in Py2 are no longer available in Py3. Some of them because they simply don't make sense on byte strings, but another few were removed because they seemed inappropriate and hard to keep alive, above all '%' formatting of byte strings, which would still have made a nice feature. There are portable ways to do it differently (e.g. the struct module), but they are more complex.

once you've learned about u'something' and 'encoding='utf-8'. It seems to be quite hard with Python3.3

It's actually quite simple. Py3.3 has backwards compatibility support for the 'u' prefix (to reduce the syntax impact when porting code), so you can just leave your Unicode strings as they are. Alternatively, strip the 'u' prefix, it's completely redundant in Py3. Now, you need to be aware when you are dealing with byte strings, but that's easy for XML because all serialised XML *is* byte strings. The encoding="unicode" feature above is the only exception, and a rather explicit one. Basically it's like this: 1) encoded byte data gets parsed into a tree 2) XML trees are completely Unicode 3) XML trees get serialised into byte strings So you're only dealing with two worlds: a nice and shiny Unicode world with parsed in-memory XML trees, and a somewhat less shiny bytes world before and after parsing. All of this being said, it's worth using Py3, but you'll have to learn something new for it, something that users of Py2 often sneaked their way around. String processing in Py2 was something seemingly simply that could quickly get you into encoding hell for no apparent reason. String processing in Py3 is way safer and cleaner, but you have to get used to being aware what you are dealing with: bytes or text. The upside is that you'll write better code when you're aware of that. Stefan

Hi, I’m a bit late to this party but Martin, if you want to write a byte string to the standard output with Python 3, use sys.stdout.buffer.write() This is what you want if eg. you’re piping the output to a file. print() on Python 3 insist on getting Unicode strings, and sys.stdout has its own logic to try and detect the appropriate encoding for showing text on the terminal. -- Simon Sapin