Hi,

since we had a lengthy discussion on whether or not non-prefixed byte strings should automatically mutate into unicode strings when compiled for Py3, here are some initial lessons from my first attempt to port lxml.

My first approach was (obviously) to import unicode_literals from __future__. This failed miserably, and even showed a couple of further bugs in Cython. :)

I then chose the route to explicitly prepend unicode strings with 'u', as I wanted to keep my source compilable with older Cython versions that do not support the 'b' prefix. Currently, I have changed about 700 lines this way in a quick walk-through, and now I'm searching the places where this was the wrong thing to do. :)

Most important evidence found: it's definitely non-trivial in a lot of places to decide what has to be unicode and what doesn't. It's non-trivial for me, and definitely not easier for Cython.

One important place where I ended up with a lot of trivial changes are docstrings. Here, I would give an almost 100% chance that the user meant a unicode string if it's not prefixed. The remaining cases, e.g. where some external tool may require binary data for some kind of configuration or analysis are rare enough to just ignore them. For exactly this reason (I think), the doctest module in Py3 ignores docstrings that are not unicode. This might be a place where an automatic conversion might make sense (although, if it's the only place, that would be some funny string semantics...)

Another important place are exception messages. Here, I'd give a real 100% for string literals, as their only purpose is to be human readable.

A field where I really had to take care is when working with byte sequences. For example, lxml has a couple of places where strings are converted into UTF-8 and then passed into re.findall() or re.sub(). When substituting, the replacement string obviously has to be a byte string, too. I also found a bug in the Py3 re module when working with byte strings in one specific case.

There are actually quite a number of places where strings are built as byte strings by combining and formatting literals, and then converted to a char*. Another place where automatic conversion must not happen.

So, while still on the way, my first real-world impression meets my original opinion. There are definitely a lot of unprefixed strings in my own code that are meant to be unicode strings. Simply switching their type in Py3 will fix a lot of them, but at the same time break many others. The things that it fixes are the trivial parts: docstrings and exceptions. Almost everything else really were byte strings, and some were non-trivial things that need real work.

If I can choose, I opt for going through this once and then having code that correctly distinguishes between byte strings and unicode strings in *both* Py2 and Py3, instead of additionally having to deal with changing string semantics for identical code in different environments. We might think about a way to simplify the transition from unprefixed docstrings and exception messages to unicode strings. As it currently stands, everything else is definitely out of scope for any automatism.

Stefan