[Python-Dev] readd u'' literal support in 3.3?

[Vinay Sajip]

Laurence Rowe <l <at> lrowe.co.uk> writes:

> The approach that most people seem to have settled on for porting  
> libraries to Python 3 is to make a single codebase that is compatible with  
> both Python 2 and Python 3, perhaps making use of the six library. If I  
> understand correctly, Chris' experience of porting WebOb was that there is  
> a large amount of manual work required in this approach in part because of  
> the many u'' strings in libraries that extensively use unicode. It should  
> be possible to automate this with the same approach as 2to3, but instead  
> of a transform from 2->3 it would transform code from 2->(2 & 3). In this  
> case the transform would only have to be run once (rather than on every  
> setup.py install) and would avoid the difficulties of debugging with  
> tracebacks that do not exactly match the source code.

I started writing a tool today, tentatively called '2to23', which aims to do
this. It's basically 2to3, but with a package of custom fixers in a package
'lib2to23.fixers' adapted from the corresponding fixers in lib2to3. It's
experimental work in progress at the moment. With a sample file like

import anything
import dummy

class CustomException(Exception):
    pass

def func1():
    a = u'abc'
    b = b'def'
    c = 'unchanged'
    c1 = u'abc' u'def'

def func2():
    try:
        d = 5L
        e = (int, long)
        f = (long, int)
        g = func3()
        if isinstance(g, basestring):
            print 'a string'
        elif isinstance(g, bytes):
            print 'some bytes'
        elif isinstance(g, unicode):
            print 'a unicode string'
        else:
            print
        for i in xrange(3):
            pass
    except Exception:
        e = sys.exc_info()
        raise CustomException, e[1], e[2]
        
class BaseClass:
    pass

class OtherBaseClass:
    pass
    
class MetaClass:
    pass

class DerivedClass(BaseClass, OtherBaseClass):
    __metaclass__ = MetaClass


2to23 gives the following suggested changes:

--- sample.py	(original)
+++ sample.py	(refactored)
@@ -1,34 +1,41 @@
 import anything
 import dummy
+from django.utils.py3 import long_type
+from django.utils.py3 import string_types
+from django.utils.py3 import binary_type
+from django.utils.py3 import b
+from django.utils.py3 import text_type
+from django.utils.py3 import u
+from django.utils.py3 import xrange
 
 class CustomException(Exception):
     pass
 
 def func1():
-    a = u'abc'
-    b = b'def'
+    a = u('abc')
+    b = b('def')
     c = 'unchanged'
-    c1 = u'abc' u'def'
+    c1 = u('abc') u('def')
 
 def func2():
     try:
-        d = 5L
+        d = long_type(5)
         e = (int, long)
         f = (long, int)
         g = func3()
-        if isinstance(g, basestring):
-            print 'a string'
-        elif isinstance(g, bytes):
-            print 'some bytes'
-        elif isinstance(g, unicode):
-            print 'a unicode string'
+        if isinstance(g, string_types):
+            print('a string')
+        elif isinstance(g, binary_type):
+            print('some bytes')
+        elif isinstance(g, text_type):
+            print('a unicode string')
         else:
-            print
+            print()
         for i in xrange(3):
             pass
     except Exception:
         e = sys.exc_info()
-        raise CustomException, e[1], e[2]
+        raise CustomException(e[1]).with_traceback(e[2])
         
 class BaseClass:
     pass
@@ -39,8 +46,8 @@
 class MetaClass:
     pass
 
-class DerivedClass(BaseClass, OtherBaseClass):
-    __metaclass__ = MetaClass
+class DerivedClass(with_metaclass(MetaClass, BaseClass, OtherBaseClass)):
+    pass

As you can see, there's still a bit of work to do, and the sample doesn't cover
all use cases yet. I'll be cross-checking it using my recent Django porting work
to confirm that it covers everything at least needed for that port, which is why
the fixers currently generate imports from django.utils.py3. Obviously, I'll
change this in due course.

Regards,

Vinay Sajip