[Idle-dev] Forward progress with full backward compatibility

[David Scherer]

[Peter writes, essentially]

>    	def some_function():
> 	    if int(time.time() / 3600) % 2:
> 	        return 123  # an integer object
> 	    else:
> 	        return 123.0 # an float object
>
>   	42 / some_function()
>
>    You see what I mean: It is impossible to change the expression above
>    to the proposed language level 2 (aka as Py3k) automatically, since you
>    would either have to use use the truncating integer division or
>    the floating point division depending on the time of day.

This is quite possible in principle, just really ugly:

def _div(a,b):
  if isinstance(a,types.IntType) and isinstance(b,types.IntType):
    return divmod(a,b)[0]
  else:
    return a/b

_div(42, some_module.some_function())

In practice, doing this would be both ugly and slow.  However, doing it *in
the interpreter* is fast.  In principle, one could put THREE operators in
the language: one with the new "float division" semantics, one that divided
only integers, and a "backward compatibility" operator with EXACTLY the old
semantics:

print 42 div 6      # 7
print 42 div 6.0	  # error?
print 42 / 6        # 7.0
print 42 olddiv 6   # 7
print 42 olddiv 6.0 # 7.0

Sprinkling "olddiv" through old programs is still ugly.  There is a better
way...

I actually implemented 1/2==0.5 in Python 1.5.2, complete with a
module-level backward compatibility flag.  The flag made an unusual use of
the "global" statement, which appears to be accepted at toplevel by 1.5.2
without any effect.  Therefore a statement like "global olddivision" will be
silently ignored by 1.5.2 and earlier, and will result in the old behavior
under my patch.  "global" even has the right sort of sound for module-level
options :)

An outline of what I did:

1.  Add new opcode BINARY_FRACTION to opcode.h and dis.py
2.  Add new flag "int c_olddivision" to struct compiling in compile.c
3.  Set c_olddivision to base->c_olddivision or 0 in jcompile
4.  Check for "global olddivision" outside a function definition in
com_global_stmt, and set c_olddivision=1
5.  Check c_olddivision in com_term, and generate either BINARY_DIVISION or
BINARY_FRACTION
6.  Add PyNumber_Fraction to abstract.h, and define it in abstract.c to
explicitly check for a pair of integers and do float division
7.  Add a BINARY_FRACTION case to ceval.c, which calls PyNumber_Fraction
instead of PyNumber_Divide.

This took me very little time, despite my total unfamiliarity with the
internals of the Python interpreter.  It's more effort than just modifying
intobject.c to do all integer divisions the new way, but is hardly a big
deal.

Any python module that ran under 1.5.2 can be made to work under these
changes by adding one line to the start: "global olddivision."  Such a
module continues to work with 1.5.2.  Inserting this automatically in a
large group of python files would be trivial.  The only problems would be
with programs that make extraordinary use of eval() or exec.

.pyc files generated with 1.5.2 will still run under my patch, since they
use the old opcode that generates the old semantics.  C modules will also
retain their behavior; if they want the new division behavior they can call
PyNumber_Fraction instead of PyNumber_Divide.

Something very similar could be done for any other kind of "pragma" - the
only difference would be where c_olddivision gets set.

> 2. What should the new Interpreter do, if he sees a source file without a
>    pragma defining the language level?  There are two possibilities:
>
>    1. Assume, it is an old source file and apply Language level 1 to it.
>       This has the disadvantage, that the pragma is required for all new
>       source files. [...]
>
>    2. Assume, it is a new source file and apply language level 2 to it.
>       This has the disadvantage, that it will break any existing code.

I think the answer is 2.  A high-quality script for adding the pragma to
existing files, with CLI and GUI interfaces, should be packaged with Python.
Running it on your existing modules would be part of the installation
process.

Long-lived modules should always have a language level, since it makes them
more robust against changes and also serves as documentation.  A version
statement could be encouraged at the top of any nontrivial script, e.g:

python 1.6
import sys, time
# etc

Such a statement, if used consistently, would permit all future changes to
the language to be made gracefully.  People who don't use it are living
dangerously, and if they install a new version of Python, they will have to
add a version statement to *.py.

Personally, I think that it makes more sense to talk about ways to
gracefully migrate individual changes into the language than to put off
every backward-incompatible change to a giant future "flag day" that will
break all existing scripts.  Versioning of some sort should be encouraged
starting *now*, and incorporated into 1.6 before it goes final.

Just my $0.02.

> BTW: This discussion is swway off-topic for IDLE.

Indeed, but Guido has spoken:

> Great ideas there, Bruce!  I hope you will post these to an
> appropriate mailing list (perhaps idle-dev, as there's no official SIG
> to discuss the Python 3000 transition yet, and python-dev is closed).

Dave