[Python-Dev] Deprecation warning on integer shifts and such

[Guido van Rossum]

> On maandag, augustus 12, 2002, at 05:37 , Guido van Rossum wrote:
> > Oops.  Darn.  You're right.  Sigh.  That's painful.  We have to add a
> > new format code (or more) to accept signed 32-bit ints but also longs
> > in range(32).  This should be added in 2.3 so extensions can start
> > using it now, and user code can start passing longs in range(2**32)
> > now.  I propose 'k' (for masK).  We should backport this to 2.2.2 as
> > well.  Plus a variant on PyInt_AsLong() with the same semantics, maybe
> > named PyInt_AsMask().
> 
> Ow, pleeeeeeeeeeeeeeeeeeeeeeeeaaaaaase........
> 
> Just before 2.1 was released (or was it 2.0?) on a whim someone 
> "fixed" the short integer handling to bother about signs, in a 
> backward-incompatible way, despite that fact that about 95% of 
> the short PyArg_Parse formats in the core were mine, and I asked 
> for some form of backward compatibility. I spent about 2 weeks 
> going over a few thousand API calls to fix this mess at a time I 
> had more than enough other work on my hands.

Oops.  That wasn't intended of course.

> Can we please make this change in a backwards-compatible way, 
> i.e. leave the i and l formats alone and use something new for 
> "range-checked-int" and "range-checked-long"?

Um, they *already* do range checking.  'i' requires that the value
(whether it comes from a Python int or a Python long) is in the range
[INT_MIN, INT_MAX].  'l' doesn't do range checking on Python ints
(because they are defined to fit in a C long), but for a Python long
it requires that the value fits in the range of a signed C long,
i.e. [-sys.maxint-1, sys.maxint].

The problem is that hex constants and shifted values may be
represented by signed Python ints, abusing the sign bit as a mask bit,
*or* by Python longs that represent corresponding values by
nonnegative values in the range [0, 2*sys.maxint+1].  Since
PyArg_Parse* doesn't know whether the value will be used as a mask or
as a real signed int, we must allow negative Python longs (in the
range [-sys.maxint-1, -1] as well.

That means that the range checking (for long values) has to be
defective from the point of view of a function that doesn't want a
mask value but really expects a signed C int or long: Python code can
pass in a Python long value that's too large, but because it's still
in the 32-bit range the Python code won't be told about the overflow
error, and the C code will happily be given a large negative value.

If we really believe that there's more code (in the world, not just in
the core CVS tree) that uses 'i' or 'l' for masks than that uses it
for signed values, we cold fix 'i' and 'l' this way, and add new codes
for code that really wants signed values.  Still, all that code would
have to be fixed somehow and we would have to track it down.

And then we'd still be stuck with PyInt_AsLong() -- should it use the
same rule?  I hope not.

> I already fear that I have to come up with some sort of a fix 
> for the range-check warning (more than 6000 lines worth of 
> constant definitions that can currently be copied verbatim from 
> C header files to Python will have to be parsed, and computed, 
> and all these things can contain references to other constants, 
> strings and who knows what more, see Mac/Lib/Carbon/*.py), I 
> really could do without more work on my plate...

Before you start to panic, can you please try to import all those
modules and see how many cause warnings?  I only found one,
Controls.py line 11, but there are many files that I can't run because
they require extension modules I don't have.  I do note that none of
them generate warnings in the parser.  I also found a SyntaxError that
contradicts your assertion about "can currently be copied verbatim
from C header files".

--Guido van Rossum (home page: http://www.python.org/~guido/)