[Python-Dev] int/long FutureWarning

[Martin v. Löwis]

"Mark Hammond" <mhammond@skippinet.com.au> writes:

> True.  But can you come up with a reasonable example?  A few of us have this
> problem, and almost everyone has agreed that hex constants are generally
> used in operations where the numeric value is not important, but the bit
> pattern is.  

The problem is that you won't know whether an integer was hex. If I do

f = open("/tmp/large","w")
f.seek(4278190080)

(which is 0xff000000l) then, under your proposed change, this might
seek to a negative offset on some systems, instead of the positive
offset.

Likewise

[None] * (1l << 32)

now gives an OverflowError. With your proposed change, it gives

[]

(actually, all calls to __mul__ will change their behaviour).

> I haven't seen a real example of where your scenario actually exists
> - not to mention that any such code would have to be new (as that
> constant does not work with "i" today) so therefore the author
> should use the new correct type code!

See, this is a central mistake: Under your proposed change, existing
code *will* change. Whether or not the constant was original
hexadecimal, or whether it was a constant at all, is not known at the
time ParseTuple makes its decision.

> I'm not sure if you are playing devil's advocate, or your contempt
> for our problem is real?

At the moment, I just want everybody to understand what the
implications of proposed changes are. It seems we are still in
disagreement about facts.

I don't know whether dropping the OverflowError test would cause real
problems, since existing code likely does not trigger these errors (or
else it would not work correctly now). 

However, I do believe that extension authors *will* complain if their
extension crashes if Python has failed to check a range, and converted
a positive value to a negative one. Some underlying C library might
react badly when it gets large negative numbers.

> No - no one is trying to say any of that at all.  The problem is
> simply breaking existing code.  That code was *not* broken in the
> first place.  We understand the rationale - just we also have lots
> of existing code.

The question is what the proposed correction should be. I'm in favour
of using stricter semantics when the semantics changes (i.e. give more
exceptions than before); authors of applications that can accept to
drop some of the built-in tests then need to make explicit changes to
drop them.

You seem to favour the reverse policy: drop the stronger checks by
default, and give authors who want them an option to add them
back. The problem with this strategy is that authors might not know
that they have to change anything. With my strategy, they will find
out automatically.

> Well, I wouldn't be happy, but it wouldn't kill me, but would have to be a
> semi-blind replace to keep me sane (Maybe I would just write a script to
> convert all my .py files with hex constants to decimal - the bitwise ops
> will still get me, but locating them may be OK.)  

That's what I did with h2py.py. It now generates decimal constants to
silence the warning, and a unary - to get the negative value, since I
cannot guesss whether the constant really was meant to be negative or
not.

> Of course, someone will have to update PyXPCOM in the Mozilla tree,
> and any thing else using hex constants in that way but not actively
> maintained may struggle and become unavailable for a little while.

That's why Python 2.3 adds the warning. There is no change to
semantics, and over a year of time to make modification (when you
update to Python 2.4).

> Plenty of extension authors not on python-dev (including plenty of
> home-grown extensions taking win32con constants, for example), and
> obviously I can't speak for them. And as Jack's latest mail points
> out, supporting multiple versions of Python becomes almost
> impossible.  The cost is real, and not able to be borne only by
> people on this list.

If you can't afford to break code now, accept the warning. If you
don't like the warning, silence it with a global filter.

I estimate that Python users confronted with the warning will go
through the same process as readers of this list:
- What is this warning?
- I want constants to be int. Can I have that back please?
  (answer: at the moment, they are still int. In the future, they
   will be long, and rightfully so).
- Ok, what do I do?
  (answer: if you want negative constants, write a positive constant,
   and add a unary -. If you want a positive constant, add an L).
- I don't care whether they are positive or negative.
  (answer: then add an L)
- I don't want to add that many Ls.
  (answer: then use the future import)
- This will break my C modules which will give OverflowErrors.
  (answer: use new formatting codes)
- This will break backwards compatibility.
  (answer: this is what the warning is for. Breaking backwards
   compatibility cannot be avoided. The warning warns you that
   something *will* break at some point. It is your choice what
   breaks, and at what point. Reconsider whether you don't care
   whether the constants are positive or negative).

When users truly understand the issues, and can make an educated
decision what to do, the warning has served its purpose.

Regards,
Martin