"Mark Hammond" writes:

Any suggestions for me?

There is no solution to this problem yet. If you add an L to the constant (or if it becomes positive in Python 2.4), you can't pass it to the "i" format to anymore, as it will cause an OverflowError. Suppressing the warning now will only defer the problem to the future. Regards, Martin

"Mark Hammond" writes:

...

Any suggestions for me?

[Martin]

There is no solution to this problem yet. If you add an L to the constant (or if it becomes positive in Python 2.4), you can't pass it to the "i" format to anymore, as it will cause an OverflowError.

Suppressing the warning now will only defer the problem to the future.

But in Mark's case (and in many other cases) there will be no problem in the future -- in Python 2.4, his C code will happily accept the positive Python longs that 0x80000000 and others will be then. I wonder if perhaps the warnings for hex/oct constants aren't so important, and we should use PendingDeprecationWarning or some similar warning that's not normally printed? They sure are a pest! BTW, this reminds me that I've long promised a set of new format codes for PyArg_ParseTuple() to specify taking the lower N bits (for N in 8, 16, 32, 64) and throwing the rest away, without range checks. If someone else can get to this first, that would be great -- I can't seem to make time for this, even though it is still my utmost desire and plan to have a 2.3a1 release ready before Xmas. --Guido van Rossum (home page: http://www.python.org/~guido/)

Martin v. Löwis wrote:

Guido van Rossum writes:

...

But in Mark's case (and in many other cases) there will be no problem in the future -- in Python 2.4, his C code will happily accept the positive Python longs that 0x80000000 and others will be then.

Can you please explain how this will happen? If you do

int x; PyArg_ParseTuple(args,"i",&x);

and args is (0x80000000,), what will be the value of x?

x should be 0x80000000. Whether that's a negative number in its decimal representation is really not all that important if you are interfacing to 32-bit bitmaps ;-) I honestly don't think that anyone would write x = 0x80000000 and then expect x < 0 to be True. People usually write hex representations when they are trying to do bit-level manipulations and these rarely deal with signed numeric data.

...

BTW, this reminds me that I've long promised a set of new format codes for PyArg_ParseTuple() to specify taking the lower N bits (for N in 8, 16, 32, 64) and throwing the rest away, without range checks.

Wouldn't Mark have to use these format codes?

-- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Jack Jansen writes:

I've been sick and tired of these warnings, especially since in 99.9% of the cases that you get the warning it is meaningless (as we are really taking about bitpatterns that have a special meaning in some C API). I personally haven't seen a single instance of the warning making sense.

I found that all those warnings are correct: in particular *when* the constant is a bit pattern in some C API. It means that your code *will* break in Python 2.4, unless you take corrective action (which you cannot take at the moment). It will break because ParseTuple will raise an OverflowError. Regards, Martin

Guido van Rossum writes:

...

But in Mark's case (and in many other cases) there will be no problem in the future -- in Python 2.4, his C code will happily accept the positive Python longs that 0x80000000 and others will be then.

Can you please explain how this will happen? If you do

int x; PyArg_ParseTuple(args,"i",&x);

and args is (0x80000000,), what will be the value of x?

...

BTW, this reminds me that I've long promised a set of new format codes for PyArg_ParseTuple() to specify taking the lower N bits (for N in 8, 16, 32, 64) and throwing the rest away, without range checks.

Wouldn't Mark have to use these format codes?

That's what I meant. --Guido van Rossum (home page: http://www.python.org/~guido/)

Jack Jansen writes:

Well.... First of all, warning people about something without giving them a way to do something about it isn't really good style.

Whom are you directing this observation at? I was merely pointing out that the warning is factually correct and indicates a real future problem; I did not consider style at all.

Second, it will *not* break in 2.4, because I'm just going to add an O& formatter PyMac_Parse32BitIntWithoutSillyComplaints, which will take any reasonable type on the Python side and just return the lower 32 bits.

It would be better if you could just contribute a patch to add such an argument parser to the standard ParseTuple implementation, instead of coming up with proprietary Mac solutions. When I said "it will break", I meant "unless action is taken", of course. Regards, Martin

Greg Ewing writes:

...

It means that your code *will* break in Python 2.4, unless you take corrective action (which you cannot take at the moment).

Pardon me, but... wouldn't it have been better to defer introducing these warnings until there *is* something that can be done about them?

That might be the case. At the time the warning was added, there was consensus that it is be easy to do something about each of them. It was only detected later that it is not easy in some cases (strictly speaking, you can correct all the warnings today, taking, for example, the approach that Jack would take, of adding a custom conversion function into your C modules). Today, I would rather hope that somebody contributes a patch to add the requested features instead of contributing a patch to disable the warning. Regards, Martin

"Mark Hammond" writes:

Except, on the flip-side, let's say I am *happy* for such contants to become longs. I really don't want to see the warning for every hex literal once I understand the impact.

Reducing the warnings to one warning per module might be reasonable. To silence the warning, you should really add "L" suffixes to all those literals - or we could provide a future statement, where you indicate that you want those literals to become positive longs *now*; you'ld need to add this statement once per module. However, when you do that, I believe you will get OverflowErrors once you pass the constants to your C API.

So, maybe we simply need finer-grained warnings - such as in PyArg_ParseTuple, and any other places where the impact will actually be felt.

Feel free to propose specific patches. I believe this specific strategy is unimplementable: When the value goes to ParseTuple, it is not known anymore whether the integer literal was hex (or whether there was any integer literal at all). Regards, Martin

Pardon me, but... wouldn't it have been better to defer introducing these warnings until there *is* something that can be done about them?

We *will* let you do something about them before we release 2.3a1. --Guido van Rossum (home page: http://www.python.org/~guido/)

I must be missing something, but I can't understand why the existing PyArg_ParseTuple codes can't take on a kind of "hybrid" approach for b/w compatibility. This could mean, assuming a 32 bit platform, that 'l':

c_func_taking_int( 0x80000000L ) -> 0x8000000 c_func_taking_int( 0xFFFFFFFFL ) -> 0xFFFFFFFF c_func_taking_int( 0x100000000L ) -> OverflowError c_func_taking_int( -0x80000001L ) -> whatever we like <wink>

We might as well not bother with range checking then, which would be backwards incompatible. :-(

The new format codes can be more precise in their handling of these objects and their sign.

I can't see a real downside to this. There is no way someone can get (+-)0x80000001L into a C int via ParseTuple("l") now, so we are not breaking anything. When the arg is an int object or a long in the currently supported range, the semantics are identical. A few cases may exist where an OverflowError would have been thrown but no longer is, but that behaviour is unlikely to be relied upon.

The only case I see this failing with is bitwise rotates. Today: c_func_taking_int( 0x80000000<<1 ) -> 0x0

but with my semantics above, it would yield

c_func_taking_int( 0x80000000L<<1 ) -> OverflowError

You get a slightly different warning for left shifts that actually lose bits, and I don't think that warning is causing you any pain.

re-defining 'l' to mean "lower 'n' bits" would solve this, but I accept this is going too far. Certainly for all my "constant files", all bitwise rotate operations are "safe" in terms of losing bits - just not in changing sign - so I believe this would still work for me.

I am sure I am missing something, but I can't see it. I am ready to feel foolish <wink>. It would certainly be easier for me to work on such a strategy than to convert all my extensions.

I don't have enough time to think about this right now (and I'm dying to go to bed), but I agree that something needs to be done. Originally the PEP didn't prescribe warnings for this. Maybe we can just disable the warnings by default? Or maybe we can use the "__future__" suggestion, since it's the only way to signal to the parser not to issue the warnings. (One problem with the warning about hex constants is that because the entire module is parsed before it is executed, putting a warnings.filterwarnings() call in the module itself is ineffective; you must put it somewhere else. This is not a problem with the warnings about <<. A __future__ statement is a signal to the parser.) --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thursday, Nov 28, 2002, at 23:12 Europe/Amsterdam, Martin v. Löwis wrote:

...

Pardon me, but... wouldn't it have been better to defer introducing these warnings until there *is* something that can be done about them?

That might be the case. At the time the warning was added, there was consensus that it is be easy to do something about each of them.

No, there was no consensus. I have screamed loudly about it, but things went ahead anyway. Note that I'm not complaining about this: that's the way it goes in a group project, but let's not try to rewrite history...

It was only detected later that it is not easy in some cases (strictly speaking, you can correct all the warnings today, taking, for example, the approach that Jack would take, of adding a custom conversion function into your C modules).

No, that does *not* help (as I have explained umpteen times). As the warning is given by the parser there is absolutely nothing you can do about it with a PyArg_ParseTuple converter. There are of course workarounds (like adding, for 2.3 only, an "L" to all the constants in addition to the custom converter) but that would be silly because things will work again in 2.4. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

No, that does *not* help (as I have explained umpteen times). As the warning is given by the parser there is absolutely nothing you can do about it with a PyArg_ParseTuple converter.

As I have explained twice as often, you need to add an L to the constants.

There are of course workarounds (like adding, for 2.3 only, an "L" to all the constants in addition to the custom converter) but that would be silly because things will work again in 2.4.

No, they won't. If you just add the L, you will get OverflowErrors, unless you also modify the the ParseTuple calls. Regards, Martin

On Fri, Nov 29, 2002, M.-A. Lemburg wrote:

2. x = 0x80000000 gives a signed integer in Python <=2.2 and long(x) results in -2147483648L; int(0x80000000L) gives an OverflowError in Python <=2.2

You sure about this? I thought the whole point was that it's *not* necessarily -2147483648L and OverflowError -- on a 64-bit platform. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "If you don't know what your program is supposed to do, you'd better not start writing it." --Dijkstra

1. 0x80000000 and similar large hex and octal constants will generate a parser warning in Python 2.3 which is only useful for developers

Correct. The same holds for any other warning, and for all uncaught exceptions: the message you get is only useful for the author of the software.

2. x = 0x80000000 gives a signed integer in Python <=2.2 and long(x) results in -2147483648L; int(0x80000000L) gives an OverflowError in Python <=2.2

Correct.

3. x = 0x80000000L gives a long integer 2147483648L in Python >=2.4; int(0x80000000L) returns a long in Python >=2.4 or generates an OverflowError (no idea ?)

Correct, although you probably meant to omit the L in both cases. I think the intention is that int(x) is x if x is long.

4. C extensions have typically used "i" to get at integer values and often don't care for the sign (that is, they use them as if they were unsigned ints)

I'm not sure about the proportions, i.e. whether the "ints are bitmaps" case is more or less often than the "ints are ints" case.

5. new parser markers would not be available in older Python versions unless they are backported

Correct.

To me the picture looks as if the warnings should only be printed on request by a developer (not per default) and that new parser markers are the only way to get everyone satisfied.

The same is true for any other warning. The problem with not printing warnings by default is that it removes much of the application for the warning: To forcefully indicate that something will change. If we would only want to tell those who want to hear, we could just write it all in the documentation - if you read the documentation, you will know, if you don't, you won't. This approach was heavily critized in the past, and lead to the introduction of warnings.

Since the new parser markers will need to mask bitmaps, I suggest to use "m#" as new marker with # being 1,2,3 or 4 representing the number of bytes to mask, i.e. "m1" gives ((unsigned long)value & 0xF), "m2" gives ((unsigned long)value & 0xFF), etc. The marker variable will have to reference an (unsigned int).

I'm not sure this would work for all cases. If you have things like fcntl operation codes which you generate from header files, the format argument depends on sizeof(long), so writing a portable fcntl module would be difficult with these format markers. Regards, Martin

Jack Jansen writes:

But note that these long/int warnings are especially obnoxious: as they are given by the parser you cannot turn them off with a filterwarning in the module itself, you have to go out and find each and every module importing them.

A patch contributing a future import statement would be appreciated. Regards, Martin

A patch contributing a future import statement would be appreciated.

If we don't have a solution that is satisfactory for Jack and Mark, we'll have to disable the warnings for the 2.3a1 release. I think it's not acceptable to issue warnings without providing a reasonable way to disable them. For the 2.3 final release (and even for 2.3b1) I want the warnings back on, but by then a solution for Jack's and Mark's problem will definitely have to be in place. Personally, I think there's nothing wrong with Jack & Mark regenerating their hex constants with a trailing 'L', even though that becomes redundant in 2.4. But who knows when 2.4 will be out. Trailing 'L' won't become illegal until 3.0. It seems that that would silence the warnings at the cost of breaking the calls, since PyArg_ParseTuple still does range checking. At least Jack knows how to also regenerate the C code that gets called. But I also have no problem with a __future__ statement. Look in Python 2.2 for how the "yield" keyword support was done. --Guido van Rossum (home page: http://www.python.org/~guido/)

Someone will have to give me a hand with this: I could probably figure out how normal future imports work (although I've never done one), but this is one of those hairy ones that needs a hook in the parser. And that's an area of Python that has always frightened me to no end...

I think Guido's recommendation of looking at the yield processing is not that helpful; yield is added as a keyword in the parser, whereas the FutureWarning occurs in compile.c. So you need to add the feature to compile.c:future_check_features, and then just check for the flag around the place where the FutureWarning is raised (and interpret the constant as long if the feature is set).

Another option was to add "k1", "k2", "k4" and "k8", to mean uint8, uint16, uint32 and uint64. "k1" and "k2" would be synonyms for "B" and "H", but this would make the k-format-family consistent.

Since this is the proposal that MAL just came up with also, it seems to be a manifest idea (unless it was also MAL who proposed this the last time around, in which case it might only be manifest to him). If that meets all requirements, go for it - I'm just not sure how I would use for plain int, since I don't know its size in advance. Regards, Martin -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev

I grabbed it from the old discussion, so there's a good chance MAL came up with it at that time. But Guido mumbled somthing about "k1"=="B" and "k2"=="H" and that being overkill, and then the discussion stopped. Hence my question: is k1/k2/k4/k8 what it's going to be?

+1. I personally don't care about the duplication, since I expect that both the implementation and the documentation can be quite compact and treat all sizes uniformly, so having one format more or less doesn't matter. Regards, Martin

[Martin]

"Mark Hammond" writes:

...

I am sure I am missing something, but I can't see it. I am ready to feel foolish <wink>. It would certainly be easier for me to work on such a strategy than to convert all my extensions.

It is a violation of the principle

Errors should never pass silently. Unless explicitly silenced.

Ah-ha - elementary now you point it out <wink>

If you have a function

my_account.add_amount_of_money(0x80000000L)

and the implementation is a C function with an "i" converter, this currently gives an indication of the error. Under your proposed change, it would remove money from the account.

True. But can you come up with a reasonable example? A few of us have this problem, and almost everyone has agreed that hex constants are generally used in operations where the numeric value is not important, but the bit pattern is. I haven't seen a real example of where your scenario actually exists - not to mention that any such code would have to be new (as that constant does not work with "i" today) so therefore the author should use the new correct type code! I'm not sure if you are playing devil's advocate, or your contempt for our problem is real?

If we think that the case "large numbers are bitmaps" is more common than the case "large numbers are large numbers", we should drop the range check for the existing converters, and add new converters in case somebody wants large numbers instead of bitmaps, which then do the range checks.

No - no one is trying to say any of that at all. The problem is simply breaking existing code. That code was *not* broken in the first place. We understand the rationale - just we also have lots of existing code. [Jack]

This would be ideal, of course. But note that for me changing the format char is a minor issue, as long as the behavior of the new format char is as expected ... But of course I don't know how this would be for Mark...

Well, I wouldn't be happy, but it wouldn't kill me, but would have to be a semi-blind replace to keep me sane (Maybe I would just write a script to convert all my .py files with hex constants to decimal - the bitwise ops will still get me, but locating them may be OK.) Of course, someone will have to update PyXPCOM in the Mozilla tree, and any thing else using hex constants in that way but not actively maintained may struggle and become unavailable for a little while. Plenty of extension authors not on python-dev (including plenty of home-grown extensions taking win32con constants, for example), and obviously I can't speak for them. And as Jack's latest mail points out, supporting multiple versions of Python becomes almost impossible. The cost is real, and not able to be borne only by people on this list. Ultimately though, the end result is likely to be everyone just mass-replaces "i" with "?" in any set of code where they *may* have the problem - but carefully avoiding code where hex constants are passed as currency values <wink> Mark.

"Mark Hammond" writes:

True. But can you come up with a reasonable example? A few of us have this problem, and almost everyone has agreed that hex constants are generally used in operations where the numeric value is not important, but the bit pattern is.

The problem is that you won't know whether an integer was hex. If I do f = open("/tmp/large","w") f.seek(4278190080) (which is 0xff000000l) then, under your proposed change, this might seek to a negative offset on some systems, instead of the positive offset. Likewise [None] * (1l << 32) now gives an OverflowError. With your proposed change, it gives [] (actually, all calls to __mul__ will change their behaviour).

I haven't seen a real example of where your scenario actually exists - not to mention that any such code would have to be new (as that constant does not work with "i" today) so therefore the author should use the new correct type code!

See, this is a central mistake: Under your proposed change, existing code *will* change. Whether or not the constant was original hexadecimal, or whether it was a constant at all, is not known at the time ParseTuple makes its decision.

I'm not sure if you are playing devil's advocate, or your contempt for our problem is real?

At the moment, I just want everybody to understand what the implications of proposed changes are. It seems we are still in disagreement about facts. I don't know whether dropping the OverflowError test would cause real problems, since existing code likely does not trigger these errors (or else it would not work correctly now). However, I do believe that extension authors *will* complain if their extension crashes if Python has failed to check a range, and converted a positive value to a negative one. Some underlying C library might react badly when it gets large negative numbers.

No - no one is trying to say any of that at all. The problem is simply breaking existing code. That code was *not* broken in the first place. We understand the rationale - just we also have lots of existing code.

The question is what the proposed correction should be. I'm in favour of using stricter semantics when the semantics changes (i.e. give more exceptions than before); authors of applications that can accept to drop some of the built-in tests then need to make explicit changes to drop them. You seem to favour the reverse policy: drop the stronger checks by default, and give authors who want them an option to add them back. The problem with this strategy is that authors might not know that they have to change anything. With my strategy, they will find out automatically.

Well, I wouldn't be happy, but it wouldn't kill me, but would have to be a semi-blind replace to keep me sane (Maybe I would just write a script to convert all my .py files with hex constants to decimal - the bitwise ops will still get me, but locating them may be OK.)

That's what I did with h2py.py. It now generates decimal constants to silence the warning, and a unary - to get the negative value, since I cannot guesss whether the constant really was meant to be negative or not.

Of course, someone will have to update PyXPCOM in the Mozilla tree, and any thing else using hex constants in that way but not actively maintained may struggle and become unavailable for a little while.

That's why Python 2.3 adds the warning. There is no change to semantics, and over a year of time to make modification (when you update to Python 2.4).

Plenty of extension authors not on python-dev (including plenty of home-grown extensions taking win32con constants, for example), and obviously I can't speak for them. And as Jack's latest mail points out, supporting multiple versions of Python becomes almost impossible. The cost is real, and not able to be borne only by people on this list.

If you can't afford to break code now, accept the warning. If you don't like the warning, silence it with a global filter. I estimate that Python users confronted with the warning will go through the same process as readers of this list: - What is this warning? - I want constants to be int. Can I have that back please? (answer: at the moment, they are still int. In the future, they will be long, and rightfully so). - Ok, what do I do? (answer: if you want negative constants, write a positive constant, and add a unary -. If you want a positive constant, add an L). - I don't care whether they are positive or negative. (answer: then add an L) - I don't want to add that many Ls. (answer: then use the future import) - This will break my C modules which will give OverflowErrors. (answer: use new formatting codes) - This will break backwards compatibility. (answer: this is what the warning is for. Breaking backwards compatibility cannot be avoided. The warning warns you that something *will* break at some point. It is your choice what breaks, and at what point. Reconsider whether you don't care whether the constants are positive or negative). When users truly understand the issues, and can make an educated decision what to do, the warning has served its purpose. Regards, Martin