I can help with the compiler changes.

Great! (Though I was hoping someone from outside PythonLabs -- we're already too busy, and you in particular.)

What about True and False? Same logic -- prevent accidental harm, prevent people from doing silly things to make them faster.

Maybe in Python 3.0. Old code could define these as variables and we don't want to break that. (I have no mercy for old code that defines None, but True and False are a difference case. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

I think it's a small project. We've got a new node type in the grammar -- None -- and we'll need a new opcode -- LOAD_NONE. The compiler just needs to generate a LOAD_NONE for every None node.

Does the parser prevent assignment to None? Or does the compiler need to check for the use of None in an assignment?

The compiler needs to check. It's just like assigning to a literal: the parser has no power to prevent this, so the compiler has to act as a safeguard. --Guido van Rossum (home page: http://www.python.org/~guido/)

Here's what I have so far. :-) *** Grammar/Grammar 15 Oct 2001 15:44:04 -0000 1.45 --- Grammar/Grammar 26 Apr 2002 12:30:36 -0000 *************** *** 80,86 **** term: factor (('*'|'/'|'%'|'//') factor)* factor: ('+'|'-'|'~') factor | power power: atom trailer* ('**' factor)* ! atom: '(' [testlist] ')' | '[' [listmaker] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING+ listmaker: test ( list_for | (',' test)* [','] ) lambdef: 'lambda' [varargslist] ':' test trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME --- 80,86 ---- term: factor (('*'|'/'|'%'|'//') factor)* factor: ('+'|'-'|'~') factor | power power: atom trailer* ('**' factor)* ! atom: '(' [testlist] ')' | '[' [listmaker] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING+ | 'None' listmaker: test ( list_for | (',' test)* [','] ) lambdef: 'lambda' [varargslist] ':' test trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME It needs to add 'None' as an alternative to all other places where currently NAME occurs. The Unix Makefile automatically regenerates Includes/graminit.h and Python/graminit.c when you touch Grammar/Grammar. For some mysterious reason, I didn't have to change the code generator at all -- the NAME terminals and the 'None' terminal look the same to it. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]

...

... It needs to add 'None' as an alternative to all other places where currently NAME occurs.

Fudge. None makes no sense anywhere in, e.g.,

dotted_name: NAME ('.' NAME)*

or

global_stmt: 'global' NAME (',' NAME)*

but I suppose it has to be allowed for in 2.3 anyway else an existing

Yes, that's (alas) the point.

import None.None.None as None

would yield a SyntaxError instead of a warning.

More realistically, def foo(a, None=None): if a is None: ... I found 4 examples of this in Zope3 alone, and MAL has some too.

Note too that we can't change the generated code at all for 2.3 unless under the control of a new future statement, else existing None-abusing code could break.

A future statement makes no sense for this, so I guess all we can do in 2.3 is warn about use of None in all contexts except in expressions. This is really too bad -- I wish we could decide to just break code (since it's a clean break) but apparently that's unacceptable to the users. --Guido van Rossum (home page: http://www.python.org/~guido/)

In that case there's really no point to changing the grammar now either, right? This is a context-sensitive warning. In that case, I'd like us to start warning about (at least) "non-expression" uses of 'True' and 'False' too. That is, if we can't change the generated code yet anyway, we may as well start warning about everything we *intend* to make keywords someday. The same context-senstive checks are needed for each such name.

Good point. Same question. Any takers? --Guido van Rossum (home page: http://www.python.org/~guido/)

Wouldn't it be nice if there were a collection of programs (perhaps those owned by registered paranoids, but whatever) that you could use to get statistics about breakage under some syntax change?

Yes. :-) But I think syntax changes aren't even the tip of the iceberg -- semantic changes are relevant too.

Clearly extending this through to run-time would add a huge testing infrastructure that would need maintaining, but allowing people to add their own codes to the syntax-checker base might mollify them a bit about the prospect of future language change.

Call it the PythOnGuard (TM) database. :-) - if your programs would break you'll be mailed before it's committed to production.

Sounds like some kind of huge test suite. The problem is, there's always *something* that breaks... :-( --Guido van Rossum (home page: http://www.python.org/~guido/)

Sorry, this is the syntax department. I think you need to speak to Mr. Tester in room 104.

I think I'll continue arguing in my spare time. :)

...

...

Clearly extending this through to run-time would add a huge testing infrastructure that would need maintaining, but allowing people to add their own codes to the syntax-checker base might mollify them a bit about the prospect of future language change.

Call it the PythOnGuard (TM) database. :-) - if your programs would break you'll be mailed before it's committed to production.

Sounds like some kind of huge test suite. The problem is, there's always *something* that breaks... :-(

OK. The hugeness of the test suite was precisely what made me stick to syntax. Do you think quality would benefit by an enlargement of the test suite to non-distributed code?

The quality of what? We need lots more test, probably 40% of the C code isn't covered by the test suite.

Couple it with a few polling buttons from "I wouldn't mind fixing this breakage" to "I'ma get my gun and come looking for you" and you might obtain a measure of resistance *from people whose code would actually be broken*. Clearly this should not necessarily be the arbiter of development, but it might allow you to tell people whose code hadn't broken so far to just PythOnGuard it and not complain until something *did* break.

I think it's too simplistic to expect much out of testing a large body of real life code. Much of the expected breakage only shows up when an application is run on real-life data. Maybe running PyChecker on a large body of code would be a good idea, except that then the code would first have to be made PyChecker-clean -- and that's a moving target.

Also good for news releases: "What breaks in this version?" (I can see Paul Rubin loving that one).

It's easy to list the things that could potentially break. It's usually impossible to find whether they are used in a particular body of code without running a thorough test suite *for that code*. --Guido van Rossum (home page: http://www.python.org/~guido/)

Patrick> ... what about something like: >>> None.__class__.__base__ <type 'object'> Patrick> Or does this not apply because None is at the front? I suspect it does apply. I doubt it's used much, but what would happen to types.NoneType? If None is a keyword, how will you generate it, or will it disappear when that happens? Skip

Tim> I don't see the problem here. It's currently generated via Tim> NoneType = type(None) duh... Sorry, my bad. Use of the term "keyword" was causing my sleep-deprived brain to think it couldn't be used in expressions. Obviously, that's not the case. All the current keywords in the language either identify statements statements (def, print, continue) or operators (or, and, in). Elevating "None" to keyword status would create a third type keyword. Maybe it should be referred to as a "reserved identifier" or something like that. Skip

[Skip Montanaro]

... All the current keywords in the language either identify statements statements (def, print, continue) or operators (or, and, in). Elevating "None" to keyword status would create a third type keyword. Maybe it should be referred to as a "reserved identifier" or something like that.

+1 from me.

[Guido van Rossum]

Nothing would change. None would be a literal keyword, but its value in an expression would be the same as before, so None.__class__ would work. Compare "foo".__class__.

...

...

''.__class__ <type 'str'> [].__class__ <type 'list'> {}.__class__ <type 'dict'> True.__class__ <type 'int'> 2.__class__ File "<input>", line 1 2.__class__ ^ SyntaxError: invalid syntax 2.2.__class__ <type 'float'> 2L.__class__ <type 'long'>

Looks like int is one of the few literals that doesn't allow this. But floats and longs do. Hmmm. Just FYI. ;-) --- Patrick K. O'Brien Orbtech

...

dotted_name: NAME ('.' NAME)*

[Patrick K. O'Brien]

I know nothing about the compiler code, but what about something like:

...

...

...

None.__class__.__base__ <type 'object'>

Or does this not apply because None is at the front?

Yup, good point, 'None' does make sense as the first NAME in the dotted_name production.

...

...

...

...

"GvR" == Guido van Rossum <guido@python.org> writes:

...

Yup, good point, 'None' does make sense as the first NAME in the dotted_name production.

GvR> No, dotted_name is only used in import statements. None is a valid package name. Jeremy

[Tim]

Yup, good point, 'None' does make sense as the first NAME in the dotted_name production.

[Guido]

No, dotted_name is only used in import statements.

[Jeremy]

None is a valid package name.

But it won't be when None becomes a keyword. We shouldn't change the grammar now anyway, so let's fight instead about which names *should* become keywords eventually (== which should generate warnings in 2.3 for non-rvalue uses). import string import __builtin__ candidates = [k for k in dir(__builtin__) if k[0] in string.ascii_uppercase and not k.endswith('Error') and not k.endswith('Warning')] for k in candidates: print k displays: Ellipsis Exception False KeyboardInterrupt None NotImplemented StopIteration SystemExit True in current CVS. Ellipsis False None NotImplemented True are the builtin singleton data constants, so I'm +1 on those. The others are exception classes; -0 from me.

...

Ellipsis False None NotImplemented True

are the builtin singleton data constants, so I'm +1 on those.

[Guido]

I'll go along with True and False, since I expect that these will be frequently used (though still not as frequently as None).

In a dynamic sense I bet they'll be used more often than None, in part because I'll steer people toward it <wink>. For years I've been steering people to manage their "dicts used as sets" via: for element in some_candidate_producer: dict_used_as_set[element] = 1 instead of using "None", for two reasons: first, I view the dict as mapping elements to "true", as in "yes, I'm present". The introduction of bool makes this very natural, as in ispresent = dict_used_as_set.get(element, False) The guts of loops like that have a high dynamic count. Second, using "1" instead of "None" is significantly faster today, and that can be important in short scripts doing data crunching (i.e., the dict setup can account for the bulk of such a program's runtime, and the faster string-keyed dict guts get, the more significant the fixed overhead of doing the "= None/1/True" part). Objective statistics <wink>: A module I've been writing for my own fun is up to 329 lines now. It contains "None" 8 times, and "True" and "False" 7 times each. The Nones are mostly used as argument defaults (so evaluated once per program run), and in one-time-per-call "was an argument specified?" tests at function starts. One None is used in an inner loop, in an assert that a certain value is not None. The dynamic count of True and False evaluations is going to swamp the dynamic count of None evaluations in this program.

Ellipsis is very rarely used (only by numeric code implementing multi-dimensional arrays), NotImplemented is used somewhat more but still pretty esoteric. So I'm -1 on those.

Fine by me. So is this consensus? Start warning for non-rvalue uses of None False True and that's it for 2.3.

Yes. I don't need no steenkin' consensus. :-)

Jeremy has started a patch: http://python.org/sf/549213

Too bad.. it was mostly working. :-) Anything else in the top of your mind!? -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ]

...

...

...

...

"GN" == Gustavo Niemeyer <niemeyer@conectiva.com> writes:

...

Note too that we can't change the generated code at all for 2.3 unless under the control of a new future statement, else existing None-abusing code could break.

GN> I'm trying to eat the cake and have it too, by detecting if None GN> is being abused and disabling the change of LOAD_GLOBAL to GN> LOAD_CONST. If it works (and probably will), we may then choose GN> to include warnings, keep it that way forever, or whatever. This can't be detected statically, because a global could be introduced dynamically: foo.py: def f(): return None

...

...

import foo foo.None = 12 foo.f() 12

Jeremy

[Guido]

...

atom: '(' [testlist] ')' | '[' [listmaker] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING+ | 'None'

It needs to add 'None' as an alternative to all other places where currently NAME occurs.

[Greg E]

Wait a minute, you're going about this all wrong!

Instead of making None a reserved word, why not just make it a non-reserved keyword when used as a variable name?

What's that?

Then you wouldn't have to change the grammar at all, and there would be no problem with the use of None in other contexts, such as attribute names.

Unfortunately, what's a variable in one context is an attribute in another. I don't think I *want* people to use None as an attribute name (or in any other context). --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido:

I don't think I *want* people to use None as an attribute name (or in any other context).

Okay, everything's fine, then. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

...

...

...

...

"FLD" == Fred L Drake, <fdrake@acm.org> writes:

FLD> Jeremy Hylton writes:

...

Does the parser prevent assignment to None? Or does the compiler need to check for the use of None in an assignment?

FLD> Inside functions, you don't even need that. You just need to FLD> use LOAD_CONST (or whatever its called now), since None is FLD> entry 0 in the constants table, used by the implied "return FLD> None" when control drops off the end. Doing this in function FLD> buys most of the performance. I don't understand what you mean, but I'll try to reply anyway :-). I assume LOAD_NONE will eliminate the need for LOAD_CONST 0 (None). Instead of this: case LOAD_CONST: x = GETCONST(oparg); Py_INCREF(x); PUSH(x); goto fast_next_opcode; we'd have this: case LOAD_NONE: Py_INCREF(Py_None); PUSH(Py_None); goto fast_next_opcode; It's probably a wee bit faster and it makes the bytecode smaller, because you don't need None in co_consts and you don't need an argument to the bytecode. Based on my cycle counter measurements before the conference, I suspect the performance impact is, well, negligible. Jeremy

Hi Fred!

Regarding changing LOAD_CONST 0 to LOAD_NONE, yes. What's more interesting are the changes of LOAD_GLOBAL 'None' to one of LOAD_CONST 0 or LOAD_NONE. That could be changed to use LOAD_CONST 0 *now*, without adding a new bytecode, and we could get a better idea of how much performance it actually buys us in practice, since we get rid of two dict lookups (globals & builtins).

It's clear now.. thanks!

That doesn't address the deprecation cycle, but it would be nice to see what the change would buy us.

I'm going to work on it, and come back with some results. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ]

I meant two say things, and tried to be clever at the expense of being understood :-(. I think the cleanest way to handle None as a keyword is via a LOAD_NONE opcode. LOAD_NONE would replace two different mechanisms that the compiler currently uses -- LOAD_GLOBAL 'None' and LOAD_CONST 0 (None). While I think LOAD_NONE would result in a tidy state of affairs, and I think it would be marginally faster, I don't think the performance effect would be measurable on any interesting benchmark. I don't care how much faster it is, though. Jeremy

Ellipsis then? It's probably used /much/ less than None, but still.

Not worth it (it's a fairly big hack in the compiler). --Guido van Rossum (home page: http://www.python.org/~guido/)

...

...

...

...

"GvR" == Guido van Rossum <guido@python.org> writes:

GvR> I just realized there's another use for None as an identifier GvR> that is currently totally legal and which would become GvR> illegal: it's conceivable that someone would use None as an GvR> attribute name, e.g. | class C: | def None(self): pass GvR> C().None() That reminds me, I wonder if we should dust off the old Jython idea of allowing keywords in locations where their non-keywordness is obvious? C().print()-ly y'rs, -Barry