__missing__ object/keyword

Several times I find myself using the following idiom: _Missing = object() # sentinel def foo(x, y=_Missing): if y is _Missing: do_this else: do_that The reason for using a "private" sentinel is that any python object, including None, might be a valid argument. Another option is using *args or **kwds instead of y but that obfuscates unnecessarily the function signature. It would be nice if a new object or keyword, say __missing__, was introduced as a canonical way to address this common scenario. Specifically, the only valid usages of __missing__ would be: 1. As a default argument in a callable. 2. In identity tests: <var> is __missing__ Anything else would raise either a SyntaxError (e.g. `x = __missing__`) or a RuntimeError/TypeError (e.g. `x = y` if y is __missing__). Only the interpreter could assign __missing__ to a name when binding objects to formal parameters. If this was to be accepted, a further generalization could be to allow the `var is __missing__` expression even if `var` is not a formal parameter. This would be equivalent to: try: var except NameError: expr = True else: expr = False Thoughts ? George

On Thu, Nov 6, 2008 at 12:31 PM, George Sakkis <george.sakkis@gmail.com> wrote:
By analogy with the necessity of __missing__ when we already have None, what if I have something that calls foo() and I want __missing__ to be a valid argument? The problem is that an infinite series of higher-level None-like objects is necessary. Is the one extra line to define `sentinel` really so bad? Cheers, Chris -- Follow the path of the Iguana... http://rebertia.com

On Thu, Nov 6, 2008 at 3:43 PM, Chris Rebert <cvrebert@gmail.com> wrote: On Thu, Nov 6, 2008 at 12:31 PM, George Sakkis <george.sakkis@gmail.com>
You won't be able to; by definition __missing__ cannot be a valid argument.
Is the one extra line to define `sentinel` really so bad?
Certainly it's not a showstopper by any means, but it's more than one extra line. If __missing__ is a regular object, a client may "cheat" by passing it explicitly. It can also help introspection and documentation tools (e.g. epydoc) and IDEs. George

On Thu, Nov 6, 2008 at 3:31 PM, George Sakkis <george.sakkis@gmail.com> wrote:
It would be nice if a new object or keyword, say __missing__, was introduced as a canonical way to address this common scenario.
Thoughts ?
This has been proposed and shot down before: http://mail.python.org/pipermail/python-dev/2007-May/073439.html The older proposal didn't give __missing__ the special run-time and compile-time error behaviors, but this actually just makes things worse. Why complicate the language merely to save a single idiomatic line of code dealing with a relatively infrequent use-case? Greg F

On Thu, Nov 6, 2008 at 3:47 PM, Greg Falcon <veloso@verylowsodium.com>wrote: On Thu, Nov 6, 2008 at 3:31 PM, George Sakkis <george.sakkis@gmail.com>
I agree that it doesn't make sense without special compile/runtime behavior; that would be like adding a second None.
I can't tell how much it would complicate the implementation of the language but the semantics are pretty straightforward. But you have a point there, None is more often than not an adequare sentinel, so if implementing __missing__ requires significant work it's probably not worth the effort. George

On 6 Nov 2008, at 20:31, George Sakkis wrote:
Here's a proof-of-concept decorator version: from inspect import getargspec def tell_missing(f): argnames = getargspec(f)[0][1:] def decorated(*args, **kwargs): missing = set(argnames[len(args):]) - set(kwargs) return f(missing, *args, **kwargs) return decorated
-- Arnaud

The problem with using a value like None as some have proposed is that None may be a valid parameter value. Thus the only safe value is a special value like a throwaway object. There are numerous problems with this including that it's not safe: I might not check for the missing value in some place and end up passing the throwaway object to another function which then passes it as one of the optional values. There *is* one guaranteed way of ensuring that I can't use a variable's value: leaving it unbound. That is, to support this, we would add a new syntax something like what George proposed with slightly different semantics: def foo(x, y=__unbound__): where if y is omitted from the call, then y is unbound. This is slightly different than what would happen if y were left out of the parameter list as in that case, y could reference a global of the same name. In this case, y can only reference an unbound local. No other changes are required to use this. If I don't check whether or not y is bound, then I'll get a NameError when I try to use it, just as with any other unbound variable. If this proposal were to be seriously considered, there are of course alternative syntaxes that could be considered, like using missing, __missing__ or def foo(x, ?y) but that's getting ahead. I agree it would be nice to have a way to check whether a variable is unbound without writing a multi-line try/except, but I think that can be discussed as a different issue. --- Bruce

This seems to be a solution in search of a problem. If you handle sentinel default values early in a function then they are rarely problematic. Michael Foord http://www.ironpythoninaction.com On 8 Nov 2008, at 05:37, "Bruce Leban" <bruce@leapyear.org> wrote:

On Sat, Nov 8, 2008 at 6:36 AM, Michael <fuzzyman@gmail.com> wrote: This seems to be a solution in search of a problem. If you handle sentinel
default values early in a function then they are rarely problematic.
The general consensus so far seems to be that None and sentinel values are good enough in practice and adding a new feature is over-engineering, but I wouldn't go as far as calling it a solution in search of problem. To me it looks similar to the motivation for keyword-only arguments; as the PEP mentions, one can effectively have keyword-only arguments with the idiom: def compare(a, b, *ignore, key=None): if ignore: # If ignore is not empty raise TypeError Thankfully, this didn't prevent the PEP from being accepted. George

On Sat, Nov 8, 2008 at 12:37 AM, Bruce Leban <bruce@leapyear.org> wrote: There *is* one guaranteed way of ensuring that I can't use a variable's
I like that, it's more straightforward than checking for identity with a __missing__ pseudo-object. if this was to be accepted, checking whether a name is bound would be more common than now, so it might be worthwhile considering a new isbound('var') builtin as a more explicit (and perhaps faster) alternative of "if 'var' in locals()". If this proposal were to be seriously considered, there are of course
alternative syntaxes that could be considered, like using missing, __missing__ or def foo(x, ?y) but that's getting ahead.
+1 for the alternative syntax, reads much better and it's at least as intuitive as *args and **kwds. George

Bruce Leban wrote:
The problem with using a value like None as some have proposed is that None may be a valid parameter value.
It usually is not, and when it is, a private object suffices.
Thus the only safe value is a special value like a throwaway object.
Yes, a private instance of object().
There *is* one guaranteed way of ensuring that I can't use a variable's value: leaving it unbound.
This Perl, etc. concept does not fit Python's name-object model. A Python mappings and sequences cannot have a key or index without a value. To propose otherwise is a major conceptual change. tjr

On Sat, Nov 8, 2008 at 12:19 PM, Terry Reedy <tjreedy@udel.edu> wrote:
What I would expect (as a programmer) is that when the parameter is omitted, the variable is simply not bound and therefore it's omitted from the mapping, not added to the mapping with a special magic value meaning unbound. However, there is an implementation complication due to the way variable scoping works in Python: if a variable is left unbound in the function locals, then references to that variable name would pick up the global if it exists which would definitely be a surprising result. So I agree that there would definitely be some complexities to implement this. --- Bruce

Bruce Leban wrote:
No, it wouldn't: Python 2.3 (#1, Aug 5 2003, 15:52:30) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information.
Python already has the machinery to make this work. All that's needed is a way to leave parameters unbound, and a syntax for asking "Would I get a NameError if I were to refer to this name now?" Perhaps something like def f(x?): if x?: ... -- Greg

On Thu, Nov 6, 2008 at 3:31 PM, George Sakkis <george.sakkis@gmail.com> wrote:
Several times I find myself using the following idiom:
_Missing = object() # sentinel
And I often catch myself trying to write def foo(x, y=object()): if y is ... # oops, I need to define the y *outside* the function. But once I've defined it outside the function, it gets tempting to reuse it for something else, such as another method in the same class with a similar need. And eventually, it may end up getting promoted to a real object, like None... I'll freely grant that this is a small use case, but ... it is one I would appreciate seeing handled better.
I assume you could also write 2a. var is not __missing__
I wouldn't do this, at least at first. I see now that it is effectively a way of checking for run-time parameters added to builtins, which can be used to control imports, etc. But that wasn't obvious as first. -jJ

On Thu, Nov 6, 2008 at 12:31 PM, George Sakkis <george.sakkis@gmail.com> wrote:
By analogy with the necessity of __missing__ when we already have None, what if I have something that calls foo() and I want __missing__ to be a valid argument? The problem is that an infinite series of higher-level None-like objects is necessary. Is the one extra line to define `sentinel` really so bad? Cheers, Chris -- Follow the path of the Iguana... http://rebertia.com

On Thu, Nov 6, 2008 at 3:43 PM, Chris Rebert <cvrebert@gmail.com> wrote: On Thu, Nov 6, 2008 at 12:31 PM, George Sakkis <george.sakkis@gmail.com>
You won't be able to; by definition __missing__ cannot be a valid argument.
Is the one extra line to define `sentinel` really so bad?
Certainly it's not a showstopper by any means, but it's more than one extra line. If __missing__ is a regular object, a client may "cheat" by passing it explicitly. It can also help introspection and documentation tools (e.g. epydoc) and IDEs. George

On Thu, Nov 6, 2008 at 3:31 PM, George Sakkis <george.sakkis@gmail.com> wrote:
It would be nice if a new object or keyword, say __missing__, was introduced as a canonical way to address this common scenario.
Thoughts ?
This has been proposed and shot down before: http://mail.python.org/pipermail/python-dev/2007-May/073439.html The older proposal didn't give __missing__ the special run-time and compile-time error behaviors, but this actually just makes things worse. Why complicate the language merely to save a single idiomatic line of code dealing with a relatively infrequent use-case? Greg F

On Thu, Nov 6, 2008 at 3:47 PM, Greg Falcon <veloso@verylowsodium.com>wrote: On Thu, Nov 6, 2008 at 3:31 PM, George Sakkis <george.sakkis@gmail.com>
I agree that it doesn't make sense without special compile/runtime behavior; that would be like adding a second None.
I can't tell how much it would complicate the implementation of the language but the semantics are pretty straightforward. But you have a point there, None is more often than not an adequare sentinel, so if implementing __missing__ requires significant work it's probably not worth the effort. George

On 6 Nov 2008, at 20:31, George Sakkis wrote:
Here's a proof-of-concept decorator version: from inspect import getargspec def tell_missing(f): argnames = getargspec(f)[0][1:] def decorated(*args, **kwargs): missing = set(argnames[len(args):]) - set(kwargs) return f(missing, *args, **kwargs) return decorated
-- Arnaud

The problem with using a value like None as some have proposed is that None may be a valid parameter value. Thus the only safe value is a special value like a throwaway object. There are numerous problems with this including that it's not safe: I might not check for the missing value in some place and end up passing the throwaway object to another function which then passes it as one of the optional values. There *is* one guaranteed way of ensuring that I can't use a variable's value: leaving it unbound. That is, to support this, we would add a new syntax something like what George proposed with slightly different semantics: def foo(x, y=__unbound__): where if y is omitted from the call, then y is unbound. This is slightly different than what would happen if y were left out of the parameter list as in that case, y could reference a global of the same name. In this case, y can only reference an unbound local. No other changes are required to use this. If I don't check whether or not y is bound, then I'll get a NameError when I try to use it, just as with any other unbound variable. If this proposal were to be seriously considered, there are of course alternative syntaxes that could be considered, like using missing, __missing__ or def foo(x, ?y) but that's getting ahead. I agree it would be nice to have a way to check whether a variable is unbound without writing a multi-line try/except, but I think that can be discussed as a different issue. --- Bruce

This seems to be a solution in search of a problem. If you handle sentinel default values early in a function then they are rarely problematic. Michael Foord http://www.ironpythoninaction.com On 8 Nov 2008, at 05:37, "Bruce Leban" <bruce@leapyear.org> wrote:

On Sat, Nov 8, 2008 at 6:36 AM, Michael <fuzzyman@gmail.com> wrote: This seems to be a solution in search of a problem. If you handle sentinel
default values early in a function then they are rarely problematic.
The general consensus so far seems to be that None and sentinel values are good enough in practice and adding a new feature is over-engineering, but I wouldn't go as far as calling it a solution in search of problem. To me it looks similar to the motivation for keyword-only arguments; as the PEP mentions, one can effectively have keyword-only arguments with the idiom: def compare(a, b, *ignore, key=None): if ignore: # If ignore is not empty raise TypeError Thankfully, this didn't prevent the PEP from being accepted. George

On Sat, Nov 8, 2008 at 12:37 AM, Bruce Leban <bruce@leapyear.org> wrote: There *is* one guaranteed way of ensuring that I can't use a variable's
I like that, it's more straightforward than checking for identity with a __missing__ pseudo-object. if this was to be accepted, checking whether a name is bound would be more common than now, so it might be worthwhile considering a new isbound('var') builtin as a more explicit (and perhaps faster) alternative of "if 'var' in locals()". If this proposal were to be seriously considered, there are of course
alternative syntaxes that could be considered, like using missing, __missing__ or def foo(x, ?y) but that's getting ahead.
+1 for the alternative syntax, reads much better and it's at least as intuitive as *args and **kwds. George

Bruce Leban wrote:
The problem with using a value like None as some have proposed is that None may be a valid parameter value.
It usually is not, and when it is, a private object suffices.
Thus the only safe value is a special value like a throwaway object.
Yes, a private instance of object().
There *is* one guaranteed way of ensuring that I can't use a variable's value: leaving it unbound.
This Perl, etc. concept does not fit Python's name-object model. A Python mappings and sequences cannot have a key or index without a value. To propose otherwise is a major conceptual change. tjr

On Sat, Nov 8, 2008 at 12:19 PM, Terry Reedy <tjreedy@udel.edu> wrote:
What I would expect (as a programmer) is that when the parameter is omitted, the variable is simply not bound and therefore it's omitted from the mapping, not added to the mapping with a special magic value meaning unbound. However, there is an implementation complication due to the way variable scoping works in Python: if a variable is left unbound in the function locals, then references to that variable name would pick up the global if it exists which would definitely be a surprising result. So I agree that there would definitely be some complexities to implement this. --- Bruce

Bruce Leban wrote:
No, it wouldn't: Python 2.3 (#1, Aug 5 2003, 15:52:30) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information.
Python already has the machinery to make this work. All that's needed is a way to leave parameters unbound, and a syntax for asking "Would I get a NameError if I were to refer to this name now?" Perhaps something like def f(x?): if x?: ... -- Greg

On Thu, Nov 6, 2008 at 3:31 PM, George Sakkis <george.sakkis@gmail.com> wrote:
Several times I find myself using the following idiom:
_Missing = object() # sentinel
And I often catch myself trying to write def foo(x, y=object()): if y is ... # oops, I need to define the y *outside* the function. But once I've defined it outside the function, it gets tempting to reuse it for something else, such as another method in the same class with a similar need. And eventually, it may end up getting promoted to a real object, like None... I'll freely grant that this is a small use case, but ... it is one I would appreciate seeing handled better.
I assume you could also write 2a. var is not __missing__
I wouldn't do this, at least at first. I see now that it is effectively a way of checking for run-time parameters added to builtins, which can be used to control imports, etc. But that wasn't obvious as first. -jJ
participants (12)
-
Arnaud Delobelle
-
Bruce Leban
-
Chris Rebert
-
George Sakkis
-
Greg Ewing
-
Greg Falcon
-
Jim Jewett
-
Leif Walsh
-
Mathias Panzenböck
-
Michael
-
Raymond Hettinger
-
Terry Reedy