[Python-ideas] Maybe/Option builtin

[Devin Jeanpierre]

On Wed, May 21, 2014 at 5:24 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Wed, May 21, 2014 at 05:11:19PM -0700, Devin Jeanpierre wrote:
>
>> Python has no way to represent "I have a value, and that value is None".
>
> Sure it has.  'a' in x and x['a'] is None

Hm, why do you bring this up? Do you think I didn't know about this?

While we're at it, you could just implement the Option type, and then
the problem is also solved!  I am sorry I did not communicate what I
meant to say.

>> e.g. what is the difference between x.get('a') and x.get('b') for x ==
>> {'a': None} ?
>
> dict.get is explicitly designed to blur the distinction between "I have
> a key and here is its value" and "I may or may not have a key, it
> doesn't matter which, return its value or this default regardless". (The
> default value defaults to None.) If you care about the difference, as
> most people do most of the time, you should avoid the get method and
> call x['a'] directly.

I was not  trying to criticize dict.get, I was noting the same
property you have agreed with: it blurs the distinction.

That distinction does not have to be blurred, and for some other APIs,
must not be blurred. I appreciate that dict.get is not the former
case, so, for the sake of making the discussion about something
important, let's choose such an API: imagine that we steal Guido's
time machine to go into the past and redesign next(). With next(), it
is absolutely vital that we are able to know when the iterator has
stopped vs when the iterator has produced a value.

API ideas:

- Use next's current primary API: raise an exception if the iterator
has ended. This is a very good API, except that if the caller forgets
to catch the exception, this can result in generators or iterators up
the call stack spontaneously and silently stopping, which can be hard
to debug - and, unfortunately, is almost never actually your fault.

- Use next's current alternate API: give next a default parameter
which is returned when the iterator stops, and follow Amber's
solution. However, you have to be careful that this sentinel value
won't ever be wanted as a return value, and that you don't
accidentally match things against the sentinel by mistake: i.e. it has
to be defined inline with a unique object and checked by identity.

- make next() return None when the generator is exhausted. How do you
differentiate None as a value vs None as a no-result sentinel? ideas:

  - give iterators a .is_exhausted attribute which is True if the None
it returned indicates end-of-iterator, False otherwise. This is
acceptable, but now, if the caller forgets to check for exhaustion,
they start silently giving None as values. This might also be hard to
debug.

  - make all values be wrapped in Some(v), so that next(it) returns
either None or Some(v). This can cause problems if you forget to
actually unwrap the Some to get at the value. Unlike the above
problems, however, such things are very easy to debug - you will get a
TypeError when you add Some(3) + 5. And, similarly, if you forget to
check for None, this will be an error when you treat it like a Some
value (e.g. 'NoneType' object has no attribute 'unwrap'). So this
option is pretty resilient against mistakes, even in a dynamically
typed language.

Stylistically, I feel like exceptions and option types give you a
clean separation of cases, where the other two options just munge data
together and let you sort it out afterwards - this feels ugly to me,
and would seem to push programmers towards not caring about the value.
(Probably not a good idea when the cases are important.)  Similarly,
it's easy to forget to catch an exception, especially since quite
often people use next in a situation where they think it will never
raise an exception. And in the particular case of StopIteration,
unlike most other exceptions, it's frequently silently caught by
various functions that might sit between you and where the exception
was raised, meaning that you might not easily identify the cause of
the problem -- especially in a badly tested codebase. (And what other
kind of codebase would miss this error? ;).  So for me personally, if
Python had all these options and they were all equally easy and
natural, I would probably choose option types. This is of course
subjective, but I hope it sort of ties together everything and
explains the utility.

If you value certain things and have a certain sense of elegance, you
might want to use option types instead of some of the alternatives, in
some circumstances, even in a dynamically typed language. Though, I am
not trying to suggest that you personally would ever want to use it,
or that it's even particularly Pythonic. I personally feel like
probably Some/None would be awkward/unpythonic to retrofit (although I
think sum types in general belong in the language, which is why I am
talking about this at all - awareness is important!).

Does that help explain how Some/None can sometimes be useful or
desirable (to some people) even without static typing?

-- Devin