On Tue, Dec 29, 2015 at 3:08 PM, Franklin? Lee wrote:

What do people think about having `max` and `min` ignore `None`?

Examples: max(1, None) == 1 min(1, None) == 1 max([1, None]) == 1 max(None, None) == max() or max(None, None) == max([])

(The last one currently throws two different errors.)

What you could do is simply filter them out: def max_without_none(seq): return max(item for item in seq if item is not None) Would that do what you need? ChrisA

On Dec 28, 2015, at 20:08, Franklin? Lee wrote:

This change would allow one to use `None` as a default value.

Actually, it might be useful to allow a default value in general. (In typed functional languages, you often specify a default, or use a type that has a default value, so max(list[A]) can always return an A.) Then again, you can write this pretty easily yourself: def my_max(iterable, *, default=_sentinel): try: return max(Iterable) except WhateverEmptyIterableRaises: if default is _sentinel: raise return default

I was hoping that my message was clear enough that I wouldn't get suggestions of alternatives. I know how to do without it. Take this example: from collections import defaultdict from math import inf def maxes(lst): bests = defaultdict(lambda: -inf) for x, y in lst: bests[x] = max(bests[x], y) return bests The proposed change would only save me an import, and I could've used `inf = float('inf')` instead. from collections import defaultdict def maxes(lst): bests = defaultdict(lambda: None) for x, y in lst: bests[x] = max(bests[x], y) return bests On Mon, Dec 28, 2015 at 11:52 PM, Terry Reedy wrote:

On 12/28/2015 11:08 PM, Franklin? Lee wrote:

...

What do people think about having `max` and `min` ignore `None`?

Examples: max(1, None) == 1 min(1, None) == 1

This amounts to saying that the comparisions 1 < None and 1 > None are both defined and both True.

Not exactly. max(1, None) == max(None, 1) == 1 There is no definable comparison to None which allows both max and min to return the correct value.

rewrite this as

def my_best(iterable): it = iter(iterable) try: best = it.next() except StopIteration: raise ValueError('Empty iterable has no maximum') for x in it: if x > best: best = x

Well, `my_best = max` is the cleanest way. It's not the point.

...

Currently, you would initialize `best` to the first element,

Since an empty iterable has no max (unless one wants to define the equivalent of float('-inf') as a default), initializing with the first element is the proper thing to do.

Mathematically, the max of the empty set is the min of the ambient set. So the max of an empty collection of natural numbers is 0, while the max of an empty collection of reals is -inf. Of course, Python doesn't know what type of elements your collection is expected to have, so (as Nick said) you would manually specify the default with a keyword argument. But that's not the point.

...

I'm concerned about this silencing some bugs which would have been caught before. I'm also worried about whether it would make sense to people learning Python.

None currently means 'no value', which means that most operations on None are senseless.

No value, like a lack of something to consider in the calculation of the maximum? PS: This change would also allow one to use a `key` function which returns None for an object that shouldn't be considered. Now _that_ might be more useful. But again, I know how to deal without it: have the key function return `inf` or `-inf` instead. I'm asking if using `None` could become the "one obvious way to do it". There is semantic meaning to initializing `best = None`, after all: "At this point, there is no best yet."

On 29 December 2015 at 21:22, Steven D'Aprano wrote:

What if the builtin max and min remained unchanged, but we added variants of them to the statistics module which treated None as a missing value, to be either ignored or propagated, as R does?

...

...

df = pandas.DataFrame([1, 2, 3, None, float("nan")]) df.min() 0 1

...

...

df.min(skipna=False) 0 NaN

If the statistics module were to start borrowing selected concepts from R, it makes sense to me to look at how those have been translated into the Python ecosystem by NumPy/SciPy/pandas first. In the case of min/max, the most relevant APIs appear to be: pandas.DataFrame.min pandas.DataFrame.max numpy.amin numpy.amax numpy.nanmin numpy.nanmax The pandas variants support a "skipna" argument, which indicates whether or not to ignore missing values (e.g. None, NaN). This defaults to true, so such null values are ignored. If you set it to False, they get included and propagate to the result: dtype: float64 dtype: float64 For NumPy, amin and amax propagate NaN/None, while nanmin/nanmax are able to filter out floating point NaN values, but emit TypeError if asked to cope with None as a value. I think the fact both NumPy and pandas support R-style handling of min() and max() counts in favour of having variants of those with additional options for handling missing data values in the standard library statistics module. Regards, Nick. P.S. Another option might be to consider the question as part of a general "data cleaning" strategy for the statistics module, similar to the one discussed for pandas at http://pandas.pydata.org/pandas-docs/stable/missing_data.html Even if the statistics module itself doesn't provide the tools to address those problems, it could provide some useful pointers on when someone may want to switch from the standard library module to a more comprehensive solution like pandas that better handles the messy complications of working with real world data (and data formats). -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Dec 29, 2015, 11:41 AM Chris Barker - NOAA Federal < chris.barker@noaa.gov> wrote:

but emit TypeError if

...

asked to cope with None as a value.

Well, sort of. Numpy arrays are homogenous, you can't have a None in an array ( other than an object style). All the Numpy "ufuncs" create an array from the input first -- that's where you get your ValueError.

But the Numpy experience is informative -- there have been years of " we need a better masked array" discussions, but no consensus on what it should be.

For floats, NaN can be used for missing values, but there is no such value for integers, and each use case has a sufferer end "obvious" interpretation. That's why it's explicit what you want with the nan* functions.

I don't think python should decide for users what None means in this context.

None is obviously the sound of one hand clapping. When you understand its proper use, you become Enlightened.

-CHB

...

I think the fact both NumPy and pandas support R-style handling of min() and max() counts in favour of having variants of those with additional options for handling missing data values in the standard library statistics module.

NumPy and Pandas have a slightly different audience than Python core. The scientific community often veers more practical than pure, in some cases to the detriment of code clarity.

Regards,

...

Nick.

P.S. Another option might be to consider the question as part of a general "data cleaning" strategy for the statistics module, similar to the one discussed for pandas at http://pandas.pydata.org/pandas-docs/stable/missing_data.html

I prefer this option. Why solve the special case of max/min when we can solve (or help solve) the general case of missing data. There's already the internal ``_coerce`` method. Maybe clean that up for public consumption, or something like it, adding drop-missing functionality? If that flies, then there might be room for an ``interpolate(sequence, method='linear')`` which would be awesome.

...

Even if the statistics module itself doesn't provide the tools to address those problems, it could provide some useful pointers on when someone may want to switch from the standard library module to a more comprehensive solution like pandas that better handles the messy complications of working with real world data (and data formats).

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Dec 30, 2015 at 11:44 PM Steven D'Aprano wrote:

On Thu, Dec 31, 2015 at 04:30:24AM +0000, Michael Selik wrote:

...

If that flies, then there might be room for an ``interpolate(sequence, method='linear')`` which would be awesome.

(I presume you're still talking about the statistics module here, not pandas.)

What did you have in mind?

While the scientific community is well-served by NumPy and Pandas, there are many users trying to do a lighter amount of data wrangling that does not include linear algebra. In my anecdotal experience, the most common tasks are: 1. drop records with missing/bad data 2. replace missing/bad values with a constant value 3. interpolate missing values with either a pad-forward or linear method While NumPy often has methods doing in-place mutation, the users I'm thinking of are generally not worried about memory size and would be better served by pure functions. Going back to the original topic of skipping None values. I'd like to add that many datasets use bizarre values like all 9s or -1 or '.' or whatever to represent missingness. So, I'm not confident there's a good general-purpose solution more simple than comprehensions.