I came across this just processing JSON data - flattening a list of 'deals' into a list of individual 'trades'. Flattening nested lists is not uncommon. And yes sum seemed the natural way to do it at the time.

Yes, you're right some people might find it convenient and use it in the REPL. It just seems like a decision has been made that that's the wrong way to do it. Hence no fast paths or optimisations.

As others have mentioned, you -could- go the opposite way and optimise for some builtins - at the expense of surprise for user-defined types. I think it would be good to make it harder for people to get it wrong.

What kind of evidence are you taking about here? There are several bug reports. Code audits? Tricky to obtain, but I could try if this is of genuine interest.

On Wed, 16 Jun 2021, 11:16 pm Steven D'Aprano, <steve@pearwood.info> wrote:
On Wed, Jun 16, 2021 at 02:50:06PM +0000, Oliver Margetts wrote:

> Hi all, I came across the following performance issue with the sum function
> while summing lists: https://bugs.python.org/issue18305 It's been discussed
> previously in this list and other issues.

Did you actually use sum to concatenate lists in real code, or is this a
theoretical issue?

What circumstances do you have where you want to concatenate a large
number of lists and thought that sum was an appropriate way to do it?


[...]
> Having non-linear complexity is not a suitable way to discourage this
> behaviour though.

We didn't put non-linear complexity in as a way to discourage the use of
sum. The non-linear complexity comes about as a natural consequence of
the behaviour of lists, it is not a "passive-aggressive runtime".

There is no practical way for us to detect ahead of time all imaginable
types where repeated addition has non-linear behaviour. And although sum
was intended for use only with numbers, there is no *hard* rule in
Python that people can't use functions for unintended purposes that make
sense to them.

It may even be that there are people who consider the convenience of sum
worth it even for lists, since quadratic runtime behaviour is still fast
for small enough N. I sometimes use it myself, in the REPL, to merge a
handful of lists. If it takes 100 ms instead of a microsecond,
I don't even notice.

The performance error for strings should be considered an anomaly, not a
feature to be extended to anything that could be used, or misused, with
non-linear behaviour. At the very least, we would probably need to see
evidence that this issue (poor performance) is a widespread problem
before breaking code which works fine for small N.


--
Steve
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FSD5TENDE4RLESDRPWN7WJ3WJMFPRASC/
Code of Conduct: http://python.org/psf/codeofconduct/