Hello, python-ideas. Trying to cover everything in one shot, so the message is long, sorry. sum() is a great function. It is the "obvious way" to add things. Unfortunately sometimes it's slower than it could be. The problem is that code: sum([[1,2,3]]*1000000, []) takes forever to complete. Let's fix that! This problem was there since the sum() introduction, but it is almost unknown among python-beginners. When people look at sum(seq,start=0) signature they most probably expect it to be like this: def sum(seq, start = 0): for item in seq: start += item return start But it is not, because in cases like: empty = [] result = sum(list_of_lists, empty) such implementation would modify content of "empty". This use-case is known, it is checked in test_builtin.py, and explicitly mentioned in comments inside sum() code. So actually sum() looks like this: def sum(seq, start = 0): for item in seq: start = start + item return start it creates a copy of the partial result on every "start + item". What I suggest is instead of making a copy for every item make just one copy and reuse it as many times as needed. For example: def sum(seq, start = 0): start = start + seq[0] for item in seq[1:]: start += item return start Patch implementing this idea attached to issue18305. Patch is simple, it just rearranges existing code. It should work for python 2.7, 3.3 and hg-tip. Except sum() becoming faster there should be no behavior change. Can this implementation break anything? Advantages: * Performance boost is like 200000% [1], nothing else becomes slower Disadvantages (inspired by Terry J. Reedy): * Other pythons (pypy, jython, older cpython versions) do not have such optimisation, people will move code that depends on the internal optimization to pythons that do not have it. And they will complain about their program 'freezing'. * It discourages people from carefully thinking about whether they actually need a concrete list or merely the iterator for a virtual list. Alternatives to sum() for this use-case: * list comprehension is 270% slower than patched sum [2] * itertools.chain is 50%-100% slower than patched sum [3] Main questions: * Whether to accept the patch * If yes, whether it should go to the new version or to a bugfix release Alternatives to this patch: * Reject the idea as a whole. Intentionally keep the sum() slow. * Accept the idea, but reject this particular implementation, instead use something different, e.g. implement special case optimization or use different approach (I thought about start=copy.copy(start)) In my opinion (looking though python changelog) performance patches were accepted for bugfix releases many times before. So as long as this patch does not break anything, it may go even to python 2.7. I think bad performance is a bug that should be fixed. Rejecting performance fixes just because older versions don't have it would mean no performance improvements to anything, ever. Sum is a great basic function. It keeps the simple code simple. This patch puts it on par with more advanced features like itertools, and allows people to use sum for simple things and itertools for complex memory-efficient algorithms. Questions that may arise if the patch is accepted: * sum() was rejecting strings because of this bug. If the bug gets fixed should another patch allow sum() to accept strings? * maybe in some distant future drop the second parameter (or make it None by default) and allow calling sum for everything, making sum() "the one obvious way" to sum up things? It would be nice if sum "just worked" for everything (e.g. sum() of empty sequence would return None, i.e. if there's nothing to sum then nothing is returned). But I think it needs more work for that, because even with this patch sum() is still ~20 times slower than "".join() [4] That's all. Any suggestions welcome. -- [1] Python 2.7.5 Before patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 10 loops, best of 3: 885 msec per loop After patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 1000 loops, best of 3: 524 usec per loop [2] Python 2.7.5 with patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "[i for l in x for i in l]" 1000 loops, best of 3: 1.94 msec per loop [3] Python 2.7.5 with patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" --setup="from itertools import chain" "list(chain.from_iterable(x))" 1000 loops, best of 3: 821 usec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" --setup="from itertools import chain" "list(chain(*x))" 1000 loops, best of 3: 1.03 msec per loop [4] Python 2.7.5 with patch and string check removed: $ ./python -mtimeit --setup="x=['a']*10000" "sum(x,'')" 100 loops, best of 3: 3.98 msec per loop $ ./python -mtimeit --setup="x=['a']*10000" "''.join(x)" 10000 loops, best of 3: 170 usec per loop
Such a function is very tiny:
import operator isum = lambda *args: reduce(operator.iadd,*args)
But this might be unexpected:
l = [] l2 = isum([[1,2,3]]*1000000, l)
l is now changed. In fact l == l2. I guess this could cause more troubles than it's good for if the user expects tha current behaviour. I don't think such an incompatible API change can ever be made. But one could maybe include isum, maybe just as recipe in the documentation or in itertools or somewhere. On 07/02/2013 08:12 PM, Sergey wrote:
Hello, python-ideas. Trying to cover everything in one shot, so the message is long, sorry.
sum() is a great function. It is the "obvious way" to add things. Unfortunately sometimes it's slower than it could be.
The problem is that code: sum([[1,2,3]]*1000000, []) takes forever to complete. Let's fix that!
This problem was there since the sum() introduction, but it is almost unknown among python-beginners.
When people look at sum(seq,start=0) signature they most probably expect it to be like this: def sum(seq, start = 0): for item in seq: start += item return start
But it is not, because in cases like: empty = [] result = sum(list_of_lists, empty) such implementation would modify content of "empty". This use-case is known, it is checked in test_builtin.py, and explicitly mentioned in comments inside sum() code.
So actually sum() looks like this: def sum(seq, start = 0): for item in seq: start = start + item return start it creates a copy of the partial result on every "start + item".
What I suggest is instead of making a copy for every item make just one copy and reuse it as many times as needed. For example: def sum(seq, start = 0): start = start + seq[0] for item in seq[1:]: start += item return start
Patch implementing this idea attached to issue18305. Patch is simple, it just rearranges existing code. It should work for python 2.7, 3.3 and hg-tip. Except sum() becoming faster there should be no behavior change. Can this implementation break anything?
Advantages: * Performance boost is like 200000% [1], nothing else becomes slower
Disadvantages (inspired by Terry J. Reedy): * Other pythons (pypy, jython, older cpython versions) do not have such optimisation, people will move code that depends on the internal optimization to pythons that do not have it. And they will complain about their program 'freezing'.
* It discourages people from carefully thinking about whether they actually need a concrete list or merely the iterator for a virtual list.
Alternatives to sum() for this use-case: * list comprehension is 270% slower than patched sum [2] * itertools.chain is 50%-100% slower than patched sum [3]
Main questions: * Whether to accept the patch * If yes, whether it should go to the new version or to a bugfix release
Alternatives to this patch: * Reject the idea as a whole. Intentionally keep the sum() slow. * Accept the idea, but reject this particular implementation, instead use something different, e.g. implement special case optimization or use different approach (I thought about start=copy.copy(start))
In my opinion (looking though python changelog) performance patches were accepted for bugfix releases many times before. So as long as this patch does not break anything, it may go even to python 2.7.
I think bad performance is a bug that should be fixed. Rejecting performance fixes just because older versions don't have it would mean no performance improvements to anything, ever.
Sum is a great basic function. It keeps the simple code simple. This patch puts it on par with more advanced features like itertools, and allows people to use sum for simple things and itertools for complex memory-efficient algorithms.
Questions that may arise if the patch is accepted: * sum() was rejecting strings because of this bug. If the bug gets fixed should another patch allow sum() to accept strings? * maybe in some distant future drop the second parameter (or make it None by default) and allow calling sum for everything, making sum() "the one obvious way" to sum up things?
It would be nice if sum "just worked" for everything (e.g. sum() of empty sequence would return None, i.e. if there's nothing to sum then nothing is returned). But I think it needs more work for that, because even with this patch sum() is still ~20 times slower than "".join() [4]
That's all. Any suggestions welcome.
On Jul 2, 2013, at 20:24, Mathias Panzenböck <grosser.meister.morti@gmx.net> wrote:
Such a function is very tiny:
import operator isum = lambda *args: reduce(operator.iadd,*args)
But this might be unexpected:
l = [] l2 = isum([[1,2,3]]*1000000, l)
l is now changed. In fact l == l2.
He explicitly suggested making one copy, before looping. So, this: sum([[1,2,3]]*1000000, l) Has to mean: reduce(operator.iadd, [[1,2,3]]*1000000, copy.copy(l)) So, it's not quite a one-liner, and it doesn't have this problem.
But one could maybe include isum, maybe just as recipe in the documentation or in itertools or somewhere.
This sounds like a good idea. Possibly even both versions, with and without the copy. While we're at it, I've always wished sum and reduce used the same name for their start/initial parameter. If we're going to be changing one (which I suspect we probably aren't, but just in case...), is this a good time to campaign for renaming the param?
On 07/02/2013 08:12 PM, Sergey wrote:
Hello, python-ideas. Trying to cover everything in one shot, so the message is long, sorry.
sum() is a great function. It is the "obvious way" to add things. Unfortunately sometimes it's slower than it could be.
The problem is that code: sum([[1,2,3]]*1000000, []) takes forever to complete. Let's fix that!
This problem was there since the sum() introduction, but it is almost unknown among python-beginners.
When people look at sum(seq,start=0) signature they most probably expect it to be like this: def sum(seq, start = 0): for item in seq: start += item return start
But it is not, because in cases like: empty = [] result = sum(list_of_lists, empty) such implementation would modify content of "empty". This use-case is known, it is checked in test_builtin.py, and explicitly mentioned in comments inside sum() code.
So actually sum() looks like this: def sum(seq, start = 0): for item in seq: start = start + item return start it creates a copy of the partial result on every "start + item".
What I suggest is instead of making a copy for every item make just one copy and reuse it as many times as needed. For example: def sum(seq, start = 0): start = start + seq[0] for item in seq[1:]: start += item return start
Patch implementing this idea attached to issue18305. Patch is simple, it just rearranges existing code. It should work for python 2.7, 3.3 and hg-tip. Except sum() becoming faster there should be no behavior change. Can this implementation break anything?
Advantages: * Performance boost is like 200000% [1], nothing else becomes slower
Disadvantages (inspired by Terry J. Reedy): * Other pythons (pypy, jython, older cpython versions) do not have such optimisation, people will move code that depends on the internal optimization to pythons that do not have it. And they will complain about their program 'freezing'.
* It discourages people from carefully thinking about whether they actually need a concrete list or merely the iterator for a virtual list.
Alternatives to sum() for this use-case: * list comprehension is 270% slower than patched sum [2] * itertools.chain is 50%-100% slower than patched sum [3]
Main questions: * Whether to accept the patch * If yes, whether it should go to the new version or to a bugfix release
Alternatives to this patch: * Reject the idea as a whole. Intentionally keep the sum() slow. * Accept the idea, but reject this particular implementation, instead use something different, e.g. implement special case optimization or use different approach (I thought about start=copy.copy(start))
In my opinion (looking though python changelog) performance patches were accepted for bugfix releases many times before. So as long as this patch does not break anything, it may go even to python 2.7.
I think bad performance is a bug that should be fixed. Rejecting performance fixes just because older versions don't have it would mean no performance improvements to anything, ever.
Sum is a great basic function. It keeps the simple code simple. This patch puts it on par with more advanced features like itertools, and allows people to use sum for simple things and itertools for complex memory-efficient algorithms.
Questions that may arise if the patch is accepted: * sum() was rejecting strings because of this bug. If the bug gets fixed should another patch allow sum() to accept strings? * maybe in some distant future drop the second parameter (or make it None by default) and allow calling sum for everything, making sum() "the one obvious way" to sum up things?
It would be nice if sum "just worked" for everything (e.g. sum() of empty sequence would return None, i.e. if there's nothing to sum then nothing is returned). But I think it needs more work for that, because even with this patch sum() is still ~20 times slower than "".join() [4]
That's all. Any suggestions welcome.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 3 July 2013 14:40, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 2, 2013, at 20:24, Mathias Panzenböck < grosser.meister.morti@gmx.net> wrote:
Such a function is very tiny:
import operator isum = lambda *args: reduce(operator.iadd,*args)
But this might be unexpected:
l = [] l2 = isum([[1,2,3]]*1000000, l)
l is now changed. In fact l == l2.
He explicitly suggested making one copy, before looping. So, this:
sum([[1,2,3]]*1000000, l)
Has to mean:
reduce(operator.iadd, [[1,2,3]]*1000000, copy.copy(l))
So, it's not quite a one-liner, and it doesn't have this problem.
But one could maybe include isum, maybe just as recipe in the documentation or in itertools or somewhere.
This sounds like a good idea. Possibly even both versions, with and without the copy.
While we're at it, I've always wished sum and reduce used the same name for their start/initial parameter. If we're going to be changing one (which I suspect we probably aren't, but just in case...), is this a good time to campaign for renaming the param?
reduce only just survived being culled completely in the Python 3 transition, so if one was going to change it would be reduce. Our perspective is that if you're considering using reduce, you should remember you're writing Python rather than a pure functional language and switch to a loop instead. (By contrast, map and filter are nice alternatives to a generator expression if you are just applying an existing function) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
03.07.13 06:24, Mathias Panzenböck написав(ла):
Such a function is very tiny:
import operator isum = lambda *args: reduce(operator.iadd,*args)
But this might be unexpected:
l = [] l2 = isum([[1,2,3]]*1000000, l)
l is now changed. In fact l == l2. I guess this could cause more troubles than it's good for if the user expects tha current behaviour. I don't think such an incompatible API change can ever be made. But one could maybe include isum, maybe just as recipe in the documentation or in itertools or somewhere.
Sergey's code is more smart. It equals to the following Python code: def sum(iterable, start=0): it = iter(iterable) try: x = next(it) except StopIteration: return start result = start + x for x in it: result += x return result
On Jul 2, 2013, at 11:12, Sergey <sergemp@mail.ru> wrote:
Questions that may arise if the patch is accepted: * sum() was rejecting strings because of this bug. If the bug gets fixed should another patch allow sum() to accept strings?
Does it actually speed up strings? I would have thought it only helps for types that have a mutating __iadd__ (and one that's faster than non mutating __add__, of course).
* maybe in some distant future drop the second parameter (or make it None by default) and allow calling sum for everything, making sum() "the one obvious way" to sum up things?
sum can't guess the unified type of all of the elements in the iterable. The best it could do is what reduce does in that case: start with the first element, and add from there. That's not always the magical DWIM you seem to be expecting. Most importantly, how could it possibly work for iterables that might be empty? ''.join(lines), or sum(lines, ''), will work when there are no lines; sum(lines) can't possibly know that you expected a string. Meanwhile, if you're going to add optional "start from the first item" functionality, I think you'll also want to make the operator/function overridable. And then you've just re-invented reduce, with a slightly different signature: sum(iterable, start=None, function=operator.iadd): return reduce(function, iterable, start)
It would be nice if sum "just worked" for everything (e.g. sum() of empty sequence would return None, i.e. if there's nothing to sum then nothing is returned). But I think it needs more work for that, because even with this patch sum() is still ~20 times slower than "".join() [4]
You could always special case it when start is a str (or when start is defaulted and the first value is a str). But then what happens if the second value is something that can be added to a str, but not actually a str? Or, for that matter, the start value?
On Jul 2, 2013 Andrew Barnert wrote: I'm trying to make sum() more SIMPLE, not more complex. Currently sum() "just works" only for numbers. So I was thinking how to make it "just work" for everything. Check this: sum([something]) sum([something1, something2]) and this: sum([], something) sum([something2], something1) First one looks obvious even for those who does not know python, while to understand the second one, you have to actually read the manuals. And even after that it may be unclear why second parameter is mandatory for non-numbers. "Readability counts." But, anyway, that's not the point of THIS bugreport/patch. This patch is to fix the slowness, that is easy to fix, without introducing any changes.
Does it actually speed up strings?
Unfortunately it does not, sum() is still much slower than join(). That's because join() was optimised for strings, it walks through the list twice, first time to calculate total size of all the strings and allocate the entire string in one shot, and second time to copy all the strings into allocated memory.
sum can't guess the unified type of all of the elements in the iterable. The best it could do is what reduce does in that case: start with the first element, and add from there. That's not always the magical DWIM you seem to be expecting.
Why not? I wouldn't expect anything else from sum(). Well, sum() is so common and useful, I was just thinking of how to turn it into "the one obvious way", something that "just works", something like: def justsum(seq): result = None try: seq = iter(seq) result = next(seq) result = result + next(seq) while True: result += next(seq) except StopIteration: return result It would work for everything:
justsum(['a', 'bb', 'ccc']) 'abbccc' justsum([1, 2, 3]) 6 justsum([[1,2], [3,4], [5,6]]) [1, 2, 3, 4, 5, 6] justsum([3, 0.1, 0.04]) 3.14
It would still work even for weird cases. For example, what should be the sum of one element? It should be the element itself, no matter what element would it be, right? For example:
justsum([list]) <type 'list'> g = (i*i for i in range(10)) justsum([g]) <generator object <genexpr> at 0xa93280>
I know, these are unusual use-cases, but they work as expected.
Most importantly, how could it possibly work for iterables that might be empty? ''.join(lines), or sum(lines, ''), will work when there are no lines; sum(lines) can't possibly know that you expected a string.
If there're no elements to sum it would return "nothing" i.e. None. That looks rather obvious to me. However I'm not Dutch, so I can be wrong. :)
Meanwhile, if you're going to add optional "start from the first item" functionality, I think you'll also want to make the operator/function overridable.
And then you've just re-invented reduce, with a slightly different signature: sum(iterable, start=None, function=operator.iadd): return reduce(function, iterable, start)
Does not work even for: sum([1, 2, 3, 4]) not to mention: lists = [[1, 2], [3, 4]] sum(lists) where reduce-based code fails, because it modifies original list.
You could always special case it when start is a str (or when start is defaulted and the first value is a str). But then what happens if the second value is something that can be added to a str, but not actually a str? Or, for that matter, the start value?
Fallback to general code. This is how sum() works right now. First it attempts to store result in integer. If that fails it tries float. If that fails too, even if it fails in the middle of the list, sum() falls back to general PyNumber_Add(). My patch only adjusts that general part so that it first tries PyNumber_InPlaceAdd(), and only if that fails uses PyNumber_Add(). Well, PyNumber_InPlaceAdd() does that for me. --
02.07.13 21:12, Sergey написав(ла):
Alternatives to sum() for this use-case: * list comprehension is 270% slower than patched sum [2] * itertools.chain is 50%-100% slower than patched sum [3]
You forgot most straightforward alternative: result = [] for x in iterable: result.extend(x) I'm sure this is most popular solution for this problem.
Questions that may arise if the patch is accepted: * sum() was rejecting strings because of this bug. If the bug gets fixed should another patch allow sum() to accept strings?
No. Strings are immutable.
* maybe in some distant future drop the second parameter (or make it None by default) and allow calling sum for everything, making sum() "the one obvious way" to sum up things?
No. sum() should work for empty sequences.
On 03/07/13 04:12, Sergey wrote:
Hello, python-ideas. Trying to cover everything in one shot, so the message is long, sorry.
sum() is a great function. It is the "obvious way" to add things. Unfortunately sometimes it's slower than it could be.
The problem is that code: sum([[1,2,3]]*1000000, []) takes forever to complete. Let's fix that!
I'm not sure that sum() is the Obvious Way to concatenate lists, and I don't think that concatenating many lists is a common thing to do. Traditionally, sum() works only on numbers, and I think we wouldn't be having this discussion if Python used & for concatenation instead of +. So I don't care that sum() has quadratic performance on lists (and tuples), and I must admit that having a simple quadratic algorithm in the built-ins is sometimes useful for teaching purposes, so I'm -0 on optimizing this case. [...]
When people look at sum(seq,start=0) signature they most probably expect it to be like this: def sum(seq, start = 0): for item in seq: start += item return start
That's not what I expect, since += risks modifying its argument in place. [...]
What I suggest is instead of making a copy for every item make just one copy and reuse it as many times as needed. For example: def sum(seq, start = 0): start = start + seq[0] for item in seq[1:]: start += item return start [...] Can this implementation break anything?
That will still have quadratic performance for tuples, and it will break if seq is an iterator. It will also be expensive to make a slice of seq if it is a huge list, and could even run out of memory to hold both the original and the slice. I would be annoyed if sum started failing with a MemoryError here: big_seq = list(range(1000))*1000000 # assume this succeeds total = sum(big_seq) # could fail [...]
Alternatives to sum() for this use-case: * list comprehension is 270% slower than patched sum [2] * itertools.chain is 50%-100% slower than patched sum [3]
How about the "Obvious Way" to concatenate lists? new = [] for x in seq: new.extend(x)
Main questions: * Whether to accept the patch * If yes, whether it should go to the new version or to a bugfix release
-0 on the general idea, -1 on the specific implementation. I'd rather have sum of lists be slow than risk sum of numbers raise MemoryError. -- Steven
On 3 July 2013 13:43, Steven D'Aprano <steve@pearwood.info> wrote:
On 03/07/13 04:12, Sergey wrote:
Hello, python-ideas. Trying to cover everything in one shot, so the message is long, sorry.
sum() is a great function. It is the "obvious way" to add things. Unfortunately sometimes it's slower than it could be.
The problem is that code: sum([[1,2,3]]*1000000, []) takes forever to complete. Let's fix that!
I'm not sure that sum() is the Obvious Way to concatenate lists, and I don't think that concatenating many lists is a common thing to do. Traditionally, sum() works only on numbers, and I think we wouldn't be having this discussion if Python used & for concatenation instead of +. So I don't care that sum() has quadratic performance on lists (and tuples), and I must admit that having a simple quadratic algorithm in the built-ins is sometimes useful for teaching purposes, so I'm -0 on optimizing this case.
This optimises all circumstances where iadd is faster than add. It makes sense to me.
What I suggest is instead of making a copy for every item make just one copy and reuse it as many times as needed. For example: def sum(seq, start = 0): start = start + seq[0] for item in seq[1:]: start += item return start
[...]
Can this implementation break anything?
That will still have quadratic performance for tuples, and it will break if seq is an iterator.
It will also be expensive to make a slice of seq if it is a huge list, and could even run out of memory to hold both the original and the slice. I would be annoyed if sum started failing with a MemoryError here:
big_seq = list(range(1000))*1000000 # assume this succeeds total = sum(big_seq) # could fail
Then you can just make it support iterators: def sum(seq, start = 0): seq = iter(seq) start = start + next(seq) for item in seq: start += item return start
Alternatives to sum() for this use-case: * list comprehension is 270% slower than patched sum [2] * itertools.chain is 50%-100% slower than patched sum [3]
How about the "Obvious Way" to concatenate lists?
new = [] for x in seq: new.extend(x)
What's wrong with sum, if sum is fast?
-0 on the general idea, -1 on the specific implementation. I'd rather have sum of lists be slow than risk sum of numbers raise MemoryError.
How about with my version? I see no obvious downsides, personally.
On 04/07/13 02:58, Joshua Landau wrote:
What's wrong with sum, if sum is fast?
sum simply cannot *always* be fast. E.g. summing tuples will still be slow even with your suggestion. Using sum() on anything other than numbers is conceptually dubious; yes, sum() is intended for addition, but it's intended for *numeric* addition. It's unfortunate that Python uses + for concatenation, we're stuck with it until Python 4000, but if you're using sum() to add lists, tuples, or other non-numbers, you're living in a state of sin. (A venal sin rather than a mortal sin.) It's allowed, but we shouldn't encourage it, and treating sum() as if it were the One Obvious Way to concatenate data is, in my strong opinion, the wrong thing to do. In my opinion, the One Obvious Way to concatenate a lot of arbitrary data is list.extend, not sum. I can't gather any enthusiasm for optimizing sum to speed up concatenation. I'm at best indifferent towards the specific proposal to speed up sum of lists; I'm hostile to any attempt to encourage people to treat sum() as the preferred way to concatenate large amounts of data, because that will surely lead them into bad habits and will end with them trying to sum() a lot of tuples or linked lists or something and getting O(n**2) performance. -- Steven
On 3 July 2013 18:58, Steven D'Aprano <steve@pearwood.info> wrote:
On 04/07/13 02:58, Joshua Landau wrote:
What's wrong with sum, if sum is fast?
sum simply cannot *always* be fast. E.g. summing tuples will still be slow even with your suggestion.
Using sum() on anything other than numbers is conceptually dubious; yes, sum() is intended for addition, but it's intended for *numeric* addition.
Sum: A quantity obtained by addition or aggregation. [http://en.wiktionary.org/wiki/sum] I don't see why it should be constrained.
It's unfortunate that Python uses + for concatenation,
Is it? I'm very happy with it myself.
we're stuck with it until Python 4000, but if you're using sum() to add lists, tuples, or other non-numbers, you're living in a state of sin.
Why? There are a lot of things it makes sense to take the sum of - a lot of which have constant-ish performance on addition.
(A venal sin rather than a mortal sin.) It's allowed, but we shouldn't encourage it, and treating sum() as if it were the One Obvious Way to concatenate data is, in my strong opinion, the wrong thing to do.
No, no. A fast sum() is not the one obvious way to concatenate data, much as sum() is not the one obvious way to add data. It just means that if conceptually you're looking for the sum, you'd be able to do it without shooting yourself in the foot. I'd also point out there are a *lot* of times when iadd is faster than add, not just contancation.
In my opinion, the One Obvious Way to concatenate a lot of arbitrary data is list.extend, not sum.
Is it? Maybe for lists. Often itertools.chain is better. It really depends on your data. I tend to use itertools.chain a lot more than list.extend, because I sum iterators more than I sum lists. Maybe I'm just weird.
I can't gather any enthusiasm for optimizing sum to speed up concatenation. I'm at best indifferent towards the specific proposal to speed up sum of lists; I'm hostile to any attempt to encourage people to treat sum() as the preferred way to concatenate large amounts of data, because that will surely lead them into bad habits and will end with them trying to sum() a lot of tuples or linked lists or something and getting O(n**2) performance.
Maybe the difference is that I look at this as an optimisation to every use of sum where iadd is faster than add, not just list contancation. If you're summing tuples, either you know what to do (sum into a list and then convert back) or you're going to do it wrong *anyway*. I can't remember the last time I had to sum tuples. Also, if you're summing linked lists, why would it be O(n**2) performance? Surely it's still O(n), given the new implementation?
On Jul 3, 2013, at 15:02, Joshua Landau <joshua.landau.ws@gmail.com> wrote:
Also, if you're summing linked lists, why would it be O(n**2) performance? Surely it's still O(n), given the new implementation?
Not for single-linked lists, where extend is O(n). (Of course maybe this is a good thing--it's a perfect opportunity for someone to show whoever wrote that code how to actually write his algorithm in python rather than trying to use python as lisp without even knowing lisp...)
On 04/07/13 08:02, Joshua Landau wrote:
On 3 July 2013 18:58, Steven D'Aprano <steve@pearwood.info> wrote:
On 04/07/13 02:58, Joshua Landau wrote:
What's wrong with sum, if sum is fast?
sum simply cannot *always* be fast. E.g. summing tuples will still be slow even with your suggestion.
Using sum() on anything other than numbers is conceptually dubious; yes, sum() is intended for addition, but it's intended for *numeric* addition.
Sum: A quantity obtained by addition or aggregation. [http://en.wiktionary.org/wiki/sum]
I don't see why it should be constrained.
I didn't say is should be constrained. But there is a big difference between "don't prevent people calling sum() on lists, if they insist" and "encourage people to use sum() on lists, in preference to list.extend".
It's unfortunate that Python uses + for concatenation,
Is it? I'm very happy with it myself.
Yes. If Python used & for concatenation, we wouldn't have to worry about sum(lists or strings or tuples) being quadratic, because people wouldn't call sum on lists, strings or tuples.
we're stuck with it until Python 4000, but if you're using sum() to add lists, tuples, or other non-numbers, you're living in a state of sin.
Why? There are a lot of things it makes sense to take the sum of - a lot of which have constant-ish performance on addition.
A "lot" of things? There are numbers (int, float, Decimal, Fraction, complex), and numpy arrays of numbers. And you probably shouldn't call sum on floats if you care about precision. (Use math.fsum instead.) Strings? Quadratic. Tuples? Quadratic. Lists? Currently quadratic, could be optimized, if we care. I can't think of anything else that supports + and has near-constant time repeated addition. What are you thinking of?
(A venal sin rather than a mortal sin.) It's allowed, but we shouldn't encourage it, and treating sum() as if it were the One Obvious Way to concatenate data is, in my strong opinion, the wrong thing to do.
No, no. A fast sum() is not the one obvious way to concatenate data,
The Original Poster seems to think it is.
much as sum() is not the one obvious way to add data. It just means that if conceptually you're looking for the sum, you'd be able to do it without shooting yourself in the foot.
I'd also point out there are a *lot* of times when iadd is faster than add, not just contancation.
Examples please.
In my opinion, the One Obvious Way to concatenate a lot of arbitrary data is list.extend, not sum.
Is it? Maybe for lists. Often itertools.chain is better. It really depends on your data. I tend to use itertools.chain a lot more than list.extend, because I sum iterators more than I sum lists. Maybe I'm just weird.
I didn't say *the Best Way*, I said *the One Obvious Way*. The obvious way to have a lot of data of some arbitrary type is a list; the obvious way to concatenate a whole lot of lists into one list is using the extend method. If you have some special type or some special need, you may be able to do better.
I can't remember the last time I had to sum tuples.
And I can't remember the last time I had to sum lists. Optimizing this case will solve exactly none of my problems.
Also, if you're summing linked lists, why would it be O(n**2) performance? Surely it's still O(n), given the new implementation?
Actually, more like O(n*m). You have n linked lists each with an average of m nodes, you have to copy each node, so that's O(n*m). Sorry, simplifying n*m to n**2 is sloppy. -- Steven
Steven D'Aprano wrote:
It's unfortunate that Python uses + for concatenation,
Is it? I'm very happy with it myself.
Yes. If Python used & for concatenation, we wouldn't have to worry about sum(lists or strings or tuples) being quadratic, because people wouldn't call sum on lists, strings or tuples.
yeah, they would try with all() :D -- ZeD
On 4 July 2013 04:35, Steven D'Aprano <steve@pearwood.info> wrote:
On 04/07/13 08:02, Joshua Landau wrote:
On 3 July 2013 18:58, Steven D'Aprano <steve@pearwood.info> wrote:
On 04/07/13 02:58, Joshua Landau wrote:
Why? There are a lot of things it makes sense to take the sum of - a lot of which have constant-ish performance on addition.
A "lot" of things? ... I can't think of anything else that supports + and has near-constant time repeated addition. What are you thinking of? ...
I'd also point out there are a *lot* of times when iadd is faster than add, not just contancation.
Examples please.
I'm going to retreat from this point as my argument is weaker than I thought it was. Mostly it comes from an internal delusion that I overload operators more than I really do -- when I do it's normally not for a final product. Since most of the following argument is based around this, I put this at the top. To be clear, I don't feel I'm wrong -- but I feel I'm less right than you.
I don't see why it should be constrained.
I didn't say is should be constrained. But there is a big difference between "don't prevent people calling sum() on lists, if they insist" and "encourage people to use sum() on lists, in preference to list.extend".
I meant constrained in a more general term. Regardless of the OP's intent, I don't think we need to encourage sum() on lists if the patch goes through. I bet you a lot of people still do it though.
It's unfortunate that Python uses + for concatenation,
Is it? I'm very happy with it myself.
Yes. If Python used & for concatenation, we wouldn't have to worry about sum(lists or strings or tuples) being quadratic, because people wouldn't call sum on lists, strings or tuples.
Optimising for the people who don't know what they're doing seems kind of backwards. I like "+" to add lists. I also think "data & [elem]" looks stupid. That's possibly just me, but it's true for me nonetheless.
A fast sum() is not the one obvious way to concatenate data,
The Original Poster seems to think it is.
I think he's wrong.
In my opinion, the One Obvious Way to concatenate a lot of arbitrary data is list.extend, not sum.
Is it? Maybe for lists. Often itertools.chain is better. It really depends on your data. I tend to use itertools.chain a lot more than list.extend, because I sum iterators more than I sum lists. Maybe I'm just weird.
I didn't say *the Best Way*, I said *the One Obvious Way*. The obvious way to have a lot of data of some arbitrary type is a list; the obvious way to concatenate a whole lot of lists into one list is using the extend method. If you have some special type or some special need, you may be able to do better.
I know what you said, and I raised a disagreement. The OOW to have a lot of data of some arbitrary type is context-dependent. Sometimes it's an iterator. Sometimes it's a list. Sometimes it's a set. Sometimes it's deque (OK, pushing it).
Not just you! I like using + for lists too! I don't see at all how '&' is any better than '+' for concatenating strings. I don't have any perf numbers to back me up, but I don't agree with the "let's leave something with poor performance in the builtins so we can encourage people not to use them like that" point. I mean, if people really shouldn't use it like that, it should throw an error. If they should use it, it should work as best it can. Having quietly, unnecessarily-O(n^2)-performing builtins just "for teaching purposes" seems absolute madness. There is a place for intentionally-crippled teaching-code and it's not in the standard library's builtins. Of course, without any perf numbers, I'm just speaking in hypotheticals. On Thu, Jul 4, 2013 at 1:27 PM, Joshua Landau <joshua.landau.ws@gmail.com>wrote:
On 4 July 2013 04:35, Steven D'Aprano <steve@pearwood.info> wrote:
On 04/07/13 08:02, Joshua Landau wrote:
On 3 July 2013 18:58, Steven D'Aprano <steve@pearwood.info> wrote:
On 04/07/13 02:58, Joshua Landau wrote:
Why? There are a lot of things it makes sense to take the sum of - a lot of which have constant-ish performance on addition.
A "lot" of things? ... I can't think of anything else that supports + and has near-constant time repeated addition. What are you thinking of? ...
I'd also point out there are a *lot* of times when iadd is faster than add, not just contancation.
Examples please.
I'm going to retreat from this point as my argument is weaker than I thought it was. Mostly it comes from an internal delusion that I overload operators more than I really do -- when I do it's normally not for a final product. Since most of the following argument is based around this, I put this at the top.
To be clear, I don't feel I'm wrong -- but I feel I'm less right than you.
I don't see why it should be constrained.
I didn't say is should be constrained. But there is a big difference between "don't prevent people calling sum() on lists, if they insist" and "encourage people to use sum() on lists, in preference to list.extend".
I meant constrained in a more general term. Regardless of the OP's intent, I don't think we need to encourage sum() on lists if the patch goes through. I bet you a lot of people still do it though.
It's unfortunate that Python uses + for concatenation,
Is it? I'm very happy with it myself.
Yes. If Python used & for concatenation, we wouldn't have to worry about sum(lists or strings or tuples) being quadratic, because people wouldn't call sum on lists, strings or tuples.
Optimising for the people who don't know what they're doing seems kind of backwards. I like "+" to add lists.
I also think "data & [elem]" looks stupid. That's possibly just me, but it's true for me nonetheless.
A fast sum() is not the one obvious way to concatenate data,
The Original Poster seems to think it is.
I think he's wrong.
In my opinion, the One Obvious Way to concatenate a lot of arbitrary data is list.extend, not sum.
Is it? Maybe for lists. Often itertools.chain is better. It really depends on your data. I tend to use itertools.chain a lot more than list.extend, because I sum iterators more than I sum lists. Maybe I'm just weird.
I didn't say *the Best Way*, I said *the One Obvious Way*. The obvious way to have a lot of data of some arbitrary type is a list; the obvious way to concatenate a whole lot of lists into one list is using the extend method. If you have some special type or some special need, you may be able to do better.
I know what you said, and I raised a disagreement. The OOW to have a lot of data of some arbitrary type is context-dependent. Sometimes it's an iterator. Sometimes it's a list. Sometimes it's a set. Sometimes it's deque (OK, pushing it). _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Jul 04, 2013 Steven D'Aprano wrote: This message is long, so here's its short summary: * Unfortunately list.extend does not look like the obvious way, and its slower than alternatives. * Original patch reduces memory, not increases it, there can be no MemoryError because of it. * sum() can *always* be fast! (patch and tests) * linked list is O(n) where n is number of lists to add * using __add__/__iadd__ for containers is a useful feature
How about the "Obvious Way" to concatenate lists? new = [] for x in seq: new.extend(x)
200% slower than patched sum, 50-100% slower than both itertools and 25% faster than list comprehension. [1] It's basically not even mentioned among most popular answers to list flattening: http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-... http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-... Slower, less known, it's much more characters to type, hey, it's not even one-liner! :) What makes you think this is the "Obvious Way"?
-0 on the general idea, -1 on the specific implementation. I'd rather have sum of lists be slow than risk sum of numbers raise MemoryError.
You must be misunderstanding something. Or maybe I've explained it poorly. Numbers have different code path in sum() that my patch does not touch. But even if it did, my patch never makes a copy of original list, it may only reduce amount of memory used, not increase it. There was an alternative idea (that I never implemented), suggesting to make a copy of `start` variable, but not the list.
sum simply cannot *always* be fast. E.g. summing tuples will still be slow even with your suggestion.
Yes, it can! That's the point of the original idea! The original patch [2] optimizes lists, because it was easy to do. But nothing stops you from optimizing other (two?) types. For example this patch [3] optimizes lists, tuples and strings. Basically it works by storing temporary result in a list, and then converts it to tuple or string in one shot if needed. Using this patch you get the O(n) sum for lists, tuples and strings [4]. And combining it with original patch you get the fastest sum() possible. Even being O(n) for strings, it's slower than ''.join(), but it is constantly slower now. I can't beat ''.join() because of function call overhead. Internally join() converts the sequence into tuple, thus saving a lot of calls, but using a lot of additional memory, that's why: ''.join('' for _ in xrange(100000000)) eats ~1GB of RAM before giving you an empty string.
I can't gather any enthusiasm for optimizing sum to speed up concatenation.
No problem, just tell what approach you think is the best and I'll try implementing it.
I'm hostile to any attempt to encourage people to treat sum() as the preferred way to concatenate large amounts of data, because that will surely lead them into bad habits and will end with them trying to sum() a lot of tuples or linked lists or something and getting O(n**2) performance.
Why not just make sum O(n) for everything? I've already done that for lists, tuples and strings. As for linked lists, the point of linked list is to insert items fast. So any decent implementation of it should store a pointer to its head and tail, should implement a O(1) __iadd__ using tail pointer, and thus falls under my first patch. There's not much sence in [single] linked list if it has no __iadd__.
Yes. If Python used & for concatenation, we wouldn't have to worry about sum(lists or strings or tuples) being quadratic, because people wouldn't call sum on lists, strings or tuples.
Heh. If Python had no sum() we wouldn't have to worry about people using it. If Python had no lists we wouldn't have to worry about people concatenating them. If there was no Python we wouldn't have to worry at all. But the world would be poor without all these great things... Seriously, I miss add for set(). I needed it when I had a dictionary like {x:set(...), ...} and needed a set of all the values from it. I wanted to use sum(dict.values()), that would be easy and obvious, but I couldn't, because set() does not support __add__. So I had to write a few lines of loop instead of a few characters. :( -- [1] Python 2.7.5 with patch, testing list.extend(): $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 1000 loops, best of 3: 531 usec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "[i for l in x for i in l]" 1000 loops, best of 3: 1.94 msec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" --setup="from itertools import chain" "list(chain.from_iterable(x))" 1000 loops, best of 3: 820 usec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" --setup="from itertools import chain" "list(chain(*x))" 1000 loops, best of 3: 1.03 msec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "l = []" "for i in x: l.extend(i)" 1000 loops, best of 3: 1.53 msec per loop [2] http://bugs.python.org/file30705/fastsum.patch [3] http://bugs.python.org/file30769/fastsum-special.patch http://bugs.python.org/issue18305#msg192281 [4] Python 2.7.5, testing new fastsum-special.patch: === Lists === No patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 10 loops, best of 3: 885 msec per loop fastsum.patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 1000 loops, best of 3: 524 usec per loop fastsum-special.patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 1000 loops, best of 3: 298 usec per loop Result: 3000 times faster. === Tuples === No patch: $ ./python -mtimeit --setup="x=[(1,2,3)]*10000" "sum(x,())" 10 loops, best of 3: 585 msec per loop fastsum.patch: $ ./python -mtimeit --setup="x=[(1,2,3)]*10000" "sum(x,())" 10 loops, best of 3: 585 msec per loop fastsum-special.patch: $ ./python -mtimeit --setup="x=[(1,2,3)]*10000" "sum(x,())" 1000 loops, best of 3: 536 usec per loop Result: 1000 times faster. === Strings === No patch (just string check removed): $ ./python -mtimeit --setup="x=['abc']*100000" "sum(x,'')" 10 loops, best of 3: 1.52 sec per loop fastsum.patch (+ string check removed): $ ./python -mtimeit --setup="x=['abc']*100000" "sum(x,'')" 10 loops, best of 3: 1.52 sec per loop fastsum-special.patch $ ./python -mtimeit --setup="x=['abc']*100000" "sum(x,'')" 10 loops, best of 3: 27.8 msec per loop join: $ ./python -mtimeit --setup="x=['abc']*100000" "''.join(x)" 1000 loops, best of 3: 1.66 msec per loop Result: 50 times faster, but still constantly slower than join.
Random thought: this kind of "different implementations for different types" seems exactly what PEP443 (http://www.python.org/dev/peps/pep-0443/) is about; it would save you having the nasty big chunk of if-elses within sum() itself, and would let other people incrementally implement special sums for their own special data types without having to muck with the std lib code. On Thu, Jul 4, 2013 at 5:54 PM, Sergey <sergemp@mail.ru> wrote:
On Jul 04, 2013 Steven D'Aprano wrote:
This message is long, so here's its short summary: * Unfortunately list.extend does not look like the obvious way, and its slower than alternatives. * Original patch reduces memory, not increases it, there can be no MemoryError because of it. * sum() can *always* be fast! (patch and tests) * linked list is O(n) where n is number of lists to add * using __add__/__iadd__ for containers is a useful feature
How about the "Obvious Way" to concatenate lists? new = [] for x in seq: new.extend(x)
200% slower than patched sum, 50-100% slower than both itertools and 25% faster than list comprehension. [1] It's basically not even mentioned among most popular answers to list flattening:
http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-...
http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-... Slower, less known, it's much more characters to type, hey, it's not even one-liner! :) What makes you think this is the "Obvious Way"?
-0 on the general idea, -1 on the specific implementation. I'd rather have sum of lists be slow than risk sum of numbers raise MemoryError.
You must be misunderstanding something. Or maybe I've explained it poorly. Numbers have different code path in sum() that my patch does not touch. But even if it did, my patch never makes a copy of original list, it may only reduce amount of memory used, not increase it. There was an alternative idea (that I never implemented), suggesting to make a copy of `start` variable, but not the list.
sum simply cannot *always* be fast. E.g. summing tuples will still be slow even with your suggestion.
Yes, it can! That's the point of the original idea!
The original patch [2] optimizes lists, because it was easy to do. But nothing stops you from optimizing other (two?) types. For example this patch [3] optimizes lists, tuples and strings.
Basically it works by storing temporary result in a list, and then converts it to tuple or string in one shot if needed.
Using this patch you get the O(n) sum for lists, tuples and strings [4]. And combining it with original patch you get the fastest sum() possible.
Even being O(n) for strings, it's slower than ''.join(), but it is constantly slower now. I can't beat ''.join() because of function call overhead. Internally join() converts the sequence into tuple, thus saving a lot of calls, but using a lot of additional memory, that's why: ''.join('' for _ in xrange(100000000)) eats ~1GB of RAM before giving you an empty string.
I can't gather any enthusiasm for optimizing sum to speed up concatenation.
No problem, just tell what approach you think is the best and I'll try implementing it.
I'm hostile to any attempt to encourage people to treat sum() as the preferred way to concatenate large amounts of data, because that will surely lead them into bad habits and will end with them trying to sum() a lot of tuples or linked lists or something and getting O(n**2) performance.
Why not just make sum O(n) for everything? I've already done that for lists, tuples and strings. As for linked lists, the point of linked list is to insert items fast. So any decent implementation of it should store a pointer to its head and tail, should implement a O(1) __iadd__ using tail pointer, and thus falls under my first patch. There's not much sence in [single] linked list if it has no __iadd__.
Yes. If Python used & for concatenation, we wouldn't have to worry about sum(lists or strings or tuples) being quadratic, because people wouldn't call sum on lists, strings or tuples.
Heh. If Python had no sum() we wouldn't have to worry about people using it. If Python had no lists we wouldn't have to worry about people concatenating them. If there was no Python we wouldn't have to worry at all. But the world would be poor without all these great things...
Seriously, I miss add for set(). I needed it when I had a dictionary like {x:set(...), ...} and needed a set of all the values from it. I wanted to use sum(dict.values()), that would be easy and obvious, but I couldn't, because set() does not support __add__. So I had to write a few lines of loop instead of a few characters. :(
--
[1] Python 2.7.5 with patch, testing list.extend(): $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 1000 loops, best of 3: 531 usec per loop
$ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "[i for l in x for i in l]" 1000 loops, best of 3: 1.94 msec per loop
$ ./python -mtimeit --setup="x=[[1,2,3]]*10000" --setup="from itertools import chain" "list(chain.from_iterable(x))" 1000 loops, best of 3: 820 usec per loop
$ ./python -mtimeit --setup="x=[[1,2,3]]*10000" --setup="from itertools import chain" "list(chain(*x))" 1000 loops, best of 3: 1.03 msec per loop
$ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "l = []" "for i in x: l.extend(i)" 1000 loops, best of 3: 1.53 msec per loop
[2] http://bugs.python.org/file30705/fastsum.patch
[3] http://bugs.python.org/file30769/fastsum-special.patch http://bugs.python.org/issue18305#msg192281
[4] Python 2.7.5, testing new fastsum-special.patch: === Lists === No patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 10 loops, best of 3: 885 msec per loop fastsum.patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 1000 loops, best of 3: 524 usec per loop fastsum-special.patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 1000 loops, best of 3: 298 usec per loop Result: 3000 times faster.
=== Tuples === No patch: $ ./python -mtimeit --setup="x=[(1,2,3)]*10000" "sum(x,())" 10 loops, best of 3: 585 msec per loop fastsum.patch: $ ./python -mtimeit --setup="x=[(1,2,3)]*10000" "sum(x,())" 10 loops, best of 3: 585 msec per loop fastsum-special.patch: $ ./python -mtimeit --setup="x=[(1,2,3)]*10000" "sum(x,())" 1000 loops, best of 3: 536 usec per loop Result: 1000 times faster.
=== Strings === No patch (just string check removed): $ ./python -mtimeit --setup="x=['abc']*100000" "sum(x,'')" 10 loops, best of 3: 1.52 sec per loop fastsum.patch (+ string check removed): $ ./python -mtimeit --setup="x=['abc']*100000" "sum(x,'')" 10 loops, best of 3: 1.52 sec per loop fastsum-special.patch $ ./python -mtimeit --setup="x=['abc']*100000" "sum(x,'')" 10 loops, best of 3: 27.8 msec per loop join: $ ./python -mtimeit --setup="x=['abc']*100000" "''.join(x)" 1000 loops, best of 3: 1.66 msec per loop Result: 50 times faster, but still constantly slower than join. _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Thu, Jul 4, 2013 at 5:31 AM, Haoyi Li <haoyi.sg@gmail.com> wrote:
Random thought: this kind of "different implementations for different types" seems exactly what PEP443 (http://www.python.org/dev/peps/pep-0443/) is about; it would save you having the nasty big chunk of if-elses within sum() itself, and would let other people incrementally implement special sums for their own special data types without having to muck with the std lib code.
Exactly. Though 443 allows dispatching only on simple type names. Python doesn't have the concept of list<int>, or list<list>. -- Juancarlo *Añez*
On 4 July 2013 11:01, Haoyi Li <haoyi.sg@gmail.com> wrote:
Random thought: this kind of "different implementations for different types" seems exactly what PEP443 (http://www.python.org/dev/peps/pep-0443/) is about; it would save you having the nasty big chunk of if-elses within sum() itself, and would let other people incrementally implement special sums for their own special data types without having to muck with the std lib code.
One thing I didn't understand from the original post was that this was a "special case". I don't think I'll support any special-casing without appropriate support from Python, such as with PEP443, first.
On Jul 4, 2013 Joshua Landau wrote:
One thing I didn't understand from the original post was that this was a "special case".
That's because it wasn't. There were too many different ideas during this thread, I'll try to collect them all together. So... Sum() is a great function. Why? Because it makes the code simple! You can always live without sum(), you can use itertools, reduce, lambdas, etc. But using sum() always makes the code easier to read. Simple Code is the goal! Faster sum() is just a tool to reach it! Sum() was designed to support different types from the beginning. For example, Tim Peters suggested to use it for the list of datetime.timedelta [1]. Unfortunately for some common types sum() is too slow. This encourages people to fallback to other solutions, write a code that is hard to read and maintain. That's why I suggest making sum a little faster. Still not the fastest thing ever, but fast enough to have a choice between "a little faster" and "easier to read". Faster sum() at least for something (e.g. lists) is already a good thing, but even the great "fast sum() for everything" goal is easy to reach. I already wrote some patches. So now there 4 suggestions/patches pending: 1. Fast sum() for lists and tuples. [2] This patch adds a special case optimisation. I wasn't going to do it that way, but sum() already have 2 special cases for ints and floats (yes, sum() already has special cases, and it has them for the last 6 years), one more special case to reach the "fast for everything" goal is not that bad. Practicality beats purity. This patch introduces no behavior changes, and may go even for python2 bugfix release. 2. Fast sum() for strings. [3] This patch is a small enhancement of #1, that makes sum() O(N) for strings. Obviously it introduces a behavior change and I do NOT suggest it for inclusion. I believe that ''.join() alternative is simple enough and don't have to be replaces with sum() for normal use cases. It is just to shows that sum() can be O(N) for strings. 3. Fast sum() for everything else. [4] This is the original patch that I suggested. It rearranges existing code so that sum() uses __iadd__ if available, making sum() fast for all built-in and custom types with fast __iadd__. Even with #1 this patch also introduces no behavior changes, and may go even for python2 bugfix release. 4. Explicit documentation of sum() complexity. If patches #1 or #3 are accepted it may be a good idea to explicitly state what sum() needs to be O(N) for user types. E.g.: """ function:: sum(iterable[, start]) Sums *start* and the items of an *iterable* from left to right and returns the total. *start* defaults to ``0``. To make advantage of faster __iadd__ implementation sum() uses it if possible. sum() has linear time complexity for built-in types and all types providing constant-time `__iadd__` or `__add__`. For some use cases, there are good alternatives to sum(). The preferred, fast way to concatenate a sequence of string is by [...] """ Patches #1 and #2 were written as an answer to "you can't make sum() fast for everything, e.g. strings and tuples". Yes, I can. :) Together these patches achieve the "fast sum() for everything" goal and make sure, that people know what is required to keep sum() fast. That's all the suggestions I have for now. Well, there's one more, but let's first decide about these. :) -- [1] http://mail.python.org/pipermail/python-dev/2003-April/034837.html [2] http://bugs.python.org/file30769/fastsum-special.patch without the string part [3] the string part of http://bugs.python.org/file30769/fastsum-special.patch [4] http://bugs.python.org/file30705/fastsum.patch
On 5 July 2013 08:41, Sergey <sergemp@mail.ru> wrote:
2. Fast sum() for strings. [3] This patch is a small enhancement of #1, that makes sum() O(N) for strings. Obviously it introduces a behavior change and I do NOT suggest it for inclusion. I believe that ''.join() alternative is simple enough and don't have to be replaces with sum() for normal use cases. It is just to shows that sum() can be O(N) for strings.
Why does this "obviously" introduce a behaviour change? If it is just a performance improvement, that's not a behaviour change. Can you clarify what behaviour will change? Also, why does there need to be a behaviour change for strings and not for any other type (you make no mention of behaviour changes anywhere else in this message that I can see). It's hard to see why there would be any argument over *pure* performance improvements. But behaviour changes, especially ones which are "needed" to get performance benefits, are a different matter. Apologies if this has already been discussed, but this thread has become quite complex, and I haven't been following the details (and thanks for the summary, BTW!) Paul
On 05/07/13 17:59, Paul Moore wrote:
On 5 July 2013 08:41, Sergey <sergemp@mail.ru> wrote:
2. Fast sum() for strings. [3] This patch is a small enhancement of #1, that makes sum() O(N) for strings. Obviously it introduces a behavior change and I do NOT suggest it for inclusion. I believe that ''.join() alternative is simple enough and don't have to be replaces with sum() for normal use cases. It is just to shows that sum() can be O(N) for strings.
Why does this "obviously" introduce a behaviour change? If it is just a performance improvement, that's not a behaviour change.
Have you tried summing strings? py> sum(['a', 'b'], '') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: sum() can't sum strings [use ''.join(seq) instead]
It's hard to see why there would be any argument over *pure* performance improvements. But behaviour changes, especially ones which are "needed" to get performance benefits, are a different matter.
Even pure performance improvements don't come for free. Adding all these extra special cases to sum increases the complexity of the code. More complex code, more documentation, more tests means more code to contain bugs, more documentation to be wrong, more tests which can fail. Simplicity is a virtue in and of itself. What benefit does this extra complexity give you? If you *only* look at the advantages, then *every* optimization looks like an obvious win. And if you *only* look at the risks, then every new piece of code looks like a problem to be avoided. The trick is to balance the two, and that relies on some sense of how often you will receive the benefit versus how often you pay the cost. Sergey's suggestion also has a more subtle behavioural change. In order to guarantee that sum is fast, i.e. O(N) instead of O(N**2), the semantics of sum have to change from "support anything which supports __add__" to "support anything which supports fast __iadd__". I don't believe that optimizing sum for non-numbers gives enough benefit to make up for the increased complexity. Who, apart from Sergey, uses sum() on large numbers of lists or tuples? Both have had quadratic behaviour for a decade, and I've never known anyone who noticed or cared. So it seems to me that optimizing these special cases will have very little benefit. I've already made it clear that I'm -0 on this: on balance, I believe the disadvantage of more complex code just slightly outweighs the benefit. But Sergey doesn't have to convince *me*, he just has to convince those who will accept or reject the patch. -- Steven
On 5 July 2013 09:36, Steven D'Aprano <steve@pearwood.info> wrote:
On 05/07/13 17:59, Paul Moore wrote:
On 5 July 2013 08:41, Sergey <sergemp@mail.ru> wrote:
2. Fast sum() for strings. [3]
This patch is a small enhancement of #1, that makes sum() O(N) for strings. Obviously it introduces a behavior change and I do NOT suggest it for inclusion. I believe that ''.join() alternative is simple enough and don't have to be replaces with sum() for normal use cases. It is just to shows that sum() can be O(N) for strings.
Why does this "obviously" introduce a behaviour change? If it is just a performance improvement, that's not a behaviour change.
Have you tried summing strings?
py> sum(['a', 'b'], '') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: sum() can't sum strings [use ''.join(seq) instead]
Ah, yes. I'd forgotten there was an explicit rejection of strings, rather than just the obvious performance trap. It's hard to see why there would be any argument over *pure* performance
improvements. But behaviour changes, especially ones which are "needed" to get performance benefits, are a different matter.
Even pure performance improvements don't come for free. Adding all these extra special cases to sum increases the complexity of the code. More complex code, more documentation, more tests means more code to contain bugs, more documentation to be wrong, more tests which can fail. Simplicity is a virtue in and of itself. What benefit does this extra complexity give you?
Good point - that was to an extent what I was trying to express by emphasizing *pure* - that very few things in practice are unqualified improvements. I didn't put it well, though. The history of all this is somewhere in the archives. I recall the original discussion (although not the details). The explicit rejection of str is clearly deliberate, and implemented by people who certainly knew how to special-case str to use ''.join(), so has Sergey researched the original discussion and explained why his proposal invalidates the conclusions reached at the time? (Again, I apologise for jumping in late - if this is all old news, I'll shut up :-))
If you *only* look at the advantages, then *every* optimization looks like an obvious win. And if you *only* look at the risks, then every new piece of code looks like a problem to be avoided. The trick is to balance the two, and that relies on some sense of how often you will receive the benefit versus how often you pay the cost.
Sergey's suggestion also has a more subtle behavioural change. In order to guarantee that sum is fast, i.e. O(N) instead of O(N**2), the semantics of sum have to change from "support anything which supports __add__" to "support anything which supports fast __iadd__".
Whoa. That's a non-trivial change. I thought the proposal was "guarantee O(N) for types with a constant-time __iadd__ and don't impact performance for other types". But that of course begs the question of what happens in pathological cases where __iadd__ is (say) O(N^3). There's no way to programatically detect whether __iadd__ or __add__ gives the best performance, so what I had thought of course cannot be the case :-(
I don't believe that optimizing sum for non-numbers gives enough benefit to make up for the increased complexity. Who, apart from Sergey, uses sum() on large numbers of lists or tuples? Both have had quadratic behaviour for a decade, and I've never known anyone who noticed or cared. So it seems to me that optimizing these special cases will have very little benefit.
I've already made it clear that I'm -0 on this: on balance, I believe the disadvantage of more complex code just slightly outweighs the benefit. But Sergey doesn't have to convince *me*, he just has to convince those who will accept or reject the patch.
Your points are good ones. Personally, I have no need for this change - like you, I never use sum() on anything other than numbers. Given the subtle implications of the change, and the fact that it explicitly reverses a deliberate decision by the core devs, I think this proposal would require a PEP to have any chance of being accepted. Paul.
Is there a better way to flatten lists than [item for sublist in l for item in sublist]? I naturally don't use sum() for much, but if i could, sum(my_list) looks much better than [item for sublist in my_list for item in sublist]. Looking at http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-..., all the alternatives seem equally obtuse and verbose. I have a lot of list flattenings I would love to use sum(my_list) for if I had the chance. On Fri, Jul 5, 2013 at 5:20 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 5 July 2013 09:36, Steven D'Aprano <steve@pearwood.info> wrote:
On 05/07/13 17:59, Paul Moore wrote:
On 5 July 2013 08:41, Sergey <sergemp@mail.ru> wrote:
2. Fast sum() for strings. [3]
This patch is a small enhancement of #1, that makes sum() O(N) for strings. Obviously it introduces a behavior change and I do NOT suggest it for inclusion. I believe that ''.join() alternative is simple enough and don't have to be replaces with sum() for normal use cases. It is just to shows that sum() can be O(N) for strings.
Why does this "obviously" introduce a behaviour change? If it is just a performance improvement, that's not a behaviour change.
Have you tried summing strings?
py> sum(['a', 'b'], '') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: sum() can't sum strings [use ''.join(seq) instead]
Ah, yes. I'd forgotten there was an explicit rejection of strings, rather than just the obvious performance trap.
It's hard to see why there would be any argument over *pure* performance
improvements. But behaviour changes, especially ones which are "needed" to get performance benefits, are a different matter.
Even pure performance improvements don't come for free. Adding all these extra special cases to sum increases the complexity of the code. More complex code, more documentation, more tests means more code to contain bugs, more documentation to be wrong, more tests which can fail. Simplicity is a virtue in and of itself. What benefit does this extra complexity give you?
Good point - that was to an extent what I was trying to express by emphasizing *pure* - that very few things in practice are unqualified improvements. I didn't put it well, though.
The history of all this is somewhere in the archives. I recall the original discussion (although not the details). The explicit rejection of str is clearly deliberate, and implemented by people who certainly knew how to special-case str to use ''.join(), so has Sergey researched the original discussion and explained why his proposal invalidates the conclusions reached at the time? (Again, I apologise for jumping in late - if this is all old news, I'll shut up :-))
If you *only* look at the advantages, then *every* optimization looks like an obvious win. And if you *only* look at the risks, then every new piece of code looks like a problem to be avoided. The trick is to balance the two, and that relies on some sense of how often you will receive the benefit versus how often you pay the cost.
Sergey's suggestion also has a more subtle behavioural change. In order to guarantee that sum is fast, i.e. O(N) instead of O(N**2), the semantics of sum have to change from "support anything which supports __add__" to "support anything which supports fast __iadd__".
Whoa. That's a non-trivial change. I thought the proposal was "guarantee O(N) for types with a constant-time __iadd__ and don't impact performance for other types". But that of course begs the question of what happens in pathological cases where __iadd__ is (say) O(N^3). There's no way to programatically detect whether __iadd__ or __add__ gives the best performance, so what I had thought of course cannot be the case :-(
I don't believe that optimizing sum for non-numbers gives enough benefit to make up for the increased complexity. Who, apart from Sergey, uses sum() on large numbers of lists or tuples? Both have had quadratic behaviour for a decade, and I've never known anyone who noticed or cared. So it seems to me that optimizing these special cases will have very little benefit.
I've already made it clear that I'm -0 on this: on balance, I believe the disadvantage of more complex code just slightly outweighs the benefit. But Sergey doesn't have to convince *me*, he just has to convince those who will accept or reject the patch.
Your points are good ones. Personally, I have no need for this change - like you, I never use sum() on anything other than numbers.
Given the subtle implications of the change, and the fact that it explicitly reverses a deliberate decision by the core devs, I think this proposal would require a PEP to have any chance of being accepted.
Paul.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 5 Jul, 2013, at 11:28, Haoyi Li <haoyi.sg@gmail.com> wrote:
Is there a better way to flatten lists than [item for sublist in l for item in sublist]? I naturally don't use sum() for much, but if i could, sum(my_list) looks much better than [item for sublist in my_list for item in sublist]. Looking at http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-..., all the alternatives seem equally obtuse and verbose. I have a lot of list flattenings I would love to use sum(my_list) for if I had the chance.
There's also list(itertools.chain.from_iterable(somelist)). I haven't benchmarked that, I don't need to flatten lists often enough that I need to worry about the performance. Ronald
On 5 July 2013 10:28, Haoyi Li <haoyi.sg@gmail.com> wrote:
Is there a better way to flatten lists than [item for sublist in l for item in sublist]? I naturally don't use sum() for much, but if i could, sum(my_list) looks much better than [item for sublist in my_list for item in sublist]. Looking at http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-..., all the alternatives seem equally obtuse and verbose. I have a lot of list flattenings I would love to use sum(my_list) for if I had the chance.
from itertools import chain def listjoin(iterable_of_iterables): return list(chain.from_iterable(iterable_of_iterables)) Oscar
On 5 July 2013 10:28, Haoyi Li <haoyi.sg@gmail.com> wrote:
Is there a better way to flatten lists than [item for sublist in l for item in sublist]? I naturally don't use sum() for much, but if i could, sum(my_list) looks much better than [item for sublist in my_list for item in sublist]. Looking at http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-..., all the alternatives seem equally obtuse and verbose. I have a lot of list flattenings I would love to use sum(my_list) for if I had the chance.
Only partially facetious, but newlist = flatten(list_of_lists) In other words, if you don't like the nested comprehension, hide it in a personal function. No need to force the functionality into an existing builtin, Not-every-one-liner-needs-to-be-a-builtin-ly y'rs, Paul
On 5 Jul, 2013, at 11:20, Paul Moore <p.f.moore@gmail.com> wrote:
On 5 July 2013 09:36, Steven D'Aprano <steve@pearwood.info> wrote:
Sergey's suggestion also has a more subtle behavioural change. In order to guarantee that sum is fast, i.e. O(N) instead of O(N**2), the semantics of sum have to change from "support anything which supports __add__" to "support anything which supports fast __iadd__".
Whoa. That's a non-trivial change. I thought the proposal was "guarantee O(N) for types with a constant-time __iadd__ and don't impact performance for other types". But that of course begs the question of what happens in pathological cases where __iadd__ is (say) O(N^3). There's no way to programatically detect whether __iadd__ or __add__ gives the best performance, so what I had thought of course cannot be the case :-(
It's not that bad, any type where __iadd__ has (significantly) worse performance that __add__ is IMHO broken, as you could just not have implemented __iadd__ in the first place. I'm only +O on using __iadd__ in sum because it is unclear how useful this would be in real-world code, but does have a simple enough implementation that is still easily explained. I'd be -1 on adding custom hooks to improve the performance of specific types (such, adding tricks for speeding up joining a sequence of tuples or strings) as that makes the implementation harder and increases the conceptual complexity of the function as well. Ronald
On 5 July 2013 11:02, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
I'm only +O on using __iadd__ in sum because it is unclear how useful this would be in real-world code, but does have a simple enough implementation that is still easily explained. I'd be -1 on adding custom hooks to improve the performance of specific types (such, adding tricks for speeding up joining a sequence of tuples or strings) as that makes the implementation harder and increases the conceptual complexity of the function as well.
This is a perfect summary of where I am on this proposal, but for slightly different reasons. If you special-case, either "sum(tuple_subclass)" is going to be asymptotically slower (and hence code that duck-types should not rely on it) or broken. Thus, special-casing is out of the window for me. Using __iadd__, btw, cannot be a "bugfix", AFAIK, because it changes behavior (subtly). But code that depends on that distinction is broken, so I'm OK with it going forward. I appreciate the speed-up, but as Steven convinced me it's not that big a deal. Hence +0.
From: Sergey <sergemp@mail.ru> Sent: Thursday, July 4, 2013 2:54 AM
On Jul 04, 2013 Steven D'Aprano wrote:
[to avoid confusion: this is Sergey's summary, not Steven's]
This message is long, so here's its short summary:
Right on some points, but you're making unwarranted assumptions for these two:
* sum() can *always* be fast! (patch and tests) * linked list is O(n) where n is number of lists to add
sum simply cannot *always* be fast. E.g. summing tuples will still be slow even with your suggestion.
Yes, it can! That's the point of the original idea!
The original patch [2] optimizes lists, because it was easy to do. But nothing stops you from optimizing other (two?) types.
It's not two other types, it's every type anyone ever has built or will build that doesn't have a fast __iadd__. If I create a new type, I can very easily make it addable, iterable, displayable, pickleable, etc., with simple Python code. But I can't make it summable without patching the builtin C function and recompiling the interpreter. So, as Steven said, sum cannot always be fast.
As for linked lists, the point of linked list is to insert items fast. So any decent implementation of it should store a pointer to its head and tail, should implement a O(1) __iadd__ using tail pointer, and thus falls under my first patch. There's not much sence in [single] linked list if it has no __iadd__.
Have you never heard of Lisp or any of its successors? A cons is a single-linked list with no tail pointer and an O(N) __iadd__. The node type is the list type; it's just a head value and a next pointer to another list. This data structure various kinds of O(1) operations (including mutating operations) that other types do not. For a trivial example: >>> a = linkedlist([0, 1, 2]) >>> b = a.next >>> b.next = linkedlist([10, 20]) >>> a linkedlist([0, 1, 10, 20]) There is no way to make this work is a holds a tail pointer. The trivial design would leave a pointing at the 2, which isn't even a member of a. Making a.next be a copying operation gives the wrong answer (and, more generally, breaks every mutating operation on cons-lists) and makes almost everything O(N) instead of O(1). Storing head and tail pointers at each node makes every mutating operation at least O(N) instead of O(1), and also means that two different lists can't share the same tail, which breaks most mutating tree operations. Storing a stack of lists that refer to each node makes everything O(N^2). Integrating that stack into the list means you have a double-linked list instead of a single-linked list. Not having a tail pointer is inherent in the cons-list data structure, and all the algorithms built for that type depend on it.
Yes. If Python used & for concatenation, we wouldn't have to worry
about sum(lists or strings or tuples) being quadratic, because people wouldn't call sum on lists, strings or tuples.
Heh. If Python had no sum() we wouldn't have to worry about people using it.
Well, there's always the Java-ish solution of adding a Summable ABC and having it only work on matching values (or just using numbers.Number). But I don't think it's a problem that there are types that are addable and aren't (fast-)summable, or that this isn't expressed directly in the language. Python doesn't avoid providing tools that are useful in their obvious uses, even if they can be harmful in silly uses. In a case like this, you don't ban the screwdriver, or pile weight onto it so it's also a good hammer; you just try to make the hammer easier to spot so people don't pick up the screwdriver. If people really do need to concatenate a bunch of tuples this often, maybe the answer is to move chain to builtins. If it's not important enough to move chain to builtins, it's probably not important enough to do anything else. If Python had no lists we wouldn't have to worry about
people concatenating them. If there was no Python we wouldn't have to worry at all. But the world would be poor without all these great things...
Seriously, I miss add for set(). I needed it when I had a dictionary like {x:set(...), ...} and needed a set of all the values from it. I wanted to use sum(dict.values()), that would be easy and obvious, but I couldn't, because set() does not support __add__. So I had to write a few lines of loop instead of a few characters. :(
So what you really want is to be able to combine things using any arbitrary operation, instead of just addition? That's the point I was making in my earlier email: What you want is reduce. It's just like sum, but you specify the operation, and you can start with next(iterator) instead of having to specify a start value. Why is sum a builtin, while reduce isn't? Sum is usable in a pretty restricted set of cases, and in most of those cases, it's the obvious way to do it. Reduce is usable in a much broader set of cases, but in most of those cases, there's a better way to do it. Broadening the usability of sum to make it more like reduce is not a positive change.
On Jul 4, 2013 Andrew Barnert wrote:
Right on some points, but you're making unwarranted assumptions for these two:
* sum() can *always* be fast! (patch and tests) * linked list is O(n) where n is number of lists to add
It's not two other types, it's every type anyone ever has built or will build that doesn't have a fast __iadd__. If I create a new type, I can very easily make it addable, iterable, displayable, pickleable, etc., with simple Python code. But I can't make it summable without patching the builtin C function and recompiling the interpreter.
Yes, you can! You just need to implement a fast __iadd__ or __add__ for your type to make it O(n) summable. And you can always do that, can't you? If that is not obvious we can make it explicit in sum() description. E.g.: """ function:: sum(iterable[, start]) Sums *start* and the items of an *iterable* from left to right and returns the total. *start* defaults to ``0``. To make advantage of faster __iadd__ implementation sum() uses it if possible. sum() has linear time complexity for built-in types and all types providing constant-time `__iadd__` or `__add__`. For some use cases, there are good alternatives to sum(). [...] """ Of course that's assuming that patches making sum O(n) are accepted.
A cons is a single-linked list with no tail pointer and an O(N) __iadd__. The node type is the list type; it's just a head value and a next pointer to another list. This data structure various kinds of O(1) operations (including mutating operations) that other types do not. For a trivial example: >>> a = linkedlist([0, 1, 2]) >>> b = a.next >>> b.next = linkedlist([10, 20]) >>> a linkedlist([0, 1, 10, 20])
Thank you for detailed explanation, now I understand what you mean. Personally I don't think such implementation would be very useful. (It's certainly unusual for python's “batteries included” philosophy). What I would call a linked-list is a separate type where your nodes are just its internal representation. Something like (second link from google "python linked list"): http://alextrle.blogspot.com/2011/05/write-linked-list-in-python.html (I would also add 'length' to 'head' and 'tail' to have O(1) len).
There is no way to make this work is a holds a tail pointer.
There is. You just need to separate the nodes from the list object, and turn nodes into internal details, invisible outside. So that your sample would turn into: >>> a = linkedlist([0, 1, 2]) >>> a.replacefrom(2, linkedlist([10, 20])) >>> a linkedlist([0, 1, 10, 20]) Or maybe even: >>> a = linkedlist([0, 1, 2]) >>> b = a[1:] >>> b[1:] = linkedlist([10, 20]) >>> a linkedlist([0, 1, 2]) >>> b linkedlist([1, 10, 20]) I don't like the idea that `a` implicitly changes when I change `b`. And yes, b=a[1:] would be O(N) since it would have to copy all the elements to make sure that change to one list won't affect another one. But that's my vision.
Making a.next be a copying operation gives the wrong answer (and, more generally, breaks every mutating operation on cons-lists) and makes almost everything O(N) instead of O(1).
Yes, you're right, this turns copy into O(N). But you can still keep other operations fast. For example to implement: a += b you only need to walk over the items of `b`. So a simple loop (and my patched sum): for x in manylists: a += x would still be O(N) where N is a total number of elements. Again, that's how *I* would implement that. But you CAN have a fast __iadd__ even for your simple a.next case! You only need to initialize `tail` before calling sum() and update it inside __iadd__, this way you get O(N) sum() where N is a total number of elements in all lists. Such __iadd__ implementation would be destructive, since it would modify all other lists pointing to those elements, but I guess this is how you wanted it do be.
If people really do need to concatenate a bunch of tuples this often, maybe the answer is to move chain to builtins.
Yes, I understand that we can restrict sum(), for example it can throw an error for non-numbers as it currently does for strings. What I can't understand is why we should restrict it if instead we can make it fast?
Seriously, I miss add for set(). I needed it when I had a dictionary like {x:set(...), ...} and needed a set of all the values from it. I wanted to use sum(dict.values()), that would be easy and obvious, but I couldn't, because set() does not support __add__. So I had to write a few lines of loop instead of a few characters. :(
So what you really want is to be able to combine things using any arbitrary operation, instead of just addition?
No! I only want to make python code simple. Simple is better than complex, right? And sum() makes code simple. But sometimes it's too slow, and cannot be used. So I wanted to make sum() faster, to make even more code simple. That's all. It's just someone said, that "+" should not be used in the first place, so I answered, that missing "+" makes code complex, and provided an example about sets.
On Jul 4, 2013 11:45 PM, "Sergey" <sergemp@mail.ru> wrote:
Or maybe even: >>> a = linkedlist([0, 1, 2]) >>> b = a[1:] >>> b[1:] = linkedlist([10, 20]) >>> a linkedlist([0, 1, 2]) >>> b linkedlist([1, 10, 20])
I don't like the idea that `a` implicitly changes when I change `b`. And yes, b=a[1:] would be O(N) since it would have to copy all the elements to make sure that change to one list won't affect another one.
If you don't like that don't use a linked list. That's like complaining about
a = [1, 2, 3] b = a a[1] = 4 b [1, 4, 3]
A linked list is not just another implementation of 'list'. --- Bruce
On 7/5/2013 2:43 AM, Sergey wrote:
I don't like the idea that `a` implicitly changes when I change `b`.
When a and b are the same things or one is part of the other, that must happen. This is the nature of mutable structures (when all or part of a structure can be named, which should be always). Here is Andrew Barnett's example translated into Python. a = [1, [2, [3, None]]] b = a[1] b[1] = [10, [20, None]] print(a)
[1, [2, [10, [20, None]]]]
The difference bettween that being a lisp list and a python list is that list sees len(a) == 4, which Python would say 2, but that is because lisp looks as simple nested structures as if there were flat, becuase the nesting is, in a sense, an internal detail. Here is a possible corresponding iadd. class Lisp(list): # the second member of each Lisp is either a Lisp or None. def __iadd__(self, other): if not instance(other, Lisp): raise TypeError while self[1] is not None: self = self[1] self[1] = other -- Terry Jan Reedy
From: Sergey <sergemp@mail.ru> Sent: Thursday, July 4, 2013 11:43 PM
On Jul 4, 2013 Andrew Barnert wrote:
Right on some points, but you're making unwarranted assumptions for these two:
* sum() can *always* be fast! (patch and tests) * linked list is O(n) where n is number of lists to add
It's not two other types, it's every type anyone ever has built or will build that doesn't have a fast __iadd__. If I create a new type, I can very easily make it addable, iterable, displayable, pickleable, etc., with simple Python code. But I can't make it summable without patching the builtin C function and recompiling the interpreter.
Yes, you can! You just need to implement a fast __iadd__ or __add__ for your type to make it O(n) summable.
Well, in Python 3.3, or even 2.3, you just need to implement a fast __add__. So, if that's an acceptable answer, then we don't need to do anything. The reason it's not acceptable to you is that sometimes, you can't implement a fast __add__. But sometimes, you can't implement a fast __iadd__, either.
And you can always do that, can't you?
No. You can't implement a fast __iadd__ for tuple. In fact, there is no single operation you could implement for tuple. To make it fast-summable, you had to modify sum so that it converts to list, extends, and converts back to tuple at the end. If that's going to be extendable to other types, you need to give them control over some external context—e.g., a way to specify how to set up the start of a sum operation before the first __iadd__ and how to conclude it after the last __iadd__.
If that is not obvious we can make it explicit in sum() description. E.g.: […] To make advantage of faster __iadd__ implementation sum() uses it if possible. sum() has linear time complexity for built-in types and all types providing constant-time `__iadd__` or `__add__`.
Note that integer addition isn't actually constant, it's linear on the word size of the integers. Which means sum isn't linear for integers. So, this is misleading right off the bat. Currently, because we don't try to specify anything except to imply that sum is an appropriately fast way to sum numbers, there's no such problem.
A cons is a single-linked list with no tail pointer and an O(N) __iadd__. The node type is the list type; it's just a head value and a next pointer to another list.
Personally I don't think such implementation would be very useful.
Given that nearly every program every written in Lisp and its successors makes heavy use of such an implementation, I submit that your intuition may be wrong.
(It's certainly unusual for python's “batteries included” philosophy).
What does "batteries included" have to do with excluding data types?
What I would call a linked-list is a separate type where your nodes are just its internal representation.
If you make the lists and nodes separate types, and the nodes private, you have to create yet a third type, like the list::iterator type in C++, which either holds a reference to a node plus a reference to a list, or can be passed to a list in the same way indices are passed to an array. Without that, you can't do anything useful with lists at all, because any operation is O(N). Just as arrays use indices and hash tables use keys, linked lists use node references.
There is no way to make this work is a holds a tail pointer.
There is. You just need to separate the nodes from the list object, and turn nodes into internal details, invisible outside.
So that your sample would turn into: >>> a = linkedlist([0, 1, 2]) >>> a.replacefrom(2, linkedlist([10, 20])) >>> a linkedlist([0, 1, 10, 20])
Or maybe even: >>> a = linkedlist([0, 1, 2]) >>> b = a[1:] >>> b[1:] = linkedlist([10, 20])
>>> a linkedlist([0, 1, 2]) >>> b linkedlist([1, 10, 20])
I don't like the idea that `a` implicitly changes when I change `b`.
Then you don't like linked lists. Saying "There is a way to make it work, just make it do something different, which is O(N) instead of O(1) and won't work with the algorithms you want to use" is not an answer. You don't have to use linked lists—there's a reason they're not built in to Python, after all. But if you want to talk about linked lists, you have to talk about linked lists, not some similar data type you've invented that has most of the disadvantages of linked lists and none of the advantages, and that no one will ever use.
And yes, b=a[1:] would be O(N) since it would have to copy all the
elements to make sure that change to one list won't affect another one. But that's my vision.
Even b=a[n] is O(N) on linked lists, even without copying. Which is exactly why linked lists provide head and next instead of indexing and slicing. Trying to use linked lists as if they were arrays is as silly as the other way around.
Making a.next be a copying operation gives the wrong answer (and, more generally, breaks every mutating operation on cons-lists) and makes almost everything O(N) instead of O(1).
Yes, you're right, this turns copy into O(N). But you can still keep other operations fast.
No, you can make one operation fast, at the expense of making every other operation slow. That's not a good tradeoff. It's like giving a tree O(1) searches by attaching a hash table that maps keys to node pointers, and then claiming that this is an improvement, and who cares that inserts are now O(N) instead of O(log N). And again, you've also ignored the fact that, performance aside, it _breaks every algorithm people use mutable linked lists for_.
But you CAN have a fast __iadd__ even for your simple a.next case! You only need to initialize `tail` before calling sum() and update it inside __iadd__
Initialize tail where exactly? In a global variable?
If people really do need to concatenate a bunch of tuples this
often, maybe the answer is to move chain to builtins.
Yes, I understand that we can restrict sum(), for example it can throw an error for non-numbers as it currently does for strings. What I can't understand is why we should restrict it if instead we can make it fast?
How is that even a response to my point? I suggest moving chain to builtins, and you ask why we should add restrictions to sum?
On Jul 5, 2013 Andrew Barnert wrote:
Yes, you can! You just need to implement a fast __iadd__ or __add__ for your type to make it O(n) summable.
Well, in Python 3.3, or even 2.3, you just need to implement a fast __add__. So, if that's an acceptable answer, then we don't need to do anything. [...]
And you can always do that, can't you?
No. You can't implement a fast __iadd__ for tuple.
Well... Yes, I can! I can't make __iadd__ faster, because tuple has no __iadd__, however I can make a faster __add__. But as long as sum() is the only (?) function suffering from this problem it was easier to do that optimization in sum() instead.
If that's going to be extendable to other types
Are there any other types (except strings, that are blocked anyway)? Looks like tuple is the only built-in type having no fast __iadd__, and sum() is the only function suffering from that. So, to make sum() "fast for everything by default" we only need to make sum use __iadd__ and write a special case for sum+tuples.
To make advantage of faster __iadd__ implementation sum() uses it if possible. sum() has linear time complexity for built-in types and all types providing constant-time `__iadd__` or `__add__`.
Note that integer addition isn't actually constant, it's linear on the word size of the integers. Which means sum isn't linear for integers.
That still makes sum O(N) with N being total number of bytes in all integers. Or how do you suggest to explain sum() complexity in docs? That text seems fair enough for me. Something like: sum() has complexity O(N)*C where N is the total number of elements and C is complexity of single __iadd__ operation or __add__ operation if __iadd__ is not supported. looks too cumbersome.
Personally I don't think such implementation would be very useful.
Given that nearly every program every written in Lisp and its successors makes heavy use of such an implementation, I submit that your intuition may be wrong.
They don't have much choice. And they're not using sum(). We're talking about python, and discussing use of sum() in python for such lists in particular. It's just you said:
I'm hostile to any attempt to encourage people to treat sum() as the preferred way to concatenate large amounts of data, because that will surely lead them into bad habits and will end with them trying to sum() a lot of tuples or linked lists or something and getting O(n**2) performance.
Which implies that using such implementation with sum could lead to O(N**2) performance. But it could not, because such implementation is not addable. So, no problem. To have a problem you must modify your implementation. And if you're changing it anyway, you have the power to solve this problem too. Explicitly writing about sum() complexity in documentation makes sure that you're aware of possible slowness.
(It's certainly unusual for python's “batteries included” philosophy).
What does "batteries included" have to do with excluding data types?
Excluding data types? What do you mean? I wasn't sure what you meant when you said about problems with linked lists, so I was thinking that you mean something like "if one day linked lists get their way into python standard libraries and people will try using sum() on them..." That's why I said that such implementation would be too skimped.
What I would call a linked-list is a separate type where your nodes are just its internal representation.
If you make the lists and nodes separate types, and the nodes private, you have to create yet a third type, like the list::iterator type in C++
If there would be a need for it, why not? Or maybe I won't need it (then I get something like collections.deque, a good tool by the way).
I don't like the idea that `a` implicitly changes when I change `b`.
Then you don't like linked lists. Saying "There is a way to make it work, just make it do something different, which is O(N) instead of O(1) and won't work with the algorithms you want to use" is not an answer.
That wasn't saying "just make it do something different". That was saying "you can have linked lists in python, that are O(N) summable".
No, you can make one operation fast, at the expense of making every other operation slow. That's not a good tradeoff.
In that implementation I'm making many operations fast. Basically, I make ALL inplace-operations fast. :) As a bonus I get a guarantee that I won't implicitly modify another variable while modifying this one. For someone it's not a good tradeoff, for others it is.
And again, you've also ignored the fact that, performance aside, it _breaks every algorithm people use mutable linked lists for_.
Those algos don't use sum(). And lists they use are not summable. To get a problem you need different lists. So it does not matter.
But you CAN have a fast __iadd__ even for your simple a.next case! You only need to initialize `tail` before calling sum() and update it inside __iadd__
Initialize tail where exactly? In a global variable?
Wherever you want. In a global variable, or in a class variable near 'next'. --
On 07/08/2013 03:22 PM, Sergey wrote:
(It's certainly unusual for python's “batteries included” philosophy).
What does "batteries included" have to do with excluding data types? Excluding data types? What do you mean?
I wasn't sure what you meant when you said about problems with linked lists, so I was thinking that you mean something like "if one day linked lists get their way into python standard libraries and people will try using sum() on them..." That's why I said that such implementation would be too skimped.
We need to keep in mind how python programmer will use the tools, "Batteries", we add to the library. Python programmers do currently write programs with linked lists. And they use what ever is in the library they think is useful... the whole point of "batteries included". -Ron
Python programmers do currently write programs with linked lists. And they use what ever is in the library they think is useful... the whole point of "batteries included".
On the other hand, python programmers do currently write programs with normal lists, and sum(), and using sum() on lists would be useful. They're BOTH in the standard library too! In fact, this seems like a perfect argument for having sum() work on lists! The objection seems like optimizing for a hypothetical, potential use case far in the future at the expense of a concrete use case now. "if you can't make it do EVERYTHING, then we shouldn't make it do ANYTHING or people will get confused!" (paraphrased and exaggerated for theatrical effect). This is all on the assumption that you consider flattening list-of-lists a concrete use case. I for one find it annoying that i have to write a verbose long thingy every time i need to flatten lists, and I have probably needed to flatten lists about a dozen times in the last 3 months and used linked lists not-at-all. Maybe you write a ton of pure-functional algorithms making great use of the persistence of singly-linked-lists for performant non-mutating head-updates, and never use vanilla lists in the code you write, and so having sum() work on linked lists is of great import. On Tue, Jul 9, 2013 at 5:58 AM, Ron Adam <ron3200@gmail.com> wrote:
On 07/08/2013 03:22 PM, Sergey wrote:
(It's certainly unusual for python's “batteries included” philosophy).
What does "batteries included" have to do with excluding data types?
Excluding data types? What do you mean?
I wasn't sure what you meant when you said about problems with linked lists, so I was thinking that you mean something like "if one day linked lists get their way into python standard libraries and people will try using sum() on them..." That's why I said that such implementation would be too skimped.
We need to keep in mind how python programmer will use the tools, "Batteries", we add to the library.
Python programmers do currently write programs with linked lists. And they use what ever is in the library they think is useful... the whole point of "batteries included".
-Ron
______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
From: Haoyi Li <haoyi.sg@gmail.com> Sent: Monday, July 8, 2013 5:39 PM
I for one find it annoying that i have to write a verbose long thingy every time i need to flatten lists
What verbose long thingy? You just need to write: flatten = itertools.chain.from_iterable Or, if you use more-itertools: from more_itertools import flatten Someone suggested moving this to builtins as "chain" or "flatten" or similar. I'm at least +0 on this. At any rate, it's about as simple as can be. Because it gives you an iterable, you don't pay the cost (in time or space) of building a list unless you need to, which means in many use cases it's actually much faster than sum, at the cost of being a little bit slower for lists when you do need a list. It also flattens any iterable of sequences, or even any iterable of iterables, in the same time—linear in the total size. With no custom optimizations, it works with tuples, blist.sortedlists, cons lists, or _anything else you can throw at it_. And it's obvious what it does. If you sum three sortedlists, do you get the first list's elements, then the second, then the third, or a merge of the three? I don't know. If you chain or flatten three sortedlists, it's obvious which one you get.
Someone suggested moving this to builtins as "chain" or "flatten" or similar. I'm at least +0 on this.
And it's obvious what it does. If you sum three sortedlists, do you get
That would be nice too; I consider both imports and third-party-modules to be part of the length-of-code for boilerplate-accounting purposes (not sure if others do), and having an additional `import itertools; flatten = blahblah` brings it up to 3 lines of code to flatten a list by that technique. My files aren't very long, so I can't just do it once at the top of everything.py and use it throughout my project. Three lines of code to flatten a list! Or two-lines with a third party package! May aswell use a for loop and accumulator. the first list's elements, then the second, then the third, or a merge of the three? I don't know. If you chain or flatten three sortedlists, it's obvious which one you get. Does python even *have* a sortedlist in the standard library? If they're third-party-module classes, then having to do some thinking to see what the builtins do to them seems acceptable to me; it's up to the sortedlist guy to document that "yeah you can sum my sortedlists and it'll do XXX". I mean you can probably *already* sum() them, and it'll do the same thing, just probably slower. <rant>I would also really like a nice object-oriented sortedlist to be part of the standard library, instead of that nasty C-style heapq stuff.</rant> I'd still rather we make use of the nice new generic dispatch stuff to make our builtins re-usable, rather than having a set of crufty only-really-works-on-builtins functions cluttering the *global* namespaces, and having to manually pick from *another* bunch of non-generic custom functions to do the same (conceptual) thing to different types. I don't think that's going to happen anytime before Python 4.0 though. This sort of "yeah, function_x() doesn't work on y, you have to use special_function_z() to concat ys and thingy_z() to concat ws" is workable, but reminds me of my bad-old php days. -Haoyi On Tue, Jul 9, 2013 at 9:02 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
From: Haoyi Li <haoyi.sg@gmail.com> Sent: Monday, July 8, 2013 5:39 PM
I for one find it annoying that i have to write a verbose long thingy every time i need to flatten lists
What verbose long thingy? You just need to write:
flatten = itertools.chain.from_iterable
Or, if you use more-itertools:
from more_itertools import flatten
Someone suggested moving this to builtins as "chain" or "flatten" or similar. I'm at least +0 on this.
At any rate, it's about as simple as can be.
Because it gives you an iterable, you don't pay the cost (in time or space) of building a list unless you need to, which means in many use cases it's actually much faster than sum, at the cost of being a little bit slower for lists when you do need a list.
It also flattens any iterable of sequences, or even any iterable of iterables, in the same time—linear in the total size. With no custom optimizations, it works with tuples, blist.sortedlists, cons lists, or _anything else you can throw at it_.
And it's obvious what it does. If you sum three sortedlists, do you get the first list's elements, then the second, then the third, or a merge of the three? I don't know. If you chain or flatten three sortedlists, it's obvious which one you get.
On Jul 8, 2013, at 18:25, Haoyi Li <haoyi.sg@gmail.com> wrote:
<rant>I would also really like a nice object-oriented sortedlist to be part of the standard library, instead of that nasty C-style heapq stuff.</rant>
I agree. heapq isn't really a sorted list because it's neither index-accessible nor key-accessible. And bisect isn't really a sorted list because everything but lookup is O(N). And both of them have C-style APIs to boot. AFAIK, two different implementations of sorted collections have been offered for the stdlib, and both were rejected because they tried to offer the wrong thing: blist was sold as a replacement for the existing list and tuple, which also happens to give you sorted list/dict/set types for free, while the other one (I forget the name) was offered as a standard binary tree library, which also happens to give you sorted list/dict/set types for free. I would be very happy to get sorted list/dict/set types added to the stdlib, ideally using the blist interface plus the views and key-slicing stuff from bintrees. And I don't care which implementation we get, as long as it's logarithmic, and there's a fast native accelerator for CPython alongside pure python code that works well in PyPy. There are multiple packages on PyPI that are 90% of the way there. I think it's just a matter of picking one, and either rallying the author to make the changes and commit to maintenance, or taking it over and committing to maintenance yourself. Either way, if you pull it off, I'll be your best friend forever and ever. (Your macro library is very cool, but a collections.sortedlist, I would use every day of my life, which makes it even cooler.)
I'd still rather we make use of the nice new generic dispatch stuff to make our builtins re-usable, rather than having a set of crufty only-really-works-on-builtins functions cluttering the *global* namespaces, and having to manually pick from *another* bunch of non-generic custom functions to do the same (conceptual) thing to different types.
The point of using chain instead of sum to concatenate sequences or iterators is that it automatically works for every kind of iterable you can imagine, so it doesn't need any type-switching or generics or anything of the sort. Also, generic dispatch wouldn't really help here, as we need to dispatch on the type(s) that the iterable yields, not the type of the iterable itself.
I don't think that's going to happen anytime before Python 4.0 though. This sort of "yeah, function_x() doesn't work on y, you have to use special_function_z() to concat ys and thingy_z() to concat ws" is workable, but reminds me of my bad-old php days.
That's exactly why I don't like expanding sum. At best, it can be the obvious way to concatenate mutable sequences with fast __iadd__ plus some builtin immutable sequences but not others... that's a mess. A chain/flatten/concat/whatever function can be (and already is, with two wasted lines) the obvious way to concatenate any iterables, no matter what. Meanwhile, sum is the obvious way to sum things that are obviously summable (numbers, matrices, etc.), and nothing else.
-Haoyi
On Tue, Jul 9, 2013 at 9:02 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
From: Haoyi Li <haoyi.sg@gmail.com> Sent: Monday, July 8, 2013 5:39 PM
I for one find it annoying that i have to write a verbose long thingy every time i need to flatten lists
What verbose long thingy? You just need to write:
flatten = itertools.chain.from_iterable
Or, if you use more-itertools:
from more_itertools import flatten
Someone suggested moving this to builtins as "chain" or "flatten" or similar. I'm at least +0 on this.
At any rate, it's about as simple as can be.
Because it gives you an iterable, you don't pay the cost (in time or space) of building a list unless you need to, which means in many use cases it's actually much faster than sum, at the cost of being a little bit slower for lists when you do need a list.
It also flattens any iterable of sequences, or even any iterable of iterables, in the same time—linear in the total size. With no custom optimizations, it works with tuples, blist.sortedlists, cons lists, or _anything else you can throw at it_.
And it's obvious what it does. If you sum three sortedlists, do you get the first list's elements, then the second, then the third, or a merge of the three? I don't know. If you chain or flatten three sortedlists, it's obvious which one you get.
Andrew Barnert writes:
Meanwhile sum is the obvious way to sum things that are obviously summable (numbers, matrices, etc.), and nothing else.
My intuition matches yours, but I find this argument (and the rest of the arguments that say that "generic sum() is unobvious and wrong") logically unsatisfactory. It would be nice if you could provide a plausible definition of "summable" other than "__add__() is implemented". I don't have one. :-(
From: Stephen J. Turnbull <stephen@xemacs.org> Sent: Monday, July 8, 2013 11:23 PM
Andrew Barnert writes:
Meanwhile sum is the obvious way to sum things that are obviously summable (numbers, matrices, etc.), and nothing else.
My intuition matches yours, but I find this argument (and the rest of the arguments that say that "generic sum() is unobvious and wrong") logically unsatisfactory. It would be nice if you could provide a plausible definition of "summable" other than "__add__() is implemented". I don't have one. :-(
As I see it, there are three possibilities. 1. sum is not appropriate when __add__ means concatenation rather than adding. If you'd use PySequence_Concat/sq_concat rather than PyNumber_Add/nb_add in porting to the C API, or if you'd use a different operator in Python 4 if concatenation stopped being written as __add__, then you shouldn't use sum. The problem is usually, you're not writing C API code or Python 4, you're writing Python 3, so it's not always obvious what the facts are. But I don't think it's ever that hard to figure out. If we used & for concatenation, list.__add__ and str.__add__ would no longer exist, but np.matrix.__add__, datetime.__add__, and quaternion.__add__ would. 2. sum is not appropriate iff chain.from_iterable makes sense.* Needing a list doesn't make chain unusable here any more than it does with map, zip, etc.; just pass the result to list. But "You can't iterate ints" or "I'm treating these np.matrix 3-vectors as atomic objects, not collections" does mean chain is unusable, so sum is the answer. 3. sum is not appropriate when 0 doesn't make sense as a start value. Summing things means, by default, starting with 0 and adding repeatedly. You can provide a non-default start value, but it should be "similar to 0" or "compatible with 0" in some way. Note that you can add 0 to an int, a float, a quaternion, a numpy.matrix, and all kinds of other types that "act like numbers". And that means that testing 0+start or start+0 is a pretty good test for summable. This is admittedly imperfect,** but it's pretty close, and very concrete and simple. Personally, I'm leaning toward 2. If you come up with a type that is addable, and isn't iterable (or does the wrong thing when iterated), why not, it's summable. Better to let some corner-case false positives slip in than the reject some corner-case false negatives (as with 3), or to make them just impossible to decide (as with 1). * This is a little too loose. Surely there are types where __add__ is not addition, but also not iterable concatenation, right? But I don't think sum is an attractive nuisance there, unlike in the case with concatenation. Meanwhile, what about strings? Actually, chain makes perfect sense with strings; it's just that usually, you're just going to want to pass the iterable to ''.join, and if you're doing that, you can just pass the original strings to ''.join. So, no problem there. ** Most notably, I think adding a sequence of timedelta objects with a datetime start makes sense, and you can't add 0 to a datetime. Really, what you need here is a way to say that start + 0 * peek(iterable) is also acceptable, not just start + 0—and you can justify that more rigorously in terms of fields—but that's nearly impossible to implement, and way too complicated to explain. So, option 3 will reject some valid types.
On 07/09/2013 03:51 AM, Andrew Barnert wrote:
From: Stephen J. Turnbull <stephen@xemacs.org>
Sent: Monday, July 8, 2013 11:23 PM
Andrew Barnert writes:
Meanwhile sum is the obvious way to sum things that are obviously summable (numbers, matrices, etc.), and nothing else.
My intuition matches yours, but I find this argument (and the rest of the arguments that say that "generic sum() is unobvious and wrong") logically unsatisfactory. It would be nice if you could provide a plausible definition of "summable" other than "__add__() is implemented". I don't have one. :-(
As I see it, there are three possibilities.
1. sum is not appropriate when __add__ means concatenation rather than adding. If you'd use PySequence_Concat/sq_concat rather than PyNumber_Add/nb_add in porting to the C API, or if you'd use a different operator in Python 4 if concatenation stopped being written as __add__, then you shouldn't use sum. The problem is usually, you're not writing C API code or Python 4, you're writing Python 3, so it's not always obvious what the facts are. But I don't think it's ever that hard to figure out. If we used & for concatenation, list.__add__ and str.__add__ would no longer exist, but np.matrix.__add__, datetime.__add__, and quaternion.__add__ would.
What if we could specify the degree of specialisation of an operator? Kind of like a cast operation, but instead, it works on the operator instead of the value? v = a +(int) b # v = int.__add__(a, b) v = a +(abc.Numbers) b # ? v = a +(str) b # v = str.__add__(a, b) v = a +(abc.Container) b # ? The default operation would be... v = a +(type(a)) b # v = type(a).__add__(a, b) # or v = a.__add__(b) I don't think ABC's are defined/refined enough to make this work as they are, but could they be? (I'm not that familiar with ABC's yet.) Ron
2. sum is not appropriate iff chain.from_iterable makes sense.* Needing a list doesn't make chain unusable here any more than it does with map, zip, etc.; just pass the result to list. But "You can't iterate ints" or "I'm treating these np.matrix 3-vectors as atomic objects, not collections" does mean chain is unusable, so sum is the answer.
3. sum is not appropriate when 0 doesn't make sense as a start value. Summing things means, by default, starting with 0 and adding repeatedly. You can provide a non-default start value, but it should be "similar to 0" or "compatible with 0" in some way. Note that you can add 0 to an int, a float, a quaternion, a numpy.matrix, and all kinds of other types that "act like numbers". And that means that testing 0+start or start+0 is a pretty good test for summable. This is admittedly imperfect,** but it's pretty close, and very concrete and simple.
Personally, I'm leaning toward 2. If you come up with a type that is addable, and isn't iterable (or does the wrong thing when iterated), why not, it's summable. Better to let some corner-case false positives slip in than the reject some corner-case false negatives (as with 3), or to make them just impossible to decide (as with 1).
* This is a little too loose. Surely there are types where __add__ is not addition, but also not iterable concatenation, right? But I don't think sum is an attractive nuisance there, unlike in the case with concatenation. Meanwhile, what about strings? Actually, chain makes perfect sense with strings; it's just that usually, you're just going to want to pass the iterable to ''.join, and if you're doing that, you can just pass the original strings to ''.join. So, no problem there.
** Most notably, I think adding a sequence of timedelta objects with a datetime start makes sense, and you can't add 0 to a datetime. Really, what you need here is a way to say that start + 0 * peek(iterable) is also acceptable, not just start + 0—and you can justify that more rigorously in terms of fields—but that's nearly impossible to implement, and way too complicated to explain. So, option 3 will reject some valid types. _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
From: Haoyi Li <haoyi.sg@gmail.com> Sent: Monday, July 8, 2013 6:25 PM
<rant>I would also really like a nice object-oriented sortedlist to be part of the standard library, instead of that nasty C-style heapq stuff.</rant>
See this: http://stupidpythonideas.blogspot.com/2013/07/sorted-collections-in-stdlib.h... I stopped short of writing up a complete proposal, because (a) there are a lot of open questions, and (b) I'd like to hear from the authors of some of the existing PyPI libraries. But the short version is: * [Mutable]Sorted[Mappingt|Set|List] and possibly Sorted[Item|Key|Value]View ABCs, which inherit from the non-Sorted and add methods like key_slice. * Sorted[Dict|Set|List] concrete classes, all sharing the same implementation, ideally borrowed from existing PyPI library. (Open questions: red-black trees, or something else? can you get timsort-like behavior for already-sorted source data, or for large updates?) * Constructor matches [dict|set|list], but with optional key parameter. (Open question: SortedDict(a=1, B=2, c=3, key=str.lower) ambiguity.)
On 07/08/2013 07:39 PM, Haoyi Li wrote:
Python programmers do currently write programs with linked lists. And they use what ever is in the library they think is useful... the whole point of "batteries included".
On the other hand, python programmers do currently write programs with normal lists, and sum(), and using sum() on lists would be useful. They're BOTH in the standard library too! In fact, this seems like a perfect argument for having sum() work on lists!
On *another* hand. These are complementary points, not opposing points. ;-)
The objection seems like optimizing for a hypothetical, potential use case far in the future at the expense of a concrete use case now. "if you can't make it do EVERYTHING, then we shouldn't make it do ANYTHING or people will get confused!" (paraphrased and exaggerated for theatrical effect).
Extra dramma noted!
This is all on the assumption that you consider flattening list-of-lists a concrete use case. I for one find it annoying that i have to write a verbose long thingy every time i need to flatten lists, ...
I think this is the reason Alex Martinelli regretted using __add__ to concatenate strings. And I agree. A method would have worked very well. s1.cat(s2, s3, s4) Or if they were long string literals: "".cat("... s1 ...", "... s2 ...", "... s3 ...") This would work very well in many situations and is a fine alternative to implicit string concatenations in my opinion. Although it doesn't solve the "don't iterate a strings characters when flattening a bunch of iterable objects problem. Cheers, Ron
...and I have probably needed to flatten lists about a dozen times in the last 3 months and used linked lists not-at-all.
Maybe you write a ton of pure-functional algorithms making great use of the persistence of singly-linked-lists for performant non-mutating head-updates, and never use vanilla lists in the code you write, and so having sum() work on linked lists is of great import.
On 09/07/13 10:39, Haoyi Li wrote:
Python programmers do currently write programs with linked lists. And they use what ever is in the library they think is useful... the whole point of "batteries included".
On the other hand, python programmers do currently write programs with normal lists, and sum(), and using sum() on lists would be useful. They're BOTH in the standard library too! In fact, this seems like a perfect argument for having sum() work on lists!
sum() does work on lists. The fact that sum(lists) has had quadratic performance since sum was first introduced in Python 2.3, and I've *never* seen anyone complain about it being slow, suggests very strongly that this is not a use-case that matters. I would bet that people simply do not sum large numbers of lists.
The objection seems like optimizing for a hypothetical, potential use case far in the future at the expense of a concrete use case now. "if you can't make it do EVERYTHING, then we shouldn't make it do ANYTHING or people will get confused!" (paraphrased and exaggerated for theatrical effect).
No no no. The objection is that complicating the implementation of a function in order to optimize a use-case that doesn't come up in real-world use is actually harmful. Maintaining sum will be harder, for the sake of some benefit that very possibly nobody will actually receive. I don't care that sum() is O(N**2) on strings, linked lists, tuples, lists. I don't think we should care. Sergey thinks we should care, and is willing to change the semantics of sum AND include as many special cases as needed in order to "guarantee" that sum will be "always fast". I don't believe that guarantee can possibly hold, and I'm dubious about the change in semantics and what I see as the standard library making misleading performance guarantees.
This is all on the assumption that you consider flattening list-of-lists a concrete use case. I for one find it annoying that i have to write a verbose long thingy every time i need to flatten lists, and I have probably needed to flatten lists about a dozen times in the last 3 months and used linked lists not-at-all.
Flattening sequences is not sum. You have to consider the question, what counts as an atomic type? Should you recursively flatten sub-sub-lists? What to do with lists that contain a reference to themselves? sum() is correspondingly trivial, you just add the items, end of story. Writing a general purpose flattener is hard, and horribly easy to either under- or over-engineer. I have in my private software collection about half a dozen half-finished flatten() utility functions, because periodically I think it "might be useful", but because I've never actually needed the damn thing, I'm never motivated to complete it. Non-recursively flattening a list of lists is trivial: flattened = [] for sublist in lists: flattened.extend(sublist) Three short lines of code. You could write that a dozen times a day, every day for a year, and not come close to the amount of discussion we've had on this sum() thread :-) -- Steven
Three short lines of code. You could write that a dozen times a day, every day for a year, and not come close to the amount of discussion we've had on this sum() thread :-)
That's very true; on the other hand, I don't have to go back and debug the discussions we've had in this thread, and I hope to have many long years of writing code ahead of me before i pass on =) -Haoyi On Tue, Jul 9, 2013 at 10:16 AM, Steven D'Aprano <steve@pearwood.info>wrote:
On 09/07/13 10:39, Haoyi Li wrote:
Python programmers do currently write programs with linked lists. And
they use what ever is in the library they think is useful... the whole point of "batteries included".
On the other hand, python programmers do currently write programs with normal lists, and sum(), and using sum() on lists would be useful. They're BOTH in the standard library too! In fact, this seems like a perfect argument for having sum() work on lists!
sum() does work on lists.
The fact that sum(lists) has had quadratic performance since sum was first introduced in Python 2.3, and I've *never* seen anyone complain about it being slow, suggests very strongly that this is not a use-case that matters.
I would bet that people simply do not sum large numbers of lists.
The objection seems like optimizing for a hypothetical, potential use case
far in the future at the expense of a concrete use case now. "if you can't make it do EVERYTHING, then we shouldn't make it do ANYTHING or people will get confused!" (paraphrased and exaggerated for theatrical effect).
No no no. The objection is that complicating the implementation of a function in order to optimize a use-case that doesn't come up in real-world use is actually harmful. Maintaining sum will be harder, for the sake of some benefit that very possibly nobody will actually receive.
I don't care that sum() is O(N**2) on strings, linked lists, tuples, lists. I don't think we should care. Sergey thinks we should care, and is willing to change the semantics of sum AND include as many special cases as needed in order to "guarantee" that sum will be "always fast". I don't believe that guarantee can possibly hold, and I'm dubious about the change in semantics and what I see as the standard library making misleading performance guarantees.
This is all on the assumption that you consider flattening list-of-lists a
concrete use case. I for one find it annoying that i have to write a verbose long thingy every time i need to flatten lists, and I have probably needed to flatten lists about a dozen times in the last 3 months and used linked lists not-at-all.
Flattening sequences is not sum. You have to consider the question, what counts as an atomic type? Should you recursively flatten sub-sub-lists? What to do with lists that contain a reference to themselves? sum() is correspondingly trivial, you just add the items, end of story. Writing a general purpose flattener is hard, and horribly easy to either under- or over-engineer. I have in my private software collection about half a dozen half-finished flatten() utility functions, because periodically I think it "might be useful", but because I've never actually needed the damn thing, I'm never motivated to complete it.
Non-recursively flattening a list of lists is trivial:
flattened = [] for sublist in lists: flattened.extend(sublist)
Three short lines of code. You could write that a dozen times a day, every day for a year, and not come close to the amount of discussion we've had on this sum() thread :-)
-- Steven
______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
On Jul 9, 2013 Steven D'Aprano wrote:
The fact that sum(lists) has had quadratic performance since sum was first introduced in Python 2.3, and I've *never* seen anyone complain about it being slow, suggests very strongly that this is not a use-case that matters.
Never seen? Are you sure? ;)
http://article.gmane.org/gmane.comp.python.general/658630 From: Steven D'Aprano @ 2010-03-29 In practical terms, does anyone actually ever use sum on more than a handful of lists? I don't believe this is more than a hypothetical problem.
This is definitely not the first time people ask to fix this O(N**2) "surprise". Maybe if the problem appears year after year it is not so hypothetical? sum() was the first answer suggested for many list-of-lists questions [1], and sometimes it wasn't even obvious for people why it might be slow [2]. IMHO, many of those who actually spot the slowness, will just think "heh, python is so slow", and won't blame sum() for bug. Why do you think sum() code has explicit comment about using sum for list of lists? Why test_builtin.py checks this case? Isn't it because this is the common case? Or do you mean that nobody suggested a patch before? Well, think about it like that: 1. How many people in the world use python? 2. How many of them need to add lists? 3. How many of them are careful enough to notice that sum is slow? 4. How many of them are experienced enough to blame sum for that? 5. How many of those are smart enough to understand how it can be fixed, not just workarounded by using another function? 6. How many of those are skilled enough to dig into python code and able to fix the bug there? 7. How many of those have enough free time to come here and start asking to accept the patch? Not too many, right? And among those someone should be the first. Well, I am. :) How many do you need?
No no no. The objection is that complicating the implementation of a function in order to optimize a use-case that doesn't come up in real-world use is actually harmful. Maintaining sum will be harder, for the sake of some benefit that very possibly nobody will actually receive.
Are you sure it complicates the implementation? Here's how sum function looks NOW, without my patches (well, it looks different in C, but idea is the same): def sum(seq, start = 0): it = iter(seq) if isinstance(start, str): raise TypeError( "sum() can't sum strings [use ''.join(seq) instead]") if isinstance(start, bytes): raise TypeError( "sum() can't sum bytearray [use b''.join(seq) instead]") # SPECIAL CASES if isinstance(start, int): i_result = int(start, overflow) if not overflow: try: start = None while start is None: item = next(it) if isinstance(item, int): b = int(item, overflow) x = i_result + b if not overflow and ((x^i_result) >= 0 or (x^b) >= 0) i_result = x continue start = i_result start = start + item except StopIteration: return i_result if isinstance(start, float): f_result = float(start) try: start = None while start is None: item = next(it) if isinstance(item, float): f_result += float(item) continue if isinstance(item, int): value = int(item, overflow) if not overflow: f_result += float(value); continue start = f_result start = start + item except StopIteration: return f_result # END OF SPECIAL CASES try: while True: item = next(it) result = result + item except StopIteration: return start So simple and obvious. :) My original patch changes the part: while True: item = next(it) result = result + item to: result = result + item while True: item = next(it) result += item (In python that effectively adds one line, in C that's 6 lines) This does not really complicate the Alternative to that patch is one more special case for "known slow types" i.e. lists and tuples. That was my second patch. In python that would be like: if isinstance(start, list) or isinstance(start, tuple): optimize_for = type(start) l_result = list(start) try: while True: item = next(it) if not isinstance(item, optimize_for): start = optimize_for(l_result) start = start + item break l_result.extend(item) except StopIteration: return optimize_for(l_result) Yes, that's not just one line, but does it really complicates existing code that much? Theoretically this code could be extended to any iterable. So, *theoretically* it could be: if start_is_iterable: optimize_for = type(start) ... same code here ... but there's a technical problem in the line: start = optimize_for(l_result) This line works for lists and tuples. It will even work for set and frozenset, if needed. But the problem is that I cannot guarantee that it will work for arbitrary type. For example it does not work for strings (even worse, it works, but not as I would want it to). In my patch I also showed how strings can be handled for that case. Basically, this very single line is the only line holding me from making my "special case" into "general case" for all iterables in the world. So if somebody knows how to solve this problem — any suggestions welcome.
I don't care that sum() is O(N**2) on strings, linked lists, tuples, lists. I don't think we should care. Sergey thinks we should care, and is willing to change the semantics of sum AND include as many special cases as needed in order to "guarantee" that sum will be "always fast". I don't believe that guarantee can possibly hold, and
Just ONE special case is needed — the one for iterables. And yes, the "guarantee" can hold, because it only affects built-in types, and my patch covers all of them, even those that are not supported by sum() yet. But it's also good for third-party types, because it gives them an option to implement a fast __iadd__. They don't have this option now.
Flattening sequences is not sum. You have to consider ...
Yet people think [1] that sum() is useful for that. Every year somebody comes and tries to use sum(), and often someone else says "Hey, don't use sum() it's slow". "BECAUSE IT'S SLOW!" All that talks about "sum() is not designed for...", "it's just because + is used for concatenation...", "sum() should not be used...", "you have to consider..." — all of these are just excuses to explain why the bug is there. Maybe it's time to stop searching for excuses and finally fix the bug? Especially if it's so easy to fix it. -- [1] Some questions about lists flattening: http://stackoverflow.com/questions/716477/join-list-of-lists-in-python http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-... http://stackoverflow.com/questions/406121/flattening-a-shallow-list-in-pytho... [2] "explain for a noob python learner please: Is O(n^2) good or bad in this case?" http://article.gmane.org/gmane.comp.python.general/441831
A fast implementation would probably allocate the output list just once and then stream the values into place with a simple index. That's what I hoped "sum" would do, but instead it barfs with a type error.
Let's split this into parts. What you have is (a) a patch to make iaddable types sum faster, (b) a patch to make tuples sum faster, and (c) 37 different half-baked ideas for how to maybe come up with a patch to make either all iterables or all addable types sum faster, most of which you later deny having ever offered. If you drop all the stuff related to sequences, and come up with a numeric case that's helped by (a), I'm pretty sure you could get that approved with little to no objection, and then come back to the rest later. And there probably are such types. Someone suggested some C-implemented quaternion library. I suggested numpy.matrix. Someone else suggested a different vector class. And there are plenty of other obvious things to try--huge pygmp ints, a BCD class, ... All you have to do is try them out and show that at least one of them gets a significant speedup. Once you get (a) accepted (or fail to find any good use for it besides lists, which I think is less likely), come back to the rest, and pick an argument and stick to it. Either (b) is good on its own because tuples are special and there's no reason to make it general, or tuples aren't special and it should be general and therefore you have a concrete proposal that actually works as expected on a variety of types that you've actually tested. On Jul 10, 2013, at 9:10, Sergey <sergemp@mail.ru> wrote:
On Jul 9, 2013 Steven D'Aprano wrote:
No no no. The objection is that complicating the implementation of a function in order to optimize a use-case that doesn't come up in real-world use is actually harmful. Maintaining sum will be harder, for the sake of some benefit that very possibly nobody will actually receive.
Are you sure it complicates the implementation?
The iadd case doesn't complicate it much. But the tuple case, halfway hacked up toward a more general solution that you haven't quite envisioned, does. Meanwhile, the fact that ints and floats _also_ complicate things isn't a good argument here. Adding ints is the paradigm use case for sum, so it's arguably worth extra complexity. Also, the fact that it's been maintained for over a decade, through the int/ long unification and py3k, means nobody has to guess whether it will be a maintenance burden.
Alternative to that patch is one more special case for "known slow types" i.e. lists and tuples. That was my second patch. In python that would be like: if isinstance(start, list) or isinstance(start, tuple): optimize_for = type(start) l_result = list(start) try: while True: item = next(it) if not isinstance(item, optimize_for): start = optimize_for(l_result) start = start + item break l_result.extend(item) except StopIteration: return optimize_for(l_result)
Yes, that's not just one line, but does it really complicates existing code that much?
Theoretically this code could be extended to any iterable. So, *theoretically* it could be: if start_is_iterable: optimize_for = type(start) ... same code here ...
No it can't. Or, rather, it would be a very bad idea. Matrices would be unraveled and concatenated into vectors instead of added, types that aren't addable for good reason like dicts and files would do various different odd things instead of raising, intentionally lazy (and potentially infinite) types would be forced strict, types that can append faster than list like deque would slow down (as would set if it became addable) and/or use much more memory, ...
but there's a technical problem in the line: start = optimize_for(l_result) This line works for lists and tuples. It will even work for set and frozenset, if needed. But the problem is that I cannot guarantee that it will work for arbitrary type. For example it does not work for strings (even worse, it works, but not as I would want it to). In my patch I also showed how strings can be handled for that case.
Basically, this very single line is the only line holding me from making my "special case" into "general case" for all iterables in the world.
And that's lucky, because as a general case it would be a very bad thing.
I don't care that sum() is O(N**2) on strings, linked lists, tuples, lists. I don't think we should care. Sergey thinks we should care, and is willing to change the semantics of sum AND include as many special cases as needed in order to "guarantee" that sum will be "always fast". I don't believe that guarantee can possibly hold, and
Just ONE special case is needed — the one for iterables. And yes, the "guarantee" can hold, because it only affects built-in types, and my patch covers all of them, even those that are not supported by sum() yet.
A guarantee that only holds for builtin types and cannot be extended in any way to other types is more misleading than useful. Today, a few people misuse sum on sequences and get quadratic behavior. That's exactly what you want to fix. But with your change, many more people would use sum on sequences and get quadratic behavior, because the whole point of your patch is to make sum the obvious way to concatenate sequences. And, while today, it's a bug in their code that can be explained with a simple link to the docs, with your change, it will be a bug in python requiring a workaround explained by a link to some FAQ or blog post. And if that isn't what you intend, then you don't intend for sum to be the obvious way to concatenate sequences, and your patch is just bending over backward to make buggy code run better. You can't have it both ways.
But it's also good for third-party types, because it gives them an option to implement a fast __iadd__. They don't have this option now.
The first patch alone does that for all fast-iaddable types, including non-sequences as well as list-like sequences. The second patch adds nothing to it, nor do any of your attempts to generalize it. It only works for mutable types that can iadd faster than they can add. You've already been given examples of types that isn't true for--any immutable type, cons lists, etc. Even if you could find a theoretical answer for those cases--which so far you haven't, but there's no harm in continuing to try--that won't actually help anyone using an actual python interpreter rather than some vague theoretically possible one.
Flattening sequences is not sum. You have to consider ...
Yet people think [1] that sum() is useful for that.
People also think that "if ('foo' or 'bar') in baz" is useful. It comes up every few days, not just a few times a year like summing tuples. It's even the basis of a running joke on StackOverflow. Does this mean we should change Python so it does what they expect? And again, your patch doesn't solve the problem, it makes it worse. If people mistakenly think that sum is useful for tuples, they're also going to think that it's useful for all kinds of other sequences. They'll still be wrong--but now the docs, and their early experiences, will tell them otherwise.
On 11/07/13 02:10, Sergey wrote:
On Jul 9, 2013 Steven D'Aprano wrote:
The fact that sum(lists) has had quadratic performance since sum was first introduced in Python 2.3, and I've *never* seen anyone complain about it being slow, suggests very strongly that this is not a use-case that matters.
Never seen? Are you sure? ;)
http://article.gmane.org/gmane.comp.python.general/658630 From: Steven D'Aprano @ 2010-03-29 In practical terms, does anyone actually ever use sum on more than a handful of lists? I don't believe this is more than a hypothetical problem.
Yes, and I stand by what I wrote back then.
Not too many, right? And among those someone should be the first. Well, I am. :) How many do you need?
No no no. The objection is that complicating the implementation of a function in order to optimize a use-case that doesn't come up in real-world use is actually harmful. Maintaining sum will be harder, for the sake of some benefit that very possibly nobody will actually receive.
Are you sure it complicates the implementation?
No, I'm not sure.
Here's how sum function looks NOW, without my patches (well, it looks different in C, but idea is the same): [snip code]
Alternative to that patch is one more special case for "known slow types" i.e. lists and tuples. That was my second patch. In python that would be like: if isinstance(start, list) or isinstance(start, tuple): optimize_for = type(start) l_result = list(start) try: while True: item = next(it) if not isinstance(item, optimize_for): start = optimize_for(l_result) start = start + item break l_result.extend(item) except StopIteration: return optimize_for(l_result)
Yes, that's not just one line, but does it really complicates existing code that much?
Of course, I understand that this is not the actual C code your patch contains. But I can see at least three problems with the above Python version, and I assume your C version will have the same flaws. 1) You test start using isinstance(start, list), but it should be "type(start) is list". If start is a subclass of list that overrides __add__ (or __iadd__), you should call the overridden methods. But your code does not, it calls list.extend instead. (Same applies for tuples.) 2) You assume that summing a sequence must return the type of the start argument. But that is not correct. See example below. 3) This can use twice as much memory as the current implementation. You build a temporary list containing the result, then you make a copy using the original type. If the result is very large, you might run out of memory trying to make the copy. So there are three potential problems with your patch. One will potentially cause code that currently works to fail with MemoryError. The other two will potentially cause code to return different results. These are exactly the sort of subtle, and unintended, changes in behaviour that I consider bugs. Here is an example of a multi-type sum: py> class A(list): ... def __add__(self, other): ... return type(self)(super().__add__(other)) ... def __radd__(self, other): ... return type(self)(other) + self ... py> result = sum([[1], [2], A([3]), [4]], []) py> type(result) <class '__main__.A'> It looks to me that your code will return a list instead of an A. By the way, Sergey, I should say that even though I have been hard on your suggestion, I do thank you for spending the time on this and value your efforts. -- Steven
On 07/10/2013 12:49 PM, Steven D'Aprano wrote:
On 11/07/13 02:10, Sergey wrote:
On Jul 9, 2013 Steven D'Aprano wrote:
The fact that sum(lists) has had quadratic performance since sum was first introduced in Python 2.3, and I've *never* seen anyone complain about it being slow, suggests very strongly that this is not a use-case that matters.
Never seen? Are you sure? ;)
http://article.gmane.org/gmane.comp.python.general/658630 From: Steven D'Aprano @ 2010-03-29 In practical terms, does anyone actually ever use sum on more than a handful of lists? I don't believe this is more than a hypothetical problem.
Yes, and I stand by what I wrote back then.
Just curious, how does your sum compare with fsum() in the math module? (Yes, I know it's specialised for floats?) It says that much in the docs. fsum(...) fsum(iterable) Return an accurate floating point sum of values in the iterable. Assumes IEEE-754 floating point arithmetic. Have you looked at it? Cheers, Ron
On 11/07/13 07:00, Ron Adam wrote:
On 07/10/2013 12:49 PM, Steven D'Aprano wrote:
On 11/07/13 02:10, Sergey wrote:
On Jul 9, 2013 Steven D'Aprano wrote:
The fact that sum(lists) has had quadratic performance since sum was first introduced in Python 2.3, and I've *never* seen anyone complain about it being slow, suggests very strongly that this is not a use-case that matters.
Never seen? Are you sure? ;)
http://article.gmane.org/gmane.comp.python.general/658630 From: Steven D'Aprano @ 2010-03-29 In practical terms, does anyone actually ever use sum on more than a handful of lists? I don't believe this is more than a hypothetical problem.
Yes, and I stand by what I wrote back then.
Just curious, how does your sum compare with fsum() in the math module?
math.fsum is a high-precision floating point sum, keeping extra precision that the built-in loses. Compare these: data = [1e-100, 1e100, 1e-100, -1e100]*1000 sum(data) math.fsum(data) The exact value for the sum is 2e-97. -- Steven
On 07/10/2013 08:03 PM, Steven D'Aprano wrote:
On 11/07/13 07:00, Ron Adam wrote:
On 07/10/2013 12:49 PM, Steven D'Aprano wrote:
On 11/07/13 02:10, Sergey wrote:
On Jul 9, 2013 Steven D'Aprano wrote:
The fact that sum(lists) has had quadratic performance since sum was first introduced in Python 2.3, and I've *never* seen anyone complain about it being slow, suggests very strongly that this is not a use-case that matters.
Never seen? Are you sure? ;)
http://article.gmane.org/gmane.comp.python.general/658630 From: Steven D'Aprano @ 2010-03-29 In practical terms, does anyone actually ever use sum on more than a handful of lists? I don't believe this is more than a hypothetical problem.
Yes, and I stand by what I wrote back then.
Just curious, how does your sum compare with fsum() in the math module?
math.fsum is a high-precision floating point sum, keeping extra precision that the built-in loses. Compare these:
data = [1e-100, 1e100, 1e-100, -1e100]*1000 sum(data) math.fsum(data)
The exact value for the sum is 2e-97.
I was thinking more on the lines of how it worked internally compared to sum. And how it handles different inputs. Of course it is quite a bit slower too.
timeit("fsum(r)", "from __main__ import fsum\nr=list(range(100))") 15.151492834091187
timeit("sum(r)", "r=list(range(100))") 2.282749891281128
So fsum will take integers, and converts (or casts) them to floats. And bytes, as they are integers.
fsum(b'12345') 255.0
But not strings, even if they can be converted to floats.
float("12.0") 12.0
fsum(['12.0', '13.0']) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: a float is required
I would like sum to (eventually) be moved to the math module and have it's API and behaviour be the same as fsum. That would have the least surprises and it reduces the mental load when two similar functions act the same and can be found near each other in the library. Cheers, Ron
On 11 July 2013 11:47, Ron Adam <ron3200@gmail.com> wrote:
On 07/10/2013 08:03 PM, Steven D'Aprano wrote:
On 11/07/13 07:00, Ron Adam wrote:
Just curious, how does your sum compare with fsum() in the math module?
math.fsum is a high-precision floating point sum, keeping extra precision that the built-in loses. Compare these: [snip]
I was thinking more on the lines of how it worked internally compared to sum. And how it handles different inputs. Of course it is quite a bit slower too.
math.fsum converts all inputs to float and then adds them using (I assume) Kahan summation [1]. A demonstration:
class A: ... pass ... a = A() math.fsum([a]) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: A instance has no attribute '__float__' class A: ... def __float__(self): ... return -1 ... a = A() math.fsum([a]) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: nb_float should return float object class A: ... def __float__(self): ... return -1.0 ... a = A() math.fsum([a]) -1.0
timeit("fsum(r)", "from __main__ import fsum\nr=list(range(100))") 15.151492834091187
timeit("sum(r)", "r=list(range(100))") 2.282749891281128
The above is not a fair comparison since fsum converts them to floats as you say. fsum is useless for integers since the additional computation is all about floating point precision. You should use a list of floats for a fair test:
timeit("fsum(r)", "from __main__ import fsum\nr=list(map(float, range(100)))") 3.432480121620099 timeit("sum(r)", "from __main__ import fsum\nr=list(map(float, range(100)))") 1.6432468412557402
[1] http://en.wikipedia.org/wiki/Kahan_summation_algorithm Oscar
On 11 July 2013 11:47, Ron Adam <ron3200@gmail.com> wrote:
I would like sum to (eventually) be moved to the math module and have it's API and behaviour be the same as fsum. That would have the least surprises and it reduces the mental load when two similar functions act the same and can be found near each other in the library.
Sum is *really* useful. You can use it for ints, for floats, for numpy arrays, for Decimals, for Fractions, for sympy.Expressions, for gmpy2... math.fsum is only useful for scalar floats but sum does so much more. There's no need to cripple it just just to stop people from summing lists. Oscar
On 07/11/2013 06:05 AM, Oscar Benjamin wrote:
On 11 July 2013 11:47, Ron Adam<ron3200@gmail.com> wrote:
I would like sum to (eventually) be moved to the math module and have it's API and behaviour be the same as fsum. That would have the least surprises and it reduces the mental load when two similar functions act the same and can be found near each other in the library.
Sum is*really* useful. You can use it for ints, for floats, for numpy arrays, for Decimals, for Fractions, for sympy.Expressions, for gmpy2...
math.fsum is only useful for scalar floats but sum does so much more. There's no need to cripple it just just to stop people from summing lists.
Isn't it just a matter of spelling it a bit different? sum(iters, []) # how often do you actually use this? vs... chain(iters) # better in most cases... list(chain(iters)) # when you actually need a combined list. I'd like to see chain as a builtin in any case. Or to look at it in another way... I wouldn't want to add the ability to sum items in a list to chain(). Note, because to get sum() to join lists requires it to be explicitly spelled with a starting list, it doesn't need to be changed. I'm +1 for doing that, only if there is a consensus for doing that. How do you feel about adding the ability of sum to sum vectors or lists of values to each other? sum([[x1, y1], [x2, y2], ...]) ---> [x1+x2, y1+y2] Cheers, Ron
On 11 July 2013 14:20, Ron Adam <ron3200@gmail.com> wrote:
On 07/11/2013 06:05 AM, Oscar Benjamin wrote:
On 11 July 2013 11:47, Ron Adam<ron3200@gmail.com> wrote:
chain(iters) # better in most cases...
I think you mean chain.from_iterable rather than chain
I'd like to see chain as a builtin in any case.
chain.from_iterable should be the builtin not chain.
How do you feel about adding the ability of sum to sum vectors or lists of values to each other?
sum([[x1, y1], [x2, y2], ...]) ---> [x1+x2, y1+y2]
That's the beauty of it. sum() already sums anything you want as long as __add__ is implemented. If I wanted to do the above with some vectors I would probably use numpy arrays which have precisely the __add__ method you want:
import numpy as np arrays = [ ... np.array([1, 2, 3]), ... np.array([2, 3, 4]), ... np.array([3, 4, 5]), ... ] arrays [array([1, 2, 3]), array([2, 3, 4]), array([3, 4, 5])] sum(arrays) array([ 6, 9, 12])
This is possible because of the simplicity of the core algorithm in sum() i.e. just calling 'total = total + item' in a loop. Anyone who wants to use sum() with their own type can already do so. Earlier you seemed to be advocating changing that by restricting the types that sum() accepts. Oscar
On Jul 11, 2013, at 7:31, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 11 July 2013 14:20, Ron Adam <ron3200@gmail.com> wrote:
On 07/11/2013 06:05 AM, Oscar Benjamin wrote:
On 11 July 2013 11:47, Ron Adam<ron3200@gmail.com> wrote:
chain(iters) # better in most cases...
I think you mean chain.from_iterable rather than chain
I'd like to see chain as a builtin in any case.
chain.from_iterable should be the builtin not chain.
This is the problem. We can't rename chain.from_iterable to chain (or, equivalently, change the API of chain) while moving it without a lot of confusion. So it seems like we probably need to name it something completely different--concat, flatten, chainiter, ... But none of those names feels right. Concat seems like it should just take two sequences, not a sequence of sequences. Flatten is something you'd only reach for when you're thinking of it as one nested sequence rather than a collection of sequences. Chainiter is ugly. Chain really would be the perfect name if it didn't already have the wrong connotation thanks to its years of life in itertools. Other than that, I love the idea. The right function for this task should return an iterator, for the same reasons map and zip should, and also because it completely avoids all of the issues with trying to define what it means for different types by only handling iterables and treating them as iterables. The docs for sum already hint that it's the obvious way to concatenate sequences, but it should be more obvious. If someone can come up with a good name (or just nudge my feeling on one of the already proposed names), I'm definitely +1 on this.
I'm not in love with it, but what about 'ichain()' following imap() and izip(), ifilter(), etc. On Thu, Jul 11, 2013 at 9:54 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 11, 2013, at 7:31, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 11 July 2013 14:20, Ron Adam <ron3200@gmail.com> wrote:
On 07/11/2013 06:05 AM, Oscar Benjamin wrote:
On 11 July 2013 11:47, Ron Adam<ron3200@gmail.com> wrote:
chain(iters) # better in most cases...
I think you mean chain.from_iterable rather than chain
I'd like to see chain as a builtin in any case.
chain.from_iterable should be the builtin not chain.
This is the problem. We can't rename chain.from_iterable to chain (or, equivalently, change the API of chain) while moving it without a lot of confusion. So it seems like we probably need to name it something completely different--concat, flatten, chainiter, ... But none of those names feels right. Concat seems like it should just take two sequences, not a sequence of sequences. Flatten is something you'd only reach for when you're thinking of it as one nested sequence rather than a collection of sequences. Chainiter is ugly. Chain really would be the perfect name if it didn't already have the wrong connotation thanks to its years of life in itertools.
Other than that, I love the idea. The right function for this task should return an iterator, for the same reasons map and zip should, and also because it completely avoids all of the issues with trying to define what it means for different types by only handling iterables and treating them as iterables. The docs for sum already hint that it's the obvious way to concatenate sequences, but it should be more obvious.
If someone can come up with a good name (or just nudge my feeling on one of the already proposed names), I'm definitely +1 on this. _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Jul 11, 2013, at 10:02, David Mertz <mertz@gnosis.cx> wrote:
I'm not in love with it, but what about 'ichain()' following imap() and izip(), ifilter(), etc.
Given that Python 3 renamed those functions to map, zip, and filter while moving them to builtins (and replacing the 2.x builtins of the same names), I don't think that's reasonable.
On Thu, Jul 11, 2013 at 9:54 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 11, 2013, at 7:31, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 11 July 2013 14:20, Ron Adam <ron3200@gmail.com> wrote:
On 07/11/2013 06:05 AM, Oscar Benjamin wrote:
On 11 July 2013 11:47, Ron Adam<ron3200@gmail.com> wrote:
chain(iters) # better in most cases...
I think you mean chain.from_iterable rather than chain
I'd like to see chain as a builtin in any case.
chain.from_iterable should be the builtin not chain.
This is the problem. We can't rename chain.from_iterable to chain (or, equivalently, change the API of chain) while moving it without a lot of confusion. So it seems like we probably need to name it something completely different--concat, flatten, chainiter, ... But none of those names feels right. Concat seems like it should just take two sequences, not a sequence of sequences. Flatten is something you'd only reach for when you're thinking of it as one nested sequence rather than a collection of sequences. Chainiter is ugly. Chain really would be the perfect name if it didn't already have the wrong connotation thanks to its years of life in itertools.
Other than that, I love the idea. The right function for this task should return an iterator, for the same reasons map and zip should, and also because it completely avoids all of the issues with trying to define what it means for different types by only handling iterables and treating them as iterables. The docs for sum already hint that it's the obvious way to concatenate sequences, but it should be more obvious.
If someone can come up with a good name (or just nudge my feeling on one of the already proposed names), I'm definitely +1 on this. _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 07/11/2013 09:31 AM, Oscar Benjamin wrote:
On 11 July 2013 14:20, Ron Adam <ron3200@gmail.com> wrote:
On 07/11/2013 06:05 AM, Oscar Benjamin wrote:
On 11 July 2013 11:47, Ron Adam<ron3200@gmail.com> wrote:
chain(iters) # better in most cases...
I think you mean chain.from_iterable rather than chain
I'd like to see chain as a builtin in any case.
chain.from_iterable should be the builtin not chain.
I agree.
How do you feel about adding the ability of sum to sum vectors or lists of values to each other?
sum([[x1, y1], [x2, y2], ...]) ---> [x1+x2, y1+y2]
That's the beauty of it. sum() already sums anything you want as long as __add__ is implemented. If I wanted to do the above with some vectors I would probably use numpy arrays which have precisely the __add__ method you want:
import numpy as np arrays = [ ... np.array([1, 2, 3]), ... np.array([2, 3, 4]), ... np.array([3, 4, 5]), ... ] arrays [array([1, 2, 3]), array([2, 3, 4]), array([3, 4, 5])] sum(arrays) array([ 6, 9, 12])
This is possible because of the simplicity of the core algorithm in sum() i.e. just calling 'total = total + item' in a loop. Anyone who wants to use sum() with their own type can already do so. Earlier you seemed to be advocating changing that by restricting the types that sum() accepts.
I think this is what it should do. I tried overiding lists __add__, but that didn't work as nice. It needs to have a starting list with all zeros in it. How does numpy get around that? Ron
On 11 July 2013 18:12, Ron Adam <ron3200@gmail.com> wrote:
I think this is what it should do.
I tried overiding lists __add__, but that didn't work as nice. It needs to have a starting list with all zeros in it. How does numpy get around that?
Like so:
import numpy a = numpy.array([1, 2, 3]) a array([1, 2, 3]) a + 0 array([1, 2, 3]) a + 1 array([2, 3, 4])
Oscar
On 07/11/2013 12:21 PM, Oscar Benjamin wrote:
On 11 July 2013 18:12, Ron Adam <ron3200@gmail.com> wrote:
I think this is what it should do.
I tried overiding lists __add__, but that didn't work as nice. It needs to have a starting list with all zeros in it. How does numpy get around that?
Like so:
import numpy a = numpy.array([1, 2, 3]) a array([1, 2, 3]) a + 0 array([1, 2, 3]) a + 1 array([2, 3, 4])
Oscar
Right answer to the wrong question. I was asking how numpy it gets around sum() needing a starting 'array' argument? Not how numpy arrays work with '+'. Cheers, Ron
On Thu, Jul 11, 2013 at 2:29 PM, Ron Adam <ron3200@gmail.com> wrote:
I was asking how numpy it gets around sum() needing a starting 'array' argument?
When you sum arrays, you don't need to start with an array:
import numpy 0 + numpy.array([1, 2, 3]) array([1, 2, 3]) sum([_, _, _]) array([3, 6, 9])
The default scalar start gets broadcast to the shape of the array.
sum([numpy.zeros((2,2))], 42) array([[ 42., 42.], [ 42., 42.]])
On 07/11/2013 11:29 AM, Ron Adam wrote:
On 07/11/2013 12:21 PM, Oscar Benjamin wrote:
On 11 July 2013 18:12, Ron Adam <ron3200@gmail.com> wrote:
I think this is what it should do.
I tried overiding lists __add__, but that didn't work as nice. It needs to have a starting list with all zeros in it. How does numpy get around that?
Like so:
import numpy a = numpy.array([1, 2, 3]) a array([1, 2, 3]) a + 0 array([1, 2, 3]) a + 1 array([2, 3, 4])
Oscar
Right answer to the wrong question.
I was asking how numpy it gets around sum() needing a starting 'array' argument? Not how numpy arrays work with '+'.
Because the default start is 0, and when you add 0 to a numpy array you get back the same* numpy array. *At least, a numpy array with all the same values. -- ~Ethan~
On 11/07/13 23:20, Ron Adam wrote:
How do you feel about adding the ability of sum to sum vectors or lists of values to each other?
sum([[x1, y1], [x2, y2], ...]) ---> [x1+x2, y1+y2]
What I think of that is that it is not backward compatible and would completely break any code relying on sum's current behaviour. We're in the middle of a long discussion arguing that a *tiny*, *subtle* shift in behaviour of sum is sufficient to disqualify the change, and you're suggesting something that completely changes the semantics from list concatenation to element-by-element addition. No offence Ron, but have you been reading the rest of the thread? If anyone wants element-by-element addition, then can either define a class that does so using the + operator (say, numpy arrays) or they define their own function. Or try to revive PEP 225: http://www.python.org/dev/peps/pep-0225/ -- Steven
On 07/11/2013 07:23 PM, Steven D'Aprano wrote:
On 11/07/13 23:20, Ron Adam wrote:
How do you feel about adding the ability of sum to sum vectors or lists of values to each other?
sum([[x1, y1], [x2, y2], ...]) ---> [x1+x2, y1+y2]
What I think of that is that it is not backward compatible and would completely break any code relying on sum's current behaviour.
That's what I think too. ;-) Although it's possible to extend an API in a backwords copatible ways. Adding a new keyword would work. (Not suggesting that.)
We're in the middle of a long discussion arguing that a *tiny*, *subtle* shift in behaviour of sum is sufficient to disqualify the change, and you're suggesting something that completely changes the semantics from list concatenation to element-by-element addition. No offence Ron, but have you been reading the rest of the thread?
No offence taken. And as you point out, we can't easily add element-by-element addition as long as it also does list concatenation.
If anyone wants element-by-element addition, then can either define a class that does so using the + operator (say, numpy arrays) or they define their own function.
I'd rather just use a function and not use the '+'.
Or try to revive PEP 225:
Interesting. I can see why it didn't get in. PEP 225 tries to do too much. I Was looking in the operator module and notice there are __concat__() and concat() methods. I don't think I've seen them anywhere else. Maybe we could just add a few new functions to the operator module? Each specialised to only numbers, strings, immutable, and mutable types. That might open the door for them being used in a wider scope while not changing too much in the near future. I sort of wish we could add alias's to oppertor methods. (In a nice way.) @__add__ def __add_number__(self, other): ... @__add__ def __add_list__(self, other): ... @__add__ def __concat_strings__(self, other): ... Multi-methods for operators(?) Where __add__ and '+' would work with all of these, but __add_lists__ would work only on lists. It's pretty much how it works now, but sense they are all spelled the same, it can be confusing when you see these used from a subclass. Cheers, Ron
On 11 Jul 2013 Steven D'Aprano wrote:
The fact that sum(lists) has had quadratic performance since sum was first introduced in Python 2.3, and I've *never* seen anyone complain about it being slow, suggests very strongly that this is not a use-case that matters. Never seen? Are you sure? ;) http://article.gmane.org/gmane.comp.python.general/658630 From: Steven D'Aprano @ 2010-03-29 In practical terms, does anyone actually ever use sum on more than a handful of lists? I don't believe this is more than a hypothetical problem.
Yes, and I stand by what I wrote back then.
Hey, you saw at least two of us complaining: me and that guy! :)
Of course, I understand that this is not the actual C code your patch contains. But I can see at least three problems with the above Python version, and I assume your C version will have the same flaws.
Thank you for reviewing the code. Even for just reviewing the "explanation".
1) You test start using isinstance(start, list), but it should be "type(start) is list". If start is a subclass of list that overrides __add__ (or __iadd__), you should call the overridden methods. But your code does not, it calls list.extend instead. (Same applies for tuples.)
2) You assume that summing a sequence must return the type of the start argument. But that is not correct. See example below.
The real C code was doing "CheckExact" for `start`, but subclass "Check" for `item`. Because that's what __add__ for list and tuple checked.
Here is an example of a multi-type sum: py> class A(list): ... def __add__(self, other): ... return type(self)(super().__add__(other)) ... def __radd__(self, other): ... return type(self)(other) + self ... py> result = sum([[1], [2], A([3]), [4]], []) py> type(result) <class '__main__.A'>
And that was a good example. I had to dig deep in sources to understand why:
type( [1] + A([2]) ) <class '__main__.A'> but: type( [1].__add__(A([2])) ) <class 'list'>
So I updated the patch [1], now it does exact type check and falls back to general code otherwise. Now it's a little less general, but more safe.
3) This can use twice as much memory as the current implementation. You build a temporary list containing the result, then you make a copy using the original type. If the result is very large, you might run out of memory trying to make the copy.
Basically, that's what str.join() is doing. You can try: ''.join('' for _ in xrange(100000000)) and watch you free memory being eaten. :) So if this is good for join I assumed that a smarter version of it for sum() should also be fine.
By the way, Sergey, I should say that even though I have been hard on your suggestion, I do thank you for spending the time on this and value your efforts.
Thank you. I appreciate your words. -- [1] http://bugs.python.org/file30897/fastsum-special-tuplesandlists.patch
First, let me summarize my position, because you're tangling up my separate points, and even tangling other people's points up with mine. I'm -1 on adding special-casing for tuples that would not be available for any other immutable type. I'm -0.75 on adding flexible special-casing that could be extended to other types from Python. I'm -1 on encouraging people to treat sum as the obvious way to concatenate any kind of sequence. I'm -0 on adding explicit code to sum to raise when start is a non-number (as Alex Martelli wanted), a sequence, or something that can't do 0+start. I'm between +1 and -0 on changing sum to use copy.copy(start) and __iadd__, depending on whether there are any types that seem like they should be obviously summable and would gain from this change. Adding a bunch of numpy.arrays starting from 0 seems like a reasonable use for sum, as does adding a bunch of timedeltas to a datetime; would either of those benefit from the __iadd__ optimization? If so, I'd be behind that. From: Sergey <sergemp@mail.ru> Sent: Monday, July 8, 2013 1:22 PM
On Jul 5, 2013 Andrew Barnert wrote:
Yes, you can! You just need to implement a fast __iadd__ or __add__ for your type to make it O(n) summable.
Well, in Python 3.3, or even 2.3, you just need to implement a fast __add__. So, if that's an acceptable answer, then we don't need to do anything. [...]
And you can always do that, can't you?
No. You can't implement a fast __iadd__ for tuple.
Well... Yes, I can!
No, you can't. You can do something different, but only by modifying the C source to sum.
I can't make __iadd__ faster, because tuple has no __iadd__, however I can make a faster __add__.
No, you can't make tuple.__add__ faster either. (If you can, please submit a patch, because that would be useful completely independent of sum.)
But as long as sum() is the only (?) function suffering from this problem it was easier to do that optimization in sum() instead.
No, because that requires putting an optimization into sum for _any_ type that's fast-summable but not by using its __add__ or __iadd__, and that's not feasible.
If that's going to be extendable to other types
Are there any other types (except strings, that are blocked anyway)?
Yes. Every single immutable type. Immutable types do not have __iadd__, for the obvious reason. Whether they're builtin, stdlib, third-party, or part of the application, they will not be fast-summable, and there will be no way to make them fast-summable without patching the interpreter. blist.btuple has the same problem. You wanted sets to be addable? Then frozenset would have this problem. Strings obviously have this problem, and any third-party immutable string type will as well. So, if you're suggesting that sum can be fast for anything reasonable, you're just wrong. Adding a custom optimization for tuple only helps tuple; it does not help immutable types in general. And that's why I think it's a bad idea to add custom code to sum for tuples.
Personally I don't think such implementation would be very useful.
Given that nearly every program every written in Lisp and its successors makes heavy use of such an implementation, I submit that your intuition may be wrong.
They don't have much choice. And they're not using sum().
Of course they have a choice. You think you can't use dynamic arrays, doubly-linked lists, balanced trees, etc. in Lisp? They use cons lists because there are some algorithms that are natural and fast with cons lists—and because those algorithms work well with appropriate fold/reduce-type functions. And Python's sum (like Common Lisp's sum) is a reduce-type function. In fact, other than the optimization for numbers, sum(iterable, start=0) really is just reduce(operator.__add__, iterable, start).
We're talking about python, and discussing use of sum() in python for such lists in particular.
No. You insisted that every collection type is O(N) summable with your design. Others argued that this isn't true. You asked for an example. I offered cons lists. Instead of accepting that, you began arguing that nobody would ever want to use such a type, and then suggesting that if you just changed the implementation and all of the performance characteristics of the type, it would become O(N) summable. Now you've come around to the very position you were arguing against. If you agree that your design is not a general solution for al sequences, then all of your misunderstandings about cons lists are a red herring, and we can drop them.
It's just you said:
I'm hostile to any attempt to encourage people to treat sum() as the preferred way to concatenate large amounts of data, because that will surely lead them into bad habits and will end with them trying to sum() a lot of tuples or linked lists or something and getting O(n**2) performance.
First, that wasn't me; please try to keep your attributions straight. But I agree with the sentiment. There is an efficient way to concatenate a lot of cons lists (basically, just fold backward instead of forward), but sum is not it. Similarly, there is an efficient way to concatenate a lot of tuples or other immutable types, but sum is not it. So, whoever said that is right—encouraging people to treat sum() as the preferred way to concatenate large amounts of data is a bad idea. Here's how to concatenate a bunch of cons lists in linear rather than quadratic time (on the resulting list length): Walk to the end of the first list, link the next pointer to the start of the second list, walk to the end of that, etc. Easy and efficient, and completely impossible to implement in terms of a repeated __add__ or __iadd__ on the head node (or separate list object).
Which implies that using such implementation with sum could lead to O(N**2) performance. But it could not, because such implementation is not addable. So, no problem.
To have a problem you must modify your implementation. And if you're changing it anyway, you have the power to solve this problem too.
No, you don't. It's very easy to add an O(N) extend or __add__ to a cons list type, but it's very hard to make it O(N) summable with the sum function.
(It's certainly unusual for python's “batteries included” philosophy).
What does "batteries included" have to do with excluding data types?
Excluding data types? What do you mean?
I wasn't sure what you meant when you said about problems with linked lists, so I was thinking that you mean something like "if one day linked lists get their way into python standard libraries and people will try using sum() on them..." That's why I said that such implementation would be too skimped.
That's nonsense. If Python were to add something like a cons list—which I don't think would ever happen, but if it did—it would certainly not be a defective version that wasn't useful with the algorithms that people want cons lists for. "Batteries included" doesn't imply that every collection has to implement the same API as list with the same performance guarantees. You clearly don't understand how people use cons lists, and therefore you don't understand why your suggested "improvement" makes them much weaker.
What I would call a linked-list is a separate type where your nodes
are just its internal representation.
If you make the lists and nodes separate types, and the nodes private, you have to create yet a third type, like the list::iterator type in C++
If there would be a need for it, why not?
Agreed. That makes the APIs a little more complicated (you need to a list and a list::iterator instead of just a node), but that's not a big deal. And, with (list, list::iterator) being equivalent to a node, it leads to exactly the same issues as you started with in having just a node type.
Or maybe I won't need it (then I get something like collections.deque, a good tool by the way).
Yes, deque is a great tool, but it's not the same tool as a linked list, and doesn't support the same algorithms. (Note that C++ has both of them, as separate types.)
I don't like the idea that `a` implicitly changes when I change `b`.
Then you don't like linked lists. Saying "There is a way to make it work, just make it do something different, which is O(N) instead of O(1) and won't work with the algorithms you want to use" is not an answer.
That wasn't saying "just make it do something different". That was saying "you can have linked lists in python, that are O(N) summable".
Which is exactly the point you were arguing against. If you now agree with everyone else, fine. There are types that can be efficiently concatenated, but not with sum. That's why everyone else thinks you shouldn't encourage people to use sum for general concatenation.
But you CAN have a fast __iadd__ even for your simple a.next case!
You only need to initialize `tail` before calling sum() and update it inside __iadd__
Initialize tail where exactly? In a global variable?
Wherever you want. In a global variable, or in a class variable near 'next'.
Using a global variable (or a class attribute, which is the same thing) means that sum isn't reentrant, or thread-safe, or generator-safe. Trying to sum two completely different iterables at the same time will break. You really think that's a sensible design? Why even have a list type; just make the head and tail into global variables and everything is fine, right?
On Jul 8, 2013 Andrew Barnert wrote:
I'm -1 on adding special-casing for tuples that would not be available for any other immutable type.
Ok, let's be honest, I don't like that special case too. :( But when I had two options: 1. Make sum faster for everything BUT tuples and write in a manual: ... sum() is fast for all built-in types except `tuple`. For tuples you have to manually convert it to list, i.e. instead of: sum(list_of_tuples, tuple()) you have to write: tuple(sum(map(list,list_of_tuples),[])) or tuple(itertools.chain.from_iterable(list_of_tuples)) ... 2. Implement just one (!) special case for the only type in python needing it and write: ... sum() is fast for all built-in types! ... I chose #2. Tuple is one of the most frequently used types in python, and it's the only type that needs such a special case. Until someone writes a better solution: Practicality beats purity. That was my motivation.
No, you can't. You can do something different, but only by modifying the C source to sum. [...]
I can't make __iadd__ faster, because tuple has no __iadd__, however I can make a faster __add__.
No, you can't make tuple.__add__ faster either. (If you can, please submit a patch, because that would be useful completely independent of sum.)
Theoretically it's possible to rewrite a tuple type to internally use list type for storage, and additionally have a `localcount` variable saying how many items from that list belong to this tuple. Then __add__ for such tuple would just create a new tuple with exactly same internal list (with incremented ref.count) extended. This way, technically, __add__ would modify all the tuples around, but due to internal `localcount` variable you won't notice that. Would you like such a patch instead? Would you want to write it? ;) It's just this patch only optimizes add, which is ONLY needed for many sequential adds, i.e. for sum(), so I thought that it would be MUCH easier to add it to sum instead.
Are there any other types (except strings, that are blocked anyway)?
Yes. Every single immutable type.
Which is just one type — tuple. There's no other summable standard types in python having O(N) __add__, is it?
blist.btuple has the same problem.
Has it? I took just a brief look in its source and it seems that it already uses blist internally, so it can implement fast __add__ on its own (i.e. using the idea I described above).
You wanted sets to be addable? Then frozenset would have this problem.
But they're not addable, so still not a problem. :)
Strings obviously have this problem, and any third-party immutable string type will as well.
And strings are blocked by sum(), so no performance problems for them. (Third-party immutable strings?)
So, if you're suggesting that sum can be fast for anything reasonable, you're just wrong.
I suggested two ways how to do that. First, one can use the approach above, i.e. use mutable type internally. Second, for internal cpython types we have a simpler option to implement optimization directly in sum(). And there may be many others, specific to the types in question.
We're talking about python, and discussing use of sum() in python for such lists in particular.
No. You insisted that every collection type is O(N) summable with your design. Others argued that this isn't true. You asked for an example. I offered cons lists.
I don't remember saying that every collection type in a world is O(N) summable, but ok. Would you agree that all summable built-in collection types of python become O(N) summable with "my design"? I.e. they were not O(N) summable before my patch, but they are O(N) after it. Then why don't you like the patch? Because somewhere in the world there could theoretically exist some other types that are still not O(N) summable? Maybe, then we (or their authors) will deal with them later (and I tried to show you the options for your single linked list example). After all, they were not O(N) summable before this patch anyway. But they MAY BECOME O(N) summable after it.
If you agree that your design is not a general solution for al sequences, then all of your misunderstandings about cons lists are a red herring, and we can drop them.
I guess you misunderstand "my design" (or whatever you call that). Let's put it like that. Currently: The only way to make a type O(N) summable is to implement fast __add__ for it. So I suggested: It is often easier to implement fast __iadd__ than fast __add__. So let's change sum so that it took advantage of __iadd__ if it exists. Then someone said: You still cannot make sum fast for everything, i.e. for tuples I understood that as: If you already changing sum() you should make it fast for tuples too, so that we could say "sum() is fast now". So I replied: Yes, that patch does not meet "sum() is fast now" goal, because there's one more type `tuple` that is still slow. So, if we want to make sum fast for all built-in types, we must make it fast for tuples too. Here's a small patch, that just adds a special case for tuples, as that is the only type that needs it. This patch can be also extended to other types, e.g. lists and strings. Yes, authors of custom types won't have that simple option. But we have it, so why not use it, if it's MUCH easier than alternatives?
It's just you said: [...] First, that wasn't me; please try to keep your attributions straight.
Oops, sorry, my mistake.
But I agree with the sentiment. There is an efficient way to concatenate a lot of cons lists (basically, just fold backward instead of forward), but sum is not it.
Hm... If you implement fast __radd__ instead of __add__ sum would use that, wouldn't it? Is that the easy way you were looking for?
So, whoever said that is right—encouraging people to treat sum() as the preferred way to concatenate large amounts of data is a bad idea.
Then, what way you suggest to be preferred? For example how would you do that in py4k? I guess you would prefer sum() to reject lists and tuples as it now rejects strings and use some other function for them instead? Or what? What is YOUR preferred way?
Agreed. That makes the APIs a little more complicated (you need to a list and a list::iterator instead of just a node), but that's not a big deal. And, with (list, list::iterator) being equivalent to a node, it leads to exactly the same issues as you started with in having just a node type.
We have 'list' and 'listiterator', 'tuple' and 'tupleiterator', 'set' and 'setiterator'. Nothing unusual here. And no issues about them.
Yes, deque is a great tool, but it's not the same tool as a linked list, and doesn't support the same algorithms.
Not all of them, but some. I.e. if you used your cons-lists as queue or stack, then deque is a good replacement.
That wasn't saying "just make it do something different". That was saying "you can have linked lists in python, that are O(N) summable".
Which is exactly the point you were arguing against. If you now agree with everyone else, fine. There are types that can be efficiently concatenated, but not with sum. That's why everyone else thinks you shouldn't encourage people to use sum for general concatenation.
Really, I don't understand that point. Are you saying, that sum() must remain slow for FREQUENTLY USED standard types just because there MAY BE some other types for which it would still be slow?
Using a global variable (or a class attribute, which is the same thing) means that sum isn't reentrant, or thread-safe, or generator-safe.
Is it now? No? Then what changes? (BTW, your __iadd__ becomes not-thread-safe, not sum) PS: Talking about all collections in the world, theoretically it may be possible to extend my "special case" to all collection types in the world, but there's a small issue with that idea that I don't know how to handle...
On 9 Jul, 2013, at 15:42, Sergey <sergemp@mail.ru> wrote:
On Jul 8, 2013 Andrew Barnert wrote:
I'm -1 on adding special-casing for tuples that would not be available for any other immutable type.
Ok, let's be honest, I don't like that special case too. :( But when I had two options:
1. Make sum faster for everything BUT tuples and write in a manual: ... sum() is fast for all built-in types except `tuple`. For tuples you have to manually convert it to list, i.e. instead of: sum(list_of_tuples, tuple()) you have to write: tuple(sum(map(list,list_of_tuples),[])) or tuple(itertools.chain.from_iterable(list_of_tuples)) ...
2. Implement just one (!) special case for the only type in python needing it and write: ... sum() is fast for all built-in types! ...
I chose #2. Tuple is one of the most frequently used types in python, and it's the only type that needs such a special case.
Until someone writes a better solution: Practicality beats purity. That was my motivation.
The better solution is to not use sum. I haven't looked at your patch, but does it deal with all edge cases (such as calling sum on a heterogenous list that happens to have a tuple as its first item)? Trying to special-case sum for a sequence of tuples is (IMHO too) magical, and getting all details right makes the code more complicated. Sum is primarily intented for summing sequences of numeric values, that it works on list and tuples as well is a more or less unintended side-effect, and btw. the documentation for sum explicitly says its mentioned to sum numbers: sum(iterable[, start]) -> value Returns the sum of an iterable of numbers (NOT strings) plus the value of parameter 'start' (which defaults to 0). When the iterable is empty, returns start. Ronald
On Tue, Jul 9, 2013 at 7:07 AM, Ronald Oussoren <ronaldoussoren@mac.com>wrote:
The better solution is to not use sum. I haven't looked at your patch, but does it deal with all edge cases (such as calling sum on a heterogenous list that happens to have a tuple as its first item)? Trying to special-case sum for a sequence of tuples is (IMHO too) magical, and getting all details right makes the code more complicated.
Sum is primarily intented for summing sequences of numeric values, that it works on list and tuples as well is a more or less unintended side-effect, and btw. the documentation for sum explicitly says its mentioned to sum numbers:
sum(iterable[, start]) -> value
Returns the sum of an iterable of numbers (NOT strings) plus the value of parameter 'start' (which defaults to 0). When the iterable is empty, returns start.
I wonder if the best solution, if sum is only intended for use on numbers, would be to move it to the math module, rather than being a built-in function. No other changes would need to be made, although the special case for strings could likely be removed then as it would become fairly obvious that the string case is not reasonably supported. I would imagine that this would not take place until Python 4 (due to the large amount of existing code it would break), and I am not really proposing it, but it would seem to be logical.
On Jul 9, 2013, at 6:42, Sergey <sergemp@mail.ru> wrote:
On Jul 8, 2013 Andrew Barnert wrote:
I'm -1 on adding special-casing for tuples that would not be available for any other immutable type.
Ok, let's be honest, I don't like that special case too. :( But when I had two options:
1. Make sum faster for everything BUT tuples and write in a manual: ... sum() is fast for all built-in types except `tuple`. For tuples you have to manually convert it to list, i.e. instead of: sum(list_of_tuples, tuple()) you have to write: tuple(sum(map(list,list_of_tuples),[])) or tuple(itertools.chain.from_iterable(list_of_tuples)) ...
2. Implement just one (!) special case for the only type in python needing it and write: ... sum() is fast for all built-in types! ...
I chose #2. Tuple is one of the most frequently used types in python, and it's the only type that needs such a special case.
Until someone writes a better solution: Practicality beats purity. That was my motivation.
#3 is for the docs to say what they currently say--sum is fast for numbers--but change the "not strings" to "not sequences", and maybe add a note saying _how_ to concatenate sequences (usually join for strings and chain for everything else). This makes sense with or without an __iadd__ patch (as long as __iadd__ is useful for some number-like types--as I said before, I expect it might be, but I don't actually know).
No, you can't. You can do something different, but only by modifying the C source to sum. [...]
I can't make __iadd__ faster, because tuple has no __iadd__, however I can make a faster __add__.
No, you can't make tuple.__add__ faster either. (If you can, please submit a patch, because that would be useful completely independent of sum.)
Theoretically it's possible to rewrite a tuple type to internally use list type for storage, and additionally have a `localcount` variable saying how many items from that list belong to this tuple. Then __add__ for such tuple would just create a new tuple with exactly same internal list (with incremented ref.count) extended. This way, technically, __add__ would modify all the tuples around, but due to internal `localcount` variable you won't notice that.
I was going to point out the side effects of such a change, but someone beat me to it.
Would you like such a patch instead? Would you want to write it? ;)
It's just this patch only optimizes add, which is ONLY needed for many sequential adds, i.e. for sum(), so I thought that it would be MUCH easier to add it to sum instead.
And it's even easier to add neither.
Are there any other types (except strings, that are blocked anyway)?
Yes. Every single immutable type.
Which is just one type — tuple. There's no other summable standard types in python having O(N) __add__, is it?
Does nobody ever use types from the stdlib, third-party libs, or their own code in your world? Builtin types are not generally magical. A function that works well for all builtin types, but not types in the stdlib, is a recipe for confusion.
So, if you're suggesting that sum can be fast for anything reasonable, you're just wrong.
I suggested two ways how to do that. First, one can use the approach above, i.e. use mutable type internally. Second, for internal cpython types we have a simpler option to implement optimization directly in sum(). And there may be many others, specific to the types in question.
Using a mutable type internally will almost always have side effects, or at least complicate the implementation. What you're suggesting is that theoretically, for some different language that placed a very strange and sometimes hard to meet requirement on all types, sum could be fast for all types. That doesn't mean sum can be fast for all Python types, because Python doesn't, and shouldn't, have such a requirement. And again, what would be the benefit? That sum could become the obvious way to do concatenation instead of just summing? That's not even desirable, much less worth bending over backward for.
We're talking about python, and discussing use of sum() in python for such lists in particular.
No. You insisted that every collection type is O(N) summable with your design. Others argued that this isn't true. You asked for an example. I offered cons lists.
I don't remember saying that every collection type in a world is O(N) summable, but ok. Would you agree that all summable built-in collection types of python become O(N) summable with "my design"?
Yes, but if it's impossible to add a new collection type that works like tuple, that makes python worse, not better.
I.e. they were not O(N) summable before my patch, but they are O(N) after it. Then why don't you like the patch?
Because somewhere in the world there could theoretically exist some other types that are still not O(N) summable?
No, because all over the world there already actually exist such types, and because it's trivial--and useful--to create a new one.
Maybe, then we (or their authors) will deal with them later (and I tried to show you the options for your single linked list example).
And you only came up with options that destroy vital features of the type, making it useless for most people who would want to use it. But, more importantly, it is very simple to design and implement new collection types in Python today. Adding a new requirement that's hard enough to reach--one that you haven't been able to pull it off for the first example you were given--implies that it would no longer be easy.
If you agree that your design is not a general solution for al sequences, then all of your misunderstandings about cons lists are a red herring, and we can drop them.
I guess you misunderstand "my design" (or whatever you call that).
Your design is the design explicitly described in your email: sum uses __iadd__, and has special casing for at least one type. The argument against this is that it makes an existing mild attractive nuisance much worse, by implying that sum is actually fast for concatenation in general. Your counter was that sum can be actually fast for concatenation in general. That's not true. If you're now saying that it can't, only for certain types, then you need a new justification for why it isn't an attractive nuisance. The one you seem to be implying is "everybody expects non-builtin types to be less usable than builtins", which is wrong.
Let's put it like that. Currently: The only way to make a type O(N) summable is to implement fast __add__ for it. So I suggested: It is often easier to implement fast __iadd__ than fast __add__. So let's change sum so that it took advantage of __iadd__ if it exists.
So far, so good--but again, it would really help your case to find number-like types that this is true for. I already suggested some (np.matrix, for example).
Then someone said: You still cannot make sum fast for everything, i.e. for tuples I understood that as: If you already changing sum() you should make it fast for tuples too, so that we could say "sum() is fast now".
Then you misunderstood it. Tuples were offered as one example, out of many, that won't be fast. Solving that one example by adding a special case in the C code doesn't help the general problem unless you're prepared to do the same for every other example, which is impossible.
So I replied: Yes, that patch does not meet "sum() is fast now" goal, because there's one more type `tuple` that is still slow. So, if we want to make sum fast for all built-in types, we must make it fast for tuples too. Here's a small patch, that just adds a special case for tuples, as that is the only type that needs it. This patch can be also extended to other types, e.g. lists and strings.
Yes, authors of custom types won't have that simple option. But we have it
Who is this "we" here? Most users of Python use custom types. That's inherent in an OO language. The stdlib is full of custom types, and so are most third party libs and most applications.
, so why not use it, if it's MUCH easier than alternatives?
The obvious alternative--just not doing it--is much easier.
It's just you said: [...] First, that wasn't me; please try to keep your attributions straight.
Oops, sorry, my mistake.
But I agree with the sentiment. There is an efficient way to concatenate a lot of cons lists (basically, just fold backward instead of forward), but sum is not it.
Hm... If you implement fast __radd__ instead of __add__ sum would use that, wouldn't it? Is that the easy way you were looking for?
First, how do you propose that sum find out whether adding or radding is faster for a given type? More importantly: that wouldn't actually do anything in this case. The key to the optimization is doing the sequence of adds in reverse order, not flipping each add.
So, whoever said that is right—encouraging people to treat sum() as the preferred way to concatenate large amounts of data is a bad idea.
Then, what way you suggest to be preferred? For example how would you do that in py4k? I guess you would prefer sum() to reject lists and tuples as it now rejects strings and use some other function for them instead? Or what? What is YOUR preferred way?
I've already answered this, but I'll repeat it. sum is not the obvious way to concatenate sequences today, and my preferred way is for that to stay true. So, I'm: * +1 on sum using __iadd__ if it helps actual sums, -0 if it only helps list concatenation. * -1 on special casing tuple. * -1 on changing the docs to imply that sum should be used for sequences. * +0 on changing the docs to say "not sequences" instead of strings, and maybe even expand on it by showing how to concatenate sequences properly. * -0 on explicitly rejecting sequences or non-numbers or whatever in sum (largely because it's too hard to determine--and explain--what it should try to reject, but also because I don't think it's a common enough problem to be worth a change). * +0 on moving chain.from_iterable to builtins and renaming it. In other words, I don't think summing tuples is enough of an attractive nuisance that it's worth bending over backward to prevent it--but that doesn't mean we should bend over backward to improve something people shouldn't be doing, especially since that would make summing many other types into an attractive nuisance that doesn't exist today.
Agreed. That makes the APIs a little more complicated (you need to a list and a list::iterator instead of just a node), but that's not a big deal. And, with (list, list::iterator) being equivalent to a node, it leads to exactly the same issues as you started with in having just a node type.
We have 'list' and 'listiterator', 'tuple' and 'tupleiterator', 'set' and 'setiterator'. Nothing unusual here. And no issues about them.
But they aren't the same kind of thing at all. I don't want to explain the differences between what C++ calls iterators and what Python calls iterators unless it's necessary. But briefly, a std::list::iterator is a mutable reference to a node. Exposing that type means--as I've already said twice--that you end up with exactly the same problems you have exposing the node directly. If you don't understand why that's true, that's fine, but please stop ignoring it completely.
Yes, deque is a great tool, but it's not the same tool as a linked list, and doesn't support the same algorithms.
Not all of them, but some. I.e. if you used your cons-lists as queue or stack, then deque is a good replacement.
Well, yes, but a dynamic array like Python's list is also a perfectly good stack. So what? I honestly can't tell at this point whether you're being deliberately obtuse, or just don't understand the basics of why we have different data structures in the first place.
That wasn't saying "just make it do something different". That was saying "you can have linked lists in python, that are O(N) summable".
Which is exactly the point you were arguing against. If you now agree with everyone else, fine. There are types that can be efficiently concatenated, but not with sum. That's why everyone else thinks you shouldn't encourage people to use sum for general concatenation.
Really, I don't understand that point. Are you saying, that sum() must remain slow for FREQUENTLY USED standard types just because there MAY BE some other types for which it would still be slow?
You're twisting the emphasis drastically, but basically yes. Today, sum is not the best way to concatenate sequences. Making it work better for some sequences but not others would mean it's still not the best way to concatenate sequences, but it would _appear_ to be. That's the very definition of an attractive nuisance.
Using a global variable (or a class attribute, which is the same thing) means that sum isn't reentrant, or thread-safe, or generator-safe.
Is it now? No? Then what changes?
Yes, it is now. So that's what changes. Again, this is something general and basic--operations that use global variables are not reentrant--and I can't tell whether you're being deliberately obtuse or whether you really don't understand that.
On 9 Jul, 2013, at 19:11, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 9, 2013, at 6:42, Sergey <sergemp@mail.ru> wrote:
On Jul 8, 2013 Andrew Barnert wrote:
I'm -1 on adding special-casing for tuples that would not be available for any other immutable type.
Ok, let's be honest, I don't like that special case too. :( But when I had two options:
1. Make sum faster for everything BUT tuples and write in a manual: ... sum() is fast for all built-in types except `tuple`. For tuples you have to manually convert it to list, i.e. instead of: sum(list_of_tuples, tuple()) you have to write: tuple(sum(map(list,list_of_tuples),[])) or tuple(itertools.chain.from_iterable(list_of_tuples)) ...
2. Implement just one (!) special case for the only type in python needing it and write: ... sum() is fast for all built-in types! ...
I chose #2. Tuple is one of the most frequently used types in python, and it's the only type that needs such a special case.
Until someone writes a better solution: Practicality beats purity. That was my motivation.
#3 is for the docs to say what they currently say--sum is fast for numbers--but change the "not strings" to "not sequences", and maybe add a note saying _how_ to concatenate sequences (usually join for strings and chain for everything else).
Good idea, I hope Guido hasn't noticed that his time machine was gone for a while ;-) <quote> sum(iterable[, start]) Sums start and the items of an iterable from left to right and returns the total. start defaults to 0. The iterable‘s items are normally numbers, and the start value is not allowed to be a string. For some use cases, there are good alternatives to sum(). The preferred, fast way to concatenate a sequence of strings is by calling ''.join(sequence). To add floating point values with extended precision, see math.fsum(). To concatenate a series of iterables, consider using itertools.chain(). </quote> This is from http://docs.python.org/3/library/functions.html (and is in the python 2.7 version as well) Ronald
On Jul 9, 2013 Andrew Barnert wrote:
1. Make sum faster for everything BUT tuples and write in a manual: ... 2. Implement just one (!) special case for the only type in python needing it and write: ...
I chose #2. Tuple is one of the most frequently used types in python, and it's the only type that needs such a special case.
Until someone writes a better solution: Practicality beats purity. That was my motivation.
#3 is for the docs to say what they currently say--sum is fast for numbers--but change the "not strings" to "not sequences", and maybe add a note saying _how_ to concatenate sequences (usually join for strings and chain for everything else).
Which effectively means do nothing and agree that slow sum is the best for python. Despite we CAN make it faster. For all cases, or for most of them, but we could at least start discussing options, instead of repeating excuses.
Theoretically it's possible to rewrite a tuple type to internally use list type for storage, and additionally have a `localcount` variable saying how many items from that list belong to this tuple. Then __add__ for such tuple would just create a new tuple with exactly same internal list (with incremented ref.count) extended. This way, technically, __add__ would modify all the tuples around, but due to internal `localcount` variable you won't notice that.
I was going to point out the side effects of such a change, but someone beat me to it.
Yes, as a side-effect it would sometimes use less memory (and it could also use more memory, however I can't think of any real-world cases where that could be a problem). But the main goal was met: it would make tuple O(N) summable. So:
Would you like such a patch instead? Would you want to write it? ;)
It's just this patch only optimizes add, which is ONLY needed for many sequential adds, i.e. for sum(), so I thought that it would be MUCH easier to add it to sum instead.
And it's even easier to add neither.
It was even easier to not create python at all, than we wouldn't have to spend our time here discussing it. It's always easy to do nothing. But sometimes it takes so much time to do something good...
Which is just one type — tuple. There's no other summable standard types in python having O(N) __add__, is it?
Does nobody ever use types from the stdlib, third-party libs, or their own code in your world? Builtin types are not generally magical. A function that works well for all builtin types, but not types in the stdlib, is a recipe for confusion.
What types from stdlib it does not work for?
So, if you're suggesting that sum can be fast for anything reasonable, you're just wrong.
I suggested two ways how to do that. First, one can use the approach above, i.e. use mutable type internally. Second, for internal cpython types we have a simpler option to implement optimization directly in sum(). And there may be many others, specific to the types in question.
Using a mutable type internally will almost always have side effects, or at least complicate the implementation.
Yes, optimization may or may not complicate implementation. Nothing new here. For example "optimizing" tuple most probably won't complicate implementation, it would just make it different, but faster to __add__.
What you're suggesting is that theoretically, for some different language that placed a very strange and sometimes hard to meet requirement on all types, sum could be fast for all types. That doesn't mean sum can be fast for all Python types, because Python doesn't, and shouldn't, have such a requirement.
No. What I suggested is that sum() could be faster for SOME types. And I provided a patch for that. But you (or someone else) said that I can't make sum fast for other types, for example for tuples. So I provided a patch doing that. Then you said that I can't do that for tuples without patching sum. And I explained how it could be done too. I never put any requirements. It's just you somewhy placed a strange requirement on sum() that it must always remain slow...
And again, what would be the benefit?
Hm. Benefits of O(N)-summable builtin types? * No more surprises "Oh, sum() is O(N**2) Why?" * People can use whatever builtin functions they want with any built-in types they want in any obvious to them way and nobody would say them that it will be too slow. A little slower, maybe, but not too much. * Programs become faster * Code becomes simpler and easier to read * Python becomes a better language * More people start using python * Everybody's happy, world piece, etc.
I don't remember saying that every collection type in a world is O(N) summable, but ok. Would you agree that all summable built-in collection types of python become O(N) summable with "my design"?
Yes, but if it's impossible to add a new collection type that works like tuple, that makes python worse, not better.
Why would it be impossible? In worst case it would be hard (but not impossible) to add a new collection type, that is as fast as tuple, yes, so what? It is already hard to do that, nothing new there.
I.e. they were not O(N) summable before my patch, but they are O(N) after it. Then why don't you like the patch?
Because somewhere in the world there could theoretically exist some other types that are still not O(N) summable?
No, because all over the world there already actually exist such types, and because it's trivial--and useful--to create a new one.
Could you show me some code where a lot of those custom summable sequences are added together? Or maybe about someone, complaining about sum being slow for them? No examples? Then, this is not a problem, right? ;)
Maybe, then we (or their authors) will deal with them later (and I tried to show you the options for your single linked list example).
And you only came up with options that destroy vital features of the type, making it useless for most people who would want to use it.
I did not. Or are you saying that STL developers also destroyed vital features of your the because they have not implemented std::list the same way? Have python destroyed those vital features with it's list? Your type is not summable. You asked for the type that is summable. I suggested one to you, you did not liked it. That's it, nothing is destroyed.
But, more importantly, it is very simple to design and implement new collection types in Python today. Adding a new requirement that's hard enough to reach--one that you haven't been able to pull it off for the first example you were given--implies that it would no longer be easy.
What requirements are you talking about?
Then you misunderstood it. Tuples were offered as one example, out of many, that won't be fast. Solving that one example by adding a special case in the C code doesn't help the general problem
That's why I explained how this can also be done without writing a special case in sum(). It's just if you have a goal and two ways to reach it, one of them is extremely complicated, and another one is easy, other than that they're equal, which one will you choose?
But I agree with the sentiment. There is an efficient way to concatenate a lot of cons lists (basically, just fold backward instead of forward), but sum is not it.
Hm... If you implement fast __radd__ instead of __add__ sum would use that, wouldn't it? Is that the easy way you were looking for?
First, how do you propose that sum find out whether adding or radding is faster for a given type?
More importantly: that wouldn't actually do anything in this case. The key to the optimization is doing the sequence of adds in reverse order, not flipping each add.
Yea, I agree, it won't solve the problem right away, since elements would still be radded left-to-right. Well, you still have at least two options for your cons-lists: either use a `tail` attribute of your list or as a some global variable. I.e. list_of_cons_lists[0].tail = find_tail(list_of_cons_lists[0]) sum(list_of_cons_lists) and use `tail` in your __add__ implementation. It's that simple. Or, since you're writing a custom type anyway, and you have to specify start class for sum() you can use it to hold a tail for you. Like: class ConsListsSum: tail = None def __add__(self, other): if self.tail is None: self.tail = other else: self.tail.next = other while self.tail.next is not None: self.tail = self.tail.next return self sum(list_of_cons_lists, ConsListsSum()) In that case to use sum you won't need __add__ in your list at all.
Then, what way you suggest to be preferred? For example how would you do that in py4k? I guess you would prefer sum() to reject lists and tuples as it now rejects strings and use some other function for them instead? Or what? What is YOUR preferred way?
I've already answered this, but I'll repeat it.
sum is not the obvious way to concatenate sequences today, and my preferred way is for that to stay true. So, I'm: [...]
You're saying what IS NOT your preferred way. But I'm asking what IS your preferred way. Do you prefer to have a slow sum in python and people asking why it's slow forever? Do you see that as a best possible case for python?
We have 'list' and 'listiterator', 'tuple' and 'tupleiterator', 'set' and 'setiterator'. Nothing unusual here. And no issues about them.
But they aren't the same kind of thing at all. I don't want to explain the differences between what C++ calls iterators and what Python calls iterators unless it's necessary. But briefly, a std::list::iterator is a mutable reference to a node.
And std::list::const_iterator is not. So?
Exposing that type means--as I've already said twice--that you end up with exactly the same problems you have exposing the node directly.
No, I won't, unless I make it mutable too. And I don't have to.
Not all of them, but some. I.e. if you used your cons-lists as queue or stack, then deque is a good replacement.
Well, yes, but a dynamic array like Python's list is also a perfectly good stack. So what?
Nothing. It's just you said:
If you make the lists and nodes separate types, and the nodes private, you have to create yet a third type, like the list::erator type in C++ So I answered that I don't always have to do that. I may not need iterator for some of cons-list use cases.
Really, I don't understand that point. Are you saying, that sum() must remain slow for FREQUENTLY USED standard types just because there MAY BE some other types for which it would still be slow?
You're twisting the emphasis drastically, but basically yes.
And that's the only reason? What if I solve that too? E.g. what if there would be a common way exposed to all the types in a world? For example imaging a special case (or is it "general case" now) like this: if hasattr(type(start), "__init_concatenable_sequence_from_iterable__"): optimize_for = type(start) l_result = list() l_result.extend(start) try: while True: item = next(it) if type(item) != optimize_for: start = optimize_for.__init_concatenable_sequence_from_iterable__(l_result) start = start + item break l_result.extend(item) except StopIteration: return optimize_for.__init_concatenable_sequence_from_iterable__(l_result) In that case every type would be able to implement an optional __init_concatenable_sequence_from_iterable__ method to benefit from optimized sum(). If they won't do that sum() would use general code. And of course it would be trivial to implement it for e.g. tuples. Is that what you want?
Today, sum is not the best way to concatenate sequences. Making it work better for some sequences but not others would mean it's still not the best way to concatenate sequences, but it would _appear_ to be. That's the very definition of an attractive nuisance.
Today python is not the best language in the world, because it still has bugs. Fixing some bugs but not others would mean that it's still not the best language in the world, but it would _appear_ to be. So, let's never fix any bugs? Don't you think that your logic is flawed? ;)
Using a global variable (or a class attribute, which is the same thing) means that sum isn't reentrant, or thread-safe, or generator-safe.
Is it now? No? Then what changes?
Yes, it is now. So that's what changes.
sum() uses Py_DECREF. Py_DECREF is not thread-safe. It means that sum() is not thread safe too. Where am I wrong?
Again, this is something general and basic--operations that use global variables are not reentrant--and I can't tell whether you're being deliberately obtuse or whether you really don't understand that.
Well, if you don't want to use a global variable — don't use it. :)
On 12 July 2013 02:34, Sergey <sergemp@mail.ru> wrote:
On Jul 9, 2013 Andrew Barnert wrote:
but we could at least start discussing options, instead of repeating excuses.
This is rude, and I'd rather you try to avoid being rude.
Theoretically it's possible to rewrite a tuple type to internally use list type for storage, and additionally have a `localcount` variable saying how many items from that list belong to this tuple. Then __add__ for such tuple would just create a new tuple with exactly same internal list (with incremented ref.count) extended. This way, technically, __add__ would modify all the tuples around, but due to internal `localcount` variable you won't notice that.
I was going to point out the side effects of such a change, but someone beat me to it.
Yes, as a side-effect it would sometimes use less memory (and it could also use more memory, however I can't think of any real-world cases where that could be a problem).
Is this similar to the shared-memory thing for dictionaries? If so, it might be a good proposal in and of itself.
Which is just one type — tuple. There's no other summable standard types in python having O(N) __add__, is it?
Does nobody ever use types from the stdlib, third-party libs, or their own code in your world? Builtin types are not generally magical. A function that works well for all builtin types, but not types in the stdlib, is a recipe for confusion.
What types from stdlib it does not work for?
Shall I list some stdlib types; you can tell me what it's fast for: # do this later
So, if you're suggesting that sum can be fast for anything reasonable, you're just wrong.
I suggested two ways how to do that. First, one can use the approach above, i.e. use mutable type internally. Second, for internal cpython types we have a simpler option to implement optimization directly in sum(). And there may be many others, specific to the types in question.
Using a mutable type internally will almost always have side effects, or at least complicate the implementation.
Yes, optimization may or may not complicate implementation. Nothing new here. For example "optimizing" tuple most probably won't complicate implementation, it would just make it different, but faster to __add__.
But you also make it complicated for everyone who subclasses, for example. It's not good to make lives harder for this.
And again, what would be the benefit?
Hm. Benefits of O(N)-summable builtin types? * No more surprises "Oh, sum() is O(N**2) Why?"
Only you haven't removed that, as when people use a list subclass instead it comes back -- in a much more devastating fashion. Then they have to rewrite their code.
* People can use whatever builtin functions they want with any built-in types they want in any obvious to them way and nobody would say them that it will be too slow. A little slower, maybe, but not too much.
No they can't. You're conflating sum() with *all* builtins.
* Programs become faster
Some. But not many.
* Code becomes simpler and easier to read
Hardly. If "my_type(chain.from_iterable(iterable))" is the hardest part of code for you to read, you're doing it wrong.
Maybe, then we (or their authors) will deal with them later (and I tried to show you the options for your single linked list example).
And you only came up with options that destroy vital features of the type, making it useless for most people who would want to use it.
I did not. Or are you saying that STL developers also destroyed vital features of your the because they have not implemented std::list the same way? Have python destroyed those vital features with it's list? Your type is not summable. You asked for the type that is summable. I suggested one to you, you did not liked it. That's it, nothing is destroyed.
I think you need to re-read those posts.
But, more importantly, it is very simple to design and implement new collection types in Python today. Adding a new requirement that's hard enough to reach--one that you haven't been able to pull it off for the first example you were given--implies that it would no longer be easy.
What requirements are you talking about?
Surely he means that extra flab you have to write to make it fast-summable. Otherwise other people's code that uses sum() because you said it was the "one true way" suddenly becomes painfully slow. Or they could have written the current method and it would've worked immediately.
Then you misunderstood it. Tuples were offered as one example, out of many, that won't be fast. Solving that one example by adding a special case in the C code doesn't help the general problem
That's why I explained how this can also be done without writing a special case in sum(). It's just if you have a goal and two ways to reach it, one of them is extremely complicated, and another one is easy, other than that they're equal, which one will you choose?
No matter which way you write it, it doesn't solve the problem.
In that case to use sum you won't need __add__ in your list at all.
Then, what way you suggest to be preferred? For example how would you do that in py4k? I guess you would prefer sum() to reject lists and tuples as it now rejects strings and use some other function for them instead? Or what? What is YOUR preferred way?
I've already answered this, but I'll repeat it.
sum is not the obvious way to concatenate sequences today, and my preferred way is for that to stay true. So, I'm: [...]
You're saying what IS NOT your preferred way. But I'm asking what IS your preferred way. Do you prefer to have a slow sum in python and people asking why it's slow forever? Do you see that as a best possible case for python?
YES. Godamnit YES. 100% true-to-form YES. Now stop asking us.
Really, I don't understand that point. Are you saying, that sum() must remain slow for FREQUENTLY USED standard types just because there MAY BE some other types for which it would still be slow?
You're twisting the emphasis drastically, but basically yes.
And that's the only reason? What if I solve that too? E.g. what if there would be a common way exposed to all the types in a world? For example imaging a special case (or is it "general case" now) ... ld be able to implement an optional __init_concatenable_sequence_from_iterable__ method to benefit from optimized sum(). If they won't do that sum() would use general code. And of course it would be trivial to implement it for e.g. tuples. Is that what you want?
So your solution to make sum faster is to make everything else harder?
Today, sum is not the best way to concatenate sequences. Making it work better for some sequences but not others would mean it's still not the best way to concatenate sequences, but it would _appear_ to be. That's the very definition of an attractive nuisance.
Today python is not the best language in the world, because it still has bugs. Fixing some bugs but not others would mean that it's still not the best language in the world, but it would _appear_ to be. So, let's never fix any bugs?
Don't you think that your logic is flawed? ;)
Does that actually seem like a good counterargument? It's a massively flawed analogy. Bugs are not the same as something being not-the-standard. A good analogy would be if Python was bad for <task>. If Python changed itself so it was good at <task> for one thing, that would be an attractive nuisance. Because as soon as you try to use Python, you find it isn't good for all things you need it for any you just implemented a useless program. If code uses sum() it needs to work for *all types* that the code receives. Python is duck-typed - so if you only want an indexable sequence then it needs to be fast for indexable sequences. There is no way to solve this without making people implement loads more code. When they could just use list(chain.from_iterable(...)) instead, sum looks like a really dumb idea.
On 12 July 2013 03:07, Joshua Landau <joshua@landau.ws> wrote:
Shall I list some stdlib types; you can tell me what it's fast for:
# do this later
I knew one-day I was going to forget to go back and finish a post. OK, here is a list of all types in the stdlib with __add__ or __iadd__: bool bytearray bytes codecs.CodecInfo collections.Counter collections.UserList collections.UserString collections.abc.MutableSequence collections.abc.MutableSequence collections.deque collections.deque complex float functools.CacheInfo functools._HashedSeq inspect.ArgInfo inspect.ArgSpec inspect.Arguments inspect.Attribute inspect.ClosureVars inspect.FullArgSpec inspect.ModuleInfo inspect.Traceback inspect._ParameterKind int list os.terminal_size os.terminal_size posix.sched_param posix.sched_param posix.stat_result posix.stat_result posix.statvfs_result posix.statvfs_result posix.times_result posix.times_result posix.uname_result posix.uname_result posix.waitid_result posix.waitid_result signal.struct_siginfo str str tokenize.TokenInfo tuple weakcallableproxy weakcallableproxy weakproxy weakproxy You tell me whether you have 100% coverage of the stdlib things that "sum" is plausible for.
Sergey writes:
Which effectively means do nothing and agree that slow sum is the best for python.
Sum is not slow for numbers, which many posters believe is the only plausible usage, despite the accident that it can be used for other types.
Despite we CAN make it faster.
Faster is not necessarily better. Several posters have claimed that "sum(list_of_lists)" is unreadable (for them). While Python is a "consenting adults" language (so "if you don't like it, don't use it" is a plausible argument), Python aspires to "universal" readability. So plausibly "sum(list_of_lists)" is a bad thing in itself. If there's another "obvious way to do it" (here, itertools.chain which is both efficient computationally and lazy), then making sum() better only encourages use. There are a couple of similar optimizations that Guido himself admits he regrets. Consider "sum(sum(list_of_lists_of_numbers))". For me, that's a WTF. I'd much rather "sum(chain(list_of_lists_of_numbers))". I don't ask you to agree, let alone expect you to do so. I would hope you can offer the same courtesy for those who disagree with you. Here I am only trying to explain why they might disagree with you, and why it may be difficult to change their minds.
For all cases, or for most of them, but we could at least start discussing options, instead of repeating excuses.
"Do nothing" is an option, and it's the one preferred by most posters at this point. If you don't like that, write a PEP, present it here, and then on python-dev, and get it approved. As has been pointed out, a PEP is needed, and the candidates for PEP czar probably are no longer listening, and none of them have argued against, so you do have a chance to convince that way. But as also has been pointed out, you haven't presented a new argument for several days, and your old ones haven't convinced. You need to either try something new, or get a new audience (by writing the PEP). Nobody else is going to do it for you. BTW, "make the TOOWTDI for sequences a built-in" is a new option recently proposed and now being discussed. (Ie, the subthread discussing that for "itertools.chain.from_iterable".) I kinda like "ichain" for the name, and I like that proposal. Regards,
On Jul 11, 2013, at 18:34, Sergey <sergemp@mail.ru> wrote:
And again, what would be the benefit?
Hm. Benefits of O(N)-summable builtin types? * No more surprises "Oh, sum() is O(N**2) Why?"
No, it would be _more_ surprises. "Oh, sum() is O(N**2). Why? The docs say it's fast, it actually is fast for tuples, my class is just like a tuple." This is exactly why people keep bringing up other types. I don't know what you're not getting here. If you want to argue that it's not a bad enough attractive nuisance to worry about, that might be a reasonable argument. But you've never made that argument; instead you just deny that the problem exists at all, going back and forth between claiming that we can make sum fast for all types and arguing that every type anyone brings up in an objection should be changed.
On Jul 11, 2013, at 18:34, Sergey <sergemp@mail.ru> wrote:
sum is not the obvious way to concatenate sequences today, and my preferred way is for that to stay true. So, I'm: [...]
You're saying what IS NOT your preferred way. But I'm asking what IS your preferred way.
You just quoted it. My preferred way to handle this is what we already do: don't encourage people to misuse sum for concatenating sequences, encourage them to use something appropriate. That could mean making the wording in the docs even stronger, or even explicitly rejecting sequences in sum, but I don't think the problem is anywhere near serious enough to do either of those. That could also mean making the obvious way more obvious. In other words: move itertools.chain.from_iterable to builtins under a new name.
Do you prefer to have a slow sum in python and people asking why it's slow forever? Do you see that as a best possible case for python?
Yes. Given that it is impossible to make sum fast for all collection types, and that it's the wrong function for concatenating collections in the first place, and that, even if it were fast, it would still be inferior to chain.from_iterable in the exact same way that 2.x map is inferior to 3.x map, I do. Also, It's less surprising this way, not more. Today, people only have to learn one thing: Don't use sum on collections. That's much easier than having to learn a complex mishmash like: Don't use sum on immutable collections, except for tuple, and also don't use it on some mutable collections, but it's hard to characterize exactly which, and also don't use it on things that are iterable but that you don't want to treat as sequences, and... One of the reasons I hate PHP is that all of its rules work that way--everything does what you expect in 60% of the cases, and does something baffling in the other 40%. Finally, I've ignored your requests for examples because in every case you've already been given examples and haven't dealt with any of them. If I want to concatenate cons lists, chain does it in linear time, your design does not; instead of answering that, you just keep arguing that you can sum a different kind of linked list in linear time. That doesn't even approach answering the objection. So what's the point of offering another one that you're just going to treat in the same way? Especially since I and others have already given you other examples and you just ignored them?
On Jul 11, 2013 Andrew Barnert wrote:
sum is not the obvious way to concatenate sequences today, and my preferred way is for that to stay true. So, I'm: [...]
You're saying what IS NOT your preferred way. But I'm asking what IS your preferred way.
You just quoted it. My preferred way to handle this is what we already do: don't encourage people to misuse sum for concatenating sequences, encourage them to use something appropriate.
I don't understand it. It makes no sense to me. Do you like having many broken tools? E.g. would you liked if someone added __mult__ to set()s, but made it O(N*N*N) so that people would not used it too often? Anyway, I've got your point. You want sum() to be O(N) for numbers and some rare/custom containers, but want it to stay O(N*N) for the most popular container types for some reasons, that are too hard for me to understand.
Do you prefer to have a slow sum in python and people asking why it's slow forever? Do you see that as a best possible case for python?
Yes. Given that it is impossible to make sum fast for all collection types
Technically it may be possible. As if technically it's possible to have no wars on Earth. It requires a great cooperation of many people, so it's not probable, but possible. So what? It's (kind of) impossible to make python fast for all programs, but it does not mean that python should not be fast for some programs, Same applies to sum(): even if it's impossible to make if fast for all collection types, it does not mean that it should not be fast for some of them, e.g. lists and tuples. After all it's quite easy to make it fast for most (if not all) commonly used cases.
Also, It's less surprising this way, not more. Today, people only have to learn one thing: Don't use sum on collections. That's much easier than having to learn a complex mishmash like: Don't use sum on immutable collections, except for tuple, and also don't use it on some mutable collections, but it's hard to characterize exactly which, and also don't use it on things that are iterable but that you don't want to treat as sequences, and...
It's much easier to just learn: don't use sum(). And it's even easier to learn: use sum(), because you don't have to learn that. ;)
Finally, I've ignored your requests for examples because in every case you've already been given examples and haven't dealt with any of them.
Oh, really? You said, I can't make sum fast for strings and tuples, so I did that and showed a patch. Then you said that I can't make sum fast for linked lists, so I suggested how to do that. You did not liked my linked lists, so I explained how you can do that for your cons-lists. You did not liked my explanation (and said some weird things about thread safety) so I showed you two code samples, both using your cons-lists and sum(). You also said that I can't make tuples __add__ faster without patching sum() so explained how you can do that (you never answered whether you like such a patch and whether you would agree to write it). I even wrote a simple fasttuple proof of concept [1]. What examples I haven't dealt with? ;)
If I want to concatenate cons lists, chain does it in linear time your design does not;
Quite the opposite, my design would give you a list, while chain won't even give you a correct list, since `next` elements of that "list" would not point to correct locations. BTW, chain would not work for your list, because it's not iterable by default, is it? And even if you implement __iter__ for it, how are you going to handle modifications of you list while you iterate it? You don't have those problems with sum(). Hm, that basically marks chain() as error-prone replacement for sum, that is sometimes harder to implement support for.
instead of answering that, you just keep arguing that you can sum a different kind of linked list in linear time. That doesn't even approach answering the objection.
If the "objection" was "you can't make it fast for linked lists" then "you can" is the exact answer to that objection. :) -- [1] http://bugs.python.org/file30917/fasttuple.py
Sergey writes:
I don't understand it. It makes no sense to me.
Just accept that many people, *for several different reasons*, dislike your proposal. The technical objections are the least of those reasons. Please just write the PEP and ask for a pronouncement. If you don't feel confident in your PEP-writing skills, ask for help. (If you don't get any, you probably should take that as "the community says No".)
Do you like having many broken tools?
And please stop this. sum() is not broken, any more than a screwdriver is broken just because it is rather inefficient when used to pound in nails.
On Jul 16, 2013 Stephen J. Turnbull wrote:
I don't understand it. It makes no sense to me.
Just accept that many people, *for several different reasons*, dislike your proposal. The technical objections are the least of those reasons.
Sure, I already did. I just tried to understand, what part of my proposal they don't like and why, in that case I could fix that part, so that they liked id. If I don't understand — I don't know what to fix.
Please just write the PEP and ask for a pronouncement. If you don't feel confident in your PEP-writing skills, ask for help. (If you don't get any, you probably should take that as "the community says No".)
Well, my current PEP-writing-skills are kind of zero, but the problem is that I don't know what to write there yet. PEP probably implies that I want to change something, but my simplest patch [1] covers most of use cases and changes nothing, except performance, and should need no PEP. The opposite to it is a unified container filling protocol. But it has lots of options, and still heavily discussed.
Do you like having many broken tools?
And please stop this. sum() is not broken, any more than a screwdriver is broken just because it is rather inefficient when used to pound in nails.
No, that would be the case if I was using sum for something that it's not intended to do, e.g. instead of: x = 60*70 I could try: x = sum([60].__mul__(70)) That would work, but would be obviously inefficient, because sum() is not intended to do multiplication. `a + b + c + .. + z` is exactly what sum is supposed to do. But due to some implementation-specific details it does that slowly for some a...z types. It's not a technical problem, it's a political (or personal ideology?) problem to allow sum being fast for types that it was slow for last 10 years. It's more like a drill, that is good for metal and bad for wood, because it was equipped with metal drill bits, but not with wood drill bits. It could be just fine for wood too, but its developer believes, that his drill should not be used to make holes in wood. Would you call such a drill "broken"? I would. This bug was known for a long time, I just raised a question of what would be the best way to fix that. Should we just equip it with a few most common wood drill bits (e.g. lip&spur + spade)? Or should we put a grinding stone in, so that anyone could make a drill bit himself? Or maybe both? -- [1]http://bugs.python.org/file30897/fastsum-special-tuplesandlists.patch
Sergey writes:
Do you like having many broken tools?
And please stop this. sum() is not broken, any more than a screwdriver is broken just because it is rather inefficient when used to pound in nails.
No, that would be the case if I was using sum for something that it's not intended to do, [...]
I'll say it one last time: this kind of answer does not help your case at all. I assure you "proof by repeated assertion" doesn't work here. The "intent" of sum() is clearly documented: it computes the sum of an iterable of numbers. From the library reference for 2.6 (it hasn't changed up to 3.4.0a, except to refer to itertools.chain() and remove the reference to reduce()): sum(iterable[, start]) Sums start and the items of an iterable from left to right and returns the total. start defaults to 0. The iterable‘s items are normally numbers, and are not allowed to be strings. The fast, correct way to concatenate a sequence of strings is by calling ''.join(sequence). Note that sum(range(n), m) is equivalent to reduce(operator.add, range(n), m) To add floating point values with extended precision, see math.fsum(). True, it does not deny that sum() *could* be used for certain non- numbers, but *intent* is clear: sum() adds up a sequence of numbers. Generalizing it to efficiently handle concatenation of sequences is an enhancement, not a bugfix. Your argument is simply that we *could* use sum() for anything that provides the __add__ method, and with __iadd__ it can be efficient in time and space in many cases. You point to the fact that some programmers do use sum() inefficiently, and suggest that we remove this pitfall by making sum() efficient for as many cases as possible. Such enhancements are certainly of interest to python-dev, but that's not sufficient. The counterclaim that matters is that "sum" is a *bad name* for functions that aggregate iterables, unless the type of the elements is (or can be coerced to) numerical. It follows that use of that name makes programs hard to read, and it should be deprecated in favor of readable idioms.[1] Until you successfully address that counterclaim, you are going to fail to persuade enough of the people who matter. Footnotes: [1] Note that backward compatibility, not weakness of the "bad name" argument, is why we compromise by deprecating in words rather than making it impossible to use sum on "wrong" types.
2013/7/19 Sergey <sergemp@mail.ru>:
Well, my current PEP-writing-skills are kind of zero, but the problem is that I don't know what to write there yet.
To write a PEP, I write 4 main sections: - Abstract - Rationale: why do you consider that something must be changed - Proposal: describe your propositon - Alternatives: list most alternatives proposed on python-ideas You may list advantages and drawbacks of each alternative. This is just a template to write a first draft ;-) I read only a few emails of the sum() threads, but I remember the following options: - only sum numbers: reject all other types - "if start is not None: x = start else x = first(items); for item in items: x += item; return x" your proposal - a new protocol (__concat__? I don't remember its name) with a fallback on "x=start; for item in items: x += item" or to your proposal - implement __iadd__ for immutable types (tuple += tuple) or something like that - etc.
PEP probably implies that I want to change something, but my simplest patch [1] covers most of use cases and changes nothing, except performance, and should need no PEP.
You don't need an implementation to write a PEP. In your case, you have an implementation. It's better to wait for a consensus on the PEP before update the implementation to the last PEP, or you may waste your time. Victor
On Tue, Jul 16, 2013 at 08:36:05AM +0300, Sergey wrote:
Anyway, I've got your point. You want sum() to be O(N) for numbers and some rare/custom containers, but want it to stay O(N*N) for the most popular container types for some reasons, that are too hard for me to understand.
Right now, sum() behaves in a certain way. There are certain things which people expect sum() to do. Some of those things are documented explicitly. Some of them are implied. Some of them have regression tests. Some of them don't. But regardless, we can tell how sum() behaves right now by running it and seeing what it does. Your suggested optimizations change that behaviour. It does not just speed sum() up, they lead to an actual semantic change. So we are not just arguing about speed, we are arguing about behaviour as well. You are worried about sum() being slow for people who call it with list arguments. That is a valid concern. Nobody here *wants* sum() to be slow. If it was a pure speed optimization, then we would all be 100% behind it. But it is not a pure speed optimization, it also changes behaviour, sometimes in subtle, hard to see ways. So there are three approaches we can take: - Do nothing. sum() continues to work exactly the same way as it currently works, even if that means sometimes it is slow. - Accept your patches. sum() changes its behaviour, which will break somebody's working code, but it will be fast, at least for some objects. - Accept a compromise position. We can make sum() faster for built-in lists, and maybe built-in tuples, while keeping the same behaviour. Everything else, including subclasses of list and tuple, keep the same behaviour, which may mean it remains slow. They are the only choices. You are concerned more about sum() being slow than you are about breaking code that today works. Some of us here disagree, and think that breaking code is worse than slow code, especially for something as uncommon as sum(list_of_lists). It's not that we want sum() to be slow. But if we have a choice between accepting your patch: # this code works now, but your patch will break it result = sum(list_of_objects) and rejecting it: # this code works now, but is slow, and will remain slow result = sum(list_of_lists) I believe that the decision is simple. Breaking code that works now for a mere optimization is unacceptable. But, a compromise patch that speeds up some code without breaking any code may be acceptable.
Same applies to sum(): even if it's impossible to make if fast for all collection types, it does not mean that it should not be fast for some of them, e.g. lists and tuples.
That is change from your previous posts where you said you could make it fast for "everything". I am glad to see you have accepted this. -- Steven
On Jul 16, 2013 Steven D'Aprano wrote:
Right now, sum() behaves in a certain way. There are certain things which people expect sum() to do. Some of those things are documented explicitly. Some of them are implied. Some of them have regression tests. Some of them don't. But regardless, we can tell how sum() behaves right now by running it and seeing what it does.
Your suggested optimizations change that behaviour. It does not just speed sum() up, they lead to an actual semantic change. So we are not just arguing about speed, we are arguing about behaviour as well.
All my suggestions produce NO behaviour change. If they lead to some semantic changes — it's a bug, that should be fixed, or the patch should be removed from suggestions list.
You are worried about sum() being slow for people who call it with list arguments. That is a valid concern. Nobody here *wants* sum() to be slow. If it was a pure speed optimization, then we would all be 100% behind it. But it is not a pure speed optimization, it also changes behaviour, sometimes in subtle, hard to see ways.
So there are three approaches we can take:
- Do nothing. sum() continues to work exactly the same way as it currently works, even if that means sometimes it is slow.
- Accept your patches. sum() changes its behaviour, which will break somebody's working code, but it will be fast, at least for some objects.
If you talk about "+=" patch then I already removed it from my suggestion list exactly because it changed the behaviour. Others should not change sum() in any way except performance.
- Accept a compromise position. We can make sum() faster for built-in lists, and maybe built-in tuples, while keeping the same behaviour. Everything else, including subclasses of list and tuple, keep the same behaviour, which may mean it remains slow.
That would probably cover most of use-cases, at least most of those I could find, but yes, it does not help other types and subclasses. (are there some known use cases of tuple subclasses additions?)
They are the only choices.
No, there're lots of others! That's what I'm trying to do here! I'm searching for the best choice to make sum performance consistent. If while doing that we'll also improve overall python performance, reduce its memory usage or find a "more obvious" way for sequences concatenation — great!
You are concerned more about sum() being slow than you are about breaking code that today works. Some of us here disagree, and think that breaking code is worse than slow code, especially for something as uncommon as sum(list_of_lists).
Then I agree with some of us. :) Because I believe that backward compatibility is more important, than speed. That's why I'm mainly focused on those options, that do not break it.
But, a compromise patch that speeds up some code without breaking any code may be acceptable.
Yes, but which one? * Special case for lists and tuples is easy to do and breaks nothing, but does nothing good for subclasses and other types. * Direct optimization of lists/tuples looks like a great idea, works for subclasses, should also change nothing, but it's a large patch, that is harder to write and test compared to a special case one. * Universal concatenation interface (which one? there were at least 3 suggested) looks interesting, but needs lots of additional polishing.
Same applies to sum(): even if it's impossible to make if fast for all collection types, it does not mean that it should not be fast for some of them, e.g. lists and tuples.
That is change from your previous posts where you said you could make it fast for "everything". I am glad to see you have accepted this.
I have not really changed my mind about that. :) I still think that for every real-world type you name I can probably find a way to make it O(N) summable with sum() patch or without it. But of course, I do not expect that any of my suggestions implemented will instantly make all the types in the world O(N) summable. It may help them become O(N) however.
I do not like that implementation, because it shares the underlying storage. This means that tuples which ought to be small will grow and grow and grow just because you have called __add__ on a different tuple.
Using Python 2.7 and your implementation above:
py> a = ft([]) # empty tuple py> len(a._store) 0 py> b = ft([1]) py> c = a + b py> d = ft([2]*10000) py> c = c + d py> len(a._store) 10001
So adding a big tuple to c changes the internal storage of a.
Yes, that's right. All 3 variables `a`, `b` and `c` share the same storage, so you effectively get 3 variables for the price of one. :) That's the concept. Why is that bad? --
On 7/17/2013 10:28 AM, Sergey wrote:
On Jul 16, 2013 Steven D'Aprano wrote:
Right now, sum() behaves in a certain way. There are certain things which people expect sum() to do. Some of those things are documented explicitly. Some of them are implied. Some of them have regression tests. Some of them don't. But regardless, we can tell how sum() behaves right now by running it and seeing what it does.
Your suggested optimizations change that behaviour. It does not just speed sum() up, they lead to an actual semantic change. So we are not just arguing about speed, we are arguing about behaviour as well.
All my suggestions produce NO behaviour change. If they lead to some semantic changes — it's a bug, that should be fixed, or the patch should be removed from suggestions list.
You are worried about sum() being slow for people who call it with list arguments. That is a valid concern. Nobody here *wants* sum() to be slow. If it was a pure speed optimization, then we would all be 100% behind it. But it is not a pure speed optimization, it also changes behaviour, sometimes in subtle, hard to see ways.
So there are three approaches we can take:
- Do nothing. sum() continues to work exactly the same way as it currently works, even if that means sometimes it is slow.
- Accept your patches. sum() changes its behaviour, which will break somebody's working code, but it will be fast, at least for some objects.
If you talk about "+=" patch then I already removed it from my suggestion list exactly because it changed the behaviour. Others should not change sum() in any way except performance.
- Accept a compromise position. We can make sum() faster for built-in lists, and maybe built-in tuples, while keeping the same behaviour. Everything else, including subclasses of list and tuple, keep the same behaviour, which may mean it remains slow.
That would probably cover most of use-cases, at least most of those I could find, but yes, it does not help other types and subclasses. (are there some known use cases of tuple subclasses additions?)
They are the only choices.
No, there're lots of others! That's what I'm trying to do here! I'm searching for the best choice to make sum performance consistent.
If while doing that we'll also improve overall python performance, reduce its memory usage or find a "more obvious" way for sequences concatenation — great!
You are concerned more about sum() being slow than you are about breaking code that today works. Some of us here disagree, and think that breaking code is worse than slow code, especially for something as uncommon as sum(list_of_lists).
Then I agree with some of us. :) Because I believe that backward compatibility is more important, than speed. That's why I'm mainly focused on those options, that do not break it.
But, a compromise patch that speeds up some code without breaking any code may be acceptable.
Yes, but which one? * Special case for lists and tuples is easy to do and breaks nothing, but does nothing good for subclasses and other types. * Direct optimization of lists/tuples looks like a great idea, works for subclasses, should also change nothing, but it's a large patch, that is harder to write and test compared to a special case one. * Universal concatenation interface (which one? there were at least 3 suggested) looks interesting, but needs lots of additional polishing.
Same applies to sum(): even if it's impossible to make if fast for all collection types, it does not mean that it should not be fast for some of them, e.g. lists and tuples.
That is change from your previous posts where you said you could make it fast for "everything". I am glad to see you have accepted this.
I have not really changed my mind about that. :) I still think that for every real-world type you name I can probably find a way to make it O(N) summable with sum() patch or without it.
But of course, I do not expect that any of my suggestions implemented will instantly make all the types in the world O(N) summable. It may help them become O(N) however.
I do not like that implementation, because it shares the underlying storage. This means that tuples which ought to be small will grow and grow and grow just because you have called __add__ on a different tuple.
Using Python 2.7 and your implementation above:
py> a = ft([]) # empty tuple py> len(a._store) 0 py> b = ft([1]) py> c = a + b py> d = ft([2]*10000) py> c = c + d py> len(a._store) 10001
So adding a big tuple to c changes the internal storage of a.
Yes, that's right. All 3 variables `a`, `b` and `c` share the same storage, so you effectively get 3 variables for the price of one. :) That's the concept. Why is that bad?
What happens to len(a._store) after del c? -- Terry Jan Reedy
On Jul 18, 2013 Terry Reedy wrote:
I do not like that implementation, because it shares the underlying storage. This means that tuples which ought to be small will grow and grow and grow just because you have called __add__ on a different tuple.
Using Python 2.7 and your implementation above:
py> a = ft([]) # empty tuple py> len(a._store) 0 py> b = ft([1]) py> c = a + b py> d = ft([2]*10000) py> c = c + d py> len(a._store) 10001
So adding a big tuple to c changes the internal storage of a.
Yes, that's right. All 3 variables `a`, `b` and `c` share the same storage, so you effectively get 3 variables for the price of one. :) That's the concept. Why is that bad?
What happens to len(a._store) after del c?
In that proof-of-concept implementation? Nothing. I tried to keep it simple, so that the idea was easier to understand. Its technically possible to have __del__ resizing internal store, but is it really needed? --
On Tue, Jul 16, 2013 at 08:36:05AM +0300, Sergey wrote:
You also said that I can't make tuples __add__ faster without patching sum() so explained how you can do that (you never answered whether you like such a patch and whether you would agree to write it). I even wrote a simple fasttuple proof of concept [1].
I do not like that implementation, because it shares the underlying storage. This means that tuples which ought to be small will grow and grow and grow just because you have called __add__ on a different tuple. Using Python 2.7 and your implementation above: py> a = ft([]) # empty tuple py> len(a._store) 0 py> b = ft([1]) py> c = a + b py> d = ft([2]*10000) py> c = c + d py> len(a._store) 10001 So adding a big tuple to c changes the internal storage of a. -- Steven
On 09/07/13 06:22, Sergey wrote:
On Jul 5, 2013 Andrew Barnert wrote:
Yes, you can! You just need to implement a fast __iadd__ or __add__ for your type to make it O(n) summable.
Well, in Python 3.3, or even 2.3, you just need to implement a fast __add__. So, if that's an acceptable answer, then we don't need to do anything. [...]
And you can always do that, can't you?
No. You can't implement a fast __iadd__ for tuple.
Well... Yes, I can! I can't make __iadd__ faster, because tuple has no __iadd__, however I can make a faster __add__.
And how do you expect to do that? Tuples are immutable, you have to create a new tuple. So when adding a sequence of N tuples together, you end up making and throwing away N-1 intermediate results.
But as long as sum() is the only (?) function suffering from this problem it was easier to do that optimization in sum() instead.
That's the big question though. Is summing a sequence of tuples important and common enough to justify special-casing it in sum? Just how many special cases can be justified?
If that's going to be extendable to other types
Are there any other types (except strings, that are blocked anyway)?
Looks like tuple is the only built-in type having no fast __iadd__,
I don't think so: py> for type in (str, bytes, bytearray, tuple, frozenset, object): ... print(type.__name__, hasattr(type, '__iadd__')) ... str False bytes False bytearray True tuple False frozenset False object False Okay, you can't sum() frozensets at all, but there's at least three types that support + that don't support __iadd__ (str, bytes, tuple), and by default anything inheriting from object will not have __iadd__ either.
and sum() is the only function suffering from that. So, to make sum() "fast for everything by default" we only need to make sum use __iadd__ and write a special case for sum+tuples.
To make advantage of faster __iadd__ implementation sum() uses it if possible. sum() has linear time complexity for built-in types and all types providing constant-time `__iadd__` or `__add__`.
Note that integer addition isn't actually constant, it's linear on the word size of the integers. Which means sum isn't linear for integers.
I don't think that strictly matters, since the N in O(N) we're talking about is the number of ints being added, not the number of bits in each int. What is true is that adding a pair of ints is only constant-time if they are sufficiently small (and maybe not even then). -- Steven
On Jul 9, 2013 Steven D'Aprano write:
Well... Yes, I can! I can't make __iadd__ faster, because tuple has no __iadd__, however I can make a faster __add__.
And how do you expect to do that? Tuples are immutable, you have to create a new tuple. So when adding a sequence of N tuples together, you end up making and throwing away N-1 intermediate results.
For example, rewrite tuple to internally store its values in a list, and have `localcount` variable saying how many items from that list belong to this tuple. Then __add__ could extend that list and reuse it for new tuple. But this patch looks too complex for me, implementing such a list storage directly in sum was much easier.
But as long as sum() is the only (?) function suffering from this problem it was easier to do that optimization in sum() instead.
That's the big question though. Is summing a sequence of tuples important and common enough to justify special-casing it in sum? Just how many special cases can be justified?
Personally, I think this use-case is not too important. But look at it from another perspective: is this special case is such a bad trade-off to say in manuals: ... sum() is fast for all built-in types. ...
Looks like tuple is the only built-in type having no fast __iadd__,
I don't think so: [...] Okay, you can't sum() frozensets at all, but there's at least three types that support + that don't support __iadd__ (str, bytes, tuple), and by default anything inheriting from object will not have __iadd__ either.
Note, that str and bytes are rejected by sum, so again we end up with just one type: tuple. And things inheriting from object will probably have fast enough __add__ anyway. :)
On 9 Jul, 2013, at 15:54, Sergey <sergemp@mail.ru> wrote:
On Jul 9, 2013 Steven D'Aprano write:
Well... Yes, I can! I can't make __iadd__ faster, because tuple has no __iadd__, however I can make a faster __add__.
And how do you expect to do that? Tuples are immutable, you have to create a new tuple. So when adding a sequence of N tuples together, you end up making and throwing away N-1 intermediate results.
For example, rewrite tuple to internally store its values in a list, and have `localcount` variable saying how many items from that list belong to this tuple. Then __add__ could extend that list and reuse it for new tuple.
That's not going to happen, not only breaks that backward compatibility for users for the C API, it has nasty side effects and is incorrect. Nasty side effect: a = (1,) b = (2,) * 1000 c = a + b del b del c With the internal list 'a' keeps alive the extra storage used for 'c'. Incorrect: a = (1,) b = a + (2,) c = a + (3,) Now 'b' and 'c' can't possibly both share storage with 'a'. Ronald
On Jul 9, 2013 Ronald Oussoren wrote:
For example, rewrite tuple to internally store its values in a list, and have `localcount` variable saying how many items from that list belong to this tuple. Then __add__ could extend that list and reuse it for new tuple.
That's not going to happen, not only breaks that backward compatibility for users for the C API,
It depends on implementation details, it's possible to keep it backward compatible. BTW, what C API do you expect to break?
it has nasty side effects and is incorrect. Nasty side effect: a = (1,) b = (2,) * 1000 c = a + b del b del c With the internal list 'a' keeps alive the extra storage used for 'c'.
Yes, technically it's possible to implement tuple so that it would realloc internal list to save some ram, but why? List does not do that when you remove elements from it, why should tuple do that? On the other hand: a = (1,) * 1000 b = a + (2,3) c = b + (4,5) And you have 3 variables for the price of one. Lot's of memory saved!
Incorrect: a = (1,) b = a + (2,) c = a + (3,) Now 'b' and 'c' can't possibly both share storage with 'a'.
Nothing incorrect here, of course __add__ should handle that, and if it cannot reuse list it would copy it. As it does now. Such tuple would never allocate more RAM than before, often it should use either same RAM or less. In some rare cases it may not free some RAM, that it could free before. But I'm not sure they worth the effort fixing. PS: I don't think anybody wants to see that implementation anyway, I guess it was just an attempt to bug me with "you can't". ——
On 11 Jul, 2013, at 0:47, Sergey <sergemp@mail.ru> wrote:
On Jul 9, 2013 Ronald Oussoren wrote:
For example, rewrite tuple to internally store its values in a list, and have `localcount` variable saying how many items from that list belong to this tuple. Then __add__ could extend that list and reuse it for new tuple.
That's not going to happen, not only breaks that backward compatibility for users for the C API,
It depends on implementation details, it's possible to keep it backward compatible. BTW, what C API do you expect to break?
If a tuple stores the values in a separate list the structure of a PyTupleObject changes. That structure is exposed to users of the C API, and hence changes shouldn't be made lightly.
it has nasty side effects and is incorrect. Nasty side effect: a = (1,) b = (2,) * 1000 c = a + b del b del c With the internal list 'a' keeps alive the extra storage used for 'c'.
Yes, technically it's possible to implement tuple so that it would realloc internal list to save some ram, but why? List does not do that when you remove elements from it, why should tuple do that?
Actually, list does resize when you remove items but only does so when the amount of free items gets too large (see list_resize in Object/listobject.c) Keeping memory blocks alive unnecessarily is bad because this can increase the amount of memory used by a script, without their being a clear reason for it when you inspect the python code. This has been one of the reasons for not making string slicing operations views on the entire string (that is, a method like aStr.partition could return objects that reference aStr for the character storage which could save memory but at the significant risk of keeping too much memory alive when aStr is discarded earlier than one of the return values)
On the other hand: a = (1,) * 1000 b = a + (2,3) c = b + (4,5) And you have 3 variables for the price of one. Lot's of memory saved!
How do you do that? As for as I know Python isn't a quantum system, item 1000 of the hidden storage list can't be both 2 and 4 at the same time.
Incorrect: a = (1,) b = a + (2,) c = a + (3,) Now 'b' and 'c' can't possibly both share storage with 'a'.
Nothing incorrect here, of course __add__ should handle that, and if it cannot reuse list it would copy it. As it does now.
And then you introduce unpredictable performance for tuple addition, currently you can reason about the performance of code that does tuple addition and with this change you no longer can (it sometimes is fast, and sometimes it is slow). That's a major gotcha that will cause confusion (string addition also has this feature, and that optimization wouldn't have gotten in with the knowlegde we now have). Anyways, I'm still +0 using += in sum, and -1 on trying to special case particular types. Ronald
On Jul 11, 2013 Ronald Oussoren wrote:
It depends on implementation details, it's possible to keep it backward compatible. BTW, what C API do you expect to break?
If a tuple stores the values in a separate list the structure of a PyTupleObject changes. That structure is exposed to users of the C API, and hence changes shouldn't be made lightly.
That would effectively add one more: void *_internal_list; field, which is backward compatible. (On the other hand it may be a good idea to break API compatibility from time to time to make sure that it's not badly misused. :)
Yes, technically it's possible to implement tuple so that it would realloc internal list to save some ram, but why? List does not do that when you remove elements from it, why should tuple do that?
Actually, list does resize when you remove items but only does so when the amount of free items gets too large (see list_resize in Object/listobject.c)
Agree. It indeed tries to reallocate lists sometimes. I was fooled because I tried: a=[i for i in xrange(50000000)] while len(a) > 10: x = a.pop() and it only released the memory when I closed the interpreter. (this brings a weak question of whether it makes sense to reallocate if memory is not released to the system)
Keeping memory blocks alive unnecessarily is bad because this can increase the amount of memory used by a script, without their being a clear reason for it when you inspect the python code. This has been one of the reasons for not making string slicing operations views on the entire string (that is, a method like aStr.partition could return objects that reference aStr for the character storage which could save memory but at the significant risk of keeping too much memory alive when aStr is discarded earlier than one of the return values)
Well, depending on the actual implementation this can be fixed. E.g. as long as we optimize things for the case of 2 objects using same list, the list could store both sizes and reallocate if they both are much smaller than allocated size. Or it can store all sizes and reallocate when all of them are small enough. Or it can use some lazy reallocation, just marking the list and allowing other alive objects to reallocate when they next access this list. Lots of options. :)
On the other hand: a = (1,) * 1000 b = a + (2,3) c = b + (4,5) And you have 3 variables for the price of one. Lot's of memory saved!
How do you do that? As for as I know Python isn't a quantum system, item 1000 of the hidden storage list can't be both 2 and 4 at the same time.
c==(1, ... 1,2,3,4,5). No need to be 2 and 4 at the same time. :)
Nothing incorrect here, of course __add__ should handle that, and if it cannot reuse list it would copy it. As it does now.
And then you introduce unpredictable performance for tuple addition, currently you can reason about the performance of code that does tuple addition and with this change you no longer can (it sometimes is fast, and sometimes it is slow).
Don't understand. Now it's always slow and unpredictable (because it depends on the current heap state, allocation map of system RAM, etc). You cannot predict anything right now. It is as much unpredictable as adding item to a list (it may or may not need to realloc), or as almost every other operation in python. It may become faster in average, but will still be as much unpredictable as it is now. E.g. you have no way to predict that the memory page you're trying to read was swapped out by the system and won't come up in next few seconds because disk is under a heavy load.
Anyways, I'm still +0 using += in sum, and -1 on trying to special case particular types.
This is not about special casing. What we discuss now is some sort of copy-on-write optimisation for python. Copy-on-write is known to save memory, it's widely used in many places (e.g. by Linux kernel when managing apps memory pages). Is it a bad thing to have such a general and well known optimisation technique in python? This approach has other benefits. For example lists and tuples could share the allocation code, that would not only make lists faster as well, but will allow instant conversion of lists to tuples and back. What I'm afraid of is that despite making many things faster I may face the argument "this also make sum() of lists faster, and this should not happen, sum() MUST be slow because this is how it must be". So after all the sum discussions I don't have much desire to even start working on implementation. ——
On 12 July 2013 15:03, Sergey <sergemp@mail.ru> wrote:
What I'm afraid of is that despite making many things faster I may face the argument "this also make sum() of lists faster, and this should not happen, sum() MUST be slow because this is how it must be". So after all the sum discussions I don't have much desire to even start working on implementation.
If this is all you have gotten from our arguments, then there was no point talking to you. You seem to be methodologically ignoring us and trivialising our arguments.
On 4 July 2013 23:25, Andrew Barnert <abarnert@yahoo.com> wrote:
If people really do need to concatenate a bunch of tuples this often, maybe the answer is to move chain to builtins. If it's not important enough to move chain to builtins, it's probably not important enough to do anything else.
Can I suggest that if chain is ever moved to builtins it should actually be chain.from_iterable that gets moved and renamed to chain (or concat or something). It has always seemed to me that chain.from_iterable is the more appropriate function and I assume that it only has second class status because it was initially overlooked. Oscar
On 04/07/13 19:54, Sergey wrote:
On Jul 04, 2013 Steven D'Aprano wrote:
This message is long, so here's its short summary: * Unfortunately list.extend does not look like the obvious way, and its slower than alternatives.
[snip] I'm not going to get into a long and tedious point-by-point argument about this, but I will just re-iterate: I don't accept that using sum on non-numbers ought to be encouraged, although I don't think it should be prohibited either. So I'm at best neutral on speeding up sum for lists. However, some things do need to be said: 1) Alex Martelli is, or at least was, against using sum on non-numbers: [quote] I was responsible for designing sum and doing its first implementation in the Python runtime, and I still wish I had found a way to effectively restrict it to summing numbers (what it's really good at) and block the "attractive nuisance" it offers to people who want to "sum" lists;-). – Alex Martelli Jun 4 '09 at 21:07 [end quote] This quote is from the Stackoverflow question you linked to earlier: http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-... 2) You should not be benchmarking against 2.7, but 3.3. On my machine, using 3.3, list.extend is just as fast as using itertools, and both are faster than a list comprehension. (I have not tested your patched version of sum.) Augmented assignment is faster, but faster still is an optimized version using extend that avoids a name look-up: The tests I ran: python3.3 -mtimeit --setup="x=[[1,2,3]]*100000" "[i for l in x for i in l]" python3.3 -mtimeit --setup="x=[[1,2,3]]*100000" \ --setup="from itertools import chain" \ "list(chain.from_iterable(x))" python3.3 -mtimeit --setup="x=[[1,2,3]]*100000" \ --setup="from itertools import chain" \ "list(chain(*x))" python3.3 -mtimeit --setup="x=[[1,2,3]]*100000" --setup="l = []" \ "for i in x: l += i" python3.3 -mtimeit --setup="x=[[1,2,3]]*100000" --setup="l = []" \ "for i in x: l.extend(i)" python3.3 -mtimeit --setup="x=[[1,2,3]]*100000" --setup="l = []" \ --setup="extend=l.extend" \ "for i in x: extend(i)" with results, from slowest to fastest: list comp: 32.8 msec per loop chain: 21 msec per loop extend: 20.4 msec per loop chain_from_iterable: 19.8 msec per loop augmented assignment: 12.8 msec per loop optimized extend: 11.7 msec per loop On your machine, results may differ. -- Steven
From: Steven D'Aprano <steve@pearwood.info> Sent: Thursday, July 4, 2013 9:06 PM
1) Alex Martelli is, or at least was, against using sum on non-numbers:
[quote] I was responsible for designing sum and doing its first implementation in the Python runtime, and I still wish I had found a way to effectively restrict it to summing numbers (what it's really good at) and block the "attractive nuisance" it offers to people who want to "sum" lists;-). – Alex Martelli Jun 4 '09 at 21:07 [end quote]
I don't know if I agree with Alex Martelli that this would be a good idea, but… if it is, there's a really easy way to do it. Just check the start parameter (if a non-default value is passed). After all, if the start value is numeric or not a sequence or whatever the rule is, presumably anything that can be added to it is also numeric enough to be summable. (If you really want to sneak in a start value that passes as numeric but knows how to add itself to a tuple and return a tuple, go for it.) We already have three checks for the builtin string types (Python/bltinmodule.c:1950); adding a PySequence_Check(start) or PyNumber_Add(zero, start) or the C equivalent of isinstance(start, numbers.Number) or whatever (the details are of course bikesheddable) wouldn't hurt performance. (In fact, because the three string checks only need to be done if the one is-numeric or is-not-sequence or whatever fails, it might even speed things up—but either way, not enough to matter.)
On 07/03/2013 07:43 AM, Steven D'Aprano wrote:
I'm not sure that sum() is the Obvious Way to concatenate lists, and I don't think that concatenating many lists is a common thing to do. Traditionally, sum() works only on numbers, and I think we wouldn't be having this discussion if Python used & for concatenation instead of +. So I don't care that sum() has quadratic performance on lists (and tuples), and I must admit that having a simple quadratic algorithm in the built-ins is sometimes useful for teaching purposes, so I'm -0 on optimizing this case.
I agree, and wished sequences used a __join__ method instead of __add__. The '&' is already used for Bitwise And. How about '++' instead? 'hello ' ++ 'world' == "hello world" [1, 3, 3] ++ [4, 5, 6] == [1, 2, 3, 4, 5, 6] While this doesn't seem like a big change, I think it would simplify code in many places that is more complicated than it really needs to be. Cheers, Ron
05.07.13 19:22, Ron Adam написав(ла):
The '&' is already used for Bitwise And. How about '++' instead?
'hello ' ++ 'world' == "hello world"
[1, 3, 3] ++ [4, 5, 6] == [1, 2, 3, 4, 5, 6]
While this doesn't seem like a big change, I think it would simplify code in many places that is more complicated than it really needs to be.
The '++' "operator" already defined for numbers.
123 ++ 456 579
On 07/05/2013 11:22 AM, Ron Adam wrote:
The '&' is already used for Bitwise And. How about '++' instead?
'hello ' ++ 'world' == "hello world"
[1, 3, 3] ++ [4, 5, 6] == [1, 2, 3, 4, 5, 6]
While this doesn't seem like a big change, I think it would simplify code in many places that is more complicated than it really needs to be.
Hmm... Nope.. '++' is valded for numbers. <shrug> Ron
On 06/07/13 02:22, Ron Adam wrote:
On 07/03/2013 07:43 AM, Steven D'Aprano wrote:
I'm not sure that sum() is the Obvious Way to concatenate lists, and I don't think that concatenating many lists is a common thing to do. Traditionally, sum() works only on numbers, and I think we wouldn't be having this discussion if Python used & for concatenation instead of +. So I don't care that sum() has quadratic performance on lists (and tuples), and I must admit that having a simple quadratic algorithm in the built-ins is sometimes useful for teaching purposes, so I'm -0 on optimizing this case.
I agree, and wished sequences used a __join__ method instead of __add__.
The '&' is already used for Bitwise And. How about '++' instead?
Nope, because it is ambiguous. Given x++y, is that a ++ binary operator, or a binary + operator followed by unary + operator? Besides, for backwards compatibility, changing the operator from + cannot happen now until Python 4000, if ever.
'hello ' ++ 'world' == "hello world"
[1, 3, 3] ++ [4, 5, 6] == [1, 2, 3, 4, 5, 6]
While this doesn't seem like a big change, I think it would simplify code in many places that is more complicated than it really needs to be.
I don't think that merely changing an operator from one token + to a very slightly different token ++ doesn't "simplify code" in any way. I suggested that & was a better token for concatenation because it avoids associating/conflating concatenation with addition, but there's no difference in code complexity whether you write x+y, x&y or x++y. -- Steven
On 07/05/2013 11:38 AM, Steven D'Aprano wrote:
On 06/07/13 02:22, Ron Adam wrote:
On 07/03/2013 07:43 AM, Steven D'Aprano wrote:
I'm not sure that sum() is the Obvious Way to concatenate lists, and I don't think that concatenating many lists is a common thing to do. Traditionally, sum() works only on numbers, and I think we wouldn't be having this discussion if Python used & for concatenation instead of +. So I don't care that sum() has quadratic performance on lists (and tuples), and I must admit that having a simple quadratic algorithm in the built-ins is sometimes useful for teaching purposes, so I'm -0 on optimizing this case.
I agree, and wished sequences used a __join__ method instead of __add__.
The '&' is already used for Bitwise And. How about '++' instead?
Nope, because it is ambiguous. Given x++y, is that a ++ binary operator, or a binary + operator followed by unary + operator?
Yes, I realised that soon after I posted.
Besides, for backwards compatibility, changing the operator from + cannot happen now until Python 4000, if ever.
Right, but if a good solution comes before then,it can be used when the time comes.
'hello ' ++ 'world' == "hello world"
[1, 3, 3] ++ [4, 5, 6] == [1, 2, 3, 4, 5, 6]
While this doesn't seem like a big change, I think it would simplify code in many places that is more complicated than it really needs to be.
I don't think that merely changing an operator from one token + to a very slightly different token ++ doesn't "simplify code" in any way. I suggested that & was a better token for concatenation because it avoids associating/conflating concatenation with addition, but there's no difference in code complexity whether you write x+y, x&y or x++y.
It's not the expressions that are improved, but the code surrounding them. Like I said it may not seem like a big change, but I think it would be an important change. Cheers, Ron
On 7/5/2013 12:22 PM, Ron Adam wrote:
On 07/03/2013 07:43 AM, Steven D'Aprano wrote:
I'm not sure that sum() is the Obvious Way to concatenate lists, and I don't think that concatenating many lists is a common thing to do.
Agree here.
Traditionally, sum() works only on numbers, and I think we wouldn't be having this discussion if Python used & for concatenation instead of +.
Since addition of numbers represented in base-1 or tally notation is the same as sequence concatenation, I think '+' is the right choice. In my experience of American English, normal people 'add' lists (though perhaps as sets, with duplicate removal), not 'concatenate' them. And while they may 'add' together multiple lists, they would not likely 'sum' them unless they are lists of numbers. When Alex said that it was not possible to determine if the start value is a number, he was talking in the context of old style classes where the type of every user class was 'Class' and the type of every user instance was 'Instance' (or something like that). In Python 3, with ABCs, isinstance(start, Number) would solve the problem as long as the requirement were documented. -- Terry Jan Reedy
Terry Reedy writes:
In my experience of American English, normal people 'add' lists
Nitpick: That somehow doesn't sit right with me. They "put lists together" or "include one in another" (assign to slice, not list as element of superior list). Adding refers to elements: "add item to list", although it's often applied iteratively (being synonymous to "include in").. I don't think it's appropriate to refer to any specific English usage here. Most native speakers, and especially programmers, have flexibility in the direction of allowing the generalization of "+" to strings to imply that "sum" also generalizes to strings, as the iterated and associative application of "+". They're even more flexible than that: they intuitively understand that although "+" (and perhaps even more so) usually imply "commutative" (as in the convention in abstract algebra), when applied to strings and other sequences this doesn't make sense (as of course it doesn't make sense for infinite sequences in analysis). I think that as far as consistency with intuition goes, it doesn't matter which we choose: we really need to ask the ultimate authority on "Pythonicity". My own preference is based on "consenting adults": if "sum(list_of_strings)" bothers you, don't use it. But I don't understand the ramifications of thoroughly implementing that in a community where intuition splits as deeply as it evidently does here. Steve
On 07/05/2013 08:53 PM, Stephen J. Turnbull wrote:
Terry Reedy writes:
In my experience of American English, normal people 'add' lists
Nitpick: That somehow doesn't sit right with me. They "put lists together" or "include one in another" (assign to slice, not list as element of superior list). Adding refers to elements: "add item to list", although it's often applied iteratively (being synonymous to "include in")..
We combine lists, and add ingredients. Or do we combine ingredients and add lists? ;-)
I don't think it's appropriate to refer to any specific English usage here.
It's one of those things that if we can find a relationship that works, great, but it just won't always work out that well.
Most native speakers, and especially programmers, have flexibility in the direction of allowing the generalization of "+" to strings to imply that "sum" also generalizes to strings, as the iterated and associative application of "+". They're even more flexible than that: they intuitively understand that although "+" (and perhaps even more so) usually imply "commutative" (as in the convention in abstract algebra), when applied to strings and other sequences this doesn't make sense (as of course it doesn't make sense for infinite sequences in analysis).
I think that as far as consistency with intuition goes, it doesn't matter which we choose: we really need to ask the ultimate authority on "Pythonicity". My own preference is based on "consenting adults": if "sum(list_of_strings)" bothers you, don't use it. But I don't understand the ramifications of thoroughly implementing that in a community where intuition splits as deeply as it evidently does here.
It helps to view these thing from two sides. On the human side, we want something that is somewhat familiar, but on the CPU side, we need to be fairly specific. Humans tend to overgeneralise, and over simplify, but we also are able to infer more specific information by including lots of extraneous information. Python is very limited in that respect. On one hand it sometimes is nice to be able to write very general routines that don't care what the input is, or will work with a wide range of input. But if the routine is too generalized, it can do the wrong thing at times. Like iterating letters in a string in a list. So having the language organised in meaningful ways can help. The question is, is there something we can change that will help with things like this? The behaviour I would like is... (only some examples.) add and subtract scaler objects (and other scaler operations.) combine and separate immutable collections (strings, tuples) Join and split mutable collections (lists, dictionaries, sets) We already get some nice error messages when try to do things that don't make sense, but sometimes, we get a wrong result instead. Cheers, Ron
On 06/07/13 06:25, Terry Reedy wrote:
When Alex said that it was not possible to determine if the start value is a number, he was talking in the context of old style classes where the type of every user class was 'Class' and the type of every user instance was 'Instance' (or something like that). In Python 3, with ABCs, isinstance(start, Number) would solve the problem as long as the requirement were documented.
For the record, it has always been possible to check if something is a number: try: x + 0 except TypeError: print "x is not a number" -- Steven
On Jul 5, 2013, at 23:32, Steven D'Aprano <steve@pearwood.info> wrote:
On 06/07/13 06:25, Terry Reedy wrote:
When Alex said that it was not possible to determine if the start value is a number, he was talking in the context of old style classes where the type of every user class was 'Class' and the type of every user instance was 'Instance' (or something like that). In Python 3, with ABCs, isinstance(start, Number) would solve the problem as long as the requirement were documented.
For the record, it has always been possible to check if something is a number:
try: x + 0 except TypeError: print "x is not a number"
This isn't a very good rule for "is a number". You can add 0 to numpy arrays, for example, and they're not numbers. But I think it is actually a good rule for "is summable". If you've got something that's not a number, but 0+x makes sense, summing probably also makes sense. Conversely, if you create some type that is numeric, but isn't addable to 0, you wouldn't be surprised if you couldn't sum it.
On 6 July 2013 08:07, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 5, 2013, at 23:32, Steven D'Aprano <steve@pearwood.info> wrote:
On 06/07/13 06:25, Terry Reedy wrote:
When Alex said that it was not possible to determine if the start value is a number, he was talking in the context of old style classes where the type of every user class was 'Class' and the type of every user instance was 'Instance' (or something like that). In Python 3, with ABCs, isinstance(start, Number) would solve the problem as long as the requirement were documented.
For the record, it has always been possible to check if something is a number:
try: x + 0 except TypeError: print "x is not a number"
This isn't a very good rule for "is a number". You can add 0 to numpy arrays, for example, and they're not numbers.
But I think it is actually a good rule for "is summable". If you've got something that's not a number, but 0+x makes sense, summing probably also makes sense. Conversely, if you create some type that is numeric, but isn't addable to 0, you wouldn't be surprised if you couldn't sum it.
Hyelll nooo (imagine I said that with distortedly high pitch while wearing a hat). What about summing vectors? You can't tell me that doesn't make sense. Why on earth would you need to implement +0 for vectors? What about summing counters? I don't like your "dogma" about summables. It's summable *if I want to sum it, and it makes sense*. So stop trying to stop me. *Runs away crying*
On Jul 6, 2013, at 11:54, Joshua Landau <joshua.landau.ws@gmail.com> wrote:
On 6 July 2013 08:07, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 5, 2013, at 23:32, Steven D'Aprano <steve@pearwood.info> wrote:
On 06/07/13 06:25, Terry Reedy wrote:
When Alex said that it was not possible to determine if the start value is a number, he was talking in the context of old style classes where the type of every user class was 'Class' and the type of every user instance was 'Instance' (or something like that). In Python 3, with ABCs, isinstance(start, Number) would solve the problem as long as the requirement were documented.
For the record, it has always been possible to check if something is a number:
try: x + 0 except TypeError: print "x is not a number"
This isn't a very good rule for "is a number". You can add 0 to numpy arrays, for example, and they're not numbers.
But I think it is actually a good rule for "is summable". If you've got something that's not a number, but 0+x makes sense, summing probably also makes sense. Conversely, if you create some type that is numeric, but isn't addable to 0, you wouldn't be surprised if you couldn't sum it.
Hyelll nooo (imagine I said that with distortedly high pitch while wearing a hat).
What about summing vectors? You can't tell me that doesn't make sense. Why on earth would you need to implement +0 for vectors?
Maybe because it makes perfect sense to treat 0 as a null vector? Or just because it comes for free (and does the right thing) if you're using numpy.array, or complex or quaternion, etc.?
What about summing counters?
You're replying to "if you create some type that is numeric, but isn't addable to 0, you wouldn't be surprised if you couldn't sum it." I can easily see considering a vector numeric, but a counter? Anyway, in not sure that I agree with Alex Martelli that sum should be restricted--but if they should, I think 0+x is a much better rule than isinstance(x, Number).
On 7 July 2013 01:30, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 6, 2013, at 11:54, Joshua Landau <joshua.landau.ws@gmail.com> wrote:
On 6 July 2013 08:07, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 5, 2013, at 23:32, Steven D'Aprano <steve@pearwood.info> wrote:
try: x + 0 except TypeError: print "x is not a number"
This isn't a very good rule for "is a number". You can add 0 to numpy arrays, for example, and they're not numbers.
But I think it is actually a good rule for "is summable". If you've got something that's not a number, but 0+x makes sense, summing probably also makes sense. Conversely, if you create some type that is numeric, but isn't addable to 0, you wouldn't be surprised if you couldn't sum it.
Hyelll nooo (imagine I said that with distortedly high pitch while wearing a hat).
What about summing vectors? You can't tell me that doesn't make sense. Why on earth would you need to implement +0 for vectors?
Maybe because it makes perfect sense to treat 0 as a null vector? Or just because it comes for free (and does the right thing) if you're using numpy.array, or complex or quaternion, etc.?
I said why would you *need* to, not why would you. You can if you want.
What about summing counters?
You're replying to "if you create some type that is numeric, but isn't addable to 0, you wouldn't be surprised if you couldn't sum it."
Well, also to:
But I think [being addable to 0] is actually a good rule for "is summable"
I can easily see considering a vector numeric, but a counter?
Anyway, in not sure that I agree with Alex Martelli that sum should be restricted--but if they should, I think 0+x is a much better rule than isinstance(x, Number).
Fair 'nuff. That wasn't clear from what you said, though.
Sergey, 02.07.2013 20:12:
sum() is a great function. It is the "obvious way" to add things. Unfortunately sometimes it's slower than it could be.
The problem is that code: sum([[1,2,3]]*1000000, []) takes forever to complete. Let's fix that!
No, please. Using sum() on lists is not more than a hack that seems to be a cool idea but isn't. Seriously - what's the sum of lists? Intuitively, it makes no sense at all to say sum(lists). What you want is the concatenation, not the sum. Please don't stuff something as simple as sum() with a nonsensical misuse case. Stefan
On Jul 5, 2013 Stefan Behnel wrote:
No, please. Using sum() on lists is not more than a hack that seems to be a cool idea but isn't. Seriously - what's the sum of lists? Intuitively, it makes no sense at all to say sum(lists).
It's the same as it is now. What else can you think about when you see: [1, 2, 3] + [4, 5] ? Seriously, why there's so much holy wars about that? I'm not asking to rewrite cpython on Java or C#. I'm not adding a bunch of new functions, I'm not even changing signatures of existing functions. It's just among hundreds of existing functions I took one and made it faster for some use-cases. That's all! It's just a minor optimization patch. If instead I optimized e.g. ConfigParser [1] then nobody would care. Then why so many people care about this one? -- [1] http://bugs.python.org/issue7113
On 07/09/2013 04:35 AM, Sergey wrote:
On Jul 5, 2013 Stefan Behnel wrote:
No, please. Using sum() on lists is not more than a hack that seems to be a cool idea but isn't. Seriously - what's the sum of lists? Intuitively, it makes no sense at all to say sum(lists). It's the same as it is now. What else can you think about when you see: [1, 2, 3] + [4, 5] ?
Seriously, why there's so much holy wars about that? I'm not asking to rewrite cpython on Java or C#. I'm not adding a bunch of new functions, I'm not even changing signatures of existing functions.
It's the nature of this particular news group. We focus on improving python, and that includes new things and improving old things, but also includes discussing any existing or potential problems. You will almost always get a mix of approval and disapproval on just about every thing here. It's not a war, it's just different people having different opinions. Quite often that leads to finding better ways to do things, and in the long run, helps avoid adding features and changes that could be counter productive to python.
It's just among hundreds of existing functions I took one and made it faster for some use-cases. That's all! It's just a minor optimization patch.
If it only makes an existing function faster and doesn't change any other behaviour, and all the tests still pass for it. Just create an issue on the tracker, with the patch posted there, and it will probably be accepted after it goes through a much more focused review process. But discussing it here will invite a lot of opinions about how it works, how it shouldn't work, what would work better, and etc... It's what this board if for. ;-) Cheers, Ron
If instead I optimized e.g. ConfigParser [1] then nobody would care. Then why so many people care about this one?
On Jul 9, 2013 Ron Adam wrote:
Seriously, why there's so much holy wars about that? I'm not asking to rewrite cpython on Java or C#. I'm not adding a bunch of new functions, I'm not even changing signatures of existing functions.
It's the nature of this particular news group. We focus on improving python, and that includes new things and improving old things, but also includes discussing any existing or potential problems.
You will almost always get a mix of approval and disapproval on just about every thing here. It's not a war, it's just different people having different opinions.
Quite often that leads to finding better ways to do things, and in the long run, helps avoid adding features and changes that could be counter productive to python.
I must agree that I was indeed inspired with some new ideas during this discussion. It's just that those "inspirations" come in a very non-constructive form of "it makes no sense", "cannot always be fast", "you can't", "everyone else thinks you shouldn't", etc. Or is that a lifehack [1] in action? I.e. "You can't make it fast for that type. Oh, you can? Then you can't make it fast for that type. Oh, you did that too? But you can't make it fast for all the types!" What if I can? ;) It's just instead of discussing what is the best way to fix a slowness, I'm spending most time trying to convince people that slowness should be fixed. — sum is slow for lists, let's fix that! — you shouldn't use sum... — why can't I use sum? — because it's slow — then let's fix that! — you shouldn't use sum... I haven't thought that somebody can truly believe that something should be slow, and will find one excuse after another instead of just fixing the slowness.
If it only makes an existing function faster and doesn't change any other behaviour, and all the tests still pass for it. Just create an issue on the tracker, with the patch posted there, and it will probably be accepted after it goes through a much more focused review process.
I've done that first [2] And there I was asked to write here. :)
But discussing it here will invite a lot of opinions about how it works, how it shouldn't work, what would work better, and etc...
And about what *I* shouldn't do, what *I* can't and what *I* need. As if I'm the bug that should be fixed. :(
It's what this board if for. ;-)
-- [1] http://bash.org/?152037 [2] http://bugs.python.org/issue18305
On Jul 10, 2013, at 14:58, Sergey <sergemp@mail.ru> wrote:
On Jul 9, 2013 Ron Adam wrote:
Or is that a lifehack [1] in action? I.e. "You can't make it fast for that type. Oh, you can? Then you can't make it fast for that type. Oh, you did that too? But you can't make it fast for all the types!" What if I can? ;)
But you haven't found a workable solution for all of the other types people have brought up--or even for a single one of them. So that's a serious misrepresentation.
It's just instead of discussing what is the best way to fix a slowness, I'm spending most time trying to convince people that slowness should be fixed. — sum is slow for lists, let's fix that! — you shouldn't use sum... — why can't I use sum? — because it's slow
But that's not why you shouldn't use sum. So, you're trying to answer people who (a) are wrong, and (b) aren't on this list instead of answering the people who are actually here. Besides, being fast for list and tuple but slow for other collection types would be an attractive nuisance. Your only response to that has been to claim that it can be fast for every possible collection type, but it can't. You haven't gotten it to work. And there are good reasons to believe it's not just hard, but impossible. So, if that's true, where does it leave you? You can't argue that sum is the obvious way to concatenate collections. You either have to say that it's the obvious way to concatenate only builtin collections, and something else should be used for everything else, or you have to argue that the benefits to novices who do the wrong thing with tuples outweighs the costs. Or, of course, accept that it's not a good idea.
— then let's fix that! — you shouldn't use sum... I haven't thought that somebody can truly believe that something should be slow, and will find one excuse after another instead of just fixing the slowness.
Calling recv(1) over and over on a socket is slow. We could fix that by adding an implicit buffer to all socket objects. Can you believe that someone might object to that patch?
If it only makes an existing function faster and doesn't change any other behaviour, and all the tests still pass for it. Just create an issue on the tracker, with the patch posted there, and it will probably be accepted after it goes through a much more focused review process.
I've done that first [2] And there I was asked to write here. :)
But discussing it here will invite a lot of opinions about how it works, how it shouldn't work, what would work better, and etc...
And about what *I* shouldn't do, what *I* can't and what *I* need. As if I'm the bug that should be fixed. :(
It's what this board if for. ;-)
-- [1] http://bash.org/?152037 [2] http://bugs.python.org/issue18305 _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Jul 10, 2013 Andrew Barnert wrote:
But you haven't found a workable solution for all of the other types people have brought up--or even for a single one of them. So that's a serious misrepresentation.
Strings, tuples (2 options), lists (3 options) and your cons-lists (2 options). What other types have I missed? Most of those has nothing to do with my suggestion, it's just that you (or not just you) asked how one can optimize XXX for O(N) sum, so I found some ways.
It's just instead of discussing what is the best way to fix a slowness, I'm spending most time trying to convince people that slowness should be fixed. — sum is slow for lists, let's fix that! — you shouldn't use sum... — why can't I use sum? — because it's slow
But that's not why you shouldn't use sum.
No, That's the MAIN reason! There's one more, about "+" (and sum) being confusing for concatenation, and I can understand that, however it's very weak as long "+" is used for joining sequences. But the main reason that most people, including you, are pushing is that sum() is slow. The whole our "you can't make it fast for" thread is the evidence. Even now:
Besides, being fast for list and tuple but slow for other collection types would be an attractive nuisance.
You're pushing "being slow" as a key argument. If due to some technical details sum() was fast from the very beginning then nobody would say that you shouldn't use it. Same as nobody is saying that you should use reduce() instead of sum() for numbers because sum() works only for __add__ while reduce works for every operator possible. People are free to use sum() to __add__ numbers, and use reduce() e.g. to __xor__ them. Nobody says "you shouldn't use sum() for numbers because there're operators that don't work with it". Nobody says that sum() is "attractive nuisance" compared to reduce(). For the same reason IF sum would be fast for list, but slow for set, people would use sum() where it's good and won't use it where it's bad. There's nothing unusual or wrong here. min() is good for list of tuples, right? max() is also good for list of tuples? Even any() is good for list of tuples (weird, but good). Then why sum() is bad for list of tuples? Because it's SLOW! So, yes, sum() being slow is the main reason. Everything else is just a "bonus", additional arguments that are supposed to convince your opponent even more. But speed is the key. That's why I'm saying so often: if this is the main reason then let's fix it.
Your only response to that has been to claim that it can be fast for every possible collection type, but it can't. You haven't gotten it to work. And there are good reasons to believe it's not just hard, but impossible.
Sorry, but can you please quote what response are you referencing?
I haven't thought that somebody can truly believe that something should be slow, and will find one excuse after another instead of just fixing the slowness.
Calling recv(1) over and over on a socket is slow. We could fix that by adding an implicit buffer to all socket objects. Can you believe that someone might object to that patch?
Not sure what you mean, but sockets do have an internal buffer, its default size is configurable with `sysctl`. PS: maybe for 10 years people got used to the slow sum() and related arguments so much that they don't want to lose them? ;) ——
From: Sergey <sergemp@mail.ru> Sent: Thursday, July 11, 2013 10:16 PM
On Jul 10, 2013 Andrew Barnert wrote:
But you haven't found a workable solution for all of the other types people have brought up--or even for a single one of them. So that's a serious misrepresentation.
Strings, tuples (2 options), lists (3 options) and your cons-lists (2 options). What other types have I missed?
Again, you have not actually made cons lists work; you'd made different types work (deques, C++-style double-linked lists, etc.), and argued that those are "better types" or something. And then there's the whole class of immutable collection types besides tuple. You've suggested that we can patch sum for each one, and that you could theoretically rewrite every immutable collection type in the world to be fast-summable, but neither of those is even remotely feasible as a solution. Those aren't the only cases that have been brought up, but it hardly matters. If you can't answer any of the cases anyone brings up, why should anyone bother giving you new ones?
It's just instead of discussing what is the best way to fix a slowness, I'm spending most time trying to convince people that slowness should be fixed. — sum is slow for lists, let's fix that! — you shouldn't use sum... — why can't I use sum? — because it's slow
But that's not why you shouldn't use sum.
No, That's the MAIN reason!
No, it really isn't. I and others have given multiple reasons. Let me try to put them all together in one place, roughly in order of importance: 1. If sum is the right way to concatenate sequences, it's ambiguous and confusing for types that are both addable and iterable, like numpy.array and other vectors. (The fact that you proposed a solution that would cause sum to flatten arrays is proof that this confusion is not just possible, but likely.) 2. The obvious way to concatenate sequences should work on iterators and other iterables; sum can't. 3. The word "sum" doesn't mean concatenating a bunch of sequences together. This isn't as much of a problem for +, because + isn't a word—people read it as "add", "plus", "and", then", etc. in different contexts, but they read "sum" as "sum". 4. Iterator interfaces are better in a variety of ways—that's why 3.x map, zip, etc. return iterators rather than lists. 5. The right way to concatenate sequences should have consistent performance characteristics; it's impossible for sum to be O(N) for all sequence types. So, that's 4 reasons that sum cannot be the right obvious way to concatenate sequences that are more important than performance. And even reason 5 is more about consistency than performance: if people learn that sum is fast, it had better be fast for their types; if it's not going to be fast for their types, we'd better not promise that it will be. And again, we already have an alternative that has none of those problems. Chain is not confusing for vectors, it works on all iterables, it clearly says what it means and means what it says, it provides an iterator interface, and it's O(N) for all iterables. (I'm ignoring the chain-vs.-chain.from_iterable issue, which is being hashed out in another thread, because it's not relevant here.)
There's one more, about "+" (and sum) being confusing for concatenation, and I can understand that, however it's very weak as long "+" is used for joining sequences. But the main reason that most people, including you, are pushing is that sum() is slow. The whole our "you can't make it fast for" thread is the evidence.
No, the problem is that _you_ keep insisting that the only problem with sum is that it's not fast for all types, you ignore everyone who offers others reasons, so people engage you on the one thing you'll talk about.
Besides, being fast for list and tuple but slow for other
collection types would be an attractive nuisance.
You're pushing "being slow" as a key argument.
No, I'm pushing the being _inconsistent_ as a key argument, as I explained above, and not even as the most important one. Also, the very first word in that sentence is "Besides"; and you ignored the real point before the aside—and you had to go back at least 5 messages to find one you could misrepresent in that way.
If due to some technical details sum() was fast from the very beginning then nobody would say that you shouldn't use it.
Nobody? How about the person who created the sum function in the first place? You've already been given the quote. The only reason you can misuse sum for sequences at all is that he couldn't figure out a good way to prevent people from making that mistake.
Same as nobody is saying that you should use reduce() instead of sum() for numbers because sum() works only for __add__ while reduce works for every operator possible.
Actually, those are very nearly the opposite. The reason to use sum instead of reduce for adding numbers is that sum is a specific, fit-for-purpose function that does exactly what you want to do and says exactly what it's doing, while reduce is an overly-general function that makes it unclear what you're trying to do. And that exact same reason means you should use chain instead of sum for concatenating sequences.
min() is good for list of tuples, right? max() is also good for list of tuples? Even any() is good for list of tuples (weird, but good). Then why sum() is bad for list of tuples? Because it's SLOW!
No. Finding the smallest tuple in a list is an obviously sensible idea that should obviously be spelled min. Flattening, a bunch of tuples is an obviously sensible idea that should obviously be spelled something like "chain" or "flatten" or "concatenate", not "sum".
Your only response to that has been to claim that it can be fast for every possible collection type, but it can't. You haven't gotten it to work. And there are good reasons to believe it's not just hard, but impossible.
Sorry, but can you please quote what response are you referencing?
Picking one of your emails at random:
Despite we CAN make it faster. For all cases, or for most of them,
Here's another:
So, if you're suggesting that sum can be fast for anything reasonable, you're just wrong.
I suggested two ways how to do that. First, one can use the approach above, i.e. use mutable type internally. Second, for internal cpython types we have a simpler option to implement optimization directly in sum(). And there may be many others, specific to the types in question.
And so on. About half your messages have some form or other of the assertion that sum can be fast for all sequence types. Half of them also have you claiming that you already solved some type (by replacing it with another type), and the other half have you denying you ever claimed such a thing. It's like talking to http://en.wikipedia.org/wiki/Nathan_Thurm. "I never said that. Who told you I said that? It's so funny you would think that."
PS: maybe for 10 years people got used to the slow sum() and related arguments so much that they don't want to lose them? ;)
More Nathan Thurm: "I'm not being defensive. You're the one who's being defensive. Do you not want children to have toys? Did you not have toys growing up? It's so funny to me that anyone would want to deprive children of toys."
On 07/10/2013 04:58 PM, Sergey wrote:
On Jul 9, 2013 Ron Adam wrote:
Seriously, why there's so much holy wars about that? I'm not asking to rewrite cpython on Java or C#. I'm not adding a bunch of new functions, I'm not even changing signatures of existing functions.
It's the nature of this particular news group. We focus on improving python, and that includes new things and improving old things, but also includes discussing any existing or potential problems.
You will almost always get a mix of approval and disapproval on just about every thing here. It's not a war, it's just different people having different opinions.
Quite often that leads to finding better ways to do things, and in the long run, helps avoid adding features and changes that could be counter productive to python.
I must agree that I was indeed inspired with some new ideas during this discussion. It's just that those "inspirations" come in a very non-constructive form of "it makes no sense", "cannot always be fast", "you can't", "everyone else thinks you shouldn't", etc.
Or is that a lifehack [1] in action? I.e. "You can't make it fast for that type. Oh, you can? Then you can't make it fast for that type. Oh, you did that too? But you can't make it fast for all the types!" What if I can? ;)
It's just instead of discussing what is the best way to fix a slowness, I'm spending most time trying to convince people that slowness should be fixed. — sum is slow for lists, let's fix that! — you shouldn't use sum... — why can't I use sum? — because it's slow — then let's fix that! — you shouldn't use sum... I haven't thought that somebody can truly believe that something should be slow, and will find one excuse after another instead of just fixing the slowness.
My advise is to not try so hard to change an individuals mind. Are you familiar with the informal voting system we use? Basically take a look though the discussion and look for +1,-1, or things inbetween or like those, and try to get a feel for how strong we feel as a group on the different suggestions. It's not perfect. What you are looking for is strong(er) indications one way or the other(more -1's or +1's), so you identify the suggestions that can be eliminated and the ones that are worth following up on.
If it only makes an existing function faster and doesn't change any other behaviour, and all the tests still pass for it. Just create an issue on the tracker, with the patch posted there, and it will probably be accepted after it goes through a much more focused review process.
I've done that first [2] And there I was asked to write here. :)
I haven't checked that issue, but I'm guessing you were referred to here either because, their was a slight API change in your patch, or you were proposing an addition that would include a change, and so they suggested discussing it here first. Or they wanted you to discuss the possibility of making some changes while you are doing this patch. You can always go back with... "there was no consensus concerning... adding or removing... ", and ask to get just the speed increase parts approved. As long as it doesn't use ugly hacks or code that is difficult to maintain, it should be ok. (and doesn't have any other side effects.) My preference on this is to not extend the API, but go ahead and make it faster if you can. Down the road, I'd like the API's of sum() and fsum() to match. And for them to only work on numbers, and possibly be extended to work on vectors. So.. make the numbers case faster, but probably don't bother changing the non numbers case. (It seems like this is the preferred view so far.) There might be some support for depreciating the non-numbers case. I'm not sugesting that *you* do that btw... see below. :-)
But discussing it here will invite a lot of opinions about how it works, how it shouldn't work, what would work better, and etc...
And about what *I* shouldn't do, what *I* can't and what *I* need. As if I'm the bug that should be fixed. :(
These things that are expressed as *YOU* usually mean ... *WE*. It's just an American English bad habit to overly use "you". Python has millions of useers, so *WE* can't do a lot of things *WE* would like to do. :-/ Cheers, Ron
It's what this board if for. ;-)
On 07/11/2013 05:23 AM, Ron Adam wrote:
So.. make the numbers case faster, but probably don't bother changing the non numbers case. (It seems like this is the preferred view so far.)
I thought the whole point was to make *non*-numbers faster? -- ~Ethan~
On 07/11/2013 11:42 AM, Ethan Furman wrote:
So.. make the numbers case faster, but probably don't bother changing the non numbers case. (It seems like this is the preferred view so far.)
I thought the whole point was to make *non*-numbers faster?
Um... yep, like what it says in the title... Somewhere I got the idea he was making the numbers case faster too. Maybe I was mistaken. I need sum() more coffee, I think. ;-)
Ron Adam <ron3200@gmail.com> writes:
Are you familiar with the informal voting system we use? Basically take a look though the discussion and look for +1,-1, or things inbetween or like those, and try to get a feel for how strong we feel as a group on the different suggestions.
Remember to *discard* any vote outside the range −1.0–1.0 since nobody gets more than that on any single issue. Hyperbole is a ValueError! -- \ “An idea isn't responsible for the people who believe in it.” | `\ —Donald Robert Perry Marquis | _o__) | Ben Finney
On 12/07/13 09:17, Ben Finney wrote:
Ron Adam <ron3200@gmail.com> writes:
Are you familiar with the informal voting system we use? Basically take a look though the discussion and look for +1,-1, or things inbetween or like those, and try to get a feel for how strong we feel as a group on the different suggestions.
Remember to *discard* any vote outside the range −1.0–1.0 since nobody gets more than that on any single issue. Hyperbole is a ValueError!
+1000 on that! But seriously, the given numbers are *strength of feeling*, not number of votes. That's why you can vote +0 or -0.5. -- Steven
On Jul 11, 2013 Ron Adam wrote:
It's just instead of discussing what is the best way to fix a slowness, I'm spending most time trying to convince people that slowness should be fixed. — sum is slow for lists, let's fix that! — you shouldn't use sum... — why can't I use sum? — because it's slow — then let's fix that! — you shouldn't use sum... I haven't thought that somebody can truly believe that something should be slow, and will find one excuse after another instead of just fixing the slowness.
My advise is to not try so hard to change an individuals mind.
I'm not trying to change someones mind (well, maybe I do, but that's just a side-effect). I'm trying to understand their mind. What I'm really trying is to find a solution that would take in account as many opinions as possible. But I need to understand them to do that. I understood when Steven said "I am uncomfortable about changing the semantics to use __iadd__ instead of __add__". I was unsure about that too, but since its not officially documented that sum() uses __add__, I was hoping that nobody is relying on it and nothing would break if this is changed. I understood Joshua when it appeared that such change can break numpy-based code, and he said "We can't rush a semantic change for code that's in popular usage... it seems I've left for the dark side." (and I agree with him, that's why I suggested [1] to record that so others would not be tempted to do that again, so we're still on the same side about that patch :)). I even understood when Stefan said that using "+" and sum() for concatenation makes no (obvious) sense. IMO, it does not matter for our case, it's just a feature that you should be aware of. Using "+" to add lists is like using 2**20 (e.g. instead of 2^20) to get a power, it's neither good nor bad, it's just how it is named here. I mean, I do not agree with that point, but I understand it. But I cannot understand Andrew. From the very beginning it seemed that he's mainly concerned about speed, he was constantly asking me how to speed up different types (or rather he was insisting that I cannot speed them up). When I explicitly asked him whether he thinks that sum() must not be optimized JUST because of possible speed issues with other types he said "Yes". But in the next email he says that speed is not the main reason... So I should either stop trying to understand him (and I don't want to, because discussing the problem with him have inspired me with new ideas) or I'm doomed to repeat the same questions over and over in different forms until I finally understand what he really wants.
Are you familiar with the informal voting system we use? Basically take a look though the discussion and look for [...]
Thank you for detailed explanation. Unfortunatelly I can't just account them once and forget about that, because I adapt my suggestion to these opinions. I.e. initially there was just one patch suggested and now there're three patches and two more ideas waiting to be discussed and, maybe, modified again.
So.. make the numbers case faster, but probably don't bother changing the non numbers case. (It seems like this is the preferred view so far.)
There might be some support for depreciating the non-numbers case. I'm not sugesting that *you* do that btw... see below. :-)
Sum is not just for numbers. It's a rather good choice to add many things, including timedeltas and different numpy types. That's why it was never restricted to work on numbers only. It only has string restriction (for historical reasons and it's more a note to newbies than a restriction because it can be easily tricked if needed). So we can't just deprecate non-numbers, well, we can but I don't think it's a good idea. -- [1] http://bugs.python.org/issue18305#msg192956 http://bugs.python.org/file30904/fastsum-iadd_warning.patch
I think I've followed every post in this long thread and the spinoffs. I don't want to try to address all the small points and sub-threads. Here's my overall take-away: * Using 'sum()' to concatenate sequences or iterators is unintuitive, inferior to other existing techniques, and any code that does so is already *slightly* "broken." (stylistically, not functionally necessarily). Given my perception in the above bullet, I would be +1 on deprecating this use altogether (i.e. raise DeprecationWarning if a good way could be found to distinguist "sequence" from "some other thing that implements .__add__() and is numerical enough"). Even if there's no practical way to make that distinction in a duck-typed language, the documentation should (continue to) say "sum() is the wrong way to concatenate sequences". I am -1 on modifying the semantics of sum() to use .__iadd__(). Even though we have TOOWTDI, in reality there are lots of Python constructs that one *can* do, but really just *shouldn't*. Using sum() on sequences feels about like ._getframe() hacks or slightly perverse generator comprehensions on can write (e.g. for side-effects). We're all adults, and we're not going to stop you (unless we do so with DeprecationWarning), but it's just not the *right* thing to do. David... -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 12 July 2013 22:33, David Mertz <mertz@gnosis.cx> wrote:
Even though we have TOOWTDI, in reality there are lots of Python constructs that one *can* do, but really just *shouldn't*. Using sum() on sequences feels about like ._getframe() hacks or slightly perverse generator comprehensions on can write (e.g. for side-effects). We're all adults, and we're not going to stop you (unless we do so with DeprecationWarning), but it's just not the *right* thing to do.
Really? As bad as a frame-hack? I'm not arguing, just surprised.
On Fri, Jul 12, 2013 at 2:40 PM, Joshua Landau <joshua@landau.ws> wrote:
On 12 July 2013 22:33, David Mertz <mertz@gnosis.cx> wrote:
Even though we have TOOWTDI, in reality there are lots of Python constructs that one *can* do, but really just *shouldn't*. Using sum() on sequences feels about like ._getframe() hacks or slightly perverse generator comprehensions on can write (e.g. for side-effects). We're all adults, and we're not going to stop you (unless we do so with DeprecationWarning), but it's just not the *right* thing to do.
Really? As bad as a frame-hack? I'm not arguing, just surprised.
I think so. I'm not trying to hang any big moral on the exact ranking of "bad habits" in Python. But ._getframe() has the virtue that it sometimes lets you do something that you simply *cannot* do using non-hacky code. These things should be nicely hidden away in modules where average users don't have to think about the magic, but for their arcane maintainers, they can be useful. Maybe throw in some metaclass programming hacks in the same category here. In contrast, there is simply *nothing* one can do by sum()'ing sequences that you can't do with other (more) straightforward code, either itertools.chain() or a small explicit loop. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 09/07/13 19:35, Sergey wrote:
On Jul 5, 2013 Stefan Behnel wrote:
No, please. Using sum() on lists is not more than a hack that seems to be a cool idea but isn't. Seriously - what's the sum of lists? Intuitively, it makes no sense at all to say sum(lists).
It's the same as it is now. What else can you think about when you see: [1, 2, 3] + [4, 5] ?
Some of us think that using + for concatenation is an abuse of terminology, or at least an unfortunate choice of operator, and are wary of anything that encourages that terminology. Nevertheless, you are right, in Python 3 both + and sum of lists is well-defined. At the moment sum is defined in terms of __add__. You want to change it to be defined in terms of __iadd__. That is a semantic change that needs to be considered carefully, it is not just an optimization.
Seriously, why there's so much holy wars about that? I'm not asking to rewrite cpython on Java or C#. I'm not adding a bunch of new functions, I'm not even changing signatures of existing functions.
Because sometimes people are cautious and conservative about new ideas. Better to be cautious, and miss out on some new function for a version or two, than to rush into it, and then regret it if it turns out to be a bad idea. I have been very careful to say that I am only a little bit against this idea, -0 not -1. I am uncomfortable about changing the semantics to use __iadd__ instead of __add__, because I expect that this will change the behaviour of sum() for non-builtins. I worry about increased complexity making maintenance harder for no good reason. It's the "for no good reason" that concerns me: you could answer some of my objections if you showed: - real code written by people who sum() large (more than 100) numbers of lists; - real code with comments like "this is a work-around for sum() being slow on lists"; - bug reports or other public complaints by people (other than you) complaining that sum(lists) is slow; or similar. That would prove that people do call sum on lists. But at the moment, I can only judge based on my own experience, both in writing code and dealing with questions on comp.lang.python, and I don't see many people doing sum(lists) or sum(tuples). Earlier in this discussion, you posted benchmarks for the patched sum using Python 2.7. Would you be willing to do it again for 3.3? And confirm that the Python test suite continues to pass? Thank you, -- Steven
On 9 July 2013 17:13, Steven D'Aprano <steve@pearwood.info> wrote:
On 09/07/13 19:35, Sergey wrote:
On Jul 5, 2013 Stefan Behnel wrote:
No, please. Using sum() on lists is not more than a hack that seems to be a cool idea but isn't. Seriously - what's the sum of lists? Intuitively, it makes no sense at all to say sum(lists).
It's the same as it is now. What else can you think about when you see: [1, 2, 3] + [4, 5] ?
Some of us think that using + for concatenation is an abuse of terminology, or at least an unfortunate choice of operator, and are wary of anything that encourages that terminology.
Nevertheless, you are right, in Python 3 both + and sum of lists is well-defined. At the moment sum is defined in terms of __add__. You want to change it to be defined in terms of __iadd__. That is a semantic change that needs to be considered carefully, it is not just an optimization.
I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code. That makes it anything but a semantic change. It's the same way people discuss the order of __hash__ calls on updates to code but no-one calls it a *semantics* change.
I am uncomfortable about changing the semantics to use __iadd__ instead of __add__, because I expect that this will change the behaviour of sum() for non-builtins.
Other than broken stuff, any guesses as to what? I'm trying to think of maybe an IO thing (directories where __add__ makes a new "directory viewer" and __iadd__ does a "cd") but none of them actually *change* behaviour.
I worry about increased complexity making maintenance harder for no good reason. It's the "for no good reason" that concerns me: you could answer some of my objections if you showed:
The move to __iadd__, in my opinion, is such a trivial thing that "maintainability" shouldn't be concerned. Overriding for multiple types is definitely going to cause a hazard, but this is adding like 1 line to the codebase.
- bug reports or other public complaints by people (other than you) complaining that sum(lists) is slow;
I don't think that is a good measure -- I've personally found cases where "sum" looks nicer but isn't the best algorithm yet I've never complained because 2-3 lines is really not that big a deal and it *felt* like sum *had* to be O(n**2). I largely don't think of sum(list_of_lists) as a nice looking construct, but that could just be a learnt opinion and I'd never think of "sum(list_of_lists, [])" as counterintuitive. I might think "OMG INEFFICIENCY" for a long time coming, but I find it so hard to agree with those of you who say it doesn't make sense. I also think that holding back potential cases where __iadd__ is better (which is every __iadd__) because you think a fast "sum(list_of_lists, [])" would encourage that construct is a bit silly. Just say "that's not the best way to do it because it's not generic enough¹ whereas chain.from_iterables is" if you really feel that way. This is especially true if others agree that "chain.from_iterables" is deserving of __builtins__.
Earlier in this discussion, you posted benchmarks for the patched sum using Python 2.7. Would you be willing to do it again for 3.3? And confirm that the Python test suite continues to pass?
Seconded. ¹ Doesn't work for anything other than mutatable, addable objects
On 10 July 2013 07:09, Joshua Landau <joshua@landau.ws> wrote:
I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code. That makes it anything but a semantic change. It's the same way people discuss the order of __hash__ calls on updates to code but no-one calls it a *semantics* change.
It will stop working for tuples (which have no __iadd__. Or were you suggesting trying __iadd__ and falling back to __add__ (that's more complex, obviously, and I don't think I'd assume it's "trivial" extra complexity) or special-caseing tuples (that's even more complex, and doesn't solve the problem for other iimmutables)? Paul
On 10 July 2013 07:33, Paul Moore <p.f.moore@gmail.com> wrote:
On 10 July 2013 07:09, Joshua Landau <joshua@landau.ws> wrote:
I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code. That makes it anything but a semantic change. It's the same way people discuss the order of __hash__ calls on updates to code but no-one calls it a *semantics* change.
It will stop working for tuples (which have no __iadd__. Or were you suggesting trying __iadd__ and falling back to __add__ (that's more complex, obviously, and I don't think I'd assume it's "trivial" extra complexity) or special-caseing tuples (that's even more complex, and doesn't solve the problem for other iimmutables)?
Surely it just does the equivalent of "a += b", which handles fallback for you. People don't write "a.__iadd__(b)" as much as they don't write "a.__add__(b)". Python's C API has an inplace addition that mimics this, btw. And I'm not supporting special-casing anyway.
On 10 Jul, 2013, at 8:33, Paul Moore <p.f.moore@gmail.com> wrote:
On 10 July 2013 07:09, Joshua Landau <joshua@landau.ws> wrote: I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code. That makes it anything but a semantic change. It's the same way people discuss the order of __hash__ calls on updates to code but no-one calls it a *semantics* change.
It will stop working for tuples (which have no __iadd__. Or were you suggesting trying __iadd__ and falling back to __add__ (that's more complex, obviously, and I don't think I'd assume it's "trivial" extra complexity) or special-caseing tuples (that's even more complex, and doesn't solve the problem for other iimmutables)?
Both "+=" in Python and its C API equivalent (PyNumber_InPlaceAdd) perform an in place addition (__iadd__) when possible and fall back to using normal addition (__add__) when the in place method is not supported. Thus barring bugs and creative misuse of in place operators using += instead of + in sum shouldn't affect the result of sum. Still-unconviced-about-the-usefulness-ly yours, Ronald
On 10/07/13 16:09, Joshua Landau wrote:
On 9 July 2013 17:13, Steven D'Aprano <steve@pearwood.info> wrote: [...]
Nevertheless, you are right, in Python 3 both + and sum of lists is well-defined. At the moment sum is defined in terms of __add__. You want to change it to be defined in terms of __iadd__. That is a semantic change that needs to be considered carefully, it is not just an optimization.
I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code.
"Always"? Immutable objects cannot define __iadd__ as an in-place __add__. In any case, sum() currently does not modify the start argument in-place.
That makes it anything but a semantic change.
__iadd__ is optional for classes that support addition. Failure to define an __iadd__ method does not make your class broken. Making __iadd__ mandatory to support sum would be a semantic change, since there will be objects (apart from strs and bytes, which are special-cased) that support addition with + but will no longer be summable since they don't define __iadd__. Even making __iadd__ optional will potentially break working code. Python doesn't *require* that __iadd__ perform the same operation as __add__. That is the normal expectation, of course, but it's not enforced. (How could it be?) We might agree that objects where __add__ and __iadd__ do different things are "broken" in some sense, but you're allowed to write broken code, and Python should (in principle) avoid making it even more broken by changing behaviour unnecessarily. But maybe the right answer there is simply "don't call sum if you don't want __iadd__ called". -- Steven
On 10 July 2013 17:29, Steven D'Aprano <steve@pearwood.info> wrote:
On 10/07/13 16:09, Joshua Landau wrote:
On 9 July 2013 17:13, Steven D'Aprano <steve@pearwood.info> wrote:
[...]
Nevertheless, you are right, in Python 3 both + and sum of lists is well-defined. At the moment sum is defined in terms of __add__. You want to change it to be defined in terms of __iadd__. That is a semantic change that needs to be considered carefully, it is not just an optimization.
I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code.
"Always"? Immutable objects cannot define __iadd__ as an in-place __add__.
In any case, sum() currently does not modify the start argument in-place.
Now you're just (badly) playing semantics. If I say that gills are always like lungs except they work underwater, would you contradict me by stating that mammals don't have gills?
That makes it anything but a semantic change.
__iadd__ is optional for classes that support addition. Failure to define an __iadd__ method does not make your class broken.
Making __iadd__ mandatory to support sum would be a semantic change, since there will be objects (apart from strs and bytes, which are special-cased) that support addition with + but will no longer be summable since they don't define __iadd__.
Why are you saying these things? I never suggested anything like that.
Even making __iadd__ optional will potentially break working code. Python doesn't *require* that __iadd__ perform the same operation as __add__. That is the normal expectation, of course, but it's not enforced. (How could it be?) We might agree that objects where __add__ and __iadd__ do different things are "broken" in some sense, but you're allowed to write broken code, and Python should (in principle) avoid making it even more broken by changing behaviour unnecessarily. But maybe the right answer there is simply "don't call sum if you don't want __iadd__ called".
Python has previously had precedents where broken code does not get to dictate the language as long as that code was very rare. This is more than very rare. Additionally, Python does (unclearly, but it does do so) define __iadd__ to be an inplace version of __add__, so the code isn't just “broken” -- it's broken.
On 11/07/13 04:21, Joshua Landau wrote:
On 10 July 2013 17:29, Steven D'Aprano <steve@pearwood.info> wrote:
On 10/07/13 16:09, Joshua Landau wrote:
On 9 July 2013 17:13, Steven D'Aprano <steve@pearwood.info> wrote:
[...]
Nevertheless, you are right, in Python 3 both + and sum of lists is well-defined. At the moment sum is defined in terms of __add__. You want to change it to be defined in terms of __iadd__. That is a semantic change that needs to be considered carefully, it is not just an optimization.
I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code.
"Always"? Immutable objects cannot define __iadd__ as an in-place __add__.
In any case, sum() currently does not modify the start argument in-place.
Now you're just (badly) playing semantics. If I say that gills are always like lungs except they work underwater, would you contradict me by stating that mammals don't have gills?
I don't actually understand your objection. You made a general statement that __iadd__ should ALWAYS be an in-place add, and I pointed out that this cannot be the case for immutable classes. What is your objection? Surely you're not suggesting that immutable classes can define in-place __iadd__? That's not a rhetorical question, I do not understand the point you are trying to make. Perhaps you should be explicit, rather than argue by analogy. [Aside: it's a poor analogy. Gills are not like lungs, they differ greatly in many ways, e.g. fluid flow is unidirectional in gills, bidirectional in lungs, the interface to the blood system is counter-current in gills, with an efficiency of about 80%, versus concurrent in lungs, with an efficiency around 25%. There are other significant differences too, and lungs evolved independently of gills, which is why lungfish have both.]
That makes it anything but a semantic change.
__iadd__ is optional for classes that support addition. Failure to define an __iadd__ method does not make your class broken.
Making __iadd__ mandatory to support sum would be a semantic change, since there will be objects (apart from strs and bytes, which are special-cased) that support addition with + but will no longer be summable since they don't define __iadd__.
Why are you saying these things? I never suggested anything like that.
You want to change sum from using __add__ to __iadd__. That means that there are two possibilities: for a class to be summable, either __iadd__ is mandatory, or it is optional with a fallback to __add__. I considered both possibilities, and they both result in changes to the behaviour of sum, that is, a semantic change. If __iadd__ becomes mandatory, then some currently summable classes will become non-summable. If __iadd__ becomes optional, but preferred over __add__, then some currently summable classes will change their behaviour (although you call those classes "broken"). In either case, this is a semantic change to sum, which is what you explicitly denied. I think that it is a reasonable position to take that we should not care about "broken" classes that define __iadd__ differently to __add__. I'm not sure that I agree, but regardless, it is a reasonable position. But arguing that the proposed change from __add__ to __iadd__ is not a semantic change to sum is simply unreasonable.
Even making __iadd__ optional will potentially break working code. Python doesn't *require* that __iadd__ perform the same operation as __add__. That is the normal expectation, of course, but it's not enforced. (How could it be?) We might agree that objects where __add__ and __iadd__ do different things are "broken" in some sense, but you're allowed to write broken code, and Python should (in principle) avoid making it even more broken by changing behaviour unnecessarily. But maybe the right answer there is simply "don't call sum if you don't want __iadd__ called".
Python has previously had precedents where broken code does not get to dictate the language as long as that code was very rare. This is more than very rare. Additionally, Python does (unclearly, but it does do so) define __iadd__ to be an inplace version of __add__, so the code isn't just “broken” -- it's broken.
Not so. The docs for __iadd__ and other augmented assignment operators state: "These methods are called to implement the augmented arithmetic assignments (+=, -=, *=, /=, //=, %=, **=, <<=, >>=, &=, ^=, |=). These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self)." So, according to the docs, "x += y" might modify x in place and return a different instance, or even a completely different value. It is normal, and expected, for "x += y" to be the same as "x = x + y", but not compulsory. Python will fall back on the usual __add__ if __iadd__ is not defined, but (say) if you define your own DSL where += has some distinct meaning, you are free to define it to do something completely different. You consider it broken if a class defines += differently to +. I consider it unusual, but permitted. I believe the docs support my interpretation. http://docs.python.org/release/3.1/reference/datamodel.html#object.__iadd__ E.g. I have a DSL where = reassigns to a data structure, += appends to an existing one, and + is not defined at all. You can say "x += value" but not "x = x + value". It makes sense in context. As I said, I am prepared to consider that the right answer to this is "well don't call sum on your data structure then", but it is a change in behaviour, not just an optimization. -- Steven
On 11 July 2013 03:36, Steven D'Aprano <steve@pearwood.info> wrote:
On 11/07/13 04:21, Joshua Landau wrote:
On 10 July 2013 17:29, Steven D'Aprano <steve@pearwood.info> wrote:
On 10/07/13 16:09, Joshua Landau wrote:
On 9 July 2013 17:13, Steven D'Aprano <steve@pearwood.info> wrote:
[...]
Nevertheless, you are right, in Python 3 both + and sum of lists is well-defined. At the moment sum is defined in terms of __add__. You want to change it to be defined in terms of __iadd__. That is a semantic change that needs to be considered carefully, it is not just an optimization.
I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code.
"Always"? Immutable objects cannot define __iadd__ as an in-place __add__.
In any case, sum() currently does not modify the start argument in-place.
Now you're just (badly) playing semantics. If I say that gills are always like lungs except they work underwater, would you contradict me by stating that mammals don't have gills?
I don't actually understand your objection. You made a general statement that __iadd__ should ALWAYS be an in-place add,
Which it should.
and I pointed out that this cannot be the case for immutable classes. What is your objection? Surely you're not suggesting that immutable classes can define in-place __iadd__?
Of course not.
That's not a rhetorical question, I do not understand the point you are trying to make. Perhaps you should be explicit, rather than argue by analogy.
Rather, I shall explain by analogy :P. I said that __iadd__ should always be a faster __add__. This is saying that all <a> have property <b>, aka. the analogy of stating that all gills are like lungs (except [that] they work underwater). You objected by saying there are some things that cannot implement __iadd__. This has *no correlation* to the previous statement - that was about the properties that all __iadd__ have. That is why I made an analogy of you contradicting me by saying that mammals don't have gills. So basically, I made no claim in any way about what objects have __iadd__, but about what __iadd_ does (which is of course for only those circumstances where it applies -- I know that's a tautology but this whole sub-discussion seems to be one).
[Aside: it's a poor analogy. Gills are not like lungs, they differ greatly in many ways,
This isn't really relevant, but alas they *are* like lungs. Sure, it's an imperfect relation, but that's why I said "like lungs" and not "are lungs".
e.g. fluid flow is unidirectional in gills, bidirectional in lungs, the interface to the blood system is counter-current in gills, with an efficiency of about 80%, versus concurrent in lungs, with an efficiency around 25%. There are other significant differences too, and lungs evolved independently of gills, which is why lungfish have both.]
I honestly didn't know that. Interesting.
That makes it anything but a semantic change.
__iadd__ is optional for classes that support addition. Failure to define an __iadd__ method does not make your class broken.
Making __iadd__ mandatory to support sum would be a semantic change, since there will be objects (apart from strs and bytes, which are special-cased) that support addition with + but will no longer be summable since they don't define __iadd__.
Why are you saying these things? I never suggested anything like that.
You want to change sum from using __add__ to __iadd__. That means that there are two possibilities: for a class to be summable, either __iadd__ is mandatory, or it is optional with a fallback to __add__. I considered both possibilities, and they both result in changes to the behaviour of sum, that is, a semantic change.
If __iadd__ becomes mandatory, then some currently summable classes will become non-summable.
I don't believe that was ever suggested; there is a good reason "+=" falls back on "+" by default.
If __iadd__ becomes optional, but preferred over __add__, then some currently summable classes will change their behaviour (although you call those classes "broken").
That is what I was doing - calling them broken.
In either case, this is a semantic change to sum, which is what you explicitly denied.
I'm not sure not supporting broken code counts as a semantic change. That is what I was debating.
I think that it is a reasonable position to take that we should not care about "broken" classes that define __iadd__ differently to __add__. I'm not sure that I agree, but regardless, it is a reasonable position. But arguing that the proposed change from __add__ to __iadd__ is not a semantic change to sum is simply unreasonable.
But that is what I am doing :P. If a spec is undefined, you don't require results to be consistent. This is what would happen. That changes nothing, as far as I am concerned -- and hence is not a semantic change.
Even making __iadd__ optional will potentially break working code. Python doesn't *require* that __iadd__ perform the same operation as __add__. That is the normal expectation, of course, but it's not enforced. (How could it be?) We might agree that objects where __add__ and __iadd__ do different things are "broken" in some sense, but you're allowed to write broken code, and Python should (in principle) avoid making it even more broken by changing behaviour unnecessarily. But maybe the right answer there is simply "don't call sum if you don't want __iadd__ called".
Python has previously had precedents where broken code does not get to dictate the language as long as that code was very rare. This is more than very rare. Additionally, Python does (unclearly, but it does do so) define __iadd__ to be an inplace version of __add__, so the code isn't just “broken” -- it's broken.
Not so. The docs for __iadd__ and other augmented assignment operators state:
"These methods are called to implement the augmented arithmetic assignments (+=, -=, *=, /=, //=, %=, **=, <<=, >>=, &=, ^=, |=). These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self)."
So, according to the docs, "x += y" might modify x in place and return a different instance, or even a completely different value. It is normal, and expected, for "x += y" to be the same as "x = x + y", but not compulsory. Python will fall back on the usual __add__ if __iadd__ is not defined, but (say) if you define your own DSL where += has some distinct meaning, you are free to define it to do something completely different.
I read it differently. I am not sure why one would do anything other that return self, but I also read "to implement the augmented arithmetic assignments" and "should attempt to do the operation in-place (modifying self) and return the result". The final qualifier only applies to circumstances, as far as I can glean, that are still *attempted in-place additions*. Good examples could be when in-place attempts fail and it would be faster to do __add__ right there and then. Other good examples are taking a while to come, but this is quite a niche area.
You consider it broken if a class defines += differently to +. I consider it unusual, but permitted. I believe the docs support my interpretation.
http://docs.python.org/release/3.1/reference/datamodel.html#object.__iadd__
E.g. I have a DSL where = reassigns to a data structure, += appends to an existing one, and + is not defined at all. You can say "x += value" but not "x = x + value". It makes sense in context. As I said, I am prepared to consider that the right answer to this is "well don't call sum on your data structure then", but it is a change in behaviour, not just an optimization.
That is... really quite a good argument. I think I may have to think on that final point, but you've probably just about won it. Why didn't you just say this from the start?
On 11 July 2013 04:05, Joshua Landau <joshua@landau.ws> wrote:
On 11 July 2013 03:36, Steven D'Aprano <steve@pearwood.info> wrote:
E.g. I have a DSL where = reassigns to a data structure, += appends to an existing one, and + is not defined at all. You can say "x += value" but not "x = x + value". It makes sense in context. As I said, I am prepared to consider that the right answer to this is "well don't call sum on your data structure then", but it is a change in behaviour, not just an optimization.
That is... really quite a good argument. I think I may have to think on that final point, but you've probably just about won it. Why didn't you just say this from the start?
This was so close to winning me over, but think about this: 1) __iadd__ is always, for working classes, an inplace __add__. The fact that some classes miss out one, the other or both is irrelevant to this. 2) The new sum requires __add__ for step 1. Hence you *couldn't* sum the DSLs. I think you would have to write a class which cheats for the first step, but I'm not sure that that is any better than broken code. I can at least assure you there are exactly 0 instances of this in the wild as of now. 3) Things that have __add__ but not __iadd__ experience no behaviour change, so the inverse of the DSLs don't change anything either.
Joshua Landau writes:
[Aside: it's a poor analogy. Gills are not like lungs, they differ greatly in many ways,
This isn't really relevant, but alas they *are* like lungs. Sure, it's an imperfect relation, but that's why I said "like lungs" and not "are lungs".
No, it is relevant. You provide a very high-level specification ("extract oxygen from ambient fluid"), and then make a slightly lower- level distinction ("ambient fluid is air vs. water"). But as far as I can see, many of the issues that make "sum(iterable of sequences)" an unattractive API are far lower-level ("counter-current vs. concurrent"). ISTM that the choice of *not* constraining the definitions of _i<binop> to be efficient, in-place versions of the corresponding _<binop> was deliberate. So you need to take into account differences that are potential and ill-defined -- and when Steven provides a real use case at this level, you start preparing to concede.
If __iadd__ becomes optional, but preferred over __add__, then some currently summable classes will change their behaviour (although you call those classes "broken").
That is what I was doing - calling them broken.
In either case, this is a semantic change to sum, which is what you explicitly denied.
I'm not sure not supporting broken code counts as a semantic change. That is what I was debating.
It does count; it's a language change. It is not a bug-fix in which the implementation is brought into line with the language definition. CPython has historically taken the position if the language definition is ambiguous, a change in CPython behavior requires that *the language definition be changed* to clarify that the changed behavior is the mandated behavior. Note also that CPython is intended to be a reference implementation. Therefore existing behavior has a special significance unless explicitly specified to be implementation- dependent. Not only applications, but other implementations, may depend on existing behavior.
But that is what I am doing :P. If a spec is undefined, you don't require results to be consistent.
No, that may be a bug in the spec: the spec is incomplete, but it does include requiring results to be consistent. That's why reference implementations exist, and why "modern" specs explicitly state that behavior is undefined, or that a certain construct "is an error [even if the implementation doesn't signal it]".
This is what would happen. That changes nothing, as far as I am concerned -- and hence is not a semantic change.
Well, I see that you do know what you're doing. My opinion (which is not authoritative) is that you are using a different definition of "semantic change" from the one used by Python (!= CPython).
Python has previously had precedents where broken code does not get to dictate the language as long as that code was very rare.
I suspect Guido would not call code "broken" unless it depended on an actual bug in the implementation, or there was another way to do it that is TOOWTDI. Here there is another way to do it that is TOOWTDI *for the case you want to support* (not the code you consider broken). So this is a losing analogy for you.
E.g. I have a DSL where = reassigns to a data structure, += appends to an existing one, and + is not defined at all.
That is... really quite a good argument. I think I may have to think on that final point, but you've probably just about won it. Why didn't you just say this from the start?
Because he thought his other arguments were even better.<wink/> (another) Steve
On 11 Jul, 2013, at 7:27, Stephen J. Turnbull <stephen@xemacs.org> wrote:
In either case, this is a semantic change to sum, which is what you explicitly denied.
I'm not sure not supporting broken code counts as a semantic change. That is what I was debating.
It does count; it's a language change. It is not a bug-fix in which the implementation is brought into line with the language definition.
That doesn't mean that using += instead of + in sum isn't a valid change to make for 3.4. BTW. This thread has been rehashing the same arguments over and over again, and it's pretty likely that most core devs have stopped following this thread because of that. It's probably time for someone to write a summary the discussion (what are the proposals and the arguments in favor and against them) Ronald
Ronald Oussoren writes:
It does count; it's a language change. It is not a bug-fix in which the implementation is brought into line with the language definition.
That doesn't mean that using += instead of + in sum isn't a valid change to make for 3.4.
Agreed, it might be. I was just addressing Joshua's statement that:
I'm not sure not supporting broken code counts as a semantic change. That is what I was debating.
AFAIK, whether some code depending on current behavior is broken is irrelevant to Python's definition of "semantic change".
It's probably time for someone to write a summary of the discussion (what are the proposals and the arguments in favor and against them)
+1 Steve
On 7/11/2013 1:53 AM, Ronald Oussoren wrote:
On 11 Jul, 2013, at 7:27, Stephen J. Turnbull <stephen@xemacs.org> wrote:
It does count; it's a language change. It is not a bug-fix in which the implementation is brought into line with the language definition.
That doesn't mean that using += instead of + in sum isn't a valid change to make for 3.4.
Breaking code in the way this would do, would require a PEP and deprecation cycle. I do not anticipate approval for a general change. A specialized change such that sum(iterable_of_lists, []) would extend rather than replace [] might be done since the result would be equal to the current result, just faster, and since [] must be nearly always passed without aliases that depend on it not changing. Even that should have a deprecation warning. Tuples could be linearly summed in a list with .extend and then converted at the end. I don't believe that would be a semantic change at all.
BTW. This thread has been rehashing the same arguments over and over again, and it's pretty likely that most core devs have stopped following this thread
Right, I just happened to pick this post because you are also a core dev.
because of that. It's probably time for someone to write a summary the discussion (what are the proposals and the arguments in favor and against them)
-- Terry Jan Reedy
On 11 Jul, 2013, at 21:20, Terry Reedy <tjreedy@udel.edu> wrote:
On 7/11/2013 1:53 AM, Ronald Oussoren wrote:
On 11 Jul, 2013, at 7:27, Stephen J. Turnbull <stephen@xemacs.org> wrote:
It does count; it's a language change. It is not a bug-fix in which the implementation is brought into line with the language definition.
That doesn't mean that using += instead of + in sum isn't a valid change to make for 3.4.
Breaking code in the way this would do, would require a PEP and deprecation cycle. I do not anticipate approval for a general change.
I agree. At the time I wrote this I didn't know that in numpy + and += have slightly different semantics w.r.t. coercion of array element types. That means that using += in sum() would change the behavior of calling sum on a sequence of numpy arrays, and given that we try hard to maintain backward compatibility this means that this particular change probably won't happen. More so because the documentation clearly indicates that sum() is intended to be used with numbers and the change is meant to change the behavior for non-numbers (in particular lists).
A specialized change such that sum(iterable_of_lists, []) would extend rather than replace [] might be done since the result would be equal to the current result, just faster, and since [] must be nearly always passed without aliases that depend on it not changing. Even that should have a deprecation warning.
I don't think that a deprecation warning would be needed, as long as start got copied before extending it.
Tuples could be linearly summed in a list with .extend and then converted at the end. I don't believe that would be a semantic change at all.
It would make the implementation of sum more complicated. There is clear historical evidency that sum is not intented to be fast for everything that supports the + operator, calling sum on a sequence of strings raises an exception instead of trying to special case this.
BTW. This thread has been rehashing the same arguments over and over again, and it's pretty likely that most core devs have stopped following this thread
Right, I just happened to pick this post because you are also a core dev.
I am, but that doesn't mean my opinion caries a lot more weight in this discussion ;-). The higher profile core devs are conspicuously absent from this discussion, which is why I wrote that someone needs to write a summary of the discussion. I'm not that someone though. Ronald
On Jul 10, 2013, at 20:05, Joshua Landau <joshua@landau.ws> wrote:
E.g. I have a DSL where = reassigns to a data structure, += appends to an existing one, and + is not defined at all. You can say "x += value" but not "x = x + value". It makes sense in context. As I said, I am prepared to consider that the right answer to this is "well don't call sum on your data structure then", but it is a change in behaviour, not just an optimization.
That is... really quite a good argument. I think I may have to think on that final point, but you've probably just about won it. Why didn't you just say this from the start?
SymPy and another expression template library were already brought up earlier in the discussion. But I suspect most Python programmers have no experience with this kind of programming and/or have never heard of the libraries in question, and therefore had no idea what the point was until now. SymPy does actually work with the add-once-then-iadd-to-that patch. I just tested summing together a list of symbolic expressions, and the result was the correct expression. Which makes sense, because it doesn't use __iadd__ anywhere (and in fact "x += y" rebinds x to the expression "x+y"). Anyway, thanks to Steven for explaining the point, and not using an example that fails to show what's intended. :)
On 11 July 2013 04:05, Joshua Landau <joshua@landau.ws> wrote:
E.g. I have a DSL where = reassigns to a data structure, += appends to an existing one, and + is not defined at all. You can say "x += value" but not "x = x + value". It makes sense in context. As I said, I am prepared to consider that the right answer to this is "well don't call sum on your data structure then", but it is a change in behaviour, not just an optimization.
That is... really quite a good argument. I think I may have to think on that final point, but you've probably just about won it. Why didn't you just say this from the start?
Another example - if I have an event class with an API modeled after the C# approach, += is used to add a listener. But + on events makes no sense and is undefined. This class is not currently summable, but would become summable. Again, the right answer is possibly "don't use sum on these objects", but it *is* a semantic change. Also, if "do't use sum here" is a valid statement in these cases, why is it so impossible for "don't use sum on containers" to be a valid argument in the current situation? Paul
On 10 July 2013 19:21, Joshua Landau <joshua@landau.ws> wrote:
Python has previously had precedents where broken code does not get to dictate the language as long as that code was very rare. This is more than very rare. Additionally, Python does (unclearly, but it does do so) define __iadd__ to be an inplace version of __add__, so the code isn't just “broken” -- it's broken.
Although I'm quoting Joshua above, there are many people here who have made the erroneous assertion that __iadd__ should always be equivalent to __add__ and that there isn't much code that could be semantically affected by the proposed change. Numpy arrays treat += differently from + in the sense that a += b coerces b to the same dtype as a and then adds in place whereas a + b uses Python style type promotion. This behaviour is by design and it is useful. It is also entirely appropriate (and not pathological) that someone would use sum() to add numpy arrays. An example where + and += give different results:
from numpy import array a1 = array([1, 2, 3], dtype=int) a1 array([1, 2, 3]) a2 = array([.5, .5, .5], dtype=float) a2 array([ 0.5, 0.5, 0.5]) a1 + a2 array([ 1.5, 2.5, 3.5]) a1 += a2 a1 array([1, 2, 3])
Oscar
On 11 Jul, 2013, at 11:07, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 10 July 2013 19:21, Joshua Landau <joshua@landau.ws> wrote:
Python has previously had precedents where broken code does not get to dictate the language as long as that code was very rare. This is more than very rare. Additionally, Python does (unclearly, but it does do so) define __iadd__ to be an inplace version of __add__, so the code isn't just “broken” -- it's broken.
Although I'm quoting Joshua above, there are many people here who have made the erroneous assertion that __iadd__ should always be equivalent to __add__ and that there isn't much code that could be semantically affected by the proposed change.
Numpy arrays treat += differently from + in the sense that a += b coerces b to the same dtype as a and then adds in place whereas a + b uses Python style type promotion. This behaviour is by design and it is useful. It is also entirely appropriate (and not pathological) that someone would use sum() to add numpy arrays.
That, and Paul's example about using += for something else than addition, pretty much kills the idea of using += in the implementation of sum() as that would break to much code. Ronald
This started out as a response to Ronald, but turned into a summary and pseudo-code implementation, so I've changed the subject line. On 11/07/13 21:15, Ronald Oussoren wrote: [...]
That, and Paul's example about using += for something else than addition, pretty much kills the idea of using += in the implementation of sum() as that would break to much code.
Not quite :-) After spending so much time explaining why I don't think sum *should* optimize the list case, I'm now going to suggest a way which (I think) sum *could* optimize the list case without changing behaviour. [puts on Devil's Advocate cap] What we have to give up is Sergey's hope to make sum fast for "everything". I don't think that is possible, but even if it is, we should not let the Perfect be the enemy of the Good. Can we make sum fast for lists, without changing behaviour? I think we can. We know that list.__iadd__ is a fast, inplace version of __add__. We can't make that assumption about every class, not even list subclasses, but we don't have to speed up every class. Let's not be greedy: incremental improvements are better than no improvements. sum already has special cases for ints and floats and strings (the string case is to just prohibit the use of sum). We could add a special case for lists: if the start parameter is an actual built-in list, not a subclass (because subclasses might override __iadd__) then we start with an accumulator list, and call __iadd__ (or extend) on that list to concatenate on it. As soon as we find an item which is not a built-in list, we drop out of the fast-code and fall back to the normal non-fast path. This doesn't make it "fast for everything", but it's still an optimization. Here's some pseudo-code, modified from the code Sergey posted earlier: # untested def sum(values, start=0): values = iter(values) [... skip special cases for int, string, float, etc.] elif type(start) is list: result = [] result += start for value in values: if type(value) is not list: start = result + value break result += value else: # No break. return result # If we get here, fall back to the non-fast path. result = start for value in values: result = result + value return result (Credit to Sergey for coming up with the original code.) Since we cannot *guarantee* that sum is fast for all classes, we should not promise what we can't guarantee. Better to deliver more than you promise, than to make grandiose promises of "fast for everything" that you cannot live up to. Arguments in favour of this suggestion: - The simple use-case of summing a list of lists will speed up. - The behaviour of sum doesn't change (except for speed): * sum still returns a new list, it doesn't modify in place; * officially, sum still relies on __add__ only, not __iadd__. Arguments against: - Somebody has to write this code, and debug it, and maintain it. It won't be me. - Optimized code is complex code, and perhaps this extra complication is too much complication. After all, the Zen says "Simple is better than complex, complex is better than complicated." - If sum of lists is fast, it will lull people into a false sense of security. sum remains an attractive nuisance, since you cannot rely on it being fast. As soon as you add one list subclass, it falls back to the slow code again. - But on the other hand, that's not much worse than the situation now. People are already lulled into a false sense of security, because "Everything is fast for small enough N". Further questions: 1) Can we be less conservative about subclasses? E.g. if a subclass does not override __add__ or __iadd__, can we safely use the fast path, and only fall back on the slow path for those that override? - In favour: that would make sum more useful, rather than less. - Against: lots more complications... 2) Sergey wants to do something similar for tuples, using a temporary list accumulator. I've raised the possibility that if the accumulated list is huge enough, you might get a MemoryError trying to copy it back to a tuple. Does anyone care about this hypothetical case? If not, then we could extend the optimization to tuples (at least those that don't override __add__). 3) And the critical question: with this (theoretical) patch in place, do the test suites still pass? I asked Sergey this same question about his patch some days ago, but haven't seen an answer. -- Steven
On 12 July 2013 01:52, Steven D'Aprano <steve@pearwood.info> wrote:
After spending so much time explaining why I don't think sum *should* optimize the list case, I'm now going to suggest a way which (I think) sum *could* optimize the list case without changing behaviour.
<explains>
Since we cannot *guarantee* that sum is fast for all classes, we should not promise what we can't guarantee. Better to deliver more than you promise, than to make grandiose promises of "fast for everything" that you cannot live up to.
That is true. however, I don't believe that a sharp cliff where there should be a gradient is the right approach.
Arguments in favour of this suggestion:
<basically just "it works">
Arguments against:
- If sum of lists is fast, it will lull people into a false sense of security. sum remains an attractive nuisance, since you cannot rely on it being fast. As soon as you add one list subclass, it falls back to the slow code again.
The subclass thing matters a lot to me. When the discussion was about "+=", I was for it -- a subclass that keeps behaviour consistent should not be treated like a second-class citizen, and so we should do everything we can to prevent people from using sum there. A three-line solution is miles better than a half-line one which bombs as soon as you make tiny should-be-irrelevant changes.
- But on the other hand, that's not much worse than the situation now. People are already lulled into a false sense of security, because "Everything is fast for small enough N".
I don't agree that that's the same thing. That's being blindly uninterested in asymptotic performance whereas this change to sum actively tricks you into thinking that your algorithm has better asymptotic performance than it does.
Further questions:
1) Can we be less conservative about subclasses? E.g. if a subclass does not override __add__ or __iadd__, can we safely use the fast path, and only fall back on the slow path for those that override?
I'm not sure how __add__ and __iadd__ are implemented -- do they call other methods on the class?
2) Sergey wants to do something similar for tuples, using a temporary list accumulator. I've raised the possibility that if the accumulated list is huge enough, you might get a MemoryError trying to copy it back to a tuple. Does anyone care about this hypothetical case? If not, then we could extend the optimization to tuples (at least those that don't override __add__).
Personally I do not care much; Python makes no guarantees about memory usage. Additionally, the alternative involves a lot of memory holding partially-added tuples which will be at worst no better than the list-accumulation method.
3) And the critical question: with this (theoretical) patch in place, do the test suites still pass? I asked Sergey this same question about his patch some days ago, but haven't seen an answer.
On 12/07/2013 01:52, Steven D'Aprano wrote:
This started out as a response to Ronald, but turned into a summary and pseudo-code implementation, so I've changed the subject line.
On 11/07/13 21:15, Ronald Oussoren wrote: [...]
That, and Paul's example about using += for something else than addition, pretty much kills the idea of using += in the implementation of sum() as that would break to much code.
Not quite :-)
After spending so much time explaining why I don't think sum *should* optimize the list case, I'm now going to suggest a way which (I think) sum *could* optimize the list case without changing behaviour.
[puts on Devil's Advocate cap]
What we have to give up is Sergey's hope to make sum fast for "everything". I don't think that is possible, but even if it is, we should not let the Perfect be the enemy of the Good. Can we make sum fast for lists, without changing behaviour? I think we can.
We know that list.__iadd__ is a fast, inplace version of __add__. We can't make that assumption about every class, not even list subclasses, but we don't have to speed up every class. Let's not be greedy: incremental improvements are better than no improvements.
sum already has special cases for ints and floats and strings (the string case is to just prohibit the use of sum). We could add a special case for lists: if the start parameter is an actual built-in list, not a subclass (because subclasses might override __iadd__) then we start with an accumulator list, and call __iadd__ (or extend) on that list to concatenate on it. As soon as we find an item which is not a built-in list, we drop out of the fast-code and fall back to the normal non-fast path. This doesn't make it "fast for everything", but it's still an optimization.
[snip] While you have your cap on, if you're going to special-case lists, then why not strings too (just passing them on to "".join())?
On 12 July 2013 02:12, MRAB <python@mrabarnett.plus.com> wrote:
While you have your cap on, if you're going to special-case lists, then why not strings too (just passing them on to "".join())?
And of course, that specific question was debated, and the decision taken to go with what we have now, when sum was first introduced. Someone who is arguing for this proposal needs to go back and research that decision, and confirm that the reasons discussed then no longer apply. I suspect many of them still do. Paul
On Jul 12, 2013 Paul Moore wrote:
On 12 July 2013 02:12, MRAB <python@mrabarnett.plus.com> wrote:
While you have your cap on, if you're going to special-case lists, then why not strings too (just passing them on to "".join())?
And of course, that specific question was debated, and the decision taken to go with what we have now, when sum was first introduced.
Someone who is arguing for this proposal needs to go back and research that decision, and confirm that the reasons discussed then no longer apply. I suspect many of them still do.
That's what Alex Martelli, author of sum(), initially did [1]:
for the simple reason that I special-case this -- when the first item is a PyBaseString_Type, I delegate to ''.join
So you can, kind of, say that sum was DESIGNED to have special cases from the very beginning. The problem appeared for mixed lists like: sum(["str1", "str2", SomeClass, "str3"]) Or: def myit(): yield "str1" yield "str2" yield SomeClass yield "str3" sum(myit()) ''.join() can't handle such cases despite SomeClass could be addable to string. So the problem is that you can't just silently delegate the argument to ''.join() because join() first exhausts entire sequence into its own temporary list, and then fails. So the sum code had to be somewhat like that: read entire sequence into a temporary list if argument is string: try ''.join() if join succeeded: prepend first element to result and return it if argument is unicode: try u''.join() if join succeeded: prepend first element to result and return it general code here That looked unnecessarily complex. To avoid the complications it was suggested to block strings a little, and point newbies in a better direction. Yes, it was known that sum is slow for some types from the beginning, but there were no attempts to make it faster for other types, I can't find any suggestions. So the sum() came in that way: slow for lists, blocked for strings. I'm not the first one to bother about sum() being unexpectedly slow [2], and I'm not the first one trying to find a solution [3]. But it looks like I'm the first to go as far as writing patches. Maybe, just maybe, if someone came up with my ideas 10 years ago, it was accepted from the beginning, and we won't be discussing it now. -- [1] http://mail.python.org/pipermail/python-dev/2003-April/034767.html [2] http://article.gmane.org/gmane.comp.python.general/441831 Paul Rubin @ 2006-01-12
A fast implementation would probably allocate the output list just once and then stream the values into place with a simple index. That's what I hoped "sum" would do, but instead it barfs with a type error.
[3] http://article.gmane.org/gmane.comp.python.general/658610 Alf P. Steinbach @ 2010-03-28
From a more practical point of view, the sum efficiency could be improved by doing the first addition using '+' and the rest using '+=', without changing the behavior.
On 12/07/2013 22:57, Sergey wrote:
On Jul 12, 2013 Paul Moore wrote:
On 12 July 2013 02:12, MRAB <python@mrabarnett.plus.com> wrote:
While you have your cap on, if you're going to special-case lists, then why not strings too (just passing them on to "".join())?
And of course, that specific question was debated, and the decision taken to go with what we have now, when sum was first introduced.
Someone who is arguing for this proposal needs to go back and research that decision, and confirm that the reasons discussed then no longer apply. I suspect many of them still do.
That's what Alex Martelli, author of sum(), initially did [1]:
for the simple reason that I special-case this -- when the first item is a PyBaseString_Type, I delegate to ''.join
So you can, kind of, say that sum was DESIGNED to have special cases from the very beginning.
The problem appeared for mixed lists like: sum(["str1", "str2", SomeClass, "str3"])
Or: def myit(): yield "str1" yield "str2" yield SomeClass yield "str3"
sum(myit())
''.join() can't handle such cases despite SomeClass could be addable to string.
So the problem is that you can't just silently delegate the argument to ''.join() because join() first exhausts entire sequence into its own temporary list, and then fails. So the sum code had to be somewhat like that: read entire sequence into a temporary list if argument is string: try ''.join() if join succeeded: prepend first element to result and return it if argument is unicode: try u''.join() if join succeeded: prepend first element to result and return it general code here
That looked unnecessarily complex. To avoid the complications it was suggested to block strings a little, and point newbies in a better direction.
Yes, it was known that sum is slow for some types from the beginning, but there were no attempts to make it faster for other types, I can't find any suggestions. So the sum() came in that way: slow for lists, blocked for strings.
I'm not the first one to bother about sum() being unexpectedly slow [2], and I'm not the first one trying to find a solution [3]. But it looks like I'm the first to go as far as writing patches.
Maybe, just maybe, if someone came up with my ideas 10 years ago, it was accepted from the beginning, and we won't be discussing it now.
What if it did this: 1. Get the first item, or the start value if provided (i.e. not None). 2. Ask the item for an 'accumulator' (item.__accum__()). 3. Add the first item to the accumulator. 4. Add the remaining items to the accumulator. 5. Ask the accumulator for the result. If there's no accumulator available (the "__accum__" method isn't implemented), then either fall back to the current behaviour or raise a TypeError like it currently does for strings.
On 12 July 2013 22:57, Sergey <sergemp@mail.ru> wrote:
That's what Alex Martelli, author of sum(), initially did [1]:
for the simple reason that I special-case this -- when the first item is a PyBaseString_Type, I delegate to ''.join
So you can, kind of, say that sum was DESIGNED to have special cases from the very beginning.
Thanks for the reference. That's the *original* implementation. So why does the current sum() not do this? You need to locate the justification for the removal of this special case, and explain why that reason no longer applies. My recollection (yes, I was round for those original discussions!) is that the key point is that "Guido said no". If I'm right, have you persuaded Guido to change his mind? To my knowledge he's not commented in this thread. Paul
On Jul 13, 2013 Paul Moore wrote:
So you can, kind of, say that sum was DESIGNED to have special cases from the very beginning.
Thanks for the reference. That's the *original* implementation. So why does the current sum() not do this? You need to locate the justification for the removal of this special case
As I already said the problem was in mixed list of strings and non-strings[1]. Later Guido suggested to check second argument and raise TypeError for strings [2], and Alex Martelli answered [3] "I like this!!!" and implemented it.
and explain why that reason no longer applies.
It still does, and it's still there — sum() still has a special case check for strings (and bytes, and bytearrays) rejecting it. But my patch does not use that approach, it does not call external join-like function, so it can correctly recover from mixed lists case. Meaning, my patch does not have the bug of initial sum(). -- [1]http://mail.python.org/pipermail/python-dev/2003-April/034854.html [2]http://mail.python.org/pipermail/python-dev/2003-April/034853.html [3]http://mail.python.org/pipermail/python-dev/2003-April/034855.html
On Jul 11, 2013, at 2:07, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Numpy arrays treat += differently from + in the sense that a += b coerces b to the same dtype as a and then adds in place whereas a + b uses Python style type promotion. This behaviour is by design and it is useful. It is also entirely appropriate (and not pathological) that someone would use sum() to add numpy arrays.
I forgot about this. I was positive on the first patch (+ first, then += for the rest) mainly because it speeds up sum for numpy. You probably won't _often_ sum arrays of different dtypes... But if you do, you certainly don't want the result to have the dtype resulting from just coercing start.dtype and iter[0].dtype. Of course this could be marked as a caveat for numpy--pass a scalar or array of the right dtype for start, and you get the right answer, after all. But if numpy is the only example of a use case for sum that's reasonable today and gets faster with the patch, I don't think the tradeoff is worth it. Since I've already asked the proponents for such examples multiple times and gotten none beyond the ones I tried to find myself, I'm assuming they are rare or unusual, so I'm -1 now.
On 11 July 2013 17:45, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 11, 2013, at 2:07, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Numpy arrays treat += differently from + in the sense that a += b coerces b to the same dtype as a and then adds in place whereas a + b uses Python style type promotion. This behaviour is by design and it is useful. It is also entirely appropriate (and not pathological) that someone would use sum() to add numpy arrays.
I forgot about this. I was positive on the first patch (+ first, then += for the rest) mainly because it speeds up sum for numpy.
Only by a constant factor. Summing numpy arrays with sum is O(N) either way. If someone wants to speed that up they can use numpy to do so i.e.: total = np.zeros(shape, dtype=float) for a in arrays: total += a is not significantly slower than sum(arrays) if the arrays themselves are large.
You probably won't _often_ sum arrays of different dtypes... But if you do, you certainly don't want the result to have the dtype resulting from just coercing start.dtype and iter[0].dtype.
It can easily happen: import numpy as np initial_velocity = np.array([1, 1, 1]) # Implicitly create an int array velocities = [initial_velocity] for n in range(1000): velocities.append(0.9 * velocities[-1]) # Append float arrays final_position = delta_t * sum(velocities) With the proposed patch all 1000 arrays after the first would count as zero in the final result so that the answer would be (delta_t * array([1, 1, 1])) instead of (delta_t * array([10.0, 10.0, 10.0]))
Of course this could be marked as a caveat for numpy--pass a scalar or array of the right dtype for start, and you get the right answer, after all.
I don't think it's acceptable to pass off a backward incompatible change of this nature in a minor release. It's the worst kind of change since there's no DeprecationWarning, no TypeError, just code that silently produces the wrong result. The change might be small in some cases (and so not immediately obvious) but then massive in others. Oscar
On 11 July 2013 18:19, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 11 July 2013 17:45, Andrew Barnert <abarnert@yahoo.com> wrote:
You probably won't _often_ sum arrays of different dtypes... But if you do, you certainly don't want the result to have the dtype resulting from just coercing start.dtype and iter[0].dtype.
It can easily happen:
import numpy as np initial_velocity = np.array([1, 1, 1]) # Implicitly create an int array velocities = [initial_velocity] for n in range(1000): velocities.append(0.9 * velocities[-1]) # Append float arrays final_position = delta_t * sum(velocities)
With the proposed patch all 1000 arrays after the first would count as zero in the final result so that the answer would be (delta_t * array([1, 1, 1])) instead of (delta_t * array([10.0, 10.0, 10.0]))
The points that have led to this point have pushed me from significantly in favour to significantly against. I'm not sure what counts as "significantly against", but without a proper deprecation cycle, PEP and whatnot (which I'm not sure would be worth the benefits of the change) I'm absolutely against this. We can't rush a semantic change for code that's in popular usage. Apologies, Sergey, but it seems I've left for the dark side.
On 07/09/2013 11:09 PM, Joshua Landau wrote:
On 9 July 2013 17:13, Steven D'Aprano <steve@pearwood.info> wrote:
On 09/07/13 19:35, Sergey wrote:
On Jul 5, 2013 Stefan Behnel wrote:
No, please. Using sum() on lists is not more than a hack that seems to be a cool idea but isn't. Seriously - what's the sum of lists? Intuitively, it makes no sense at all to say sum(lists).
It's the same as it is now. What else can you think about when you see: [1, 2, 3] + [4, 5] ?
Some of us think that using + for concatenation is an abuse of terminology, or at least an unfortunate choice of operator, and are wary of anything that encourages that terminology.
Nevertheless, you are right, in Python 3 both + and sum of lists is well-defined. At the moment sum is defined in terms of __add__. You want to change it to be defined in terms of __iadd__. That is a semantic change that needs to be considered carefully, it is not just an optimization.
I agree it's not totally backward-compatible, but AFAICT that's only for broken code. __iadd__ should always just be a faster, in-place __add__ and so this change should never cause problems in properly written code. That makes it anything but a semantic change. It's the same way people discuss the order of __hash__ calls on updates to code but no-one calls it a *semantics* change.
Currently, sum() does not modify its arguments. You (or whoever) are suggesting that it should modify one of them. That makes it a semantic change, and a bad one. -1 -- ~Ethan~
On 11/07/13 02:50, Ethan Furman wrote:
Currently, sum() does not modify its arguments.
You (or whoever) are suggesting that it should modify one of them.
That makes it a semantic change, and a bad one.
-1
Actually, Sergey's suggestion is a bit more clever than that. I haven't tested his C version, but the intention of his pure-Python demo code is to make a temporary list, modify the temporary list in place for speed, and then convert to whatever type is needed. That will avoid modifying any of the arguments[1]. So credit to Sergey for avoiding that trap. [1] Except possibly in truly pathological cases which I for one don't care about. -- Steven
On Jul 10, 2013, at 10:20, Steven D'Aprano <steve@pearwood.info> wrote:
On 11/07/13 02:50, Ethan Furman wrote:
Currently, sum() does not modify its arguments.
You (or whoever) are suggesting that it should modify one of them.
That makes it a semantic change, and a bad one.
-1
Actually, Sergey's suggestion is a bit more clever than that. I haven't tested his C version, but the intention of his pure-Python demo code is to make a temporary list, modify the temporary list in place for speed, and then convert to whatever type is needed. That will avoid modifying any of the arguments[1]. So credit to Sergey for avoiding that trap.
Actually, he has two versions. The first does a + once and then a += repeatedly on the result. This solves the problem neatly (except with empty iterables, but that's trivial to fix, and I think his C code actually doesn't have that problem...). There's no overhead, it automatically falls back to __add__ if __iadd__ is missing, and the only possible semantic differences are for types that are already broken. The second makes a list of the argument (which means copying it if it's already a list), then calls extend repeatedly on the result, then converts back. This doesn't solve the problem in many cases, does the wrong thing in many others, and always adds overhead. And that's exactly why I think it's worth splitting into separate pieces. It's very easy for people to see problems with the second version and wrongly assume they also apply to the first (and the way he presents and argues for his ideas doesn't help). As far as I know, nobody has yet found any problem with the first version, except for the fact that it would encourage people to use sum on lists. I don't think that's a serious problem--the docs already say not to do it--and if it's a useful optimization for any number-like types, I think it's worth having. It's the second version, together with all of the attempts to make it fully generally for any concatenable type--or, alternatively, to argue that only builtin concatenable types matter--that I have a problem with.
On 10 July 2013 21:14, Andrew Barnert <abarnert@yahoo.com> wrote:
[Sergey] has two versions.
The first does a + once and then a += repeatedly on the result. This solves the problem neatly (except with empty iterables, but that's trivial to fix, and I think his C code actually doesn't have that problem...). There's no overhead, it automatically falls back to __add__ if __iadd__ is missing, and the only possible semantic differences are for types that are already broken.
The second makes a list of the argument (which means copying it if it's already a list), then calls extend repeatedly on the result, then converts back. This doesn't solve the problem in many cases, does the wrong thing in many others, and always adds overhead.
And that's exactly why I think it's worth splitting into separate pieces. It's very easy for people to see problems with the second version and wrongly assume they also apply to the first (and the way he presents and argues for his ideas doesn't help).
As far as I know, nobody has yet found any problem with the first version, except for the fact that it would encourage people to use sum on lists. I don't think that's a serious problem--the docs already say not to do it--and if it's a useful optimization for any number-like types, I think it's worth having.
It's the second version, together with all of the attempts to make it fully generally for any concatenable type--or, alternatively, to argue that only builtin concatenable types matter--that I have a problem with.
If Sergey doesn't do this separation, would it be fine if I did it? I like the idea for __iadd__ sum, and I don't want Sergey block progress on the issue.
On 07/10/2013 01:45 PM, Joshua Landau wrote:
On 10 July 2013 21:14, Andrew Barnert <abarnert@yahoo.com> wrote:
[Sergey] has two versions.
The first does a + once and then a += repeatedly on the result. This solves the problem neatly (except with empty iterables, but that's trivial to fix, and I think his C code actually doesn't have that problem...). There's no overhead, it automatically falls back to __add__ if __iadd__ is missing, and the only possible semantic differences are for types that are already broken.
The second makes a list of the argument (which means copying it if it's already a list), then calls extend repeatedly on the result, then converts back. This doesn't solve the problem in many cases, does the wrong thing in many others, and always adds overhead.
And that's exactly why I think it's worth splitting into separate pieces. It's very easy for people to see problems with the second version and wrongly assume they also apply to the first (and the way he presents and argues for his ideas doesn't help).
As far as I know, nobody has yet found any problem with the first version, except for the fact that it would encourage people to use sum on lists. I don't think that's a serious problem--the docs already say not to do it--and if it's a useful optimization for any number-like types, I think it's worth having.
It's the second version, together with all of the attempts to make it fully generally for any concatenable type--or, alternatively, to argue that only builtin concatenable types matter--that I have a problem with.
If Sergey doesn't do this separation, would it be fine if I did it? I like the idea for __iadd__ sum, and I don't want Sergey block progress on the issue.
Make a patch and add it to the tracker. A word of warning/advice: keep the __add__ fallback or it won't fly. __iadd__ is /optional/. If the new sum() suddenly stops working on classes it worked fine with before, it will not be accepted. Mind you, I haven't checked if PyNumber_InPlaceAdd will fall back to PyNumber_Add on its own. -- ~Ethan~
On 10 July 2013 22:03, Ethan Furman <ethan@stoneleaf.us> wrote:
Make a patch and add it to the tracker.
Gah! Now I need to learn C... :P
A word of warning/advice: keep the __add__ fallback or it won't fly. __iadd__ is /optional/. If the new sum() suddenly stops working on classes it worked fine with before, it will not be accepted. Mind you, I haven't checked if PyNumber_InPlaceAdd will fall back to PyNumber_Add on its own.
It does.
On 07/10/2013 02:27 PM, Joshua Landau wrote:
On 10 July 2013 22:03, Ethan Furman <ethan@stoneleaf.us> wrote:
Make a patch and add it to the tracker.
Gah! Now I need to learn C... :P
Heh. I'm in a similar boat trying to update the json module. :/ -- ~Ethan~
On 10 July 2013 22:03, Ethan Furman <ethan@stoneleaf.us> wrote:
On 07/10/2013 01:45 PM, Joshua Landau wrote:
On 10 July 2013 21:14, Andrew Barnert <abarnert@yahoo.com> wrote:
[Sergey] has two versions.
The first does a + once and then a += repeatedly on the result. This solves the problem neatly (except with empty iterables, but that's trivial to fix, and I think his C code actually doesn't have that problem...). There's no overhead, it automatically falls back to __add__ if __iadd__ is missing, and the only possible semantic differences are for types that are already broken.
The second makes a list of the argument (which means copying it if it's already a list), then calls extend repeatedly on the result, then converts back. This doesn't solve the problem in many cases, does the wrong thing in many others, and always adds overhead.
And that's exactly why I think it's worth splitting into separate pieces. It's very easy for people to see problems with the second version and wrongly assume they also apply to the first (and the way he presents and argues for his ideas doesn't help).
As far as I know, nobody has yet found any problem with the first version, except for the fact that it would encourage people to use sum on lists. I don't think that's a serious problem--the docs already say not to do it--and if it's a useful optimization for any number-like types, I think it's worth having.
It's the second version, together with all of the attempts to make it fully generally for any concatenable type--or, alternatively, to argue that only builtin concatenable types matter--that I have a problem with.
If Sergey doesn't do this separation, would it be fine if I did it? I like the idea for __iadd__ sum, and I don't want Sergey block progress on the issue.
Make a patch and add it to the tracker.
Actually, there is already a bug on the tracker at http://bugs.python.org/issue18305 and the response was "discuss it on Python-Ideas". Hence, I want to discuss it on Python ideas. So should I spawn it off onto a seperate thread about *just* the __iadd__ enhancement?
On 07/10/2013 02:36 PM, Joshua Landau wrote:
On 10 July 2013 22:03, Ethan Furman <ethan@stoneleaf.us> wrote:
On 07/10/2013 01:45 PM, Joshua Landau wrote:
On 10 July 2013 21:14, Andrew Barnert <abarnert@yahoo.com> wrote:
[Sergey] has two versions.
The first does a + once and then a += repeatedly on the result. This solves the problem neatly (except with empty iterables, but that's trivial to fix, and I think his C code actually doesn't have that problem...). There's no overhead, it automatically falls back to __add__ if __iadd__ is missing, and the only possible semantic differences are for types that are already broken.
The second makes a list of the argument (which means copying it if it's already a list), then calls extend repeatedly on the result, then converts back. This doesn't solve the problem in many cases, does the wrong thing in many others, and always adds overhead.
And that's exactly why I think it's worth splitting into separate pieces. It's very easy for people to see problems with the second version and wrongly assume they also apply to the first (and the way he presents and argues for his ideas doesn't help).
As far as I know, nobody has yet found any problem with the first version, except for the fact that it would encourage people to use sum on lists. I don't think that's a serious problem--the docs already say not to do it--and if it's a useful optimization for any number-like types, I think it's worth having.
It's the second version, together with all of the attempts to make it fully generally for any concatenable type--or, alternatively, to argue that only builtin concatenable types matter--that I have a problem with.
If Sergey doesn't do this separation, would it be fine if I did it? I like the idea for __iadd__ sum, and I don't want Sergey block progress on the issue.
Make a patch and add it to the tracker.
Actually, there is already a bug on the tracker at http://bugs.python.org/issue18305 and the response was "discuss it on Python-Ideas".
Hence, I want to discuss it on Python ideas. So should I spawn it off onto a seperate thread about *just* the __iadd__ enhancement?
A separate thread on Python Ideas is probably appropriate, but you can add your __iadd__ only patch to that issue. I would think it would have a better chance of acceptance since it would be a smaller change. -- ~Ethan~
On 10 July 2013 22:52, Ethan Furman <ethan@stoneleaf.us> wrote:
On 07/10/2013 02:36 PM, Joshua Landau wrote:
On 10 July 2013 22:03, Ethan Furman <ethan@stoneleaf.us> wrote:
On 07/10/2013 01:45 PM, Joshua Landau wrote:
If Sergey doesn't do this separation, would it be fine if I did it? I like the idea for __iadd__ sum, and I don't want Sergey block progress on the issue.
Make a patch and add it to the tracker.
Actually, there is already a bug on the tracker at http://bugs.python.org/issue18305 and the response was "discuss it on Python-Ideas".
Hence, I want to discuss it on Python ideas. So should I spawn it off onto a seperate thread about *just* the __iadd__ enhancement?
A separate thread on Python Ideas is probably appropriate, but you can add your __iadd__ only patch to that issue. I would think it would have a better chance of acceptance since it would be a smaller change.
A cursory glance yields that that is the original patch.
On Jul 10, 2013, at 15:02, Joshua Landau <joshua@landau.ws> wrote:
On 10 July 2013 22:52, Ethan Furman <ethan@stoneleaf.us> wrote:
On 07/10/2013 02:36 PM, Joshua Landau wrote:
On 10 July 2013 22:03, Ethan Furman <ethan@stoneleaf.us> wrote:
On 07/10/2013 01:45 PM, Joshua Landau wrote:
If Sergey doesn't do this separation, would it be fine if I did it? I like the idea for __iadd__ sum, and I don't want Sergey block progress on the issue.
Make a patch and add it to the tracker.
Actually, there is already a bug on the tracker at http://bugs.python.org/issue18305 and the response was "discuss it on Python-Ideas".
Hence, I want to discuss it on Python ideas. So should I spawn it off onto a seperate thread about *just* the __iadd__ enhancement?
A separate thread on Python Ideas is probably appropriate, but you can add your __iadd__ only patch to that issue. I would think it would have a better chance of acceptance since it would be a smaller change.
A cursory glance yields that that is the original patch.
Exactly. I believe Sergey's first patch already gets the __iadd__ thing exactly right. Of course it's worth reviewing the patch, and testing it, and writing a pure Python version that other implementations can use, and discussing whether there should be any doc changes, and finding cases that are clearly summing number-like things that benefit (seriously, why has nobody who's +1 on this done the simple test with numpy.matrix yet?) so nobody can complain that it's useless, ... But you don't need to write C to do any of that.
On Jul 10 2013, Steven D'Aprano wrote:
No, please. Using sum() on lists is not more than a hack that seems to be a cool idea but isn't. Seriously - what's the sum of lists? Intuitively, it makes no sense at all to say sum(lists).
It's the same as it is now. What else can you think about when you see: [1, 2, 3] + [4, 5] ?
Some of us think that using + for concatenation is an abuse of terminology, or at least an unfortunate choice of operator, and are wary of anything that encourages that terminology.
Nevertheless, you are right, in Python 3 both + and sum of lists is well-defined. At the moment sum is defined in terms of __add__. You want to change it to be defined in terms of __iadd__. That is a semantic change that needs to be considered carefully, it is not just an optimization.
Yes, I understand that. On the other hand is this documented anywhere? Does anything says that sum() actually uses __add__, not __iadd__ or something else... I'm not trying to say that we can change that freely. I'm just trying to find out how tough could be such a change.
I have been very careful to say that I am only a little bit against this idea, -0 not -1. I am uncomfortable about changing the semantics to use __iadd__ instead of __add__, because I expect that this will change the behaviour of sum() for non-builtins. I worry about increased complexity making maintenance harder for no good reason. It's the "for no good reason" that concerns me: you could answer some of my objections if you showed:
- real code written by people who sum() large (more than 100) numbers of lists;
That would be hard. You're asking to find someone using a bad function in exactly the case where it's bad. :) Even if nobody uses a bad function that does not mean it should stay bad.
- real code with comments like "this is a work-around for sum() being slow on lists";
Even I probably wouldn't do that, I would just silently use another function.
- bug reports or other public complaints by people (other than you) complaining that sum(lists) is slow;
This is easier. sum() is constantly suggested as an option to list additions in those questions I could find, and often someone comes later and says "be careful, it may be slow". So this is kind of common attempt. Examples [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]
or similar. That would prove that people do call sum on lists.
I can't say they do, but they definitely try.
Earlier in this discussion, you posted benchmarks for the patched sum using Python 2.7. Would you be willing to do it again for 3.3?
Well, I posted them for Python 2.7 because I expected that patch would not introduce any changes, so it could be applied for 2.7 too. The patch itself works for both 2.7 and 3.3. Python 3.3.2 + fastsum-special-tuplesandlists.patch [11] Before patch: list compr: 14.5 chain: 10.5 chain.from_iterable: 7.44 aug.assignment: 5.8 regular extend: 9.34 optimized extend: 5.59 sum: infinite After patch: list compr: 14.5 chain: 10.5 chain.from_iterable: 7.44 aug.assignment: 5.79 regular extend: 9.47 optimized extend: 5.53 sum: 2.58 I used the same tests as you [12] except a minor bug fixed: you did "l = []" only at start, while I do it for every iteration. I hope this patch would not introduce any changes. :)
And confirm that the Python test suite continues to pass?
Yes, they do. But they don't test many cases. And they definitely test nothing like: http://bugs.python.org/issue18305#msg192919 -- [1] http://stackoverflow.com/questions/406121/flattening-a-shallow-list-in-pytho... [2] http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-... [3] http://stackoverflow.com/questions/3021641/concatenation-of-many-lists-in-py... [4] http://stackoverflow.com/questions/7895449/merging-a-list-of-lists [5] http://stackoverflow.com/questions/10424219/combining-lists-into-one [6] http://stackoverflow.com/questions/11331908/how-to-use-reduce-with-list-of-l... [7] http://stackoverflow.com/questions/17142101/concatenating-sublists-python [8] http://stackoverflow.com/questions/716477/join-list-of-lists-in-python 2009-04-04 CMS x = [["a","b"], ["c"]] result = sum(x, []) 2010-09-04 habnabit O(n^2) complexity yaaaaaay. [9] http://article.gmane.org/gmane.comp.python.general/658537 The mildly surprising part of sum() is that is does add vs. add-in- place, which leads to O(N) vs. O(1) for the inner loop calls, for certain data structures, notably lists, even though none of the intermediate results get used by the caller. For lists, you could make a more efficient variant of sum() that clones the start value and does add-in-place. [10] http://article.gmane.org/gmane.comp.python.general/441831
A fast implementation would probably allocate the output list just once and then stream the values into place with a simple index. That's what I hoped "sum" would do, but instead it barfs with a type error.
[11] http://bugs.python.org/file30897/fastsum-special-tuplesandlists.patch [12] Two test numbers are before and after patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*100000" \ "[i for l in x for i in l]" 100 loops, best of 3: 14.5 msec per loop 100 loops, best of 3: 14.5 msec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*100000" \ --setup="from itertools import chain" \ "list(chain(*x))" 100 loops, best of 3: 10.5 msec per loop 100 loops, best of 3: 10.5 msec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*100000" \ --setup="from itertools import chain" \ "list(chain.from_iterable(x))" 100 loops, best of 3: 7.44 msec per loop 100 loops, best of 3: 7.44 msec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*100000" \ "l = []" \ "for i in x: l += i" 100 loops, best of 3: 5.8 msec per loop 100 loops, best of 3: 5.79 msec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*100000" \ "l = []" \ "for i in x: l.extend(i)" 100 loops, best of 3: 9.34 msec per loop 100 loops, best of 3: 9.47 msec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*100000" \ "l = []" \ "extend=l.extend" \ "for i in x: extend(i)" 100 loops, best of 3: 5.59 msec per loop 100 loops, best of 3: 5.53 msec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*100000" \ "sum(x,[])" infinite 100 loops, best of 3: 2.58 msec per loop
Maybe what you actually want would be list.extend to accept *args?
l = [] l.extend(*([[1,2,3]]*1000000))
Or something similar. I think "sum" is about mathematical sums. This would be list concatenation and not building sums. After all, what does addition in the context of lists even mean? In the context of sets it might be meaningful, but for lists?
On 07/06/2013 04:41 PM, Joshua Landau wrote:
On 6 July 2013 21:17, Mathias Panzenböck<grosser.meister.morti@gmx.net> wrote:
After all, what does addition in the context of lists even mean? What it currently does.
What is everyone so confused about?
They aren't confused. It just isn't a clear cut issue. Ideally... And being "ideal' isn't practical in this case because it will need too many changes. Take the following example. def add_to_values(vs, v): return [n + v for n in vs] Now what do you suppose this should do? Well it it depends on what vs and v are. It might add a value to each item in a list of values, it might add a value to each byte in a bytes string, it might concatenate a string to each string in a list, or it might join a sequence to each sequence in a list. Sounds reasonable doesn't it? Now consider that in some companies, programmers are required to take great care to be sure that the routines that they write can't do the wrong thing with lots of testing to back that up. That simple routine "ideally" should be dead simple, but now it requires some added care to be sure it can't do the wrong thing. Which could also slow it down. :/ Generalised routines are very nice and can save a lot of work, but it is easier to add behaviours than it is to limit unwanted behaviours. The problem here is that the different behaviours use a similar operator at a very low level. But, can we change this? Probably not any time soon. It would mean changing __add__, __iadd__, __mul__, __rmul__, __imul__, and possibly a few others for a lot of different objects to get a clean separation of the behaviours. And we would need new symbols and method names to replace those. So the question becomes how we be more specific in a case like this and avoid the extra conditional expression. This is one way...
def add_value_to_many(value, many): ... return [int.__add__(x, value) for x in many] ... add_value_to_many(4, [3, 6, 0]) [7, 10, 4] add_value_to_many("abc", "efg") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in add_value_to_many File "<stdin>", line 2, in <listcomp> TypeError: descriptor '__add__' requires a 'int' object but received a 'str'
It rejects the unwanted cases without the test. But different operators would have been a bit nicer. Cheers, Ron
On 07/07/13 12:14, Ron Adam wrote:
Take the following example.
def add_to_values(vs, v): return [n + v for n in vs]
I don't actually see the point of this example, but I'm willing to bear with you.
Now what do you suppose this should do?
Well it it depends on what vs and v are. It might add a value to each item in a list of values, it might add a value to each byte in a bytes string, it might concatenate a string to each string in a list, or it might join a sequence to each sequence in a list. Sounds reasonable doesn't it?
Now consider that in some companies, programmers are required to take great care to be sure that the routines that they write can't do the wrong thing with lots of testing to back that up.
That simple routine "ideally" should be dead simple, but now it requires some added care to be sure it can't do the wrong thing. Which could also slow it down. :/
Can you give an actual example of the above add_to_values function doing the wrong thing?
Generalised routines are very nice and can save a lot of work, but it is easier to add behaviours than it is to limit unwanted behaviours. The problem here is that the different behaviours use a similar operator at a very low level.
But, can we change this? Probably not any time soon. It would mean changing __add__, __iadd__, __mul__, __rmul__, __imul__, and possibly a few others for a lot of different objects to get a clean separation of the behaviours. And we would need new symbols and method names to replace those.
So the question becomes how we be more specific in a case like this and avoid the extra conditional expression. This is one way...
def add_value_to_many(value, many): ... return [int.__add__(x, value) for x in many] ... add_value_to_many(4, [3, 6, 0]) [7, 10, 4] add_value_to_many("abc", "efg") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in add_value_to_many File "<stdin>", line 2, in <listcomp> TypeError: descriptor '__add__' requires a 'int' object but received a 'str'
It rejects the unwanted cases without the test. But different operators would have been a bit nicer.
This doesn't make sense to me. Above, you just said that "concatenate a string to each string in a list" was reasonable, and here you prohibit it. -- Steven
On 07/06/2013 09:32 PM, Steven D'Aprano wrote:
On 07/07/13 12:14, Ron Adam wrote:
Take the following example.
def add_to_values(vs, v): return [n + v for n in vs]
I don't actually see the point of this example, but I'm willing to bear with you.
Yes, It's over simplified to highlight a concept rather than have a specific programming problem to solve.
Now what do you suppose this should do?
Well it it depends on what vs and v are. It might add a value to each item in a list of values, it might add a value to each byte in a bytes string, it might concatenate a string to each string in a list, or it might join a sequence to each sequence in a list. Sounds reasonable doesn't it?
Now consider that in some companies, programmers are required to take great care to be sure that the routines that they write can't do the wrong thing with lots of testing to back that up.
That simple routine "ideally" should be dead simple, but now it requires some added care to be sure it can't do the wrong thing. Which could also slow it down. :/
Can you give an actual example of the above add_to_values function doing the wrong thing?
It was kept simple to demonstrate a concept. What I was trying to demonstrate is it has to do with the context it's used in, and not something wrong with the example it self. It will do what it it says it will do. Sometimes we want very general behaviour, and sometime we don't want that. Both are good, but it should be easy to do both in a simple way. That is the point I was trying to make.
Generalised routines are very nice and can save a lot of work, but it is easier to add behaviours than it is to limit unwanted behaviours. The problem here is that the different behaviours use a similar operator at a very low level.
But, can we change this? Probably not any time soon. It would mean changing __add__, __iadd__, __mul__, __rmul__, __imul__, and possibly a few others for a lot of different objects to get a clean separation of the behaviours. And we would need new symbols and method names to replace those.
So the question becomes how we be more specific in a case like this and avoid the extra conditional expression. This is one way...
def add_value_to_many(value, many): ... return [int.__add__(x, value) for x in many] ... add_value_to_many(4, [3, 6, 0]) [7, 10, 4] add_value_to_many("abc", "efg") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in add_value_to_many File "<stdin>", line 2, in <listcomp> TypeError: descriptor '__add__' requires a 'int' object but received a 'str'
It rejects the unwanted cases without the test. But different operators would have been a bit nicer.
This doesn't make sense to me. Above, you just said that "concatenate a string to each string in a list" was reasonable, and here you prohibit it.
You missed the question mark right after "reasonable". Yes, it's an example of how to do that if you need to do it. Otherwise you need to add a isinstance() check or something else to get that. Cheers, Ron
On 7 July 2013 03:14, Ron Adam <ron3200@gmail.com> wrote:
On 07/06/2013 04:41 PM, Joshua Landau wrote:
On 6 July 2013 21:17, Mathias Panzenböck<grosser.meister.morti@gmx.net> wrote:
After all, what does addition in the context of lists even mean?
What it currently does.
What is everyone so confused about?
They aren't confused. It just isn't a clear cut issue.
Ideally... And being "ideal' isn't practical in this case because it will need too many changes.
Take the following example.
def add_to_values(vs, v): return [n + v for n in vs]
Now what do you suppose this should do?
Well it it depends on what vs and v are. It might add a value to each item in a list of values, it might add a value to each byte in a bytes string, it might concatenate a string to each string in a list, or it might join a sequence to each sequence in a list. Sounds reasonable doesn't it?
Now consider that in some companies, programmers are required to take great care to be sure that the routines that they write can't do the wrong thing with lots of testing to back that up.
So why are they using a duck-typed language? It would imply that they would never be allowed to use operators, call functions they are given or... do anything. It seems hard to work like that.
On 07/06/2013 09:35 PM, Joshua Landau wrote:
Now consider that in some companies, programmers are required to take great care to be sure that the routines that they write can't do the wrong thing with lots of testing to back that up.
So why are they using a duck-typed language? It would imply that they would never be allowed to use operators, call functions they are given or... do anything. It seems hard to work like that.
That is the other extreme, sometime you want both. Cheers, Ron
participants (26)
-
Alexander Belopolsky -
Andrew Barnert -
Ben Finney -
Bruce Leban -
Chris Kaynor -
David Mertz -
Ethan Furman -
Haoyi Li -
Joshua Landau -
Joshua Landau -
Juancarlo Añez -
Mathias Panzenböck -
MRAB -
Nick Coghlan -
Oscar Benjamin -
Paul Moore -
Ron Adam -
Ronald Oussoren -
Sergey -
Serhiy Storchaka -
Stefan Behnel -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy -
Victor Stinner -
Vito De Tullio