Alternative spelling for list.append()
[by request I've made new subject and summary of proposal] The idea is to introduce new syntax for the list.append() method. Syntax: Variant 1. Use special case of index, namely omitted index: mylist[] = item Examples: mylist = [1, 2, 3] --> [1, 2, 3] mylist[] = x --> [1, 2, 3, x] mylist[] = [foo, bar] --> [1, 2, 3, [foo, bar]] instead of current: mylist = [1, 2, 3] mylist.append(x) mylist.append([foo, bar]) Variant 2. Use one of the augmented assignment operators. For example using ^= operator: mylist = [1, 2, 3] mylist ^= x mylist ^= [foo, bar] For example using >>= operator: mylist = [1, 2, 3] mylist >>= x mylist >>= [foo, bar] Other operators may be considerd as well. Motivation ----------- 1. Assignment form reduces text amount in statements and makes the right-hand part, namely the item, clean of additional brackets, which is improtant especially by more complex items which can also contain brackets or quotes. For example: mylist.append([[foo, bar], [] ]) Could be written as: mylist[] = [[foo, bar], [] ] Which preserves the original item form and has generally more balanced look. 2. Method form has one general issue, especially by longer variable names. Example from https://docs.python.org/3/tutorial/datastructures.html ... for row in matrix: transposed_row.append (row[i]) transposed.append (transposed_row) It becomes hard to read because of lack of spacing between variable names and list names. With the new syntax it could be written: for row in matrix: transposed_row[] = row[i] transposed[] = transposed_row 3. Item appending is very frequent operation. In current syntax, extend() method has dedicated syntax. mylist1 += mylist2 One of important aspects of proposal is that it should discourage usage of += for appending an item. Namely the idea is to discourage this form : mylist += [item] Because it is clunky and may cause additional confusion. E.g. adding brackets is often used and understood as operation of increasing dimension of a list - so in case of variable (not literal) the immediate reaction to this form is that there might be intention to increase the dimension of list, and not just create a list from say, an integer. But of course, increasing the dimension may be also the intention, so this construct causes extra brain load. Also in current syntax it is possible to add item to a dictionary with assignment statement: mydict[key] = item Despite it is also possible via update() method. In both cases assignment form gained popularity and used very often.
On Sun, Jun 17, 2018, 10:01 AM Mikhail V <mikhailwas@gmail.com> wrote:
The idea is to introduce new syntax for the list.append() method.
While you have summarized your proposal, you haven't included a summary of the criticism. Also, one thing that's very common for proposals to change syntax and create new uses for operators is to show real code from major libraries that benefit from the change.
On Mon, Jun 18, 2018 at 3:01 AM, Mikhail V <mikhailwas@gmail.com> wrote:
Creation of syntax cannot be done for just one type. So what would this mean (a) for other core data types, and (b) in the protocols? What dunder will be called, and with what arguments? For example: class Foo: def __setitem__(self, key, val): print("Setting", key, "to", val) x = Foo() x[] = 1 What should be printed? Will it come through __setitem__ or are you requiring a completely different dunder method? Regardless, I am still a strong -1 on introducing another way to spell list.append(). ChrisA
I understand the view from the poster, most basic list operations are using brackets, ie reading and writing with [], delete with del L[], why not append ? And being used extensively, that brackets are annoying. And yes, += [] is more "concise" than .append() so some people would think it's more clear because "it's smaller" (they'd be wrong as the OP mentioned). But it would break the "There is only one Obvious way to do it" principle. People would take time to figure out "okay, what should I write ? What's the most Pythonic ?" If someone wants to use their own list class, it's doable with the current syntax : class List(list): def __lshift__(self, x): self.append(x) return self a = List([1,2,7,2]) a = a << 1 << 5 a <<= 0 2018-06-18 1:36 GMT+02:00 Clint Hepner <clint.hepner@gmail.com<mailto:clint.hepner@gmail.com>>:
That’s false. @ was added solely for matrix multiplication.
Regardless, I am still a strong -1 on introducing another way to spell list.append().
-1 as well. Has any one but the proposer shown any support for it yet? — Clint _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Yes - maybe a proposal to have the MutableSequence protocol to define "<<" as "append" would be something with more traction than the original proposal here. No chanegs needed to to the language core, and a = [1,2, 3] a <<= 4 resulting in a == [1, 2, 3, 4] is quite readable, syntactically valid, unambiguous (and can be implemented in two LoC for a new Sequence class). I know that at least Brython used the "<<=" enhanced assignement operator to manipulate HTML DOM, and it often results in quite compact code when compared with javascript-equivalente manipulations. On Mon, 18 Jun 2018 at 01:32, Robert Vanden Eynde <robertvandeneynde@hotmail.com> wrote:
On Mon, Jun 18, 2018 at 12:56 PM Mikhail V <mikhailwas@gmail.com> wrote:
Numpy arrays have also append() and insert() methods,
In [2]: np.arange(1).append(2) AttributeError: 'numpy.ndarray' object has no attribute 'append' In [3]: np.arange(1).insert AttributeError: 'numpy.ndarray' object has no attribute 'insert' So one syntax generalization possible is towards insert() method.
Why would you even want to encourage inserting into a list? It's slow and should be *discouraged*. Although IMO it could be made into nice syntax only by introducing
Indeed, changing the syntax to add a special meaning to caret operator inside of index assignment is a *major* change. Not worthwhile for such dubious benefit. On Mon, Jun 18, 2018 at 1:08 PM Joao S. O. Bueno <jsbueno@python.org.br> wrote:
MutableSequence protocol to define "<<" as "append"
No one has demonstrated with any realistic examples why code would look better with ``a <<= b`` instead of ``a.append(b)``. Perhaps a language in its infancy could toy with new spellings, but Python is old now and should be more cautious with change. I know that at least Brython used the "<<=" enhanced assignement
operator to manipulate HTML DOM, and it often results in quite compact code when compared with javascript-equivalente manipulations.
That sounds great for a dedicated HTML DOM manipulation tool, but not for the core list type. On Sun, Jun 17, 2018 at 6:12 PM Chris Angelico <rosuav@gmail.com> wrote:
The fact that signed 2's complement integers don't support negative zero does cause some problems for both indexing and slicing. However, to me it seems this is a fundamental human problem with zero. Thus the ever-present off-by-one error. I can't think of a solution that doesn't cause more harm than good.
On Mon, Jun 18, 2018 at 11:43 PM, Michael Selik <mike@selik.org> wrote:
https://docs.scipy.org/doc/numpy/reference/generated/numpy.append.html
I don't - but I have seen it in real projects and modules. and ther is Deque.appendleft()
So you have 2 separate inquiries in one: explaining why and where is the example. Why it is so - I have tried to explain several times and also in the summary, with a small example (see first post in this thread - 'transposed_row' example). As for examples - below is one example from 'pyparsing' module. But I don't advise to think about "why" but rather just relax and try to 'traverse' the code back and forth several times. (and sometimes it's better to treat things just as an advice if you doubt you can figure it out by yourself - that's not adressed to you but just general life observation) ---------------- with <<= out = [] NL = '\n' out <<= indent + _ustr(self.asList()) if full: if self.haskeys(): items = sorted((str(k), v) for k,v in self.items()) for k,v in items: if out: out <<= NL out <<= "%s%s- %s: " % (indent,(' '*depth), k) if isinstance (v,ParseResults): if v: out <<= v.dump(indent,depth+1) else: out <<= _ustr(v) else: out <<= repr(v) ----------------- with .append() out = [] NL = '\n' out.append( indent+_ustr(self.asList()) ) if full: if self.haskeys(): items = sorted((str(k), v) for k,v in self.items()) for k,v in items: if out: out.append(NL) out.append( "%s%s- %s: " % (indent,(' '*depth), k) ) if isinstance(v,ParseResults): if v: out.append( v.dump(indent,depth+1) ) else: out.append(_ustr(v)) else: out.append(repr(v))
On Mon, Jun 18, 2018 at 2:55 PM Mikhail V <mikhailwas@gmail.com> wrote:
Perhaps NumPy chose not to provide an append method to discourage repeated appends, since it's terribly slow. It's a good design philosophy: make inefficient code look appropriately ugly.
That's interesting advice. I contrast, I find it's best to aggressively ask "Why?" and to rewrite the code in order to fully understand it. I think I found the chunk of code you're referring to. https://github.com/pyparsing/pyparsing/blob/master/pyparsing.py#L848 First, I'll note that ``dump`` usually indicates dumping to a file, while ``dumps`` is dumping to a str. Regardless, I think this ``dump`` function could be improved a bit, and a revision reduces the benefit of a dedicated ``append`` operator. if not full: return indent + _ustr(self.asList()) lines = [_ustr(self.asList())] if self.haskeys(): fmt = ' ' * depth + '- %s: %s' for k, v in sorted((str(k), v) for k, v in self.items()): if isinstance(v, ParseResults): if v: s = v.dump(indent, depth + 1) else: s = _ustr(v) else: s = repr(v) lines.append(fmt % (k, s)) elif any(isinstance(v, ParseResults) for v in self): for i, v in enumerate(self): lines.append(' ' * depth + '[%d]:' % i) if isinstance(v, ParseResults): s = v.dump(indent, depth + 1) else: s = _ustr(v) lines.append(' ' * (depth + 1) + s) return ('\n' + indent).join(lines)
On Mon, Jun 18, 2018, 6:59 PM Michael Selik <mike@selik.org> wrote:
On the 2nd thought, that nested if is ugly. Much better as if not isinstance(v, ParseResults): s = repr(v) elif v: s = v.dump(indent, depth + 1) else: s = _ustr(v) lines.append(fmt % (k, s)) It might seem like this is going off topic. What I'm trying to demonstrate is that cases where an append operator might help are really in need of more thorough revision, not just a little sugar.
On Mon, Jun 18, 2018 at 05:07:45PM -0300, Joao S. O. Bueno wrote:
Not as readable as a.append(4), which works today and doesn't require the user to memorise the special case that <<= works on lists but (presumably) << doesn't. Or perhaps it would. What would a bare << mean for lists? b = a << 4 could mean the same as b = a + [4] but we already have a spelling for that, do we really need another one? -- Steve
On Mon, Jun 18, 2018 at 9:36 AM, Clint Hepner <clint.hepner@gmail.com> wrote:
Ah, confusing bit of language there. The @ operator was created for the benefit of a small number of types (not just one, I think), but it MUST be available to all types, including custom classes.
And that's what I was talking about. If "lst[] = X" is to be syntactically valid, there needs to be a protocol that implements it (as with "__matmul__" for @). (It's also worth noting that the @ operator is unique in being created solely for the benefit of third-party types. Every other operator is supported by the core types - usually by many of them. Support for a new operator (or a new form of assignment) would be far greater if multiple use-cases can be shown. So even interpreted the way I hadn't intended, my statement isn't completely out of left field; it just weakens from "cannot" to "is not generally". But syntactically, "cannot" is still true.)
-1 as well. Has any one but the proposer shown any support for it yet?
Not to my knowledge. However, this would be far from the first time that a much-hated-on proposal leads to one that has actual support, perhaps by restricting it, or maybe by generalizing it, or something. ChrisA
On Mon, Jun 18, 2018 at 10:07:07AM +1000, Chris Angelico wrote:
It certainly is. You are talking about *types* and Clint is talking about *semantics*. As you point out below, you are correct: the @ operator works for any type which defines the correct dunder method:
Clint's point that the *motivation* was a single use-case, matrix multiplication in numpy, is a separate issue. Mikhail's motivation might solely be appending to lists, but we would expect this to be a protocol with a dunder method that any type could opt into. [...]
That's not quite correct: although it isn't strictly speaking an operator, extended slice notation x[a:b:c] was invented for numpy, and for a while (Python 1.4 I think?) no core type supported it. Similarly for using Ellipsis ... in slices. As far as I know, there is still no core type which supports that. -- Steve
On Mon, Jun 18, 2018 at 10:50 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Ellipsis is now just a special form of literal, so it's no longer magical in any way (there's no difference between x[...] and x(...) in the syntax). Extended slice notation - interesting that no core type supported it originally. But, again, both of them are implemented using a standard protocol: an object represents the entire thing between the brackets, and that object is passed to __getitem__. So this might need a new object meaning "emptiness", or else it is defined that x[]=y is the same as x[None]=y, which would have confusing implications. Actually, maybe the problem here is that there's no easy way to represent "-0" in a slice. Consider:
Indexing is perfectly parallel, as long as you understand that "-2" mirrors "1". In fact, we could write these using boolean Not, if we wanted to.
Slicing from the beginning works tidily too.
Great. Now let's try slicing from the end:
So you use -1 in slices to parallel 1 (unlike using ~1 as with indexing), and everything works *except zero*. Which means that the slice-assignment form of insert is easy to write, but the slice-assignment form of append isn't. Mikhail, if it were possible to append using slice assignment, would that meet the case? ChrisA
On Mon, Jun 18, 2018 at 4:12 AM, Chris Angelico <rosuav@gmail.com> wrote:
On Mon, Jun 18, 2018 at 10:50 AM, Steven D'Aprano <steve@pearwood.info> wrote:
How? like this: items[-0:] = [item] One of main motivation is actually to have just 'item' on the right side. So your idea is to have special syntax for 'last item' so as to avoid something like: c = len(L) L[c:c] = [...] But IIUC you're still about _iterable_ on the right-hand part?
I think it is currently quite restricted. So imo it's rather generalizing that could make change. Namely the idea for 'L[] = item' seems plausible for append() method only, and for the types that have this method. Python array type has similar method set as lists. Numpy arrays have also append() and insert() methods, but they do not change arrays in-place. So just for keeping it more or less focused - assume we're considering the 'L[] = item' approach, say for mutable types. So one syntax generalization possible is towards insert() method. Although IMO it could be made into nice syntax only by introducing some symbol into the index notation. For example: L[] = item append(item) L[^] = item ? appendleft(item) L[^0] = item insert (0, item) ... L[^i] = item insert(i, x) Note, I am not responsible for the adequacy of the technical part - since it may be not plausible technically. So just speculating here: if that was added, then could it be linked to corresponding methods? For example for collections.deque type: deq[] = item deq.append(item) deq[^] = item deq.appendleft(item) ... deq[^i] = item deq.insert(i, item) So syntactically - I would say it makes more or less complete picture. Because insert() and append() come hand in hand semantically. And with operator-only variant or without special symbol - it seems it is not possible, (at least not with bareable index notation).
On Tue, Jun 19, 2018 at 5:51 AM, Mikhail V <mikhailwas@gmail.com> wrote:
Well, yes, except for the part where -0 is indistinguishable from 0.
Yes, for consistency with other slice assignment. I don't think a dedicated syntax for slotting a single item into it has any chance of being accepted. (PLEASE NOTE: I am not Guido. [citation needed]) But I'm trying to find some germ of an idea inside what you're asking for, something that IS plausible. ChrisA
On Sun, Jun 17, 2018 at 08:01:09PM +0300, Mikhail V wrote:
The idea is to introduce new syntax for the list.append() method.
Before trying to justify any specific syntax, you need to justify the idea of using syntax in the first place.
Reducing human-readable words like "append" in favour of cryptic symbols like "[] =" is not a motivation that I agree with. I think this syntax will make a simple statement like mylist.append(x) harder to read, harder to teach, and harder to get right: mylist[] = x Using a *named method* is a Good Thing. Replacing named methods with syntax needs to be carefully justified, not just assumed that our motive should be to reduce the number of words. On the contrary: we should be trying to keep the amount of symbols fairly small. Not zero, but each new symbol and each new syntactic form using symbols needs to be justified. Why should this be syntax if a method will work?
The original version preserves the original item form too, and I disagree that the replacement looks "more balanced".
There's nothing wrong with that example except you have stuck a space between the method name and the opening parenthises.
It becomes hard to read because of lack of spacing between variable names and list names.
That's your opinion. I think that's unjustified, but even if it were justified, there are *hundreds* or *thousands* of method calls and function calls where you might make the same claim. Should we invent syntax for every single method call?
3. Item appending is very frequent operation.
Not that frequent.
In current syntax, extend() method has dedicated syntax.
mylist1 += mylist2
No, that is wrong. The += syntax applies to *any* type, *any* value which supports the plus operator. It is NOT dedicated syntax for the extend syntax.
That should be discouraged because it uses special syntax instead of a self-explanatory method call. Again, I disagree with both this motivation and the supposed solution.
E.g. adding brackets is often used and understood as operation of increasing dimension of a list
Can you provide an example of somebody who made this error, or did you just make it up? Mikhail, I disagree with your motivation, and I disagree with your supposed solution. As far as I am concerned: - I disagree that list.append method is a problem to be fixed; - but even if it were a problem that needs fixing, your suggested syntax would be *worse* than the problem. -- Steve
On Mon, Jun 18, 2018 at 5:52 PM Juancarlo Añez <apalala@gmail.com> wrote:
Unfortunately, that would create yet another special case of operators breaking the rules. Most operators invoke magic methods. This would prevent ``+=`` from invoking ``__iadd__`` for lists, since the right-hand side would need to be compiled differently. It's similar to why ``and`` and ``or`` keywords can't have magic methods.
It seems that the optimization is already in place: import timeit COUNT = 10000 REPS = 10000 def using_append(): result = [] for i in range(COUNT): result.append(i) return result def using_concat(): result = [] for i in range(COUNT): result += [i] return result def using_iadd(): result = [] for i in range(COUNT): result.__iadd__([i]) return result def main(): print(timeit.timeit('using_append()', globals=globals(), number=REPS)) print(timeit.timeit('using_concat()', globals=globals(), number=REPS)) print(timeit.timeit('using_iadd()', globals=globals(), number=REPS)) if __name__ == '__main__': main() -- Juancarlo *Añez*
On Mon, Jun 18, 2018 at 6:20 PM Juancarlo Añez <apalala@gmail.com> wrote:
I'm not intimately familiar with the opcodes, but I believe that any code involving the expression ``[x]`` will build a list. In [2]: dis.dis("a += [x]") 1 0 LOAD_NAME 0 (a) 2 LOAD_NAME 1 (x) 4 BUILD_LIST 1 6 INPLACE_ADD 8 STORE_NAME 0 (a) 10 LOAD_CONST 0 (None) 12 RETURN_VALUE I didn't run the timings, but I wouldn't be surprised if building a one-element list is faster than looking up an attribute. Or vice-versa. I thought you meant that you wanted to change the syntax such that ``a += [x]`` would, if the left-hand is a list, use different opcodes.
On Mon, Jun 18, 2018 at 09:20:25PM -0400, Juancarlo Añez wrote:
What if mylist doesn't have an append method? Just because the variable is *called* "mylist" doesn't mean it actually is a list. Since Python is dynamically typed, the compiler has no clue ahead of time whether mylist is a list, so the best it could do is compile the equivalent of: if builtins.type(mylist) is builtins.list: call mylist.append(item) else: call mylist.__iadd__([item]) which I suppose it possible, but that's the sort of special-case optimization which CPython has avoided. (And when it has been tried, has often been very disappointing.) Better to move the optimization into list.__iadd__, but doing that still pays the cost of making a one-item list. Now that's likely to be fast, but if the aim is to avoid making that one-item list (can you say "premature optimization"?) then it is a failure. Besides, it's hardly worthwhile: += is already virtually as fast as calling append. [steve@ando ~]$ python3.5 -m timeit -s "L = []" "L.append(1)" 1000000 loops, best of 3: 0.21 usec per loop [steve@ando ~]$ python3.5 -m timeit -s "L = []" "L += [1]" 1000000 loops, best of 3: 0.294 usec per loop [...]
It seems that the optimization is already in place:
Not in 3.5 it isn't. [steve@ando ~]$ python3.5 -c "import dis; dis.dis('L.append(1)')" 1 0 LOAD_NAME 0 (L) 3 LOAD_ATTR 1 (append) 6 LOAD_CONST 0 (1) 9 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 12 RETURN_VALUE [steve@ando ~]$ python3.5 -c "import dis; dis.dis('L += [1]')" 1 0 LOAD_NAME 0 (L) 3 LOAD_CONST 0 (1) 6 BUILD_LIST 1 9 INPLACE_ADD 10 STORE_NAME 0 (L) 13 LOAD_CONST 1 (None) 16 RETURN_VALUE
import timeit [...]
That times a large amount of irrelevant code that has nothing to do with either += or append. -- Steve
On Tue, Jun 19, 2018 at 3:52 AM, Juancarlo Añez <apalala@gmail.com> wrote:
From what I've read on SO about += ,there is not much penalty in comparison to append() when using one item.
And besides, if your idea is to promote += [item] spelling instead of .append() method - then might be you have a totally different idea than mine - so probably start new discussion thread. I suspect though it has little sense because such spelling is imo nothing but obfuscation in terms of readability, and thankfully I haven't seen a lot of such usage. (and this all was many times discussed here too, so please not again)
-- Juancarlo Añez
On Sun, Jun 17, 2018, 10:01 AM Mikhail V <mikhailwas@gmail.com> wrote:
The idea is to introduce new syntax for the list.append() method.
While you have summarized your proposal, you haven't included a summary of the criticism. Also, one thing that's very common for proposals to change syntax and create new uses for operators is to show real code from major libraries that benefit from the change.
On Mon, Jun 18, 2018 at 3:01 AM, Mikhail V <mikhailwas@gmail.com> wrote:
Creation of syntax cannot be done for just one type. So what would this mean (a) for other core data types, and (b) in the protocols? What dunder will be called, and with what arguments? For example: class Foo: def __setitem__(self, key, val): print("Setting", key, "to", val) x = Foo() x[] = 1 What should be printed? Will it come through __setitem__ or are you requiring a completely different dunder method? Regardless, I am still a strong -1 on introducing another way to spell list.append(). ChrisA
I understand the view from the poster, most basic list operations are using brackets, ie reading and writing with [], delete with del L[], why not append ? And being used extensively, that brackets are annoying. And yes, += [] is more "concise" than .append() so some people would think it's more clear because "it's smaller" (they'd be wrong as the OP mentioned). But it would break the "There is only one Obvious way to do it" principle. People would take time to figure out "okay, what should I write ? What's the most Pythonic ?" If someone wants to use their own list class, it's doable with the current syntax : class List(list): def __lshift__(self, x): self.append(x) return self a = List([1,2,7,2]) a = a << 1 << 5 a <<= 0 2018-06-18 1:36 GMT+02:00 Clint Hepner <clint.hepner@gmail.com<mailto:clint.hepner@gmail.com>>:
That’s false. @ was added solely for matrix multiplication.
Regardless, I am still a strong -1 on introducing another way to spell list.append().
-1 as well. Has any one but the proposer shown any support for it yet? — Clint _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Yes - maybe a proposal to have the MutableSequence protocol to define "<<" as "append" would be something with more traction than the original proposal here. No chanegs needed to to the language core, and a = [1,2, 3] a <<= 4 resulting in a == [1, 2, 3, 4] is quite readable, syntactically valid, unambiguous (and can be implemented in two LoC for a new Sequence class). I know that at least Brython used the "<<=" enhanced assignement operator to manipulate HTML DOM, and it often results in quite compact code when compared with javascript-equivalente manipulations. On Mon, 18 Jun 2018 at 01:32, Robert Vanden Eynde <robertvandeneynde@hotmail.com> wrote:
On Mon, Jun 18, 2018 at 12:56 PM Mikhail V <mikhailwas@gmail.com> wrote:
Numpy arrays have also append() and insert() methods,
In [2]: np.arange(1).append(2) AttributeError: 'numpy.ndarray' object has no attribute 'append' In [3]: np.arange(1).insert AttributeError: 'numpy.ndarray' object has no attribute 'insert' So one syntax generalization possible is towards insert() method.
Why would you even want to encourage inserting into a list? It's slow and should be *discouraged*. Although IMO it could be made into nice syntax only by introducing
Indeed, changing the syntax to add a special meaning to caret operator inside of index assignment is a *major* change. Not worthwhile for such dubious benefit. On Mon, Jun 18, 2018 at 1:08 PM Joao S. O. Bueno <jsbueno@python.org.br> wrote:
MutableSequence protocol to define "<<" as "append"
No one has demonstrated with any realistic examples why code would look better with ``a <<= b`` instead of ``a.append(b)``. Perhaps a language in its infancy could toy with new spellings, but Python is old now and should be more cautious with change. I know that at least Brython used the "<<=" enhanced assignement
operator to manipulate HTML DOM, and it often results in quite compact code when compared with javascript-equivalente manipulations.
That sounds great for a dedicated HTML DOM manipulation tool, but not for the core list type. On Sun, Jun 17, 2018 at 6:12 PM Chris Angelico <rosuav@gmail.com> wrote:
The fact that signed 2's complement integers don't support negative zero does cause some problems for both indexing and slicing. However, to me it seems this is a fundamental human problem with zero. Thus the ever-present off-by-one error. I can't think of a solution that doesn't cause more harm than good.
On Mon, Jun 18, 2018 at 11:43 PM, Michael Selik <mike@selik.org> wrote:
https://docs.scipy.org/doc/numpy/reference/generated/numpy.append.html
I don't - but I have seen it in real projects and modules. and ther is Deque.appendleft()
So you have 2 separate inquiries in one: explaining why and where is the example. Why it is so - I have tried to explain several times and also in the summary, with a small example (see first post in this thread - 'transposed_row' example). As for examples - below is one example from 'pyparsing' module. But I don't advise to think about "why" but rather just relax and try to 'traverse' the code back and forth several times. (and sometimes it's better to treat things just as an advice if you doubt you can figure it out by yourself - that's not adressed to you but just general life observation) ---------------- with <<= out = [] NL = '\n' out <<= indent + _ustr(self.asList()) if full: if self.haskeys(): items = sorted((str(k), v) for k,v in self.items()) for k,v in items: if out: out <<= NL out <<= "%s%s- %s: " % (indent,(' '*depth), k) if isinstance (v,ParseResults): if v: out <<= v.dump(indent,depth+1) else: out <<= _ustr(v) else: out <<= repr(v) ----------------- with .append() out = [] NL = '\n' out.append( indent+_ustr(self.asList()) ) if full: if self.haskeys(): items = sorted((str(k), v) for k,v in self.items()) for k,v in items: if out: out.append(NL) out.append( "%s%s- %s: " % (indent,(' '*depth), k) ) if isinstance(v,ParseResults): if v: out.append( v.dump(indent,depth+1) ) else: out.append(_ustr(v)) else: out.append(repr(v))
On Mon, Jun 18, 2018 at 2:55 PM Mikhail V <mikhailwas@gmail.com> wrote:
Perhaps NumPy chose not to provide an append method to discourage repeated appends, since it's terribly slow. It's a good design philosophy: make inefficient code look appropriately ugly.
That's interesting advice. I contrast, I find it's best to aggressively ask "Why?" and to rewrite the code in order to fully understand it. I think I found the chunk of code you're referring to. https://github.com/pyparsing/pyparsing/blob/master/pyparsing.py#L848 First, I'll note that ``dump`` usually indicates dumping to a file, while ``dumps`` is dumping to a str. Regardless, I think this ``dump`` function could be improved a bit, and a revision reduces the benefit of a dedicated ``append`` operator. if not full: return indent + _ustr(self.asList()) lines = [_ustr(self.asList())] if self.haskeys(): fmt = ' ' * depth + '- %s: %s' for k, v in sorted((str(k), v) for k, v in self.items()): if isinstance(v, ParseResults): if v: s = v.dump(indent, depth + 1) else: s = _ustr(v) else: s = repr(v) lines.append(fmt % (k, s)) elif any(isinstance(v, ParseResults) for v in self): for i, v in enumerate(self): lines.append(' ' * depth + '[%d]:' % i) if isinstance(v, ParseResults): s = v.dump(indent, depth + 1) else: s = _ustr(v) lines.append(' ' * (depth + 1) + s) return ('\n' + indent).join(lines)
On Mon, Jun 18, 2018, 6:59 PM Michael Selik <mike@selik.org> wrote:
On the 2nd thought, that nested if is ugly. Much better as if not isinstance(v, ParseResults): s = repr(v) elif v: s = v.dump(indent, depth + 1) else: s = _ustr(v) lines.append(fmt % (k, s)) It might seem like this is going off topic. What I'm trying to demonstrate is that cases where an append operator might help are really in need of more thorough revision, not just a little sugar.
On Mon, Jun 18, 2018 at 05:07:45PM -0300, Joao S. O. Bueno wrote:
Not as readable as a.append(4), which works today and doesn't require the user to memorise the special case that <<= works on lists but (presumably) << doesn't. Or perhaps it would. What would a bare << mean for lists? b = a << 4 could mean the same as b = a + [4] but we already have a spelling for that, do we really need another one? -- Steve
On Mon, Jun 18, 2018 at 9:36 AM, Clint Hepner <clint.hepner@gmail.com> wrote:
Ah, confusing bit of language there. The @ operator was created for the benefit of a small number of types (not just one, I think), but it MUST be available to all types, including custom classes.
And that's what I was talking about. If "lst[] = X" is to be syntactically valid, there needs to be a protocol that implements it (as with "__matmul__" for @). (It's also worth noting that the @ operator is unique in being created solely for the benefit of third-party types. Every other operator is supported by the core types - usually by many of them. Support for a new operator (or a new form of assignment) would be far greater if multiple use-cases can be shown. So even interpreted the way I hadn't intended, my statement isn't completely out of left field; it just weakens from "cannot" to "is not generally". But syntactically, "cannot" is still true.)
-1 as well. Has any one but the proposer shown any support for it yet?
Not to my knowledge. However, this would be far from the first time that a much-hated-on proposal leads to one that has actual support, perhaps by restricting it, or maybe by generalizing it, or something. ChrisA
On Mon, Jun 18, 2018 at 10:07:07AM +1000, Chris Angelico wrote:
It certainly is. You are talking about *types* and Clint is talking about *semantics*. As you point out below, you are correct: the @ operator works for any type which defines the correct dunder method:
Clint's point that the *motivation* was a single use-case, matrix multiplication in numpy, is a separate issue. Mikhail's motivation might solely be appending to lists, but we would expect this to be a protocol with a dunder method that any type could opt into. [...]
That's not quite correct: although it isn't strictly speaking an operator, extended slice notation x[a:b:c] was invented for numpy, and for a while (Python 1.4 I think?) no core type supported it. Similarly for using Ellipsis ... in slices. As far as I know, there is still no core type which supports that. -- Steve
On Mon, Jun 18, 2018 at 10:50 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Ellipsis is now just a special form of literal, so it's no longer magical in any way (there's no difference between x[...] and x(...) in the syntax). Extended slice notation - interesting that no core type supported it originally. But, again, both of them are implemented using a standard protocol: an object represents the entire thing between the brackets, and that object is passed to __getitem__. So this might need a new object meaning "emptiness", or else it is defined that x[]=y is the same as x[None]=y, which would have confusing implications. Actually, maybe the problem here is that there's no easy way to represent "-0" in a slice. Consider:
Indexing is perfectly parallel, as long as you understand that "-2" mirrors "1". In fact, we could write these using boolean Not, if we wanted to.
Slicing from the beginning works tidily too.
Great. Now let's try slicing from the end:
So you use -1 in slices to parallel 1 (unlike using ~1 as with indexing), and everything works *except zero*. Which means that the slice-assignment form of insert is easy to write, but the slice-assignment form of append isn't. Mikhail, if it were possible to append using slice assignment, would that meet the case? ChrisA
On Mon, Jun 18, 2018 at 4:12 AM, Chris Angelico <rosuav@gmail.com> wrote:
On Mon, Jun 18, 2018 at 10:50 AM, Steven D'Aprano <steve@pearwood.info> wrote:
How? like this: items[-0:] = [item] One of main motivation is actually to have just 'item' on the right side. So your idea is to have special syntax for 'last item' so as to avoid something like: c = len(L) L[c:c] = [...] But IIUC you're still about _iterable_ on the right-hand part?
I think it is currently quite restricted. So imo it's rather generalizing that could make change. Namely the idea for 'L[] = item' seems plausible for append() method only, and for the types that have this method. Python array type has similar method set as lists. Numpy arrays have also append() and insert() methods, but they do not change arrays in-place. So just for keeping it more or less focused - assume we're considering the 'L[] = item' approach, say for mutable types. So one syntax generalization possible is towards insert() method. Although IMO it could be made into nice syntax only by introducing some symbol into the index notation. For example: L[] = item append(item) L[^] = item ? appendleft(item) L[^0] = item insert (0, item) ... L[^i] = item insert(i, x) Note, I am not responsible for the adequacy of the technical part - since it may be not plausible technically. So just speculating here: if that was added, then could it be linked to corresponding methods? For example for collections.deque type: deq[] = item deq.append(item) deq[^] = item deq.appendleft(item) ... deq[^i] = item deq.insert(i, item) So syntactically - I would say it makes more or less complete picture. Because insert() and append() come hand in hand semantically. And with operator-only variant or without special symbol - it seems it is not possible, (at least not with bareable index notation).
On Tue, Jun 19, 2018 at 5:51 AM, Mikhail V <mikhailwas@gmail.com> wrote:
Well, yes, except for the part where -0 is indistinguishable from 0.
Yes, for consistency with other slice assignment. I don't think a dedicated syntax for slotting a single item into it has any chance of being accepted. (PLEASE NOTE: I am not Guido. [citation needed]) But I'm trying to find some germ of an idea inside what you're asking for, something that IS plausible. ChrisA
On Sun, Jun 17, 2018 at 08:01:09PM +0300, Mikhail V wrote:
The idea is to introduce new syntax for the list.append() method.
Before trying to justify any specific syntax, you need to justify the idea of using syntax in the first place.
Reducing human-readable words like "append" in favour of cryptic symbols like "[] =" is not a motivation that I agree with. I think this syntax will make a simple statement like mylist.append(x) harder to read, harder to teach, and harder to get right: mylist[] = x Using a *named method* is a Good Thing. Replacing named methods with syntax needs to be carefully justified, not just assumed that our motive should be to reduce the number of words. On the contrary: we should be trying to keep the amount of symbols fairly small. Not zero, but each new symbol and each new syntactic form using symbols needs to be justified. Why should this be syntax if a method will work?
The original version preserves the original item form too, and I disagree that the replacement looks "more balanced".
There's nothing wrong with that example except you have stuck a space between the method name and the opening parenthises.
It becomes hard to read because of lack of spacing between variable names and list names.
That's your opinion. I think that's unjustified, but even if it were justified, there are *hundreds* or *thousands* of method calls and function calls where you might make the same claim. Should we invent syntax for every single method call?
3. Item appending is very frequent operation.
Not that frequent.
In current syntax, extend() method has dedicated syntax.
mylist1 += mylist2
No, that is wrong. The += syntax applies to *any* type, *any* value which supports the plus operator. It is NOT dedicated syntax for the extend syntax.
That should be discouraged because it uses special syntax instead of a self-explanatory method call. Again, I disagree with both this motivation and the supposed solution.
E.g. adding brackets is often used and understood as operation of increasing dimension of a list
Can you provide an example of somebody who made this error, or did you just make it up? Mikhail, I disagree with your motivation, and I disagree with your supposed solution. As far as I am concerned: - I disagree that list.append method is a problem to be fixed; - but even if it were a problem that needs fixing, your suggested syntax would be *worse* than the problem. -- Steve
On Mon, Jun 18, 2018 at 5:52 PM Juancarlo Añez <apalala@gmail.com> wrote:
Unfortunately, that would create yet another special case of operators breaking the rules. Most operators invoke magic methods. This would prevent ``+=`` from invoking ``__iadd__`` for lists, since the right-hand side would need to be compiled differently. It's similar to why ``and`` and ``or`` keywords can't have magic methods.
It seems that the optimization is already in place: import timeit COUNT = 10000 REPS = 10000 def using_append(): result = [] for i in range(COUNT): result.append(i) return result def using_concat(): result = [] for i in range(COUNT): result += [i] return result def using_iadd(): result = [] for i in range(COUNT): result.__iadd__([i]) return result def main(): print(timeit.timeit('using_append()', globals=globals(), number=REPS)) print(timeit.timeit('using_concat()', globals=globals(), number=REPS)) print(timeit.timeit('using_iadd()', globals=globals(), number=REPS)) if __name__ == '__main__': main() -- Juancarlo *Añez*
On Mon, Jun 18, 2018 at 6:20 PM Juancarlo Añez <apalala@gmail.com> wrote:
I'm not intimately familiar with the opcodes, but I believe that any code involving the expression ``[x]`` will build a list. In [2]: dis.dis("a += [x]") 1 0 LOAD_NAME 0 (a) 2 LOAD_NAME 1 (x) 4 BUILD_LIST 1 6 INPLACE_ADD 8 STORE_NAME 0 (a) 10 LOAD_CONST 0 (None) 12 RETURN_VALUE I didn't run the timings, but I wouldn't be surprised if building a one-element list is faster than looking up an attribute. Or vice-versa. I thought you meant that you wanted to change the syntax such that ``a += [x]`` would, if the left-hand is a list, use different opcodes.
On Mon, Jun 18, 2018 at 09:20:25PM -0400, Juancarlo Añez wrote:
What if mylist doesn't have an append method? Just because the variable is *called* "mylist" doesn't mean it actually is a list. Since Python is dynamically typed, the compiler has no clue ahead of time whether mylist is a list, so the best it could do is compile the equivalent of: if builtins.type(mylist) is builtins.list: call mylist.append(item) else: call mylist.__iadd__([item]) which I suppose it possible, but that's the sort of special-case optimization which CPython has avoided. (And when it has been tried, has often been very disappointing.) Better to move the optimization into list.__iadd__, but doing that still pays the cost of making a one-item list. Now that's likely to be fast, but if the aim is to avoid making that one-item list (can you say "premature optimization"?) then it is a failure. Besides, it's hardly worthwhile: += is already virtually as fast as calling append. [steve@ando ~]$ python3.5 -m timeit -s "L = []" "L.append(1)" 1000000 loops, best of 3: 0.21 usec per loop [steve@ando ~]$ python3.5 -m timeit -s "L = []" "L += [1]" 1000000 loops, best of 3: 0.294 usec per loop [...]
It seems that the optimization is already in place:
Not in 3.5 it isn't. [steve@ando ~]$ python3.5 -c "import dis; dis.dis('L.append(1)')" 1 0 LOAD_NAME 0 (L) 3 LOAD_ATTR 1 (append) 6 LOAD_CONST 0 (1) 9 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 12 RETURN_VALUE [steve@ando ~]$ python3.5 -c "import dis; dis.dis('L += [1]')" 1 0 LOAD_NAME 0 (L) 3 LOAD_CONST 0 (1) 6 BUILD_LIST 1 9 INPLACE_ADD 10 STORE_NAME 0 (L) 13 LOAD_CONST 1 (None) 16 RETURN_VALUE
import timeit [...]
That times a large amount of irrelevant code that has nothing to do with either += or append. -- Steve
On Tue, Jun 19, 2018 at 3:52 AM, Juancarlo Añez <apalala@gmail.com> wrote:
From what I've read on SO about += ,there is not much penalty in comparison to append() when using one item.
And besides, if your idea is to promote += [item] spelling instead of .append() method - then might be you have a totally different idea than mine - so probably start new discussion thread. I suspect though it has little sense because such spelling is imo nothing but obfuscation in terms of readability, and thankfully I haven't seen a lot of such usage. (and this all was many times discussed here too, so please not again)
-- Juancarlo Añez
participants (10)
-
Alexandre Brault -
Chris Angelico -
Clint Hepner -
Joao S. O. Bueno -
Juancarlo Añez -
Michael Selik -
Mikhail V -
Robert Vanden Eynde -
Robert Vanden Eynde -
Steven D'Aprano