From stephen at xemacs.org Thu Dec 1 02:38:21 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 01 Dec 2011 10:38:21 +0900 Subject: [Python-ideas] Is there a reason why file.readlines() doesn't/can't return an iterator? In-Reply-To: References: <4ED650FF.9020705@netwok.org> <4ED65DF6.3060900@netwok.org> Message-ID: <87d3c9nkaa.fsf@uwakimon.sk.tsukuba.ac.jp> Peter Otten writes: > ?ric Araujo wrote: > > > Okay. You can open a report on bugs.python.org to ask that the doc for > > readlines mention list(fp) as an alternative. > > That would make sense to me only if readlines() were deprecated. It makes sense to me. Something like Note that "for line in fp:" (to iterate over the lines of the file) and "list(fp)" (to get a list of all lines in a file) are idiomatic Python. to remind readers (especially new users) of the iterator concept, which is even today not so familiar to beginning programmers. From tjreedy at udel.edu Fri Dec 2 04:13:03 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 01 Dec 2011 22:13:03 -0500 Subject: [Python-ideas] Is there a reason why file.readlines() doesn't/can't return an iterator? In-Reply-To: <87d3c9nkaa.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4ED650FF.9020705@netwok.org> <4ED65DF6.3060900@netwok.org> <87d3c9nkaa.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 11/30/2011 8:38 PM, Stephen J. Turnbull wrote: > Peter Otten writes: > > ?ric Araujo wrote: > > > > > Okay. You can open a report on bugs.python.org to ask that the doc for > > > readlines mention list(fp) as an alternative. > > > > That would make sense to me only if readlines() were deprecated. > > It makes sense to me. Something like > > Note that "for line in fp:" (to iterate over the lines of the > file) and "list(fp)" (to get a list of all lines in a file) are > idiomatic Python. > > to remind readers (especially new users) of the iterator concept, > which is even today not so familiar to beginning programmers. See http://bugs.python.org/issue13510 , where I suggested something like that. -- Terry Jan Reedy From matthew at woodcraft.me.uk Fri Dec 2 22:02:05 2011 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Fri, 02 Dec 2011 21:02:05 +0000 Subject: [Python-ideas] Is there a reason why file.readlines() doesn't/can't return an iterator? In-Reply-To: References: Message-ID: On 2011-11-30 13:59, Peter Otten wrote: > My observation on the Tutor mailing list is that there are no valid uses of > readlines(). It's just easier to discover the readlines() method than to > find out that you can iterate over the file directly. In 2.x, iterating directly can behave unexpectedly if the file object is something other than a regular file; see eg http://bugs.python.org/issue1633941 http://bugs.python.org/issue3907 http://utcc.utoronto.ca/~cks/space/blog/python/FileIteratorProblems readline() and readlines() don't have these problems. -M- From tjreedy at udel.edu Fri Dec 2 22:55:23 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 02 Dec 2011 16:55:23 -0500 Subject: [Python-ideas] Is there a reason why file.readlines() doesn't/can't return an iterator? In-Reply-To: References: Message-ID: On 12/2/2011 4:02 PM, Matthew Woodcraft wrote: > On 2011-11-30 13:59, Peter Otten wrote: >> My observation on the Tutor mailing list is that there are no valid uses of >> readlines(). It's just easier to discover the readlines() method than to >> find out that you can iterate over the file directly. > > In 2.x, iterating directly can behave unexpectedly if the file object is > something other than a regular file; see eg > http://bugs.python.org/issue1633941 > http://bugs.python.org/issue3907 Both are fixed in 3.x > http://utcc.utoronto.ca/~cks/space/blog/python/FileIteratorProblems > > readline() and readlines() don't have these problems. > > -M- -- Terry Jan Reedy From bauertomer at gmail.com Sat Dec 3 01:12:58 2011 From: bauertomer at gmail.com (T.B.) Date: Sat, 03 Dec 2011 02:12:58 +0200 Subject: [Python-ideas] Different bases format specification Message-ID: <4ED9698A.6010609@gmail.com> An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sat Dec 3 02:26:30 2011 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 03 Dec 2011 01:26:30 +0000 Subject: [Python-ideas] Different bases format specification In-Reply-To: <4ED9698A.6010609@gmail.com> References: <4ED9698A.6010609@gmail.com> Message-ID: <4ED97AC6.9000209@mrabarnett.plus.com> On 03/12/2011 00:12, T.B. wrote: > I will start by stating that it's not my original idea, but taken from > Erlang. See Erlang's io:format documentation here: > http://www.erlang.org/doc/man/io.html#format-1 and notice the 'B' > control sequence. > > I would like to have an easy built-in way to print integers in different > bases (radices). There are so many half baked solutions out there: > http://bugs.python.org/issue6783 > http://stackoverflow.com/questions/2267362/convert-integer-to-a-string-in-a-given-numeric-base-in-python > http://stackoverflow.com/questions/2063425/python-elegant-inverse-function-of-intstring-base > http://code.activestate.com/recipes/65212/ > > I suggest using the precision field in the format specification for > integers for that. [snip] I think that the precision field should be used only for the precision and that sometimes using it for something completely different is a bad idea. From ncoghlan at gmail.com Sat Dec 3 02:31:37 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Dec 2011 11:31:37 +1000 Subject: [Python-ideas] Different bases format specification In-Reply-To: <4ED9698A.6010609@gmail.com> References: <4ED9698A.6010609@gmail.com> Message-ID: On Sat, Dec 3, 2011 at 10:12 AM, T.B. wrote: > I suggest using the precision field in the format specification for integers > for that. Supporting arbitrary bases for string formatting has been discussed and rejected in the past (both in the context of PEP 3101's introduction of new string formatting and on other occasions). Nobody has ever produced convincing use cases for natively supporting formatting with bases other than binary, octal, decimal and hexadecimal. Accordingly, those 4 are supported explicitly via the 'b', 'o', 'd' and 'x'/'X' formatting codes, while other formats still require an explicit conversion function. As for "Why Not?" 1. 'd' stands for decimal. If support for arbitrary bases were added, it would need to be as a separate format code (e.g. 'i' for integer) 2. The explicit 'b', 'o' and 'x' codes are related to integer literal notation (0b10, 0o777, 0x1F), not to the second argument to int() 3. The use cases just aren't that strong. When you start dealing with base36 and base64, you're not talking about formatting numbers for human readers any more, you're talking about encoding numbers as short pieces of text. Better to let people decide exactly the behaviour they want by coding it themselves. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From bruce at leapyear.org Sat Dec 3 03:19:24 2011 From: bruce at leapyear.org (Bruce Leban) Date: Fri, 2 Dec 2011 18:19:24 -0800 Subject: [Python-ideas] Different bases format specification In-Reply-To: <4ED9698A.6010609@gmail.com> References: <4ED9698A.6010609@gmail.com> Message-ID: On Fri, Dec 2, 2011 at 4:12 PM, T.B. wrote: > I suggest using the precision field in the format specification for > integers for that. Examples: > > "{:.16d}".format(31) #Prints '1f' > > "{:.2d}".format(-19) # Prints '-10011' > > > I don't think this belongs in format. > P.S. Bonus question: What "{:.-909d}".format(42) would print? > Any proposal which includes an inscrutable example doesn't bode well for the usability of the feature. :-) Sure, negative bases are mathematically meaningful but are they useful in Python? And why not complex bases then? Or did you have something else strange in mind? If there's enough need for encoding in different bases, including a standard version of format_integer_in_base makes a lot more sense. We could write format_integer_in_base(15, 16) to get "F" and format_integer_in_base(64, "A23456789TJQK") to get "4K". But note that standard base 64 encoding is not at all the same -- this function encodes starting at LSB while base 64 encodes at word boundaries. Finally, note that if you really want to mangle format strings you can do it without changing the library. Just write it this way "{:.16d}".format(arbitrarybase(31)) where you have defined class arbitrarybase: def __format__(self, format_spec): return format_integer_in_base(parse format spec etc.) --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bauertomer at gmail.com Sat Dec 3 03:45:59 2011 From: bauertomer at gmail.com (T.B.) Date: Sat, 03 Dec 2011 04:45:59 +0200 Subject: [Python-ideas] Different bases format specification In-Reply-To: References: <4ED9698A.6010609@gmail.com> Message-ID: <4ED98D67.8070006@gmail.com> Tip for self: No more HTML e-mails. On 2011-12-03 04:19, Bruce Leban wrote: > >> P.S. Bonus question: What "{:.-909d}".format(42) would print? > > Any proposal which includes an inscrutable example doesn't bode well for > the usability of the feature. :-) Sure, negative bases are > mathematically meaningful but are they useful in Python? And why not > complex bases then? Or did you have something else strange in mind? > My intention will be clear after reading http://bugs.python.org/issue2844. It will also ruin the surprise of figuring out alone. Regards, TB From bauertomer at gmail.com Sat Dec 3 04:16:00 2011 From: bauertomer at gmail.com (T.B.) Date: Sat, 03 Dec 2011 05:16:00 +0200 Subject: [Python-ideas] Different bases format specification In-Reply-To: References: <4ED9698A.6010609@gmail.com> Message-ID: <4ED99470.20205@gmail.com> On 2011-12-03 03:31, Nick Coghlan wrote: > On Sat, Dec 3, 2011 at 10:12 AM, T.B. wrote: >> I suggest using the precision field in the format specification for integers >> for that. > > Supporting arbitrary bases for string formatting has been discussed > and rejected in the past (both in the context of PEP 3101's > introduction of new string formatting and on other occasions). > > Nobody has ever produced convincing use cases for natively supporting > formatting with bases other than binary, octal, decimal and > hexadecimal. Accordingly, those 4 are supported explicitly via the > 'b', 'o', 'd' and 'x'/'X' formatting codes, while other formats still > require an explicit conversion function. For weird math scenarios I know there are already many modules and packages. But what about ternary? en.wikipedia.org/wiki/Ternary_numeral_system has some points that include that base 9 and 27 are used [no citation]. > As for "Why Not?" > > 1. 'd' stands for decimal. If support for arbitrary bases were added, > it would need to be as a separate format code (e.g. 'i' for integer) > > 2. The explicit 'b', 'o' and 'x' codes are related to integer literal > notation (0b10, 0o777, 0x1F), not to the second argument to int() > That one reason I wrote: "It might be a nice mnemonic using 'b' instead, standing for 'base'. Then the default base will be 2." Anyway, I think there should be 'B' and 'O' presentation types, that will be used for outputting '0B' and '0O' prefixes. Thanks for your reply, TB From ncoghlan at gmail.com Sat Dec 3 05:31:51 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Dec 2011 14:31:51 +1000 Subject: [Python-ideas] Different bases format specification In-Reply-To: <4ED99470.20205@gmail.com> References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> Message-ID: On Sat, Dec 3, 2011 at 1:16 PM, T.B. wrote: > For weird math scenarios I know there are already many modules and packages. > But what about ternary? en.wikipedia.org/wiki/Ternary_numeral_system has > some points that include that base 9 and 27 are used [no citation]. Weird math scenarios are no justification for changing the behaviour of a builtin type. b/o/d/x/X cover all the common use cases, everything else can be handled by libraries (including providing custom string.Formatter subclasses). Code and functionality are not free - we need solid gains in easier (or otherwise improved) coding and maintenance for real world problems before we add more of either. >> 2. The explicit 'b', 'o' and 'x' codes are related to integer literal >> notation (0b10, 0o777, 0x1F), not to the second argument to int() >> > That one reason I wrote: "It might be a nice mnemonic using 'b' instead, > standing for 'base'. Then the default base will be 2." And the symmetry with the integer literal codes will still be lost. > Anyway, I think there should be 'B' and 'O' presentation types, that > will be used for outputting '0B' and '0O' prefixes. If you really want that (Why would you?) and so long as the numbers aren't negative: "0B{:b}".format(number) "0O{:o}".format(number) The only reason 'X' is provided for hexadecimal formatting is to capitalize the letters that appear within the number itself. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From bauertomer at gmail.com Sat Dec 3 06:06:42 2011 From: bauertomer at gmail.com (T.B.) Date: Sat, 03 Dec 2011 07:06:42 +0200 Subject: [Python-ideas] Different bases format specification In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> Message-ID: <4ED9AE62.5030303@gmail.com> On 2011-12-03 06:31, Nick Coghlan wrote: >>> 2. The explicit 'b', 'o' and 'x' codes are related to integer literal >>> notation (0b10, 0o777, 0x1F), not to the second argument to int() >>> >> That one reason I wrote: "It might be a nice mnemonic using 'b' instead, >> standing for 'base'. Then the default base will be 2." > > And the symmetry with the integer literal codes will still be lost. All is not lost. "{:b}".format(num) would still print numbers in binary notion. Only when using an optional field it will change the output. One far-reaching solution for the symmetry break is allowing ALL integer literal codes to have that optional field, and each literal code will have its own default base: b->2, o->8, d->10, x->16. >> Anyway, I think there should be 'B' and 'O' presentation types, that >> will be used for outputting '0B' and '0O' prefixes. Horrors such as an uppercase-only file/shell/whatever still exist. They are rare and you'll usually call upper() before sending a message/writing to a file to them, but it might worth adding 'B'. > If you really want that (Why would you?) and so long as the numbers > aren't negative: > > "0B{:b}".format(number) > "0O{:o}".format(number) > > The only reason 'X' is provided for hexadecimal formatting is to > capitalize the letters that appear within the number itself. 0B111 and -0O755 are *current* valid python tokens. There is no "elegant" way of outputting those tokens. I suggested 'B' and 'O' as a side-effect, because bases > 10 have some digits as letters. regards, TB From ncoghlan at gmail.com Sat Dec 3 06:16:06 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Dec 2011 15:16:06 +1000 Subject: [Python-ideas] Different bases format specification In-Reply-To: <4ED9AE62.5030303@gmail.com> References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <4ED9AE62.5030303@gmail.com> Message-ID: On Sat, Dec 3, 2011 at 3:06 PM, T.B. wrote: >>> Anyway, I think there should be 'B' and 'O' presentation types, that >>> will be used for outputting '0B' and '0O' prefixes. > > Horrors such as an uppercase-only file/shell/whatever still exist. They are > rare and you'll usually call upper() before sending a message/writing to a > file to them, but it might worth adding 'B'. You're setting your bar for "hey, let's change the way a builtin type works" *way* too low. It's *OK* if obscure corner cases like bases outside the main four (2, 8, 10, 16), or "upper case only" environments require extra code. "Make easy things easy and hard things possible" is the goal, not "build in complex features to handle special cases that only arise on rare occasions and can already be dealt with using the vast array of general purpose programming tools Python provides". It's not that supporting arbitrary bases is a terrible idea - it's that it doesn't come up often enough as a general programming problem to be worth going to the effort of making the change. There's no such thing as a "trivial" change to a Python builtin - they *all* have significant repercussions, as the update ripples out through the Python ecosystem over the course of several years (see http://www.boredomandlaziness.org/2011/02/status-quo-wins-stalemate.html). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Sat Dec 3 17:45:00 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Dec 2011 17:45:00 +0100 Subject: [Python-ideas] Negative hexes References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> Message-ID: <20111203174500.49810bb1@pitrou.net> On Sat, 3 Dec 2011 14:31:51 +1000 Nick Coghlan wrote: > > If you really want that (Why would you?) and so long as the numbers > aren't negative: > > "0B{:b}".format(number) > "0O{:o}".format(number) > > The only reason 'X' is provided for hexadecimal formatting is to > capitalize the letters that appear within the number itself. By the way, any reason why hex output represents negative number with a negative sign (instead of the more usual 2s-complement representation)? It's not too difficult to normalize by hand (e.g. add 2**32 if you know the number is a 32-bit one) but it always irks me that Python doesn't do it by default. I cannot think of a situation where the "sign" is relevant when printing a hex number: hex is about the raw binary representation of the number. Regards Antoine. From guido at python.org Sat Dec 3 18:05:02 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 3 Dec 2011 09:05:02 -0800 Subject: [Python-ideas] Negative hexes In-Reply-To: <20111203174500.49810bb1@pitrou.net> References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> Message-ID: On Sat, Dec 3, 2011 at 8:45 AM, Antoine Pitrou wrote: > On Sat, 3 Dec 2011 14:31:51 +1000 > Nick Coghlan wrote: > > > > If you really want that (Why would you?) and so long as the numbers > > aren't negative: > > > > "0B{:b}".format(number) > > "0O{:o}".format(number) > > > > The only reason 'X' is provided for hexadecimal formatting is to > > capitalize the letters that appear within the number itself. > > By the way, any reason why hex output represents negative number with a > negative sign (instead of the more usual 2s-complement representation)? > > It's not too difficult to normalize by hand (e.g. add 2**32 if you know > the number is a 32-bit one) but it always irks me that Python doesn't > do it by default. I cannot think of a situation where the "sign" is > relevant when printing a hex number: hex is about the raw binary > representation of the number. > This is because Python's integers are not limited to 32 bits or 64 bits. If you read PEP 237, you'll see that this was one of the hardest differences between ints and longs to be resolved. You'd have to include an infinite number of leading 'F' characters to format a negative long this way... -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Dec 3 18:07:47 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Dec 2011 18:07:47 +0100 Subject: [Python-ideas] Negative hexes References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> Message-ID: <20111203180747.37c296e6@pitrou.net> On Sat, 3 Dec 2011 09:05:02 -0800 Guido van Rossum wrote: > On Sat, Dec 3, 2011 at 8:45 AM, Antoine Pitrou wrote: > > > On Sat, 3 Dec 2011 14:31:51 +1000 > > Nick Coghlan wrote: > > > > > > If you really want that (Why would you?) and so long as the numbers > > > aren't negative: > > > > > > "0B{:b}".format(number) > > > "0O{:o}".format(number) > > > > > > The only reason 'X' is provided for hexadecimal formatting is to > > > capitalize the letters that appear within the number itself. > > > > By the way, any reason why hex output represents negative number with a > > negative sign (instead of the more usual 2s-complement representation)? > > > > It's not too difficult to normalize by hand (e.g. add 2**32 if you know > > the number is a 32-bit one) but it always irks me that Python doesn't > > do it by default. I cannot think of a situation where the "sign" is > > relevant when printing a hex number: hex is about the raw binary > > representation of the number. > > > > This is because Python's integers are not limited to 32 bits or 64 bits. If > you read PEP 237, you'll see that this was one of the hardest differences > between ints and longs to be resolved. You'd have to include an infinite > number of leading 'F' characters to format a negative long this way... That's a fair point :) Regards Antoine. From atmb4u at gmail.com Sat Dec 3 20:12:30 2011 From: atmb4u at gmail.com (Anoop Thomas Mathew) Date: Sun, 4 Dec 2011 00:42:30 +0530 Subject: [Python-ideas] Portable Continuations in Python Message-ID: Hi All, I was wondering whether we can have a feature like, portable continuations (as present in Scala 2.8) is possible in Python. It is a feature, by which executions can be stopped in between, and transferred to another computer and resume the process. This is a standard feature in scala 2.8. Something similar is present in Stackless Python, but it doesn't support 'Portable'. I am not sure how can we do that. But, if it is possible, and if some one can throw some light how to achieve that, I am ready to volunteer to dig deeper. Awaiting comments, Anoop Thomas Mathew atm ___ Life is short, Live it hard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Dec 4 02:13:26 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Dec 2011 11:13:26 +1000 Subject: [Python-ideas] Negative hexes In-Reply-To: <20111203180747.37c296e6@pitrou.net> References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> Message-ID: On Sun, Dec 4, 2011 at 3:07 AM, Antoine Pitrou wrote: >> This is because Python's integers are not limited to 32 bits or 64 bits. If >> you read PEP 237, you'll see that this was one of the hardest differences >> between ints and longs to be resolved. You'd have to include an infinite >> number of leading 'F' characters to format a negative long this way... > > That's a fair point :) Random thought... could we use the integer precision field to fix *that*, by having it indicate the intended number of bytes in the integer? That is, currently: >>> "{:.4x}".format(31) Traceback (most recent call last): File "", line 1, in ValueError: Precision not allowed in integer format specifier What if instead that produced: >>> "{:.4X}".format(31) 0000001F >>> "{:.4X}".format(-31) FFFFFFE1 Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sun Dec 4 02:16:48 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Dec 2011 11:16:48 +1000 Subject: [Python-ideas] Portable Continuations in Python In-Reply-To: References: Message-ID: On Sun, Dec 4, 2011 at 5:12 AM, Anoop Thomas Mathew wrote: > Hi All, > > I was wondering whether we can have a feature like, portable continuations > (as present in Scala 2.8) is possible in Python. It is a feature, by which > executions can be stopped in between, and transferred to another computer > and resume the process. This is a standard feature in scala 2.8. Something > similar is present in Stackless Python, but it doesn't support 'Portable'. I > am not sure how can we do that. But, if it is possible, and if some one can > throw some light how to achieve that, I am ready to volunteer to dig deeper. Hi Anoop, Take a look at the 'greenlets' extension module (which has the same portability limitations as Stackless, since that's where it came from), the 'cofunctions' PEP (PEP 3152), and the extensive discussions regarding the latter in the archives of this list. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Sun Dec 4 02:17:56 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 3 Dec 2011 17:17:56 -0800 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> Message-ID: On Sat, Dec 3, 2011 at 5:13 PM, Nick Coghlan wrote: > On Sun, Dec 4, 2011 at 3:07 AM, Antoine Pitrou > wrote: > >> This is because Python's integers are not limited to 32 bits or 64 > bits. If > >> you read PEP 237, you'll see that this was one of the hardest > differences > >> between ints and longs to be resolved. You'd have to include an infinite > >> number of leading 'F' characters to format a negative long this way... > > > > That's a fair point :) > > Random thought... could we use the integer precision field to fix > *that*, by having it indicate the intended number of bytes in the > integer? > > That is, currently: > > >>> "{:.4x}".format(31) > Traceback (most recent call last): > File "", line 1, in > ValueError: Precision not allowed in integer format specifier > > What if instead that produced: > > >>> "{:.4X}".format(31) > 0000001F > >>> "{:.4X}".format(-31) > FFFFFFE1 > Usually that field is measured in characters/digits, so this should probably produce FFE1; you'd need {:.8X} to produce FFFFFFE1. This would then logically extend to binary and octal, in each case measuring characters/digits in the indicated base. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sun Dec 4 02:32:43 2011 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 04 Dec 2011 01:32:43 +0000 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> Message-ID: <4EDACDBB.1040607@mrabarnett.plus.com> On 04/12/2011 01:17, Guido van Rossum wrote: > On Sat, Dec 3, 2011 at 5:13 PM, Nick Coghlan > wrote: > > On Sun, Dec 4, 2011 at 3:07 AM, Antoine Pitrou > wrote: > >> This is because Python's integers are not limited to 32 bits or > 64 bits. If > >> you read PEP 237, you'll see that this was one of the hardest > differences > >> between ints and longs to be resolved. You'd have to include an > infinite > >> number of leading 'F' characters to format a negative long this > way... > > > > That's a fair point :) > > Random thought... could we use the integer precision field to fix > *that*, by having it indicate the intended number of bytes in the > integer? > > That is, currently: > > >>> "{:.4x}".format(31) > Traceback (most recent call last): > File "", line 1, in > ValueError: Precision not allowed in integer format specifier > > What if instead that produced: > > >>> "{:.4X}".format(31) > 0000001F > >>> "{:.4X}".format(-31) > FFFFFFE1 > > Usually that field is measured in characters/digits, so this should > probably produce FFE1; you'd need {:.8X} to produce FFFFFFE1. This would > then logically extend to binary and octal, in each case measuring > characters/digits in the indicated base. > +1 Not only would that be more consistent, it would also have a use-case. From guido at python.org Sun Dec 4 02:51:48 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 3 Dec 2011 17:51:48 -0800 Subject: [Python-ideas] Negative hexes In-Reply-To: <4EDACDBB.1040607@mrabarnett.plus.com> References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <4EDACDBB.1040607@mrabarnett.plus.com> Message-ID: On Sat, Dec 3, 2011 at 5:32 PM, MRAB wrote: > On 04/12/2011 01:17, Guido van Rossum wrote: > >> On Sat, Dec 3, 2011 at 5:13 PM, Nick Coghlan > > wrote: >> >> On Sun, Dec 4, 2011 at 3:07 AM, Antoine Pitrou > > wrote: >> >> This is because Python's integers are not limited to 32 bits or >> 64 bits. If >> >> you read PEP 237, you'll see that this was one of the hardest >> differences >> >> between ints and longs to be resolved. You'd have to include an >> infinite >> >> number of leading 'F' characters to format a negative long this >> way... >> > >> > That's a fair point :) >> >> Random thought... could we use the integer precision field to fix >> *that*, by having it indicate the intended number of bytes in the >> integer? >> >> That is, currently: >> >> >>> "{:.4x}".format(31) >> Traceback (most recent call last): >> File "", line 1, in >> ValueError: Precision not allowed in integer format specifier >> >> What if instead that produced: >> >> >>> "{:.4X}".format(31) >> 0000001F >> >>> "{:.4X}".format(-31) >> FFFFFFE1 >> >> Usually that field is measured in characters/digits, so this should >> probably produce FFE1; you'd need {:.8X} to produce FFFFFFE1. This would >> then logically extend to binary and octal, in each case measuring >> characters/digits in the indicated base. >> >> +1 > > Not only would that be more consistent, it would also have a use-case. OTOH I'm not sure what should happen if the number (negative or positive!) doesn't fit in the precision. How common is this use case? Personally I'm fine with writing x & (2**N - 1) where N is e.g. 32 or 64. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Dec 4 03:06:50 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Dec 2011 12:06:50 +1000 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> Message-ID: On Sun, Dec 4, 2011 at 11:17 AM, Guido van Rossum wrote: > On Sat, Dec 3, 2011 at 5:13 PM, Nick Coghlan wrote: >> >> On Sun, Dec 4, 2011 at 3:07 AM, Antoine Pitrou >> wrote: >> >> This is because Python's integers are not limited to 32 bits or 64 >> >> bits. If >> >> you read PEP 237, you'll see that this was one of the hardest >> >> differences >> >> between ints and longs to be resolved. You'd have to include an >> >> infinite >> >> number of leading 'F' characters to format a negative long this way... >> > >> > That's a fair point :) >> >> Random thought... could we use the integer precision field to fix >> *that*, by having it indicate the intended number of bytes in the >> integer? >> >> That is, currently: >> >> >>> "{:.4x}".format(31) >> Traceback (most recent call last): >> ?File "", line 1, in >> ValueError: Precision not allowed in integer format specifier >> >> What if instead that produced: >> >> >>> "{:.4X}".format(31) >> 0000001F >> >>> "{:.4X}".format(-31) >> FFFFFFE1 > > > Usually that field is measured in characters/digits, so this should probably > produce FFE1; you'd need {:.8X} to produce FFFFFFE1. This would then > logically extend to binary and octal, in each case measuring > characters/digits in the indicated base. True, I guess it's just a matter of dividing the bit width by 2 for binary, 3 for octal (rounding up) and 4 for binary. (My brain was locked into bytes mode for some reason, so converting to character counts seemed overly complicated - of course, if you go directly from bits to characters, it's no more complicated than converting to a bytes count). ".4d" would still raise an exception, though - I don't know of any obvious way to make two's complement notation meaningful in base 10. For numbers that didn't fit in the specified precision, I'd also suggest raising ValueError. This would be tinkering with the behaviour of builtin, so I guess it would need a PEP? (I already have too many of those in train... although I did just tidy that up a bit by officially deferring consideration of the ImportEngine PEP until 3.4 at the earliest) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python at mrabarnett.plus.com Sun Dec 4 03:08:45 2011 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 04 Dec 2011 02:08:45 +0000 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <4EDACDBB.1040607@mrabarnett.plus.com> Message-ID: <4EDAD62D.8010307@mrabarnett.plus.com> On 04/12/2011 01:51, Guido van Rossum wrote: > On Sat, Dec 3, 2011 at 5:32 PM, MRAB > wrote: > > On 04/12/2011 01:17, Guido van Rossum wrote: > > On Sat, Dec 3, 2011 at 5:13 PM, Nick Coghlan > >> wrote: > > On Sun, Dec 4, 2011 at 3:07 AM, Antoine Pitrou > > >> wrote: > >> This is because Python's integers are not limited to 32 bits or > 64 bits. If > >> you read PEP 237, you'll see that this was one of the hardest > differences > >> between ints and longs to be resolved. You'd have to include an > infinite > >> number of leading 'F' characters to format a negative long this > way... > > > > That's a fair point :) > > Random thought... could we use the integer precision field > to fix > *that*, by having it indicate the intended number of bytes > in the > integer? > > That is, currently: > > >>> "{:.4x}".format(31) > Traceback (most recent call last): > File "", line 1, in > ValueError: Precision not allowed in integer format specifier > > What if instead that produced: > > >>> "{:.4X}".format(31) > 0000001F > >>> "{:.4X}".format(-31) > FFFFFFE1 > > Usually that field is measured in characters/digits, so this should > probably produce FFE1; you'd need {:.8X} to produce FFFFFFE1. > This would > then logically extend to binary and octal, in each case measuring > characters/digits in the indicated base. > > +1 > > Not only would that be more consistent, it would also have a use-case. > > > OTOH I'm not sure what should happen if the number (negative or > positive!) doesn't fit in the precision. > Well, the width is treated as the minimum width, so perhaps the precision should be the minimum precision in this case. > How common is this use case? Personally I'm fine with writing x & (2**N > - 1) where N is e.g. 32 or 64. > From python at mrabarnett.plus.com Sun Dec 4 03:14:56 2011 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 04 Dec 2011 02:14:56 +0000 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> Message-ID: <4EDAD7A0.5020300@mrabarnett.plus.com> On 04/12/2011 02:06, Nick Coghlan wrote: > On Sun, Dec 4, 2011 at 11:17 AM, Guido van Rossum wrote: >> On Sat, Dec 3, 2011 at 5:13 PM, Nick Coghlan wrote: >>> >>> On Sun, Dec 4, 2011 at 3:07 AM, Antoine Pitrou >>> wrote: >>>>> This is because Python's integers are not limited to 32 bits or 64 >>>>> bits. If >>>>> you read PEP 237, you'll see that this was one of the hardest >>>>> differences >>>>> between ints and longs to be resolved. You'd have to include an >>>>> infinite >>>>> number of leading 'F' characters to format a negative long this way... >>>> >>>> That's a fair point :) >>> >>> Random thought... could we use the integer precision field to fix >>> *that*, by having it indicate the intended number of bytes in the >>> integer? >>> >>> That is, currently: >>> >>>>>> "{:.4x}".format(31) >>> Traceback (most recent call last): >>> File "", line 1, in >>> ValueError: Precision not allowed in integer format specifier >>> >>> What if instead that produced: >>> >>>>>> "{:.4X}".format(31) >>> 0000001F >>>>>> "{:.4X}".format(-31) >>> FFFFFFE1 >> >> >> Usually that field is measured in characters/digits, so this should probably >> produce FFE1; you'd need {:.8X} to produce FFFFFFE1. This would then >> logically extend to binary and octal, in each case measuring >> characters/digits in the indicated base. > > True, I guess it's just a matter of dividing the bit width by 2 for > binary, 3 for octal (rounding up) and 4 for binary. (My brain was > locked into bytes mode for some reason, so converting to character > counts seemed overly complicated - of course, if you go directly from > bits to characters, it's no more complicated than converting to a > bytes count). > > ".4d" would still raise an exception, though - I don't know of any > obvious way to make two's complement notation meaningful in base 10. > It wouldn't be two's complement in that case, it would be ten's complement. (Ever used BCD arithmetic in assembly language? :-)) Actually, would it really be two's complement in octal or hexadecimal either? Wouldn't it be eight's complement and sixteen's complement respectively? > For numbers that didn't fit in the specified precision, I'd also > suggest raising ValueError. > > This would be tinkering with the behaviour of builtin, so I guess it > would need a PEP? (I already have too many of those in train... > although I did just tidy that up a bit by officially deferring > consideration of the ImportEngine PEP until 3.4 at the earliest) > From tjreedy at udel.edu Sun Dec 4 04:05:54 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 03 Dec 2011 22:05:54 -0500 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> Message-ID: On 12/3/2011 9:06 PM, Nick Coghlan wrote: > This would be tinkering with the behaviour of builtin, It is a backwards compatible augmentation: (as I understand the proposal) .n for ints means print negatives with n digits in baseNs-complement notation instead of raising an exception. -- Terry Jan Reedy From ncoghlan at gmail.com Sun Dec 4 05:57:32 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Dec 2011 14:57:32 +1000 Subject: [Python-ideas] Negative hexes In-Reply-To: <4EDAD7A0.5020300@mrabarnett.plus.com> References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <4EDAD7A0.5020300@mrabarnett.plus.com> Message-ID: On Sun, Dec 4, 2011 at 12:14 PM, MRAB wrote: > On 04/12/2011 02:06, Nick Coghlan wrote: >> ".4d" would still raise an exception, though - I don't know of any >> obvious way to make two's complement notation meaningful in base 10. >> > It wouldn't be two's complement in that case, it would be ten's > complement. (Ever used BCD arithmetic in assembly language? :-)) > > Actually, would it really be two's complement in octal or hexadecimal > either? Wouldn't it be eight's complement and sixteen's complement > respectively? For hexadecimal, it doesn't make much difference, since the number of bits per digit is a power of 2: 2**32 == (2**4)**8 For octal, you're right, since it would be 2**33 == (2**3)**11 I quite like Terry's definition, which does extend cleanly to 'd' (as well as giving the expected answer for 'b', 'o', 'x' and 'X'): For integers, the precision field would be used to specify the expected maximum number of digits in the answer, and to switch the representation of negative values to the appropriate 'complement' form (e.g. two's complement for binary numbers). When a precision ".prec" is specified for an integer formatting code (b, o, d, x or X), the value to be displayed would be calculated as follows: _BASES = dict(b=2,o=8,d=10,x=16,X=16) _BASE_NAMES = dict(b='binary',o='octal',d='decimal',x='hexadecimal',X='hexadecimal') _base = _BASES[format_code] _prec_bound = _base ** prec _max_value = _prec_bound / 2 if value < -_max_value or value >= _max_value: _code = _BASE_NAMES[format_code] raise ValueError("Integer {} too large for {} precision of {}".format(value, _code, n)) _value = _prec_bound - value However, I'm not sure that qualifies as *useful* behaviour - while the bounds checking aspect could be useful for decimal, the complement form of negative numbers is almost never going to be what anyone wants. If we decide to improve things in terms of Python-level handling of two's complement arithmetic, perhaps it would make more sense to just provide a method on int objects that calculates the two's complement of a number for a given bit length? Something like: def as_twos_complement(self, bits): if self.bit_length() >= bits: raise ValueError("int {} too large for {}-bit signed precision".format(self, bits)) if self >= 0: return self return 2**bits + self # self is known to be negative at this point As for whether this is worth doing or not... I think so. While Python integers may be limited in size solely by available memory, it's going to be a fact of computing life for quite some time that there are going to be fixed size signed and unsigned integers under the hood *somewhere*. We already provide a mechanism to find out how many bits a given integer needs, this would be about providing a standard, efficient, mechanism to convert negative integers to their two's complement positive equivalents for a given number of bits. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From storchaka at gmail.com Mon Dec 5 12:58:44 2011 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 05 Dec 2011 13:58:44 +0200 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> Message-ID: 04.12.11 04:06, Nick Coghlan ???????(??): > ".4d" would still raise an exception, though - I don't know of any > obvious way to make two's complement notation meaningful in base 10. Obviously, '{0:.4d}'.format(-31) == '{0:4d}'.format((-31)%10**4) == '9969'. > This would be tinkering with the behaviour of builtin, so I guess it > would need a PEP? No, no! Please, do not obfuscate Python formatting. From solipsis at pitrou.net Mon Dec 5 18:15:32 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Dec 2011 18:15:32 +0100 Subject: [Python-ideas] Negative hexes References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> Message-ID: <20111205181532.1214a70f@pitrou.net> On Sun, 4 Dec 2011 11:13:26 +1000 Nick Coghlan wrote: > On Sun, Dec 4, 2011 at 3:07 AM, Antoine Pitrou wrote: > >> This is because Python's integers are not limited to 32 bits or 64 bits. If > >> you read PEP 237, you'll see that this was one of the hardest differences > >> between ints and longs to be resolved. You'd have to include an infinite > >> number of leading 'F' characters to format a negative long this way... > > > > That's a fair point :) > > Random thought... could we use the integer precision field to fix > *that*, by having it indicate the intended number of bytes in the > integer? > > That is, currently: > > >>> "{:.4x}".format(31) > Traceback (most recent call last): > File "", line 1, in > ValueError: Precision not allowed in integer format specifier This is so poorly discoverable that I don't think it's worth it. Guido's approach looks sufficient to me. Regards Antoine. From ncoghlan at gmail.com Mon Dec 5 23:19:53 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 Dec 2011 08:19:53 +1000 Subject: [Python-ideas] Negative hexes In-Reply-To: <20111205181532.1214a70f@pitrou.net> References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <20111205181532.1214a70f@pitrou.net> Message-ID: On Tue, Dec 6, 2011 at 3:15 AM, Antoine Pitrou wrote: >> That is, currently: >> >> >>> "{:.4x}".format(31) >> Traceback (most recent call last): >> ? File "", line 1, in >> ValueError: Precision not allowed in integer format specifier > > This is so poorly discoverable that I don't think it's worth it. > Guido's approach looks sufficient to me. I agree the formatting approach is way too obscure, but did you see my later suggestion of an "as_twos_complement(bit_length)" conversion method on int objects? (we could actually provide a more generic version on numbers.Integer as well) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Tue Dec 6 00:10:54 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Dec 2011 00:10:54 +0100 Subject: [Python-ideas] Negative hexes References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <20111205181532.1214a70f@pitrou.net> Message-ID: <20111206001054.2cbba034@pitrou.net> On Tue, 6 Dec 2011 08:19:53 +1000 Nick Coghlan wrote: > On Tue, Dec 6, 2011 at 3:15 AM, Antoine Pitrou wrote: > >> That is, currently: > >> > >> >>> "{:.4x}".format(31) > >> Traceback (most recent call last): > >> ? File "", line 1, in > >> ValueError: Precision not allowed in integer format specifier > > > > This is so poorly discoverable that I don't think it's worth it. > > Guido's approach looks sufficient to me. > > I agree the formatting approach is way too obscure, but did you see my > later suggestion of an "as_twos_complement(bit_length)" conversion > method on int objects? (we could actually provide a more generic > version on numbers.Integer as well) Ah, right. Yes, I think that would be useful, although a shorter name would be nicer. to_unsigned() perhaps? Regards Antoine. From ncoghlan at gmail.com Tue Dec 6 01:14:03 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 Dec 2011 10:14:03 +1000 Subject: [Python-ideas] Negative hexes In-Reply-To: <20111206001054.2cbba034@pitrou.net> References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <20111205181532.1214a70f@pitrou.net> <20111206001054.2cbba034@pitrou.net> Message-ID: On Tue, Dec 6, 2011 at 9:10 AM, Antoine Pitrou wrote: > On Tue, 6 Dec 2011 08:19:53 +1000 > Nick Coghlan wrote: >> On Tue, Dec 6, 2011 at 3:15 AM, Antoine Pitrou wrote: >> >> That is, currently: >> >> >> >> >>> "{:.4x}".format(31) >> >> Traceback (most recent call last): >> >> ? File "", line 1, in >> >> ValueError: Precision not allowed in integer format specifier >> > >> > This is so poorly discoverable that I don't think it's worth it. >> > Guido's approach looks sufficient to me. >> >> I agree the formatting approach is way too obscure, but did you see my >> later suggestion of an "as_twos_complement(bit_length)" conversion >> method on int objects? (we could actually provide a more generic >> version on numbers.Integer as well) > > Ah, right. Yes, I think that would be useful, although a shorter name > would be nicer. to_unsigned() perhaps? Hmm, in the sense that the answer we're getting is the same answer you would get with a cast to an unsigned type at the C level? I think that's a little misleading - conceptually, the number is still signed, we're just representing it differently (i.e. explicitly using the twos complement form, rather than the the normal sign bit). I'd be OK with dropping the explicit 'twos' qualifier, though - then the method name could just be "to_complement()". I guess we'd also want a "to_signed()" to reverse the process: def to_complement(self, bits): "Convert this integer to its unsigned two's complement equivalent for the given bit length" if self.bit_length() >= bits: raise ValueError("{} is too large for {}-bit two's complement precision".format(self, bits)) if self >= 0: return self return 2**bits + self # self is known to be negative at this point def to_signed(self, bits): "Convert an integer in two's complement format to its signed equivalent for the given bit length" if self < 0: raise ValueError("{} is already signed".format(self)) if self.bit_length() > bits: raise ValueError("{} is too large for {}-bit two's complement precision".format(self, bits)) upper_bound = 2**bits if self < (upper_bound / 2): return self return upper_bound - self Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Tue Dec 6 01:16:59 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 5 Dec 2011 16:16:59 -0800 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <20111205181532.1214a70f@pitrou.net> <20111206001054.2cbba034@pitrou.net> Message-ID: On Mon, Dec 5, 2011 at 4:14 PM, Nick Coghlan wrote: > On Tue, Dec 6, 2011 at 9:10 AM, Antoine Pitrou > wrote: > > On Tue, 6 Dec 2011 08:19:53 +1000 > > Nick Coghlan wrote: > >> On Tue, Dec 6, 2011 at 3:15 AM, Antoine Pitrou < > solipsis-xNDA5Wrcr86sTnJN9+BGXg at public.gmane.org> wrote: > >> >> That is, currently: > >> >> > >> >> >>> "{:.4x}".format(31) > >> >> Traceback (most recent call last): > >> >> File "", line 1, in > >> >> ValueError: Precision not allowed in integer format specifier > >> > > >> > This is so poorly discoverable that I don't think it's worth it. > >> > Guido's approach looks sufficient to me. > >> > >> I agree the formatting approach is way too obscure, but did you see my > >> later suggestion of an "as_twos_complement(bit_length)" conversion > >> method on int objects? (we could actually provide a more generic > >> version on numbers.Integer as well) > > > > Ah, right. Yes, I think that would be useful, although a shorter name > > would be nicer. to_unsigned() perhaps? > > Hmm, in the sense that the answer we're getting is the same answer you > would get with a cast to an unsigned type at the C level? I think > that's a little misleading - conceptually, the number is still signed, > we're just representing it differently (i.e. explicitly using the twos > complement form, rather than the the normal sign bit). > I don't think there's a better term available. As long as the return value of to_unsigned() is never negative I think it's a fine name. > I'd be OK with dropping the explicit 'twos' qualifier, though - then > the method name could just be "to_complement()". > > I guess we'd also want a "to_signed()" to reverse the process: > Sure. > def to_complement(self, bits): > "Convert this integer to its unsigned two's complement equivalent > for the given bit length" > if self.bit_length() >= bits: > raise ValueError("{} is too large for {}-bit two's complement > precision".format(self, bits)) > if self >= 0: > return self > return 2**bits + self # self is known to be negative at this point > > def to_signed(self, bits): > "Convert an integer in two's complement format to its signed > equivalent for the given bit length" > if self < 0: > raise ValueError("{} is already signed".format(self)) > if self.bit_length() > bits: > raise ValueError("{} is too large for {}-bit two's complement > precision".format(self, bits)) > upper_bound = 2**bits > if self < (upper_bound / 2): > return self > return upper_bound - self > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leapyear.org Tue Dec 6 01:24:45 2011 From: bruce at leapyear.org (Bruce Leban) Date: Mon, 5 Dec 2011 16:24:45 -0800 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <20111205181532.1214a70f@pitrou.net> <20111206001054.2cbba034@pitrou.net> Message-ID: On Mon, Dec 5, 2011 at 4:14 PM, Nick Coghlan wrote: > > I'd be OK with dropping the explicit 'twos' qualifier, though - then > the method name could just be "to_complement()". > That name's ambiguous since there's also one's complement as well as ten's and nine's complements. Also, perhaps from_twos_complement might be a clearer name than to_signed. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 6 01:52:14 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 Dec 2011 10:52:14 +1000 Subject: [Python-ideas] Negative hexes In-Reply-To: References: <4ED9698A.6010609@gmail.com> <4ED99470.20205@gmail.com> <20111203174500.49810bb1@pitrou.net> <20111203180747.37c296e6@pitrou.net> <20111205181532.1214a70f@pitrou.net> <20111206001054.2cbba034@pitrou.net> Message-ID: On Tue, Dec 6, 2011 at 10:16 AM, Guido van Rossum wrote: >> Hmm, in the sense that the answer we're getting is the same answer you >> would get with a cast to an unsigned type at the C level? I think >> that's a little misleading - conceptually, the number is still signed, >> we're just representing it differently (i.e. explicitly using the twos >> complement form, rather than the the normal sign bit). > > I don't think there's a better term available. As long as the return value > of to_unsigned() is never negative I think it's a fine name. Recorded the RFE here (using "to_signed()/to_unsigned()"): http://bugs.python.org/issue13535 I was sold on the name when I read my own docstring: "Convert this integer to its unsigned two's complement equivalent for the given bit length" Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Dec 7 15:53:44 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Dec 2011 00:53:44 +1000 Subject: [Python-ideas] Exploring the 'strview' concept further Message-ID: With encouragement from me (and others) Armin Ronacher recently attempted to articulate his problems in dealing with the migration to Python 3 [1]. They're actually quite similar to the feelings I had during my early attempts at restoring the ability of the URL parsing APIs to deal directly with ASCII-encoded binary data, rather than requiring that the application developer explicitly decode it to text first [2]. Now, I clearly disagree with Armin on at least one point: there already *is* "one true way" to have unified text processing code in Python 3. That way is the way the Python 3.2 urllib.parse module handles it: as soon as it is handed something that isn't a string, it attempts to decode it using a default assumed encoding (specifically 'ascii', at least for now). It keeps track of whether or not the arguments were decoded from bytes and, if they were, encodes the return value on output [3]. If you're pipelining such interfaces, it's obviously more efficiently to just decode once before invoking the pipeline and then (optionally) encoding again at the end (just as is the case in Python 2), but you can still make your APIs largely polymorphic with respect to bytes and text without massive internal code duplication. So, that's always one of my first suggestions to people struggling with Python 3's unicode model: I ask if they have tried putting aside any concerns they may have about possible losses of efficiency, and just tried the decode-on-input-and-return-an-output-coercion-function, coerce-on-output approach. Python used to do this implicitly for you at every string operation (minus the 'coerce on output' part), but now it is asking that you do it manually, and decide for *yourself* on an appropriate encoding, instead of the automatic assumption of ASCII text that is present in Python 2 (we'll leave aside the issue of platform-specific defaults in various contexts - that's a whole different question and one I'm not at all equipped to answer. I don't think I've ever even had to work on a system with any locale other than en_US or en_GB). Often this actually resolves their problem (since they're no longer fighting the new Unicode model, and instead embracing it), and this is why PEP 393 is going to be such a big deal when Python 3.3 is released next year. Protocol developers are *right* to be worried about a four-fold increase in memory usage (and the flow on effects on CPU usage and cache misses) when going from bytes data to the UCS4 internal Unicode format used on most distro-provided Python builds for Linux. With PEP 393's flexible internal representations, the amount of memory used will be as little as possible while still allowing straightforward O(1) lookup of individual code points. However, that urllib.urlparse code also highlights another one of Armin's complaints: like much of the stdlib (and core interpreter!), it doesn't ducktype 'str'. Instead, it demands the real thing and accepts no substitutes (not even collections.UserString). This kind of behaviour is quite endemic - the coupling between the interpreter and the details of the string implementation is, in general, even tighter than that between the interpreter and the dict implementation used for namespaces. With PEP 3118, we introduced the concept of 'memoryview' to make allowance for the fact that it is often useful to look at the same chunk of memory in multiple ways, *without* incurring the costs of making multiple copies. In a discussion back in June [4], I briefly mentioned the idea of a 'strview' type that would extend those concepts to providing a str-like view of a region of memory, *without* necessarily making a copy of the entire thing. DISCLAIMERS: 1. I don't know yet if this is a good idea. It may in fact be a terrible idea. I think it is, at least, an idea worth discussing further. 2. Making this concept work may require actually *classifying* our codecs to some degree (for attributes like 'ASCII-compatible', 'stateless', 'fixed width', etc). That might be tedious, but doesn't seem completely infeasible. 3. There are issues with memoryview itself that should be accounted for if pursuing this idea [5] 4. There is an issue with CPython's operand coercion for sequence concatenation and repetition that may affect attempts to implement this idea, although you should be fine so long as you implement the number methods in addition to the sequence ones (which happens automatically for classes written in Python) [6] So, how might a 'strview' object work? 1. The basic construction would be "strview(object, encoding, errors)". For convenience, actual str objects would just be returned unmodified (alternatively: a factory function could be provided with that behaviour) 2. A 'strview' *wouldn't* try to pass itself off as a real string for all purposes. Instead, it would support a new String ABC (more on that below). 4. The encode() method would work like a string's normal encode() method, decoding the original object to a str, then encoding that to the desired encoding. If the encodings match, then an optimised fast path of simply calling bytes() on the underlying object would be used. 5. If asked to index, slice or iterate over the underlying string, the strview would use the incremental decoder for the relevant codec to build an efficient mapping from code point indices to byte indices and then return real strings (various strategies for doing this have been posted to this list in the past). Alternatively, if codecs were classified to explicitly indicate when they implemented stateless fixed width encodings, then strview could simply be restricted to only working with that subset of possible encodings. The latter strategy might be needed to get around issues with stateful encodings like ShiftJIS and ITA2 - those are hard (impossible?) to index and interpret efficiently without fully decoding them and storing the result. 6. The new type would implement the various binary operators supported by strings, promoting itself to a real string type whenever needed 7. The new type would similarly support the full string API, returning actual string objects rather than any kind of view. What might a String ABC provide? For a very long time, slice indices had to be real integers - we didn't allow other "integer like" types. The reason was that floats implemented __int__, so ducktyping on that method would have allowed binary floating point numbers in functions where we didn't want to permit them. The answer, ultimately, was to introduce __index__ (and, eventually, numbers.Integral) to mark "true" integers, allowing things like NumPy scalars to be used directly as slice indices without inheriting from int. An explicit String ABC, even if not supported for performance critical core functionality like identifiers, would allow the implementation of code like that in urllib.urlparse to be updated to avoid keying behaviour on the concrete builtin str type - instead, it would check against the String ABC, allowing for all the usual explicit type registration goodies that ABCs support (and that make them much better for type checking than concrete types). Just as much of the old UserDict functionality is now available on Mapping and MutableMapping, so much of the existing UserString functionality could be moved to the hypothetical String ABC. Hopefully-the-rambling-isn't-too-incoherent'ly-yours, Nick. [1] http://lucumr.pocoo.org/2011/12/7/thoughts-on-python3/ [2] http://bugs.python.org/issue9873 [3] http://hg.python.org/cpython/file/default/Lib/urllib/parse.py#l74 [4] http://mail.python.org/pipermail/python-ideas/2011-June/010439.html [5] http://bugs.python.org/issue10181 [6] http://bugs.python.org/issue11477 -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Wed Dec 7 16:39:58 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Dec 2011 16:39:58 +0100 Subject: [Python-ideas] Exploring the 'strview' concept further References: Message-ID: <20111207163958.5ac4a9fe@pitrou.net> On Thu, 8 Dec 2011 00:53:44 +1000 Nick Coghlan wrote: > > 3. There are issues with memoryview itself that should be accounted > for if pursuing this idea [5] These issues are related to complex buffer types (strided, multi-dimensional, etc.). They wouldn't apply to a hypothetical "linear unicode buffer". > 1. The basic construction would be "strview(object, encoding, > errors)". For convenience, actual str objects would just be returned > unmodified (alternatively: a factory function could be provided with > that behaviour) The factory function is a better idea than silent pass-through, IMO. > 5. If asked to index, slice or iterate over the underlying string, the > strview would use the incremental decoder for the relevant codec to > build an efficient mapping from code point indices to byte indices and > then return real strings (various strategies for doing this have been > posted to this list in the past). Be careful, the incremental decoders use a layer of pure Python wrappers. You want to call them on big blocks (at least 4 or 8KB, as TextIOWrapper does) if you don't want to lose a lot of speed. So building a mapping may not be easy. Even bypassing the Python layer would still incur the overhead of repeatedly calling a standalone function, instead of having a tight loop such as the following: http://hg.python.org/cpython/file/e49220f4c31f/Objects/unicodeobject.c#l4228 And of course, your mapping must be space-efficient enough that it's much smaller than the full decoded string. I think that for small strings (< 1024 bytes?), decoding and storing the decoded string are not a big deal. Decoding once is *much* faster (especially for optimized encodings such as latin-1 or utf-8, and only them will be left in a few years) than trying to do it piecewise. strview would only be a win for rather large strings. Which makes it useless for URL parsing ;) > Alternatively, if codecs were > classified to explicitly indicate when they implemented stateless > fixed width encodings, then strview could simply be restricted to only > working with that subset of possible encodings. From an usability POV this seems undesireable. On the other hand, if complete decoding is required, calling str() is just as cheap. > 7. The new type would similarly support the full string API, returning > actual string objects rather than any kind of view. Even for slicing? From barry at python.org Wed Dec 7 20:50:57 2011 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Dec 2011 14:50:57 -0500 Subject: [Python-ideas] Exploring the 'strview' concept further References: Message-ID: <20111207145057.3522180f@resist.wooz.org> On Dec 08, 2011, at 12:53 AM, Nick Coghlan wrote: >With encouragement from me (and others) Armin Ronacher recently >attempted to articulate his problems in dealing with the migration to >Python 3 [1]. They're actually quite similar to the feelings I had >during my early attempts at restoring the ability of the URL parsing >APIs to deal directly with ASCII-encoded binary data, rather than >requiring that the application developer explicitly decode it to text >first [2]. I've just finished a port of dbus-python to Python 3, submitting the patches upstream, although they haven't been reviewed yet. It was an interesting exercise for many reasons, and I have my own thoughts the state of porting which I'll post at another time. I agree with some of the issues that Armin brought up, and disagree with others. ;) At the C level, dbus defines its strings as UTF-8 encoded char*'s. My first crack at this (despite upstream's thinking otherwise) was to use bytes to represent these objects. That turned out to be completely infeasible for several reasons. The biggest problem is that some of the core callback dispatch code was doing slicing and comparisons of these objects against literals, or externally registered objects. So where you might see something like: >>> s = ':1/123' >>> s[:1] == ':' True this doesn't work when `s` is a bytes object. Given the number of places internally that would have to be fixed, and the cost of imposing a potentially huge number of changes to clients of the library, I ultimately decided that upstream's suggestion to model these things as unicodes was right after all. Once I made that change, the port went relatively easily, as judged by the amount of time it took to get the test suite completing successfully. ;) At the Python level, many of the interfaces which accept 8-bit strings in Python 2, accept bytes or strs in Python 3, where the bytes must be utf-8 encoded. Internally, I had to do many more type checks for PyUnicodes, and then decode them to bytes before I could pass them to do the dbus C API. In almost all cases though, returning data from the dbus C API involved decoding the char*'s to unicodes, not bytes. This means rather than returning bytes if bytes were given, at the Python layer, unicode is always returned. This I think causes the least disruption in user code. Well, we'll see as I'm now going to be porting some dbus-based applications. >Now, I clearly disagree with Armin on at least one point: there >already *is* "one true way" to have unified text processing code in >Python 3. That way is the way the Python 3.2 urllib.parse module >handles it: as soon as it is handed something that isn't a string, it >attempts to decode it using a default assumed encoding (specifically >'ascii', at least for now). It keeps track of whether or not the >arguments were decoded from bytes and, if they were, encodes the >return value on output [3]. If you're pipelining such interfaces, it's >obviously more efficiently to just decode once before invoking the >pipeline and then (optionally) encoding again at the end (just as is >the case in Python 2), but you can still make your APIs largely >polymorphic with respect to bytes and text without massive internal >code duplication. It's certainly an interesting idea, although I can't decide whether this is more implicit or more explicit. I'm not sure it would have helped me with dbus-python since most of the porting work happens at the interface between Python and an existing C API with clearly defined semantics. There's also a large body of existing code that uses the library, so an important goal is to make *their* porting easier. Have you had any experience porting applications which use the new urllib and does it make your life easier? >So, that's always one of my first suggestions to people struggling >with Python 3's unicode model: I ask if they have tried putting aside >any concerns they may have about possible losses of efficiency, and >just tried the decode-on-input-and-return-an-output-coercion-function, >coerce-on-output approach. Python used to do this implicitly for you >at every string operation (minus the 'coerce on output' part), but now >it is asking that you do it manually, and decide for *yourself* on an >appropriate encoding, instead of the automatic assumption of ASCII >text that is present in Python 2 (we'll leave aside the issue of >platform-specific defaults in various contexts - that's a whole >different question and one I'm not at all equipped to answer. I don't >think I've ever even had to work on a system with any locale other >than en_US or en_GB). I also wonder if some of these ideas would help with RDM's re-imagining of the email package. email is notoriously quantum in its bytes/str duality - sometimes the same email data needs to act like a str or a bytes at different times. The email-sig design attempts to address this with a layered approach, although I think we're still wondering whether this is going to work in practice. (I'm not sure about the current state of the email package work.) One difference here is that email usually tells you explicitly what the encoding is. Assuming the Content-Types don't lie or specify charsets that are unknown to Python, I think it would be nice to pass them through the APIs for better idempotency. >However, that urllib.urlparse code also highlights another one of >Armin's complaints: like much of the stdlib (and core interpreter!), >it doesn't ducktype 'str'. Instead, it demands the real thing and >accepts no substitutes (not even collections.UserString). This kind of >behaviour is quite endemic - the coupling between the interpreter and >the details of the string implementation is, in general, even tighter >than that between the interpreter and the dict implementation used for >namespaces. In the dbus port, this is mostly not a problem, since the dbus-python data types derive from PyBytes or PyUnicode. There are one or two places that do _CheckExacts() for various reasons, but I think mostly everything works properly with _Check() calls. I know that's not duck-typing though, and I agree with you about the problem at the interpreter level. (It turned out to be a bigger PITA with int/long. An important requirement was to not change the API for Python 2, so types which derived from PyInts in Python 2 had to derive from PyLongs in Python 3. Liberal use of #ifdefs mostly handles the complexity, but porting clients to Python 3 will definitely see API differences, e.g. in inheritance structure and such.) I've rambled on enough, but I think you and Armin bring up some good points that we really need to address in Python 3.3. I'm fairly convinced that there's little we could have done different before now, and most of these issues are cropping up because people are doing actual real-world ports now, which is a good thing! I also strongly disagree with Armin that a Python 2.8 could help in any way. Much better to expose these problems now and help make Python 3.3 the best it can be. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Dec 8 00:51:45 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Dec 2011 09:51:45 +1000 Subject: [Python-ideas] Exploring the 'strview' concept further In-Reply-To: <20111207163958.5ac4a9fe@pitrou.net> References: <20111207163958.5ac4a9fe@pitrou.net> Message-ID: On Thu, Dec 8, 2011 at 1:39 AM, Antoine Pitrou wrote: >> Alternatively, if codecs were >> classified to explicitly indicate when they implemented stateless >> fixed width encodings, then strview could simply be restricted to only >> working with that subset of possible encodings. > > >From an usability POV this seems undesireable. On the other hand, if > complete decoding is required, calling str() is just as cheap. Yeah, that's kind of where I'm going with this. For stateless encodings, views make sense - it's really just a memory view with a particular way of interpreting the individual characters and providing the string API rather than the bytes one. For the multitude of ASCII-compatible single byte codings and the various fixed-width encodings, that could be very useful. With some fiddling, you could support BOM and signature encodings, too (just by offsetting your view a bit and adjusting your interpretation of the individual code points). But for the fully general case of stateful encodings (including all variable width encodings) it is basically impossible to do O(1) indexing (which is the whole reason the Unicode model is the way it is). Especially once PEP 393 is in place, you rapidly reach a point of diminishing returns where converting the whole shebang to Unicode code points and working directly on the code point array is the right answer (and, if it isn't, you're clearly doing something sufficiently sophisticated that you're going to be OK with rolling your own tools to deal with the problem). In those terms, I'm actually wondering if it might be appropriate to extract some of the tools I created for the urllib.parse case and publish them via the string module. 1. Provide a string.Text ABC (why *did* we put UserString in collections, anyway?) 2. Provide a "coerce_to_str" helper: def coerce_to_str(*args, encoding, errors='strict'): # Invokes decode if necessary to create str args # and returns the coerced inputs along with # an appropriate result coercion function # - a noop for str inputs # - encoding function otherwise # False inputs (including None) are all coerced to the empty string args_are_text = isinstance(args[0], Text) if args_are_text: def _encode_result(obj): return obj else: def _encode_result(obj): return obj.encode(encoding, errors) def _decode(obj): if not obj: return '' if isinstance(obj, Text): return str(obj) return obj.decode(encoding, errors) def _decode_args(args): return tuple(map(_decode, args)) for arg in args[1:]: # We special-case False values to support the relatively common # use of None and the empty string as default arguments if arg and args_are_text != isinstance(arg, Text): raise TypeError("Cannot mix text and non-text arguments") return _decode_args(args) + (_encode_result,) Note the special-casing of None would be sufficient to support arbitrary defaults in binary/text polymorphic APIs: def f(a, b=None): (a_str, b_str), _coerce_result = coerce_to_str(a, b, 'utf-8') if b is None: b_str = "Default text" Cheers, Nick. >> 7. The new type would similarly support the full string API, returning >> actual string objects rather than any kind of view. > > Even for slicing? If we restricted strview to stateless encodings, then slicing could also return views (there wouldn't be any point in returning a view for iteration or indexing though - the view object would be bigger than any single-character string. In fact, we could probably figure out a cutoff whereby real strings are returned for sufficiently small slices, too). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Dec 8 01:22:32 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Dec 2011 10:22:32 +1000 Subject: [Python-ideas] Exploring the 'strview' concept further In-Reply-To: <20111207145057.3522180f@resist.wooz.org> References: <20111207145057.3522180f@resist.wooz.org> Message-ID: On Thu, Dec 8, 2011 at 5:50 AM, Barry Warsaw wrote: > This means rather than returning bytes if bytes were given, at the Python > layer, unicode is always returned. ?This I think causes the least disruption > in user code. ?Well, we'll see as I'm now going to be porting some dbus-based > applications. If you look at the way the os APIs are defined, they mostly work on a "bytes in, bytes out" model (e.g. os.listdir(), os.walk()). Where there is no input type to reliably determine the expected output type, then a 'b' variant is added (e.g. os.getcwdb(), os.environb). (And yes, I do occasionally wonder if we should have a builtin "openb" shorthand for "open(name, 'b')" with a signature that omits all the parameters that are only valid for text mode files: "openb(file, mode, buffering, closefd)") An "always unicode out" model can work, too, but you need to be sure your clients can cope with that. One thing I like about my proposed string.coerce_to_str() API is that you can use it to implement either approach. If you want bytes->bytes, then you call the result coercion function, if you want to always emit unicode, then you skip that step. >>Now, I clearly disagree with Armin on at least one point: there >>already *is* "one true way" to have unified text processing code in >>Python 3. That way is the way the Python 3.2 urllib.parse module >>handles it: as soon as it is handed something that isn't a string, it >>attempts to decode it using a default assumed encoding (specifically >>'ascii', at least for now). It keeps track of whether or not the >>arguments were decoded from bytes and, if they were, encodes the >>return value on output [3]. If you're pipelining such interfaces, it's >>obviously more efficiently to just decode once before invoking the >>pipeline and then (optionally) encoding again at the end (just as is >>the case in Python 2), but you can still make your APIs largely >>polymorphic with respect to bytes and text without massive internal >>code duplication. > > It's certainly an interesting idea, although I can't decide whether this is > more implicit or more explicit. ?I'm not sure it would have helped me with > dbus-python since most of the porting work happens at the interface between > Python and an existing C API with clearly defined semantics. ?There's also a > large body of existing code that uses the library, so an important goal is to > make *their* porting easier. ?Have you had any experience porting applications > which use the new urllib and does it make your life easier? No, the API updates were in response to user requests (and extended discussions here and on python-dev). However, the use case of getting URL components off the wire, manipulating them and putting them back on the wire, all in an RFC compliant strict ASCII encoding made a lot of sense, which is why we ended up going with this model (that and the os module precedent). > I also wonder if some of these ideas would help with RDM's re-imagining of the > email package. ?email is notoriously quantum in its bytes/str duality - > sometimes the same email data needs to act like a str or a bytes at different > times. ?The email-sig design attempts to address this with a layered approach, > although I think we're still wondering whether this is going to work in > practice. ?(I'm not sure about the current state of the email package work.) The email model in 3.2 (which actually makes it somewhat usable) is quite similar to the way urllib.urlparse works (although I believe there may be a bit more internal code duplication). The two API updates actually co-evolved as part of the same set of discussions, which is how they came to share the bytes->bytes, str->str philosophy. I'm not sure about the status of email6 either, though - has anyone heard from RDM lately? > I've rambled on enough, but I think you and Armin bring up some good points > that we really need to address in Python 3.3. ?I'm fairly convinced that > there's little we could have done different before now, and most of these > issues are cropping up because people are doing actual real-world ports now, > which is a good thing! ?I also strongly disagree with Armin that a Python 2.8 > could help in any way. ?Much better to expose these problems now and help make > Python 3.3 the best it can be. Indeed - it's *already* the case that ports that can drop 2.5 support have a much easier time of things, and those that can also drop 2.6 have it easier still (I believe the latter is uncommon though, since most people want their stuff to run on platforms like the RHEL6 system Python installation). Whether or not 2.8 exists won't magically make the need to support 2.5 (or even earlier versions!) go away - the only thing that will make that happen is time, as people upgrade their underlying OS installations. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Thu Dec 8 04:22:37 2011 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Dec 2011 22:22:37 -0500 Subject: [Python-ideas] Exploring the 'strview' concept further In-Reply-To: References: <20111207145057.3522180f@resist.wooz.org> Message-ID: <20111207222237.42c88605@limelight.wooz.org> On Dec 08, 2011, at 10:22 AM, Nick Coghlan wrote: >Indeed - it's *already* the case that ports that can drop 2.5 support >have a much easier time of things, and those that can also drop 2.6 >have it easier still (I believe the latter is uncommon though, since >most people want their stuff to run on platforms like the RHEL6 system >Python installation). I think the ability to drop anything less than 2.6 is a huge win, but the ability to drop 2.6 itself is a smaller win. Yes, there are things in 2.7 that help make porting to Python 3 easier still, but at least IME so far, not so much as to be overwhelmingly compelling. Python 2.6 as a minimum *is* very compelling. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From jimjjewett at gmail.com Thu Dec 8 17:13:53 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 8 Dec 2011 11:13:53 -0500 Subject: [Python-ideas] Exploring the 'strview' concept further In-Reply-To: References: <20111207163958.5ac4a9fe@pitrou.net> Message-ID: On Wed, Dec 7, 2011 at 6:51 PM, Nick Coghlan wrote: > For stateless > encodings, views make sense - it's really just a memory view with a > particular way of interpreting the individual characters and providing > the string API rather than the bytes one. For the multitude of > ASCII-compatible single byte codings and the various fixed-width > encodings, that could be very useful. With some fiddling, you could > support BOM and signature encodings, too (just by offsetting your view > a bit and adjusting your interpretation of the individual code > points). I really like PEP 393, and it has gotten much better even since the initial proposal, but this one objection has been bugging me the whole time -- I just can't find a good way to explain it. But with the concrete code, I will take a stab now... I want the ability to use a more efficient string representation when I know one exists -- such as when I could be using a single-byte charset other than Latin-1, or when the underlying data is bytes, but I want to treat it as text temporarily without copying the whole buffer. PyUnicode_Kind already supports the special case of PyUnicode_WCHAR_KIND (also known as "legacy string, not ready" -- http://hg.python.org/cpython/file/174fbbed8747/Include/unicodeobject.h around line 247). I would like to see another option for "custom subtype", and to accept that strings might stay in this state longer. A custom type need not allow direct access to the buffer as an array, so it would have to provide its own access functions. I accept that using these subtype-specific functions might be slower, but I think the downside for "normal" strings can be limited to an extra case statement in places like PyUnicode_WRITE (at http://hg.python.org/cpython/file/174fbbed8747/Include/unicodeobject.h lines 487-508; currently the default case asserts PyUnicode_4BYTE_KIND). Looing at Barry's example: >>> s = ':1/123' >>> s[:1] == ':' True Modelling this as bytes with a unicode view on top, this would work fine (so long as you sliced the view, rather than the original bytes object), but creating that string view wouldn't require copying the buffer. (Of course, the subtype's implementation of PyUnicode_Substring might well copy parts of the buffer.) I would expect bytes in particular to grow an as_string(encoding="Latin-1") method, which could be used to deprecate the various string-related methods. A type for an alternate one-byte encoding could be as simple as using the 1Byte variants to create a string of the same type when large strings are called for, and a translation function when individual characters are requested. -jJ From solipsis at pitrou.net Thu Dec 8 21:34:55 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 08 Dec 2011 21:34:55 +0100 Subject: [Python-ideas] Exploring the 'strview' concept further In-Reply-To: References: <20111207163958.5ac4a9fe@pitrou.net> Message-ID: <1323376495.3274.16.camel@localhost.localdomain> > But with the concrete code, I will take a stab now... > > I want the ability to use a more efficient string representation when > I know one exists -- such as when I could be using a single-byte > charset other than Latin-1, or when the underlying data is bytes, but > I want to treat it as text temporarily without copying the whole > buffer. How long is your buffer? Have you timed how long it takes to "copy" (or decode) it? > PyUnicode_Kind already supports the special case of > PyUnicode_WCHAR_KIND (also known as "legacy string, not ready" -- > http://hg.python.org/cpython/file/174fbbed8747/Include/unicodeobject.h > around line 247). I would like to see another option for "custom > subtype", and to accept that strings might stay in this state longer. The unicode implementation is already complicated enough. I think adding one further option will be a tough sell, if it doesn't exhibit major benefits. (note PyUnicode_WCHAR_KIND is deprecated and supposed to be removed some day, perhaps Python 4 :-)) > I would expect bytes in particular to grow an > as_string(encoding="Latin-1") method, which could be used to deprecate > the various string-related methods. Why deprecate useful functionality? Regards Antoine. From stephen at xemacs.org Fri Dec 9 05:41:41 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 09 Dec 2011 13:41:41 +0900 Subject: [Python-ideas] Exploring the 'strview' concept further In-Reply-To: References: <20111207163958.5ac4a9fe@pitrou.net> Message-ID: <878vmm1hmi.fsf@uwakimon.sk.tsukuba.ac.jp> Jim Jewett writes: > I want the ability to use a more efficient string representation when > I know one exists -- such as when I could be using a single-byte > charset other than Latin-1, For most people all of the time, and for almost all people most of the time, this is a YAGNI, and gets more so every year. As a facility of the language, it is an attractive nuisance for developers many of whom will undoubtedly go searching for truffles and end up consuming Knuth's root of all error, and will attract lots of one-off RFEs to deal with specific use cases that break with a minimal implementation. N.B. Emacs has just given up on a 15-year experiment with such a minimal facility (the execrable "string-as-unibyte" toggle). Use of multiple internal text encodings is really a can of worms, as the Emacs experience demonstrates (they were unable to even write Latin-1 files properly, with repeated regressions of the so-called "\201 bug" that I know of 1995-2008, mostly because of misuse of string-as-unibyte). XEmacs, with a proper character type, eliminated the "\201 bug" *before* its multilingual version stopped crashing in the codecs. But even there, because the internal character type is based on ISO-2022, it sucks, and we consistent have issues with bogus decoding and the like that is hard to get around at the app level because there's way too much generality at the underlying level that we try to handle "transparently". That's where you're going; maybe you can do better than XEmacs levels of suckiness :-), but (for a general facility) it won't be easy. Better to do that at the application level, which can decide for itself what safeguards are needed. > or when the underlying data is bytes, but I want to treat it as > text temporarily without copying the whole buffer. That ship has sailed AFAICS. If the "copy-the-whole-buffer" style of polymorphism isn't good enough, you have special knowledge of the data and/or the application, and it's a layering violation to ask Python to manage that data for you because Python's model of text is str. It will result in unexpected UnicodeErrors. > A custom type need not allow direct access to the buffer as an array, > so it would have to provide its own access functions. I accept that > using these subtype-specific functions might be slower, but I think > the downside for "normal" strings can be limited to an extra case > statement in places like PyUnicode_WRITE I expect that library code that must be robust against UnicodeError (eg, email) will need to be prepared for gratuitous errors from custom types. Since that's at least a desideratum for all stdlib code, this could be rather more expensive than you suggest. > A type for an alternate one-byte encoding could be as simple as using > the 1Byte variants to create a string of the same type when large > strings are called for, and a translation function when individual > characters are requested. This would require substantial analysis in some cases I would expect to be common to determine whether it wasn't a pessimization. I suppose that in many use cases, you will be implicitly creating many strings, and the space overhead of the implicit strings may be greater than the size of the single string. In cases where the analysis is simple (eg, parsing an RFC 822 message header out of the middle of a huge mbox file), the analysis that shows that this could be done efficiently with a custom type can easily and efficiently be converted to an implementation based on converting only the bytes needed. I understand the attraction of such facilities for simplifying user code, but given my somewhat extensive experience with maintaining them, I recommend that Python core Just Say No. It's just too hard to maintain "text invariants" when you might be processing a few million bytes from /dev/urandom. If one (as an application programmer) knows better, and of course she does, then shouldn't she DTRT at the application code level? From dirkjan at ochtman.nl Fri Dec 9 21:48:58 2011 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Fri, 9 Dec 2011 21:48:58 +0100 Subject: [Python-ideas] [Python-Dev] [PATCH] Adding braces to __future__ In-Reply-To: <20111209202629.GB2319@slate.Speedport_W_723V_Typ_A> References: <20111209202629.GB2319@slate.Speedport_W_723V_Typ_A> Message-ID: On Fri, Dec 9, 2011 at 21:26, Cedric Sodhi wrote: > (not proof-read) A good sign that it's too long to inflict upon others... Also, I scrolled and scrolled but failed to see any example of the kind of change you seem to want. Also, if you really really had read all the prior discussions, you'd have known that this post belongs on python-ideas, not python-dev. Redirecting there. Cheers, Dirkjan From ncoghlan at gmail.com Sat Dec 10 06:06:13 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 10 Dec 2011 15:06:13 +1000 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) Message-ID: A more appropriate way to phrase this suggestion would be one that clearly states the problems with the status quo and explores them, rather than assuming an answer, and indulging in a long rambling diatribe against those that will shout you down for adding nothing new to the discussion. Step 1: post to the right mailing list (i.e. python-ideas, not python-dev) Step 2: don't assume the conclusion in the subject line. Ask a question, don't state an imperative. For example: "An Alternate Suite Delineation Syntax For Python?" Step 3: respect the time of others, by trimming your message to its core essence For example: An Alternate Suite Delineation Syntax For Python? ================================================= Python's whitespace based delineation of suites is one of its greatest strengths. It aligns what the human reader perceives with what the computer is actually executing, reducing the frequency of semantic errors due to mismatches between the use of separate block delimiters and the human readable indentation. However, this benefit comes at quite a high price: it is effectively impossible to embed arbitrary Python statements into any environment where leading whitespace is *not* significant, including Python's own expression syntax. It can be argued that this restriction has led directly to the introduction of "expression friendly" variants of several Python top level constructs (for example, lambda expressions, conditional expressions and as a contributing factor in creating the various forms of comprehension). It is also one of the reasons Python-based templating languages almost always create their own custom syntax - embedding Python's own whitespace sensitive statement syntax into environments where leading whitespace is either ignored or forms a significant part of the template output is a formidable challenge. In other languages, this kind of issue is handled by using explicit suite and statement delimiters (often braces and semi-colons, respectively) to allow full suites to be used as expressions. While Python uses braces for another purpose (dictionary and set definitions), it is already the case that semi-colons (';') can be used as statement terminators, both optionally at the end of any simple statement, and also to combine multiple simple statements into a single larger statement (e.g. "x += y; print(x)"). It seems that this existing feature could be combined with a brace-based notation to create an unambiguous "suite expression" syntax that would enjoy the same semantics as ordinary Python suites (i.e. doesn't create a new scope, doesn't directly affect control flow), but allows *all* Python statements to be embedded inside expressions. Currently, the character sequence "{:" is a Syntax Error: you are attempting to end a compound statement header line while an opening brace remains unmatched, or else trying to build a dictionary without specifying the key value. This creates an opportunity to re-use braces for a suite expression syntax without conflicting with their use for set and dictionary construction. Specifically, it should be possible to create a variant of the top-level Python syntax that: 1. Explicitly delimits suites using the notation "{:" to open the suite and "}" to end it 2. Requires the use of ";" to separate simple statements (i.e. newline characters would not end a statement, since we would be inside an expression) 3. Requires that all subordinate suites also be suite expressions (i.e. leading whitespace would not be significant, since we would be inside an expression) 4. Returns the value of the last statement in the suite as the result of the suite expression (since return statements would affect the containing scope) 5. Anonymous class and function definitions would be permitted in a suite expression (but accepting only a suite expression instead of an ordinary suite) (Ruby block notation and C's comma expressions are pretty much direct inspiration for the above list) Some examples: Raise expressions: x = y if y is not None else {: raise ValueError("y must not be None!")} Try expressions: x = {: try {: y.hello} except AttributeError {: "world!"}} With expressions: data = {: with open(fname) as f {: f.read()}} Embedded assignments: if {: m = pat.search(data); m is not None}: # do something with m else: # No match! In-order conditional expressions: {: if a {:b} else {:c}} One-liner accumulator function: def acc(n) {: s=n; return {: def (i) {: nonlocal s; s += i; return s}}} Cheers, Nick. P.S. I actually think the above idea is kinda cryptic and ugly. It is, however, extraordinarily powerful, especially when it comes to embedding Python code inside other environments that may not be friendly to significant leading whitespace. Applying the excellent set of criteria suggested by Mike Meyer: 1. What's the use case? Cleanly embedding arbitrary Python statements inside environments that are not generally sensitive to leading whitespace, such as templating languages, Python expressions and Python strings. 2. Does it make such code more readable? Yes, I believe it does, by providing "one obvious way to do it". Currently there are a mish-mash of alternatives out there (or else people give up and use something else entirely, like Javascript). 3. Does it encourage writing unreadable code? Perhaps. You certainly end up with a lot of braces, colons and semi-colons floating around. However, that's also something that can be addressed by style guides - if people are using the embedded syntax when the indented syntax would work fine, the problem isn't really due to the embedded syntax, it's due to people not caring about maintainability. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Sat Dec 10 06:08:57 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 10 Dec 2011 00:08:57 -0500 Subject: [Python-ideas] [Python-Dev] [PATCH] Adding braces to __future__ In-Reply-To: References: <20111209202629.GB2319@slate.Speedport_W_723V_Typ_A> Message-ID: On 12/9/2011 3:48 PM, Dirkjan Ochtman wrote: > On Fri, Dec 9, 2011 at 21:26, Cedric Sodhi wrote: >> (not proof-read) > > A good sign that it's too long to inflict upon others... > > Also, I scrolled and scrolled but failed to see any example of the > kind of change you seem to want. > > Also, if you really really had read all the prior discussions, you'd > have known that this post belongs on python-ideas, not python-dev. > Redirecting there. Rude trollish rants on something that isn't going to change do not belong here either. They belong on python-list or a personal blog. FWIW I use Python *because* it uses indents and not braces. -- Terry Jan Reedy From cs at zip.com.au Sat Dec 10 08:30:24 2011 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 10 Dec 2011 18:30:24 +1100 Subject: [Python-ideas] Adding braces to __future__ In-Reply-To: <4EE2A140.2010509@pearwood.info> References: <4EE2A140.2010509@pearwood.info> Message-ID: <20111210073023.GA31670@cskk.homeip.net> On 10Dec2011 11:01, Steven D'Aprano wrote: | Guido van Rossum wrote: | >Point of order (repeated), please move this thread to python-ideas. | | Isn't that cruel to the people reading python-ideas? That's what it's _for_! -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ A monkey has the right to copy what he sees other monkeys doing. Shouldn't humans have equivalent rights? - Alien Being From ron3200 at gmail.com Sat Dec 10 08:33:01 2011 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 10 Dec 2011 01:33:01 -0600 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: Message-ID: <1323502381.19113.36.camel@Gutsy> On Sat, 2011-12-10 at 15:06 +1000, Nick Coghlan wrote: > One-liner accumulator function: > def acc(n) {: s=n; return {: def (i) {: nonlocal s; s += i; return > s}}} >>> def acc(n):s=n; return lambda i, s=s: s+i ... >>> a = acc(2) >>> a(1) 3 >>> a(3) 5 >>> a(7) 9 ;-) In your example returning an unnamed function is a new feature in addition to the braces. It would probably need to be more like this... def acc(n){:s=n; def add(i){:nonlocal s; s+=i; return s};return add} There are a couple of issues to work out. Statements (def, class, for, while, and with) don't work after simi-colons. There may be others. Outside those issues, all you really need is an exit suit token. The colon can still be the open suit. Two semi-colons could do it. def acc(n):s=n; def add(i):nonlocal s; s+=i; return s;; return add The real issues are in getting the statements to work. Cheers, Ron From p.f.moore at gmail.com Sat Dec 10 12:32:46 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 10 Dec 2011 11:32:46 +0000 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: Message-ID: On 10 December 2011 05:06, Nick Coghlan wrote: > A more appropriate way to phrase this suggestion would be one that > clearly states the problems with the status quo and explores them, > rather than assuming an answer, and indulging in a long rambling > diatribe against those that will shout you down for adding nothing new > to the discussion. > > Step 1: post to the right mailing list (i.e. python-ideas, not python-dev) > > Step 2: don't assume the conclusion in the subject line. Ask a > question, don't state an imperative. For example: "An Alternate Suite > Delineation Syntax For Python?" > > Step 3: respect the time of others, by trimming your message to its core essence > > For example: [...] This, along with the original posting on python-dev, is a spectacular example of how to propose a new feature, and how not to. It would make an excellent example for a HOWTO document on proposing changes (with the OP's permission - he might not like his posting being held op as the canonical "how not to do this" example...) I barely read the original posting, even though I tried to follow it - my mind was screaming "ain't never going to happen" at me throughout. And yet Nick's rewording is pretty persuasive, I can see the benefits and the power of the idea. My only major objection is that the resulting code is pretty ugly. I could easily imagine something along these lines being accepted into Python. Thanks, Nick, for doing this - and Cedric for the original idea, even though I didn't appreciate it at the time :-) It certainly got me thinking. As regards the actual proposal, some comments: - It's got serious power (if, try and with expressions, etc etc) - It would answer a number of requests that keep coming up over and over again. - I suspect the semantics in corner cases will be hard to pin down, but the basic principle looks simple enough - The code is pretty ugly, and certainly an attractive nuisance for obfuscated code, but I can't think of a better syntax - Given that it effectively replaces lambdas and if expressions, it partly violates OOWTDI - and it does so in a way that could encourage people to still propose try or with expressions ("they are as common as conditionals, so why not?") Paul. From masklinn at masklinn.net Sat Dec 10 12:50:08 2011 From: masklinn at masklinn.net (Masklinn) Date: Sat, 10 Dec 2011 12:50:08 +0100 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: Message-ID: <0624B9B6-23B1-488A-BB80-D1EE4C578935@masklinn.net> On 2011-12-10, at 12:32 , Paul Moore wrote: > - The code is pretty ugly, and certainly an attractive nuisance for > obfuscated code, but I can't think of a better syntax There are always the possibilities of keeping the colon as a start token and using an other un-matched token as an end. An other possibility (with some precedent in generator comprehensions) would be to declare current statements to be expressions (and define their expressive semantics) and surround them with parens for disambiguation purposes when necessary. > - Given that it effectively replaces lambdas and if expressions, it > partly violates OOWTDI For lambdas, as far as I can tell `def` would still require `return`, it is pretty common for languages to provide a shorter form for single-expression bodies. Of course, this could also be resolved by converting defs to implicit returns and removing `lambda` altogether. This change, however, would break a lot of code. > - and it does so in a way that could encourage > people to still propose try or with expressions ("they are as common > as conditionals, so why not?") Considering these are both included in Nick's proposal (which is a general transformation on existing statements, not a special-cased expressive form of some), I don't think that is a valid objection: On 2011-12-10, at 06:06 , Nick Coghlan wrote: > Some examples: > > Try expressions: > > x = {: try {: y.hello} except AttributeError {: "world!"}} > > With expressions: > > data = {: with open(fname) as f {: f.read()}} From richard.prosser at mail.com Sat Dec 10 15:16:21 2011 From: richard.prosser at mail.com (Richard Prosser) Date: Sat, 10 Dec 2011 14:16:21 +0000 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial Message-ID: Although I love Python there are some aspects of the language design which are disappointing and which can even lead to problems in some cases. A classic example is a mutable default argument having the potential to produce unexpected side-effects, as a consequence of the non-intuitive scoping rules. Another awkward 'feature' is the requirement for a trailing comma in singleton tuples, due I believe to the use of expression parentheses rather than (say) the use of special brackets like chevrons. Something that I personally wish for is the ability to declare variable types 'up front' but that facility is missing from Python. This is an important issue, so I propose that the Python tutorial be updated to highlight such problems. I would be willing to write a draft section myself but obviously it would need to be reviewed. I am not sure if this is the appropriate place to make such a comment but it seems to be a good starting point. Any advice on making a more formal proposal would be welcome. Cheers, Richard Prosser PS Is it too late to fix such warts in version 3? -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn at masklinn.net Sat Dec 10 15:41:14 2011 From: masklinn at masklinn.net (Masklinn) Date: Sat, 10 Dec 2011 15:41:14 +0100 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: Message-ID: <28FB8C6C-D4B0-45E5-9535-6D8F6B37FCB5@masklinn.net> On 2011-12-10, at 15:16 , Richard Prosser wrote: > > A classic example is a mutable default argument having the potential to > produce unexpected side-effects, as a consequence of the non-intuitive > scoping rules. As far as I know, mutable default arguments have nothing to do with scoping, they have to do with the "toplevel" being evaluated fully, so as the function declaration is evaluated (to create the function object) so are its default arguments. This is independent from Python's scoping issues unless I misunderstood what you meant by "scoping". But this is definitely something which trips people. However, there is already a note (though a pretty low-key one, it should probably use an actual warning directive instead of just bolding it, you should submit a documentation patch) in the tutorial on that subject[0]. > Another awkward 'feature' is the requirement for a trailing comma in > singleton tuples, due I believe to the use of expression parentheses rather > than (say) the use of special brackets like chevrons. For tuples, there are no matching operators left, as literal sets have been added. And technically, the irregularity with tuples is probably the empty tuple `()` as parens in other tuple arities are only necessary for disambiguation (much like parens around generator expressions): the "tuple constructor" is the comma, not the parens, a = 1, b = 1, 2 c = 1, 2, 3 are all valid and generate respectively a singleton, a pair and a triple. In that context, the trailing comma for singletons makes sense. If you want regularity, you can even add a trailing comma to the pair and the triple (as you can in e.g. a list or a dict): a = 1, b = 1, 2, c = 1, 2, 3, I'd rather have a lone comma (with or without parens, depending on the context) create a null tuple. > Something that I personally wish for is the ability to declare variable > types 'up front' but that facility is missing from Python. I fail to see how this is a "gotcha": since Python is dynamically typed names don't have types (well technically Python 3 added documentary type specs to arguments, but they're not used by any implementation I know of though some third-party tools may already have started using them) [0] http://docs.python.org/tutorial/controlflow.html#default-argument-values From fuzzyman at gmail.com Sat Dec 10 16:27:20 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Sat, 10 Dec 2011 15:27:20 +0000 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: Message-ID: On 10 December 2011 14:16, Richard Prosser wrote: > Although I love Python there are some aspects of the language design which > are disappointing and which can even lead to problems in some cases. > > A classic example is a mutable default argument having the potential to > produce unexpected side-effects, as a consequence of the non-intuitive > scoping rules. > The default argument "problem" is not intuitive (and does need highlighting to people new to Python). But it is still better than the alternatives. It isn't to do with scoping but when the default arguments are evaluated. If instead the default values were evaluated at call time, what would the following code do: a = 3 def function(arg=a): pass del a function() In addition you have added extra overhead cost to each function call (argument evaluation). Plus Python exposes the default arguments to introspection. If the default arguments aren't evaluated until call time you lose that capability. > > Another awkward 'feature' is the requirement for a trailing comma in > singleton tuples, due I believe to the use of expression parentheses rather > than (say) the use of special brackets like chevrons. > > Well, parentheses are only part of the syntax for an empty tuple: () For non-empty tuple literals it is *only* the commas that are significant. single = 1, double = 2, 3 So although the syntax for a single member tuple does feel a bit awkward I wonder what alternative you would suggest? > Something that I personally wish for is the ability to declare variable > types 'up front' but that facility is missing from Python. > > That runs counter to the basic principles of Python where types are determined at runtime. What effect would declaring types have and how would that interact with the rest of the language? (Do you want every assignment to do a runtime type check?) > > This is an important issue, so I propose that the Python tutorial be > updated to highlight such problems. I would be willing to write a draft > section myself but obviously it would need to be reviewed. > > This is a good idea however. :-) Just do it. Create a documentation patch (for Python 3) and attach it to an issue in the bug tracker: http://bugs.python.org/ > I am not sure if this is the appropriate place to make such a comment but > it seems to be a good starting point. Any advice on making a more formal > proposal would be welcome. > > > Cheers, > > Richard Prosser > PS Is it too late to fix such warts in version 3? > For the issues you've raised, yes. Unless you have specific proposals that don't break backwards compatibility. All the best, Michael Foord > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Dec 10 18:45:54 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 Dec 2011 18:45:54 +0100 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial References: Message-ID: <20111210184554.475bac43@pitrou.net> Hello Richard, On Sat, 10 Dec 2011 14:16:21 +0000 Richard Prosser wrote: > > This is an important issue, so I propose that the Python tutorial be > updated to highlight such problems. I would be willing to write a draft > section myself but obviously it would need to be reviewed. I think documenting "gotchas" can be useful indeed. However, I'm not sure the tutorial is the right place: it should present an easy to grasp view of the language, not digress about edge cases. So perhaps a FAQ, for example, would be more appropriate. In any case, feel free to propose a draft on http://bugs.python.org You can take a look at http://docs.python.org/dev/documenting/ if you are not familiar with the process. Regards Antoine. From ned at nedbatchelder.com Sat Dec 10 19:34:15 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sat, 10 Dec 2011 13:34:15 -0500 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: Message-ID: <4EE3A627.8010002@nedbatchelder.com> On 12/10/2011 9:16 AM, Richard Prosser wrote: > Although I love Python there are some aspects of the language design > which are disappointing and which can even lead to problems in some cases. > > A classic example is a mutable default argument having the potential > to produce unexpected side-effects, as a consequence of the > non-intuitive scoping rules. > > Another awkward 'feature' is the requirement for a trailing comma in > singleton tuples, due I believe to the use of expression parentheses > rather than (say) the use of special brackets like chevrons. > > Something that I personally wish for is the ability to declare > variable types 'up front' but that facility is missing from Python. > One of these things is not like the others. Mutable default arguments, and singleton tuples are surprises that make sense once you understand things on a deeper level. It makes sense to call them out as a "gotcha": a common stumbling block for learners. But "no type declarations" is not a wart, it's a fundamental feature of the language that is immediately apparent from the first lesson. --Ned. From greg.ewing at canterbury.ac.nz Sat Dec 10 22:55:06 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Dec 2011 10:55:06 +1300 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: Message-ID: <4EE3D53A.1030804@canterbury.ac.nz> Paul Moore wrote: > - The code is pretty ugly, and certainly an attractive nuisance for > obfuscated code, but I can't think of a better syntax Maybe that's actually an advantage? It would help to discourage people from using it in code intended for humans. -- Greg From mwm at mired.org Sun Dec 11 01:16:08 2011 From: mwm at mired.org (Mike Meyer) Date: Sat, 10 Dec 2011 16:16:08 -0800 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: Message-ID: <20111210161608.13313643@bhuda.mired.org> On Sat, 10 Dec 2011 15:06:13 +1000 Nick Coghlan wrote: > An Alternate Suite Delineation Syntax For Python? > ================================================= > > However, this benefit comes at quite a high price: it is effectively > impossible to embed arbitrary Python statements into any environment > where leading whitespace is *not* significant, including Python's own > expression syntax. Since you sent it to Python-ideas, I'm going to assume you were at least semi-serious. So let me add the use case I came up with while reading the original post, also often seen in template languages: Writing code to generate python. With significant whitespace, you have to keep explicit track of the current nesting level in order to generate the appropriate indent. With real delimiters, you can drop that. Compare: def generate(self): yield '{keyword} {expr}{:'.format(keyword=self.name, expr=self.expresion) for s in self.suite: for l in s.generate(): yield l With: def generate(self, level): yield '{0}{keyword} {expr}:'.format(' ' * level, keyword=self.name, expr=self.expresion) for s in self.suite: for l in s.generate(level + 1): yield l A minor improvement in just that one method, but it will show up in every generate method of every class, making it significant in total. > Applying the excellent set of criteria suggested by Mike Meyer: "Suggested" may be a bit to much credit. I've observed the community applying these criteria to pretty much every idea. Some ideas get adopted in spite of failing one or more of them, but a *lot* of ideas get dropped after failing one of them. In this case: 1. What's the use case? Writing code that's going to write python code. > 2. Does it make such code more readable? It makes the code generating code less complicated, which means more readable. > 3. Does it encourage writing unreadable code? Perhaps. You certainly > end up with a lot of braces, colons and semi-colons floating around. > However, that's also something that can be addressed by style guides - > if people are using the embedded syntax when the indented syntax would > work fine, the problem isn't really due to the embedded syntax, it's > due to people not caring about maintainability. I think not. As someone else observed, Haskell has both options available - and the rules for the indented version are a lot less straightforward than they are in Python (at least I haven't seen as clean a statement of them). About the only use of the delimited version in real code is turning a short suite into a one-line suite. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From tjreedy at udel.edu Sun Dec 11 01:46:11 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 10 Dec 2011 19:46:11 -0500 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: Message-ID: On 12/10/2011 12:06 AM, Nick Coghlan wrote: > 4. Returns the value of the last statement in the suite as the > result of the suite expression (since return statements would affect > the containing scope) I believe you mean return the value of the last statement that is executed, as in the try-except example or the conditional example: {: if a {:b} else {:c}} I must congratulate you, Nick, for being the first person in my 14 years with Python to make a brace idea look half-way sensible. I can see that algorithmically generated code (which includes templates) might be a real use case. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sun Dec 11 02:03:02 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Dec 2011 14:03:02 +1300 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: Message-ID: <4EE40146.20904@canterbury.ac.nz> Nick Coghlan suggested: > x = y if y is not None else {: raise ValueError("y must not be None!")} I'd want to see {: closed with :} for symmetry. -- Greg From ncoghlan at gmail.com Sun Dec 11 02:21:01 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Dec 2011 11:21:01 +1000 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: Message-ID: On Sun, Dec 11, 2011 at 10:46 AM, Terry Reedy wrote: > On 12/10/2011 12:06 AM, Nick Coghlan wrote: > >> ? 4. Returns the value of the last statement in the suite as the >> result of the suite expression (since return statements would affect >> the containing scope) > > > I believe you mean return the value of the last statement that is executed, > as in the try-except example or the conditional example: > > ? ?{: if a {:b} else {:c}} Indeed, that was what I meant, albeit not what I wrote :) > I must congratulate you, Nick, for being the first person in my 14 years > with Python to make a brace idea look half-way sensible. I can see that > algorithmically generated code (which includes templates) might be a real > use case. I actually surprised myself - I started writing that purely as a devil's advocate "if you're going to rant about something to that extent, you may as well *try* to be constructive" post, but the end result was significantly less terrible than I originally expected :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python at mrabarnett.plus.com Sun Dec 11 02:33:27 2011 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 11 Dec 2011 01:33:27 +0000 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <4EE40146.20904@canterbury.ac.nz> References: <4EE40146.20904@canterbury.ac.nz> Message-ID: <4EE40867.2000504@mrabarnett.plus.com> On 11/12/2011 01:03, Greg Ewing wrote: > Nick Coghlan suggested: > >> x = y if y is not None else {: raise ValueError("y must not be None!")} > > I'd want to see {: closed with :} for symmetry. > That's just what I was thinking. (Honest! :-)) From steve at pearwood.info Sun Dec 11 02:55:51 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 11 Dec 2011 12:55:51 +1100 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <20111210161608.13313643@bhuda.mired.org> References: <20111210161608.13313643@bhuda.mired.org> Message-ID: <4EE40DA7.4020702@pearwood.info> Mike Meyer wrote: > Since you sent it to Python-ideas, I'm going to assume you were at > least semi-serious. So let me add the use case I came up with while > reading the original post, also often seen in template languages: > > Writing code to generate python. > > With significant whitespace, you have to keep explicit track of the > current nesting level in order to generate the appropriate > indent. With real delimiters, you can drop that. Compare: That's hardly an onerous requirement. As you show, the two code snippets are almost identical, except that the indent version takes an explicit level argument, while your braces version neglects to close suites at all. A fair comparison should show the close-suite code, even if it is only a single line. > def generate(self): > yield '{keyword} {expr}{:'.format(keyword=self.name, > expr=self.expresion) > for s in self.suite: > for l in s.generate(): > yield l I guess you would need to yield '}' after the outer for loop. > With: > > def generate(self, level): > yield '{0}{keyword} {expr}:'.format(' ' * level, > keyword=self.name, > expr=self.expresion) > for s in self.suite: > for l in s.generate(level + 1): > yield l > > > A minor improvement in just that one method, but it will show up in > every generate method of every class, making it significant in total. I wouldn't use the term "minor improvement" so much as "trivial difference". But even if I grant that it is an improvement, how many generate methods do you expect there to be? Presumably most of them will inherit their generate method from one or two parent classes. Even if every block keyword gets an independent copy in violation of DRY, there's still fewer than a dozen classes needed: def class for while if elif else try except finally with. So the total significance is still likely to be small. But wait... now you have your code generator, which generates ugly, unreadable code. Code needs to be read more often than written, even generated code, so now you need a code beautifier to reformat it using indentation for the human reader. Which means keeping track of the indent level. You're right back where you started, only worse, because now you have to parse the generated code to reformat it, instead of formatting it correctly the first time. So to save a dozen lines in your code generator, you end up with an extra hundred lines in a code beautifier. A classic example of code pessimation (of LOC rather than speed in this case). I think it is suggestive that we will need to warn people not to use this hypothetical syntax. As Nick wrote: >> However, that's also something that can be addressed by style guides - >> if people are using the embedded syntax when the indented syntax would >> work fine, the problem isn't really due to the embedded syntax, it's >> due to people not caring about maintainability. People who don't care about maintainability don't care about readability and they certainly don't read style guides unless they are forced to. > I think not. As someone else observed, Haskell has both options > available - and the rules for the indented version are a lot less > straightforward than they are in Python (at least I haven't seen as > clean a statement of them). About the only use of the delimited > version in real code is turning a short suite into a one-line suite. When you defend a feature on the basis that "people will hardly ever use it, except for trivial pieces of code that don't need it", that speaks volumes. -- Steven From julian at grayvines.com Sun Dec 11 06:07:23 2011 From: julian at grayvines.com (Julian Berman) Date: Sun, 11 Dec 2011 00:07:23 -0500 Subject: [Python-ideas] contextlib.maybe Message-ID: <20111211050723.GA12197@air> Twice recently I've found myself wanting to write the following code: def fn(a_file=None): responsible_for_closing = False if a_file is None: a_file = open(a_default_location) responsible_for_closing = True do_stuff(a_file) if responsible_for_closing: a_file.close() which can be written slightly shorter I know, but it's still a tiny bit messy and repetitive. What I'd prefer to write is something more like: def fn(a_file=None): with contextlib.maybe(a_file, open, default) as a_file: do_stuff(a_file) where `maybe` takes an object and conditionally runs a context manager if a check fails. Implementation would be: @contextlib.contextmanager def maybe(got, contextfactory, *args, checkif=bool, **kwargs): if checkif(got): yield got else: with contextfactory(*args, **kwargs) as got: yield got It's hard to gauge utility for such simple functions (though contextlib already has closing(), so I figured it'd be worth asking at least). Would this be useful to others? Or perhaps I'm completely missing something and you've got suggestions on how to better have an API where an argument can be fetched if not provided but a context manager would preferably need to be run to do so. From tjreedy at udel.edu Sun Dec 11 07:02:30 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 11 Dec 2011 01:02:30 -0500 Subject: [Python-ideas] contextlib.maybe In-Reply-To: <20111211050723.GA12197@air> References: <20111211050723.GA12197@air> Message-ID: On 12/11/2011 12:07 AM, Julian Berman wrote: > Twice recently I've found myself wanting to write the following code: > > def fn(a_file=None): > responsible_for_closing = False > > if a_file is None: > a_file = open(a_default_location) > responsible_for_closing = True > > do_stuff(a_file) > > if responsible_for_closing: > a_file.close() > > which can be written slightly shorter I know, but it's still a tiny bit > messy and repetitive. What I'd prefer to write is something more like: > > def fn(a_file=None): > with contextlib.maybe(a_file, open, default) as a_file: > do_stuff(a_file) Expecting contextlib to have such a specialized context manager that does exactly what you want is perhaps too much. However, you should be able to write a class yourself that keeps the flag and does the conditional opening and closing in the __enter__ and __exit__ methods. -- Terry Jan Reedy From masklinn at masklinn.net Sun Dec 11 09:44:17 2011 From: masklinn at masklinn.net (Masklinn) Date: Sun, 11 Dec 2011 09:44:17 +0100 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <20111210161608.13313643@bhuda.mired.org> References: <20111210161608.13313643@bhuda.mired.org> Message-ID: <6048C036-E479-4BFA-A643-F79EECD9C2A7@masklinn.net> On 2011-12-11, at 01:16 , Mike Meyer wrote: > A minor improvement in just that one method, but it will show up in > every generate method of every class, making it significant in total. If users are not supposed to read the generated code, why not generate pyc files directly? That would avoid the issue would it not? And if users are supposed to edit (therefore read) the generated code, wouldn't you want to make the file actually readable, therefore use indentation-based blocks? > I think not. As someone else observed, Haskell has both options > available - and the rules for the indented version are a lot less > straightforward than they are in Python (at least I haven't seen as > clean a statement of them). The short version is: http://www.haskell.org/onlinereport/lexemes.html#lexemes-layout * when a brace is omitted after a `where`, `let`, `of` or `do` keyword, a brace is inserted and the indentation of the next lexeme is remembered * if the next lexeme is indented below the keyword, immediately insert a closing brace (before the lexeme itself) * otherwise, for each subsequent line - if it contains only whitespace or is indented further than the current level, continue the current expression - if it is indented at the current level, insert a semicolon - if it is indented below the current level, insert a closing brace You can find the complete and more formal definition at http://www.haskell.org/onlinereport/syntax-iso.html#layout From ncoghlan at gmail.com Sun Dec 11 11:30:45 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Dec 2011 20:30:45 +1000 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <4EE40DA7.4020702@pearwood.info> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> Message-ID: On Sun, Dec 11, 2011 at 11:55 AM, Steven D'Aprano wrote: > Mike Meyer wrote: > >> Since you sent it to Python-ideas, I'm going to assume you were at >> least semi-serious. So let me add the use case I came up with while >> reading the original post, also often seen in template languages: >> >> ? ?Writing code to generate python. >> >> With significant whitespace, you have to keep explicit track of the >> current nesting level in order to generate the appropriate >> indent. With real delimiters, you can drop that. Compare: > > > That's hardly an onerous requirement. As you show, the two code snippets are > almost identical, except that the indent version takes an explicit level > argument, while your braces version neglects to close suites at all. A fair > comparison should show the close-suite code, even if it is only a single > line. Indeed, that's a reason I didn't use straight up code generation as my motivating use case - the significant whitespace simply isn't that painful in that case. The reason significant leading whitespace can be a PITA is due to two main circumstances: 1. Attempting to transport it through a channel that either strips leading and trailing whitespace from lines, or else consolidates certain whitespace sequences into a single whitespace character (generally sequences of spaces and tabs becoming a single space). Python source code simply cannot be passed through such channels correctly - if they don't offer an escaping mechanism, or that mechanism is not applied correctly, the code *will* be corrupted. Explicitly delimited code, on the other hand, can be passed through without semantic alteration (even if the details of the whitespace change) and a pretty printer can fix it at the far end. 2. Attempting to transport it through a channel where leading whitespace already has another meaning. This comes up primarily with templating languages - your whitespace is generally part of the template output, so it becomes problematic to break up your Python code across multiple code snippets while keeping the suite groupings clear. With explicit delimiters, though, you can just ignore the template parts, and pretend the code snippets are all part of a single string (the following is based on an example a colleague sent me [1], with the closing delimiter adjusted to mirror the opening one as a couple of folks suggested): <% if danger_level > 3 {: %>
<% if danger_level == 5 {: %>EXTREME <% :} %>DANGER ALERT!
<% elif danger_level > 0 {: %>
Some chance of danger
<% else {: %>
No danger
<% :} %> <% for a in ['cat', 'dog', 'rabbit'] {: %>

<%= a %>

<%= describe_animal(a) %>

<% :} %> [1] https://gist.github.com/1455210 Is such code as beautiful and readable as normal Python code? No, of course not. But it would serve a purpose in defining a *standard* way to embed Python code in environments where the leading whitespace causes problems. In terms of how such a delimited syntax would relate to currently legal Python code, the transformation is basically the reverse of the Haskell one: 1. If you encounter a ":" to open a new suite, replace it and any trailing whitespace with "{:". 2. All whitespace between statements in such a suite is replaced with ";" characters 3. Any trailing whitespace after the last statement in the suite is replaced with ":};" The rest of the changes (i.e. a suite's expression value being the result of the last statement executed, anonymous function and class definitions) then follow from the fact that you now have a suite syntax that can potentially be embedded as an expression, so a range of new possibilities open up. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From sven at marnach.net Sun Dec 11 20:06:16 2011 From: sven at marnach.net (Sven Marnach) Date: Sun, 11 Dec 2011 19:06:16 +0000 Subject: [Python-ideas] contextlib.maybe In-Reply-To: References: <20111211050723.GA12197@air> Message-ID: <20111211190615.GB3760@pantoffel-wg.de> Terry Reedy schrieb am So, 11. Dez 2011, um 01:02:30 -0500: > Expecting contextlib to have such a specialized context manager that > does exactly what you want is perhaps too much. I don't think this is a very specialised need. It would overcome a fundamental limitation of context managers compared to some clean-up constructs in other languages. An example is Go's 'defer', which is Go's replacement for try/finally blocks and context managers. It defers a function call to the time the current function returns, and guarantees the deferred call will be executed no matter how control leaves the current function -- just like a 'finally' clause. The advantage of 'defer' over 'with' and try/finally blocks is that it is not a compound statement, so you can "conditionally add a 'finally' clause". Returning to the example from the original post, let's see how it would look in Go: func Fn(a_file *os.File) (err os.Error) { if a_file == nil { a_file, err = os.Open(a_default_location, os.O_RDONLY, 0) if err != nil { return } defer a_file.Close() } // do stuff return } to achieve exactly what is desired. The line 'defer a_file.Close()' is only executed when we actually need to close the file. Note that the disadvantage of 'defer' is also that it is not a compound statement -- it is bound to the block defined by the current function and in this regard less flexible than 'with' blocks. We could add our own version of 'defer' to Python by offering a context manager 'Deferrer' with a 'push()' method to push a clean-up call-back on the Deferrer's stack. This design would offer the combined advantages of both approaches described above. I, for one, do think that this would make a very worthwhile addition to the 'contextlib' module. We discussed such a context manager less than two weeks ago on this list http://mail.python.org/pipermail/python-ideas/2011-October/012418.html and Jan Kaliszewski even provided an implementation: http://mail.python.org/pipermail/python-ideas/2011-October/012463.html What do you think? Cheers, Sven From richard.prosser at mail.com Sun Dec 11 22:30:54 2011 From: richard.prosser at mail.com (Richard Prosser) Date: Sun, 11 Dec 2011 21:30:54 +0000 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial Message-ID: Ned, I accept your comment about the tutorial - I wasn't sure about that. Thanks. However I still feel that there are some aspects of the language which are not in the true spirit of Python (i.e. 'intuitive'). The discussion about default mutable types is one of these. It seems to me that the essential problem is that of assignment in general, which (I believe) creates a reference on the LHS to the object on the RHS, rather than having a copy operation to make the two objects completely separate. That can be confusing in other contexts, not just with default parameters. If I am to write a 'gotchas' FAQ or whatever then I would like to understand the reasoning behind such design decisions but I can't find any 'deep' explanations at present - just several posts about people being puzzled! A similar comment applies to the lack of type declarations. So if you or anyone else can explain exactly why such odditties are implemented I would be grateful. Unfortunately it is almost certainly too late to propose fixes (if appropriate) for such quirks in Python 3 but at least I should be able provide arguments as to why things are done the way they are. Richard On 11 December 2011 01:21, wrote: > Message: 2 > Date: Sat, 10 Dec 2011 13:34:15 -0500 > From: Ned Batchelder > To: Richard Prosser > Cc: python-ideas at python.org > Subject: Re: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' > section to the tutorial > Message-ID: <4EE3A627.8010002 at nedbatchelder.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 12/10/2011 9:16 AM, Richard Prosser wrote: > > Although I love Python there are some aspects of the language design > > which are disappointing and which can even lead to problems in some > cases. > > > > A classic example is a mutable default argument having the potential > > to produce unexpected side-effects, as a consequence of the > > non-intuitive scoping rules. > > > > Another awkward 'feature' is the requirement for a trailing comma in > > singleton tuples, due I believe to the use of expression parentheses > > rather than (say) the use of special brackets like chevrons. > > > > Something that I personally wish for is the ability to declare > > variable types 'up front' but that facility is missing from Python. > > > One of these things is not like the others. Mutable default arguments, > and singleton tuples are surprises that make sense once you understand > things on a deeper level. It makes sense to call them out as a > "gotcha": a common stumbling block for learners. But "no type > declarations" is not a wart, it's a fundamental feature of the language > that is immediately apparent from the first lesson. > > --Ned. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwm at mired.org Sun Dec 11 22:40:56 2011 From: mwm at mired.org (Mike Meyer) Date: Sun, 11 Dec 2011 13:40:56 -0800 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <4EE40DA7.4020702@pearwood.info> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> Message-ID: <20111211134056.25d40c34@bhuda.mired.org> On Sun, 11 Dec 2011 09:44:17 +0100 Masklinn wrote: > On 2011-12-11, at 01:16 , Mike Meyer wrote: > > A minor improvement in just that one method, but it will show up in > > every generate method of every class, making it significant in total. > If users are not supposed to read the generated code, why not generate > pyc files directly? That would avoid the issue would it not? Users generally aren't expected to read generated code. It's pretty ugly for systems that generate python now. The reason you generate python instead of pyc is that backward compatibility of python source is a priority for python development, so with a little care your generated code will work across multiple versions of python. The same is not true for byte code files (and if that's changed with 3.x, I couldn't find anything saying so), so your code generator is liable to depend on the version of the target python interpreter. On Sun, 11 Dec 2011 12:55:51 +1100 Steven D'Aprano wrote: > Mike Meyer wrote: > > That's hardly an onerous requirement. As you show, the two code snippets are > almost identical, except that the indent version takes an explicit level > argument, while your braces version neglects to close suites at all. A fair > comparison should show the close-suite code, even if it is only a single line. Correct. I thought it, but forgot to add it. > > A minor improvement in just that one method, but it will show up in > > every generate method of every class, making it significant in total. > I wouldn't use the term "minor improvement" so much as "trivial difference". I wouldn't refer to any change that involves adding a variable as "trivial". Especially one that's part of the API. Part of the issue is that I understated the problem - this change is ubiquitous, and large enough that it adds up. > But even if I grant that it is an improvement, how many generate methods do > you expect there to be? Because it's in the API, not only does this affect every generate method, but anything that might call them. You have to thread the indent level to anything that might want to write code. It might be better to make it a global (i.e. - instance or module variable) value of some sort. Basically, this "trivial difference" needs to be taken into consideration when designing the system architecture. You came close to the real reason for rejecting this here: > When you defend a feature on the basis that "people will hardly ever use it, > except for trivial pieces of code that don't need it", that speaks volumes. Two use cases are at or over the edge. Writing in something that lets you embed python code? You arguably aren't writing python. Writing code that generates python? *You* may not be writing python at all, even if your code is. That leaves the case of turning statements into expressions. While suggestions/requests for similar things keep coming up, they aren't all that frequent, and the tools we have for that have been acceptable so far. While Nick is to be congratulated for providing a solution to that problem that's much better than it's predecessors, I don't think it's worth the complexity it looks like it would require in the compiler. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From masklinn at masklinn.net Mon Dec 12 00:31:11 2011 From: masklinn at masklinn.net (Masklinn) Date: Mon, 12 Dec 2011 00:31:11 +0100 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: Message-ID: On 2011-12-11, at 22:30 , Richard Prosser wrote: > However I still feel that there are some aspects of the language which are > not in the true spirit of Python (i.e. 'intuitive'). I have to note that, as far as I know, "intuitive" is not quite part of the "true spirit of Python". The Zen even has a stanza on the subject: > Although that way may not be obvious at first unless you're Dutch. > It seems to me > that the essential problem is that of assignment in general, which (I > believe) creates a reference on the LHS to the object on the RHS, rather > than having a copy operation to make the two objects completely separate. I don't see this as a problem, it's the normal semantics of languages using reference-values for object types. > A similar comment applies to the lack of type declarations. > > So if you or anyone else can explain exactly why such odditties are > implemented I would be grateful. I don't understand, why do you consider dynamic typing to be "an oddity"? There is nothing odd about it, and it's in fact older than computer science itself. The default arguments issue is an unfortunate interaction of Python's core reference-value semantics and default arguments being implemented as attributes to the function object, evaluated when said function object is created (the feature itself is as old as Python, according to the logs it was added for the 1.0.2 release back in 1994, the changelog seems to peg it to April 14, so your only chance for an explanation of why it was implemented with these semantics is hoping Guido has a very, very good memory. Numerous post-hoc rationalization exist, but only him may hold the true reason). > Unfortunately it is almost certainly too late to propose fixes (if > appropriate) for such quirks in Python 3 Most of these have been core semantic attributes of the language for almost two decades now, so even if you had proposed these changes during the Python 3 design cycle I think it's very unlikely they'd have passed: they don't just change Python, they create a very different language with a syntax similar to Python's. From tjreedy at udel.edu Mon Dec 12 01:13:23 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 11 Dec 2011 19:13:23 -0500 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: Message-ID: On 12/11/2011 4:30 PM, Richard Prosser wrote: > However I still feel that there are some aspects of the language which > are not in the true spirit of Python (i.e. 'intuitive'). While 'intuitive' may be part of the 'true spirit of Python', it is not included in the Zen of Python, perhaps because it it so slippery and person dependent. Python is more in tune with everyday life and therefore with naive intuition than some other languages. > The discussion about default mutable types is one of these. It seems to > me that the essential problem is that of assignment in general, which (I > believe) creates a reference on the LHS to the object on the RHS, rather > than having a copy operation to make the two objects completely > separate. That can be confusing in other contexts, not just with default > parameters. When an organization reassign the role 'president' to a new person, they do not copy the person. Neither does Python. We use aliases (multiple ways to refer to the same entity) all the time in real life. Python is quite consistent. Expressions evaluate to objects, either pre-existing or new. "Target = expression" binds the target to the object resulting from evaluating the expression. "f(expression)" binds the first parameter name of f to the expression object. > If I am to write a 'gotchas' FAQ or whatever then I would like to > understand the reasoning behind such design decisions but I can't find > any 'deep' explanations at present - just several posts about people > being puzzled! A similar comment applies to the lack of type declarations. Both behaviors reflect the fact that Python is a named object language, rather than a named memory block language, and that in Python types are a property of objects rather than of names. This is similar to at least some uses of names in everyday life. For instance, the name Buddy could be bound by a particular person to a person, dog, other pet, story, boat, or even a rifle. -- Terry Jan Reedy From ned at nedbatchelder.com Mon Dec 12 01:34:07 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sun, 11 Dec 2011 19:34:07 -0500 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: Message-ID: <4EE54BFF.3050603@nedbatchelder.com> Richard, I don't think I can provide you with a "why" for dynamic typing. It's a choice Guido made early on, one that is central to the language, and one that I think pays off in the long run. Values are objects, which are typed. Names are untyped, and can refer to any value, regardless of what value they referred to in the past. Therefore, it doesn't make sense to talk about the type of a name, which is all that a type declaration could do. This is very different than some other programming languages, but I don't know if it could be called unintuitive. I'm not sure anything about programming could truly be called intuitive, I think the closest we can get is "familiar". Certainly if you've worked with statically typed languages before, then dynamic typing is unfamiliar. I worry that you are still placing dynamic typing into a category you call "gotchas" or "quirks", with the word "fixes" nearby. Dynamic typing cannot be "fixed", it is central to the language. I think it is great to write something to help those new to Python, but you should be sure that you fully understand Python before you undertake it. --Ned. On 12/11/2011 4:30 PM, Richard Prosser wrote: > Ned, > > I accept your comment about the tutorial - I wasn't sure about that. > Thanks. > > However I still feel that there are some aspects of the language which > are not in the true spirit of Python (i.e. 'intuitive'). > > The discussion about default mutable types is one of these. It seems > to me that the essential problem is that of assignment in general, > which (I believe) creates a reference on the LHS to the object on the > RHS, rather than having a copy operation to make the two objects > completely separate. That can be confusing in other contexts, not just > with default parameters. > > If I am to write a 'gotchas' FAQ or whatever then I would like to > understand the reasoning behind such design decisions but I can't find > any 'deep' explanations at present - just several posts about people > being puzzled! A similar comment applies to the lack of type declarations. > > So if you or anyone else can explain exactly why such odditties are > implemented I would be grateful. > > Unfortunately it is almost certainly too late to propose fixes (if > appropriate) for such quirks in Python 3 but at least I should be able > provide arguments as to why things are done the way they are. > > > Richard > > > On 11 December 2011 01:21, > wrote: > > Message: 2 > Date: Sat, 10 Dec 2011 13:34:15 -0500 > From: Ned Batchelder > > To: Richard Prosser > > Cc: python-ideas at python.org > Subject: Re: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' > section to the tutorial > Message-ID: <4EE3A627.8010002 at nedbatchelder.com > > > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 12/10/2011 9:16 AM, Richard Prosser wrote: > > Although I love Python there are some aspects of the language design > > which are disappointing and which can even lead to problems in > some cases. > > > > A classic example is a mutable default argument having the potential > > to produce unexpected side-effects, as a consequence of the > > non-intuitive scoping rules. > > > > Another awkward 'feature' is the requirement for a trailing comma in > > singleton tuples, due I believe to the use of expression parentheses > > rather than (say) the use of special brackets like chevrons. > > > > Something that I personally wish for is the ability to declare > > variable types 'up front' but that facility is missing from Python. > > > One of these things is not like the others. Mutable default > arguments, > and singleton tuples are surprises that make sense once you understand > things on a deeper level. It makes sense to call them out as a > "gotcha": a common stumbling block for learners. But "no type > declarations" is not a wart, it's a fundamental feature of the > language > that is immediately apparent from the first lesson. > > --Ned. > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Dec 12 04:21:55 2011 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Dec 2011 19:21:55 -0800 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <20111211134056.25d40c34@bhuda.mired.org> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <20111211134056.25d40c34@bhuda.mired.org> Message-ID: On Sun, Dec 11, 2011 at 1:40 PM, Mike Meyer wrote: > Users generally aren't expected to read generated code. It's pretty > ugly for systems that generate python now. If the point really is just generated code, we should just invent some really ugly but easy-to-parse and 100%-guaranteed backwards compatible set of brackets for just that purpose. But I don't believe this is about generated code. -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Mon Dec 12 06:21:26 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Dec 2011 18:21:26 +1300 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <4EE40DA7.4020702@pearwood.info> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> Message-ID: <4EE58F56.10507@canterbury.ac.nz> > Mike Meyer wrote: > >> With significant whitespace, you have to keep explicit track of the >> current nesting level in order to generate the appropriate >> indent. With real delimiters, you can drop that. Yes, but with an appropriate design you can arrange for that to be done in only *one* place. There's no need for every method that generates code to know about indentation levels. For example, you can create a code-output-stream object that you pass to your code-generating methods. You give it begin_block() and end-block() methods that you call whenever you would otherwise have written opening and closing delimiters. It keeps track of the indentation level internally and emits the appropriate amount of whitespace before each line. -- Greg From greg.ewing at canterbury.ac.nz Mon Dec 12 06:27:22 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Dec 2011 18:27:22 +1300 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <20111211134056.25d40c34@bhuda.mired.org> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <20111211134056.25d40c34@bhuda.mired.org> Message-ID: <4EE590BA.6080305@canterbury.ac.nz> Mike Meyer wrote: > The reason you generate python instead of pyc is that backward > compatibility of python source is a priority for python development, > so with a little care your generated code will work across multiple > versions of python. Another approach would be to generate an AST (either an actual Python AST or one of your own devising) and then traverse it to generate indented source code. This is another way of isolating knowledge of the indentation level to one part of the code. -- Greg From mwm at mired.org Mon Dec 12 06:45:29 2011 From: mwm at mired.org (Mike Meyer) Date: Sun, 11 Dec 2011 21:45:29 -0800 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <4EE58F56.10507@canterbury.ac.nz> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> Message-ID: <20111211214529.53d6dd34@bhuda.mired.org> On Mon, 12 Dec 2011 18:21:26 +1300 Greg Ewing wrote: > > Mike Meyer wrote: > >> With significant whitespace, you have to keep explicit track of the > >> current nesting level in order to generate the appropriate > >> indent. With real delimiters, you can drop that. > Yes, but with an appropriate design you can arrange for that to > be done in only *one* place. Exactly. This change causes you to modify your architecture to deal with it. In my book, changes that cause you to rethink the architecture of your system aren't "trivial". http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From mwm at mired.org Mon Dec 12 06:59:54 2011 From: mwm at mired.org (Mike Meyer) Date: Sun, 11 Dec 2011 21:59:54 -0800 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: <4EE54BFF.3050603@nedbatchelder.com> References: <4EE54BFF.3050603@nedbatchelder.com> Message-ID: <20111211215954.79f33232@bhuda.mired.org> On Sun, 11 Dec 2011 19:34:07 -0500 Ned Batchelder wrote: > Richard, I don't think I can provide you with a "why" for dynamic > typing. And this is the wrong place to ask. Dynamic typing and naming objects dates back to the precursors to LISP in the mid 50s. You should be asking the people who made that decision. By the same token, have you asked anyone why C/Java/etc. have static typing and name locations? It's an equally valid question. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From bruce at leapyear.org Mon Dec 12 07:09:06 2011 From: bruce at leapyear.org (Bruce Leban) Date: Sun, 11 Dec 2011 22:09:06 -0800 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <4EE58F56.10507@canterbury.ac.nz> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> Message-ID: On Sun, Dec 11, 2011 at 9:21 PM, Greg Ewing wrote: > Mike Meyer wrote: >> >> With significant whitespace, you have to keep explicit track of the >>> current nesting level in order to generate the appropriate >>> indent. With real delimiters, you can drop that. >>> >> > Yes, but with an appropriate design you can arrange for that to > be done in only *one* place. There's no need for every method > that generates code to know about indentation levels. > I've written this code (for HTML) and it's not only straightforward, it makes the code easier to write and more robust. And you can make the closing automatic with a context manager. Here's how you might write some HTML: with html_tag('table'): with html_tag('tr'): for x in data: with html_tag('td'): # do stuff with x On the other hand, this might not help with template languages as Nick points out: On Sun, Dec 11, 2011 at 2:30 AM, Nick Coghlan wrote: > <% if danger_level > 3 {: %> >
<% if danger_level == 5 {: %>EXTREME <% :} > %>DANGER ALERT!
> > <% elif danger_level > 0 {: %> >
Some chance of danger
> > <% else {: %> >
No danger
> > <% :} %> > > <% for a in ['cat', 'dog', 'rabbit'] {: %> >

<%= a %>

>

<%= describe_animal(a) %>

> > <% :} %> While I thought his example was compelling at first, after thinking it through I realized that any existing template language that generates Python has to be aware of indentation and it's going to have to modify the indentation of the included Python code. So it would need to either read the indentation or read the {: and :} tokens. So what if we think in terms of modifying the template language instead of Python? In the code below, I've added two extra tokens to the template language: %{> which increases the indentation and <}% which decreases it. All other leading spaces would be stripped. Here's that example rewritten: <% if danger_level > 3: %{>
<% if danger_level == 5: %{>EXTREME <}% %>DANGER ALERT!
<% elif danger_level > 0: %{>
Some chance of danger
<% else: %{>
No danger
<}%> <% for a in ['cat', 'dog', 'rabbit']: %{>

<%= a %>

<%= describe_animal(a) %>

<}%> This isn't a perfect design. In particular %{> wouldn't be necessary if the template language parses the embedded code, but it's sufficient to illustrate the point. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Dec 12 07:40:00 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Dec 2011 19:40:00 +1300 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <20111211214529.53d6dd34@bhuda.mired.org> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <20111211214529.53d6dd34@bhuda.mired.org> Message-ID: <4EE5A1C0.8090604@canterbury.ac.nz> Mike Meyer wrote: > Exactly. This change causes you to modify your architecture to deal > with it. It's not about "modifying" an architecture, it's about considering the requirements and designing an appropriate architecture from the beginning to accommodate them. If you set out to generate Python code, write most of your program and then say "Oh, bugger, what am I going to do about indentation?", then you just haven't thought far enough ahead. Also, the particular architecture I suggested was only an example. If you put your mind to it, you should be able to come up with many ways of skinning this snake. If you want to keep your generator style, you could yield special "indent" and "dedent" tokens that the top level recognises and does the appropriate thing with. You might even use special strings such as "{:" and ":}" embedded in the text to represent these. Then your code generating methods wouldn't look all that much different from how they would if you were generating the delimited form of python that you are arguing for... As a side note, I would argue that it's a good idea to adopt an architecture that can manage indentation in any case, whether you're generating indentation-sensitive code or not. I use this kind of architecture in Pyrex to generate C code, even though indentation isn't required in C. I do it because I want to generate properly-indented code anyway, for reasons of readability. Even if the output is normally never seen, it's a big help when debugging. -- Greg From ncoghlan at gmail.com Mon Dec 12 07:42:09 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Dec 2011 16:42:09 +1000 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> Message-ID: On Mon, Dec 12, 2011 at 4:09 PM, Bruce Leban wrote: > This isn't a perfect design. In particular %{> wouldn't be necessary if the > template language parses the embedded code, but it's sufficient to > illustrate the point. Sure, but that's just perpetuating the status quo - you can fairly easily create a Python *inspired* templating language right now by building your suite delimeters into your template syntax (e.g. Django and Jinja2 favour explicit suite closing snippets like "{% endfor %}"). It isn't that Python *can't* be tunnelled through whitespace insensitive environments - it's that there's no *standard* mechanism for doing so, so everyone that has the need reinvents the wheel in their own idiosyncratic way. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Mon Dec 12 07:47:30 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Dec 2011 19:47:30 +1300 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> Message-ID: <4EE5A382.4030901@canterbury.ac.nz> Nick Coghlan wrote: > It isn't that Python *can't* be tunnelled through whitespace > insensitive environments - it's that there's no *standard* mechanism > for doing so, so everyone that has the need reinvents the wheel in > their own idiosyncratic way. So perhaps an appropriate solution would be a PEP setting out such a standard, and maybe a library module implementing the transformation to and from? -- Greg From ncoghlan at gmail.com Mon Dec 12 07:56:03 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Dec 2011 16:56:03 +1000 Subject: [Python-ideas] An Alternate Suite Delineation Syntax For Python? (was Re: [Python-Dev] [PATCH] Adding braces to __future__) In-Reply-To: <4EE5A382.4030901@canterbury.ac.nz> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <4EE5A382.4030901@canterbury.ac.nz> Message-ID: On Mon, Dec 12, 2011 at 4:47 PM, Greg Ewing wrote: > Nick Coghlan wrote: > >> It isn't that Python *can't* be tunnelled through whitespace >> insensitive environments - it's that there's no *standard* mechanism >> for doing so, so everyone that has the need reinvents the wheel in >> their own idiosyncratic way. > > > So perhaps an appropriate solution would be a PEP setting out > such a standard, and maybe a library module implementing the > transformation to and from? Sure, that would be one way to go. It doesn't give you multi-line lambda support or any of the other goodies, though. Personally, I don't intend to take this idea any further than the writeup I posted over on RTFD: http://readthedocs.org/docs/ncoghlan_devs-python-notes/en/latest/pep_ideas/suite_expr.html If someone else wants to pursue it, either as a Python reformatting library or as an actual modification to the language grammar, more power to 'em. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Mon Dec 12 06:11:32 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Dec 2011 18:11:32 +1300 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: Message-ID: <4EE58D04.10403@canterbury.ac.nz> Richard Prosser wrote: > It seems to > me that the essential problem is that of assignment in general, which (I > believe) creates a reference on the LHS to the object on the RHS, > > I would like to > understand the reasoning behind such design decisions but I can't find > any 'deep' explanations at present > > So if you or anyone else can explain exactly why such odditties are > implemented I would be grateful. Python's assignment semantics are only an "oddity" to people whose prior exposure to programming languages is very limited. To anyone familiar with almost any other dynamic language -- such as Lisp, Scheme, Smalltalk, or Javascript -- it's not only unsurprising, it's the *obvious* thing to do. So I wouldn't class it as a "gotcha" in the same sense as truly Python-specific features like default argument evaluation and list comprehension variable scope. As for rationale, it comes down to something like this: Copying large chunks of data is expensive, so it makes sense to do it only when you really need to. And experience shows that most of the time you *don't* need to copy things. Furthermore, copying some kinds of things automatically and not others (as some other languages such as VB and Java do) makes the rules needlessly complicated and difficult to remember. So Python does the simplest possible thing and doesn't copy anything by default. If you want a copy, you need to do something explicit to make it happen. > Unfortunately it is almost certainly too late to propose fixes > (if appropriate) for such quirks in Python 3 Python's assignment behaviour is most definitely *not* something that needs "fixing"! -- Greg From tjreedy at udel.edu Mon Dec 12 09:55:57 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Dec 2011 03:55:57 -0500 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: <20111211215954.79f33232@bhuda.mired.org> References: <4EE54BFF.3050603@nedbatchelder.com> <20111211215954.79f33232@bhuda.mired.org> Message-ID: On 12/12/2011 12:59 AM, Mike Meyer wrote: > On Sun, 11 Dec 2011 19:34:07 -0500 > Ned Batchelder wrote: > >> Richard, I don't think I can provide you with a "why" for dynamic >> typing. > > And this is the wrong place to ask. Dynamic typing and naming objects > dates back to the precursors to LISP in the mid 50s. You should be > asking the people who made that decision. List was designed for writing algorithms. > By the same token, have you asked anyone why C/Java/etc. have static > typing and name locations? It's an equally valid question. C was designed for writing a computing machine operating system with mutable sequential memory slots numbered from 0 to 2**n - 1. In this respect, Python is much more like List than C, even though its syntax is more like C. -- Terry Jan Reedy From masklinn at masklinn.net Mon Dec 12 09:59:16 2011 From: masklinn at masklinn.net (Masklinn) Date: Mon, 12 Dec 2011 09:59:16 +0100 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: <4EE58D04.10403@canterbury.ac.nz> References: <4EE58D04.10403@canterbury.ac.nz> Message-ID: <5E0C7E67-23DA-4F38-84ED-B3EB7137320F@masklinn.net> On 2011-12-12, at 06:11 , Greg Ewing wrote: > Richard Prosser wrote: >> It seems to me that the essential problem is that of assignment in general, which (I believe) creates a reference on the LHS to the object on the RHS, >> I would like to understand the reasoning behind such design decisions but I can't find any 'deep' explanations at present >> So if you or anyone else can explain exactly why such odditties are implemented I would be grateful. > > Python's assignment semantics are only an "oddity" to people > whose prior exposure to programming languages is very limited. > To anyone familiar with almost any other dynamic language -- > such as Lisp, Scheme, Smalltalk, or Javascript -- it's not > only unsurprising It's not even a question of "dynamic languages", Java and C# reference types have exactly the same semantics (assignment copies the value part of a type, which is the reference itself). The only people I'd see confused by this are those with significant C++ experience, where assignment of references does indeed go through a copy of the object itself. > As for rationale, it comes down to something like this: Copying > large chunks of data is expensive, so it makes sense to do it > only when you really need to. And experience shows that most of > the time you *don't* need to copy things. Of course technically that copy could very well be performed "on write". This would significantly complexify the runtime as well. From masklinn at masklinn.net Mon Dec 12 10:00:59 2011 From: masklinn at masklinn.net (Masklinn) Date: Mon, 12 Dec 2011 10:00:59 +0100 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: <4EE54BFF.3050603@nedbatchelder.com> <20111211215954.79f33232@bhuda.mired.org> Message-ID: <7D6FA2B6-E7CE-4025-9867-DD80C279AEA9@masklinn.net> On 2011-12-12, at 09:55 , Terry Reedy wrote: > On 12/12/2011 12:59 AM, Mike Meyer wrote: >> On Sun, 11 Dec 2011 19:34:07 -0500 >> Ned Batchelder wrote: >>> Richard, I don't think I can provide you with a "why" for dynamic >>> typing. >> >> And this is the wrong place to ask. Dynamic typing and naming objects >> dates back to the precursors to LISP in the mid 50s. You should be >> asking the people who made that decision. > List was designed for writing algorithms. I think you meant "lisp" here ;) >> By the same token, have you asked anyone why C/Java/etc. have static >> typing and name locations? It's an equally valid question. > C was designed for writing a computing machine operating system with mutable sequential memory slots numbered from 0 to 2**n - 1. On the other hand, there are very few typed assemblies. From phd at phdru.name Sat Dec 10 15:42:30 2011 From: phd at phdru.name (Oleg Broytman) Date: Sat, 10 Dec 2011 18:42:30 +0400 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: References: Message-ID: <20111210144230.GA16801@iskra.aviel.ru> On Sat, Dec 10, 2011 at 02:16:21PM +0000, Richard Prosser wrote: > Although I love Python there are some aspects of the language design which > are disappointing and which can even lead to problems in some cases. What really is disappointing is the number of people who criticize Python without knowing it. > Another awkward 'feature' is the requirement for a trailing comma in > singleton tuples, due I believe to the use of expression parentheses rather > than (say) the use of special brackets like chevrons. You do not understand the syntax. Parens do not construct tuples - commas do. So for every tuple - even of length 1 - you must have a comma. The only exception is an empty tuple (of length 0). > Something that I personally wish for is the ability to declare variable > types 'up front' but that facility is missing from Python. You can use annotations. See PEP 3107. Twas implemented in Python 3.0. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From masklinn at masklinn.net Mon Dec 12 13:38:29 2011 From: masklinn at masklinn.net (Masklinn) Date: Mon, 12 Dec 2011 13:38:29 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <20111210144230.GA16801@iskra.aviel.ru> References: <20111210144230.GA16801@iskra.aviel.ru> Message-ID: <5D96D1D2-ABE1-4C04-A0C2-6BC52AFE9D2D@masklinn.net> On 2011-12-10, at 15:42 , Oleg Broytman wrote: > You can use annotations. See PEP 3107. Twas implemented in Python 3.0. It's not used by the runtime though (I think IDEA/PyCharm uses it, but I wouldn't bet on it), so it's mostly documentary at this point. From ubershmekel at gmail.com Mon Dec 12 13:41:26 2011 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 12 Dec 2011 14:41:26 +0200 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <5D96D1D2-ABE1-4C04-A0C2-6BC52AFE9D2D@masklinn.net> References: <20111210144230.GA16801@iskra.aviel.ru> <5D96D1D2-ABE1-4C04-A0C2-6BC52AFE9D2D@masklinn.net> Message-ID: On Mon, Dec 12, 2011 at 2:38 PM, Masklinn wrote: > On 2011-12-10, at 15:42 , Oleg Broytman wrote: > > You can use annotations. See PEP 3107. Twas implemented in Python 3.0. > It's not used by the runtime though (I think IDEA/PyCharm uses it, but > I wouldn't bet on it), so it's mostly documentary at this point. > > pyopt uses annotations as types for parsing command line options if that counts: http://code.google.com/p/pyopt/ --Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Mon Dec 12 14:00:28 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 12 Dec 2011 08:00:28 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <20111210144230.GA16801@iskra.aviel.ru> References: <20111210144230.GA16801@iskra.aviel.ru> Message-ID: <4EE5FAEC.1080907@nedbatchelder.com> On 12/10/2011 9:42 AM, Oleg Broytman wrote: > On Sat, Dec 10, 2011 at 02:16:21PM +0000, Richard Prosser wrote: >> Although I love Python there are some aspects of the language design which >> are disappointing and which can even lead to problems in some cases. > What really is disappointing is the number of people who criticize > Python without knowing it. > >> Another awkward 'feature' is the requirement for a trailing comma in >> singleton tuples, due I believe to the use of expression parentheses rather >> than (say) the use of special brackets like chevrons. > You do not understand the syntax. Parens do not construct tuples - > commas do. So for every tuple - even of length 1 - you must have a > comma. The only exception is an empty tuple (of length 0). I don't think we have to go as far as blaming the user. Tuple syntax is a little tricky, people often trip up on (x,) as a single-item tuple. You and I understand why it is, and there isn't a better alternative, but that one-item syntax sticks out when compared to the others: (), (x, y), (x, y, z), etc. This is a true "gotcha" as Richard originally expressed it. > Oleg. --Ned. From anacrolix at gmail.com Mon Dec 12 14:06:07 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 13 Dec 2011 00:06:07 +1100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EE5FAEC.1080907@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> Message-ID: (a,) is the real gotcha, not () On Tue, Dec 13, 2011 at 12:00 AM, Ned Batchelder wrote: > On 12/10/2011 9:42 AM, Oleg Broytman wrote: >> >> On Sat, Dec 10, 2011 at 02:16:21PM +0000, Richard Prosser wrote: >>> >>> Although I love Python there are some aspects of the language design >>> which >>> are disappointing and which can even lead to problems in some cases. >> >> ? ?What really is disappointing is the number of people who criticize >> Python without knowing it. >> >>> Another awkward 'feature' is the requirement for a trailing comma in >>> singleton tuples, due I believe to the use of expression parentheses >>> rather >>> than (say) the use of special brackets like chevrons. >> >> ? ?You do not understand the syntax. Parens do not construct tuples - >> commas do. So for every tuple - even of length 1 - you must have a >> comma. The only exception is an empty tuple (of length 0). > > I don't think we have to go as far as blaming the user. ?Tuple syntax is a > little tricky, people often trip up on (x,) as a single-item tuple. ?You and > I understand why it is, and there isn't a better alternative, but that > one-item syntax sticks out when compared to the others: (), (x, y), (x, y, > z), etc. ?This is a true "gotcha" as Richard originally expressed it. >> >> Oleg. > > --Ned. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- ?_? From solipsis at pitrou.net Mon Dec 12 14:07:58 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Dec 2011 14:07:58 +0100 Subject: [Python-ideas] Python Users Aren't Perfect References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> Message-ID: <20111212140758.301e7431@pitrou.net> On Mon, 12 Dec 2011 08:00:28 -0500 Ned Batchelder wrote: > On 12/10/2011 9:42 AM, Oleg Broytman wrote: > > On Sat, Dec 10, 2011 at 02:16:21PM +0000, Richard Prosser wrote: > >> Although I love Python there are some aspects of the language design which > >> are disappointing and which can even lead to problems in some cases. > > What really is disappointing is the number of people who criticize > > Python without knowing it. > > > >> Another awkward 'feature' is the requirement for a trailing comma in > >> singleton tuples, due I believe to the use of expression parentheses rather > >> than (say) the use of special brackets like chevrons. > > You do not understand the syntax. Parens do not construct tuples - > > commas do. So for every tuple - even of length 1 - you must have a > > comma. The only exception is an empty tuple (of length 0). > I don't think we have to go as far as blaming the user. Tuple syntax is > a little tricky, people often trip up on (x,) as a single-item tuple. > You and I understand why it is, and there isn't a better alternative, > but that one-item syntax sticks out when compared to the others: (), (x, > y), (x, y, z), etc. This is a true "gotcha" as Richard originally > expressed it. I think it would be more of a gotcha if parentheses were enough to create a tuple, though. Parentheses are useful to group operations, either for stylistic / syntactic support (think multi-line statements), or to work around operator precedence. Creating a tuple by mistake because you put some parentheses where not necessary would be really annoying. Regards Antoine. From ned at nedbatchelder.com Mon Dec 12 14:15:45 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 12 Dec 2011 08:15:45 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <20111212140758.301e7431@pitrou.net> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> Message-ID: <4EE5FE81.3060304@nedbatchelder.com> On 12/12/2011 8:07 AM, Antoine Pitrou wrote: > On Mon, 12 Dec 2011 08:00:28 -0500 > Ned Batchelder > wrote: >> On 12/10/2011 9:42 AM, Oleg Broytman wrote: >>> On Sat, Dec 10, 2011 at 02:16:21PM +0000, Richard Prosser wrote: >>>> Although I love Python there are some aspects of the language design which >>>> are disappointing and which can even lead to problems in some cases. >>> What really is disappointing is the number of people who criticize >>> Python without knowing it. >>> >>>> Another awkward 'feature' is the requirement for a trailing comma in >>>> singleton tuples, due I believe to the use of expression parentheses rather >>>> than (say) the use of special brackets like chevrons. >>> You do not understand the syntax. Parens do not construct tuples - >>> commas do. So for every tuple - even of length 1 - you must have a >>> comma. The only exception is an empty tuple (of length 0). >> I don't think we have to go as far as blaming the user. Tuple syntax is >> a little tricky, people often trip up on (x,) as a single-item tuple. >> You and I understand why it is, and there isn't a better alternative, >> but that one-item syntax sticks out when compared to the others: (), (x, >> y), (x, y, z), etc. This is a true "gotcha" as Richard originally >> expressed it. > I think it would be more of a gotcha if parentheses were enough to > create a tuple, though. Parentheses are useful to group operations, > either for stylistic / syntactic support (think multi-line statements), > or to work around operator precedence. Creating a tuple by mistake > because you put some parentheses where not necessary would be really > annoying. Believe me, I understand the issues. It is true, though that the single-element tuple syntax is often a surprise to people, and often well into their Python learning experience. We often repeat, "it isn't parens that make a tuple, but a comma." Then why when displaying a tuple does Python insist on using parens around it? >>> 1, 2, 3 (1, 2, 3) I'm not saying it shouldn't, it's a rhetorical question. The repr of a tuple always includes parens, even though "parens don't make a tuple." It's the best of all the options, but let's face it: it's confusing. --Ned. > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From solipsis at pitrou.net Mon Dec 12 14:19:19 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Dec 2011 14:19:19 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EE5FE81.3060304@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> Message-ID: <1323695959.21067.21.camel@localhost.localdomain> Le lundi 12 d?cembre 2011 ? 08:15 -0500, Ned Batchelder a ?crit : > Believe me, I understand the issues. It is true, though that the > single-element tuple syntax is often a surprise to people, and often > well into their Python learning experience. We often repeat, "it isn't > parens that make a tuple, but a comma." Then why when displaying a > tuple does Python insist on using parens around it? > > >>> 1, 2, 3 > (1, 2, 3) I would say: - because it's easier to read (subjectively so, I guess) - because it's easier to copy/paste into an expression without running into precedence problems Regards Antoine. From masklinn at masklinn.net Mon Dec 12 14:25:42 2011 From: masklinn at masklinn.net (Masklinn) Date: Mon, 12 Dec 2011 14:25:42 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EE5FE81.3060304@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> Message-ID: <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> On 2011-12-12, at 14:15 , Ned Batchelder wrote: > > Believe me, I understand the issues. It is true, though that the single-element tuple syntax is often a surprise to people, and often well into their Python learning experience. We often repeat, "it isn't parens that make a tuple, but a comma." Then why when displaying a tuple does Python insist on using parens around it? > > >>> 1, 2, 3 > (1, 2, 3) > > I'm not saying it shouldn't, it's a rhetorical question. The repr of a tuple always includes parens, even though "parens don't make a tuple." It's the best of all the options, but let's face it: it's confusing. An alternative would be to just drop the literal unary tuple. I would be kind-of sad as I'd lose the ability to unpack singleton iterables, but it would "fix" the issue. FWIW, Haskell does not have a literal singleton (the standard defines "unit" `()` and 2-tuple through 15-tuple) From ned at nedbatchelder.com Mon Dec 12 14:26:16 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 12 Dec 2011 08:26:16 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <1323695959.21067.21.camel@localhost.localdomain> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <1323695959.21067.21.camel@localhost.localdomain> Message-ID: <4EE600F8.8010801@nedbatchelder.com> On 12/12/2011 8:19 AM, Antoine Pitrou wrote: > Le lundi 12 d?cembre 2011 ? 08:15 -0500, Ned Batchelder a ?crit : >> Believe me, I understand the issues. It is true, though that the >> single-element tuple syntax is often a surprise to people, and often >> well into their Python learning experience. We often repeat, "it isn't >> parens that make a tuple, but a comma." Then why when displaying a >> tuple does Python insist on using parens around it? >> >> >>> 1, 2, 3 >> (1, 2, 3) > I would say: > - because it's easier to read (subjectively so, I guess) > - because it's easier to copy/paste into an expression without running > into precedence problems Yes, those are good reasons. And you can see why Python's insistence on showing tuples with parens contributes to the gotcha that the parens are a red herring, and it's the commas that are important. Also, who hasn't said this to a beginner: "lists and tuples are very similar, lists use square brackets, tuples use parens"? Somehow, in a list, the commas don't make a tuple... It's complicated. I welcome Richard's help in explaining this issue to beginners. --Ned. > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From anacrolix at gmail.com Mon Dec 12 14:41:41 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 13 Dec 2011 00:41:41 +1100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EE600F8.8010801@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <1323695959.21067.21.camel@localhost.localdomain> <4EE600F8.8010801@nedbatchelder.com> Message-ID: Here's some crazy ideas: * Don't allow iteration on tuples. for i in tuple * Make the singleton tuple the default object representation: (), a, (a, b), (a, b, c), ... * Steal STM, Channels, and sparks from Haskell * Profit. On Tue, Dec 13, 2011 at 12:26 AM, Ned Batchelder wrote: > On 12/12/2011 8:19 AM, Antoine Pitrou wrote: >> >> Le lundi 12 d?cembre 2011 ? 08:15 -0500, Ned Batchelder a ?crit : >>> >>> Believe me, I understand the issues. ?It is true, though that the >>> single-element tuple syntax is often a surprise to people, and often >>> well into their Python learning experience. ?We often repeat, "it isn't >>> parens that make a tuple, but a comma." ?Then why when displaying a >>> tuple does Python insist on using parens around it? >>> >>> ?>>> ?1, 2, 3 >>> ? ? ?(1, 2, 3) >> >> I would say: >> - because it's easier to read (subjectively so, I guess) >> - because it's easier to copy/paste into an expression without running >> into precedence problems > > Yes, those are good reasons. ?And you can see why Python's insistence on > showing tuples with parens contributes to the gotcha that the parens are a > red herring, and it's the commas that are important. ?Also, who hasn't said > this to a beginner: ?"lists and tuples are very similar, lists use square > brackets, tuples use parens"? ?Somehow, in a list, the commas don't make a > tuple... ?It's complicated. > > I welcome Richard's help in explaining this issue to beginners. > > --Ned. > >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- ?_? From anacrolix at gmail.com Mon Dec 12 14:44:46 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 13 Dec 2011 00:44:46 +1100 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: <5E0C7E67-23DA-4F38-84ED-B3EB7137320F@masklinn.net> References: <4EE58D04.10403@canterbury.ac.nz> <5E0C7E67-23DA-4F38-84ED-B3EB7137320F@masklinn.net> Message-ID: > The only people I'd see confused by this are those with significant> C++ experience, where assignment of references does indeed go through> a copy of the object itself. ?_? On Mon, Dec 12, 2011 at 7:59 PM, Masklinn wrote: > On 2011-12-12, at 06:11 , Greg Ewing wrote: >> Richard Prosser wrote: >>> It seems to me that the essential problem is that of assignment in general, which (I believe) creates a reference on the LHS to the object on the RHS, >>> I would like to understand the reasoning behind such design decisions but I can't find any 'deep' explanations at present >>> So if you or anyone else can explain exactly why such odditties are implemented I would be grateful. >> >> Python's assignment semantics are only an "oddity" to people >> whose prior exposure to programming languages is very limited. >> To anyone familiar with almost any other dynamic language -- >> such as Lisp, Scheme, Smalltalk, or Javascript -- it's not >> only unsurprising > It's not even a question of "dynamic languages", Java and C# > reference types have exactly the same semantics (assignment copies > the value part of a type, which is the reference itself). > > The only people I'd see confused by this are those with significant > C++ experience, where assignment of references does indeed go through > a copy of the object itself. > >> As for rationale, it comes down to something like this: Copying >> large chunks of data is expensive, so it makes sense to do it >> only when you really need to. And experience shows that most of >> the time you *don't* need to copy things. > Of course technically that copy could very well be performed > "on write". This would significantly complexify the runtime as well. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- ?_? From tjreedy at udel.edu Mon Dec 12 17:27:08 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Dec 2011 11:27:08 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EE5FE81.3060304@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> Message-ID: On 12/12/2011 8:15 AM, Ned Batchelder wrote: > parens that make a tuple, but a comma." Then why when displaying a tuple > does Python insist on using parens around it? > > >>> 1, 2, 3 > (1, 2, 3) > > I'm not saying it shouldn't, it's a rhetorical question. Actually, I would agree that it would be better to not mislead by not printing the parens unless necessary. (If done from the beginning.) For something like >>> t = 1,2 >>> l = list(t) >>> t,l ((1, 2), [1, 2]) the outer parens are also unnecessary (while the inner ones are needed) and make the result less easy to read. But the easiest way to not print them would probably be to test the output string after it is constructed for beginning with '(' and ending with ')' and strip them off if so. -- Terry Jan Reedy From tjreedy at udel.edu Mon Dec 12 17:40:03 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Dec 2011 11:40:03 -0500 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: <7D6FA2B6-E7CE-4025-9867-DD80C279AEA9@masklinn.net> References: <4EE54BFF.3050603@nedbatchelder.com> <20111211215954.79f33232@bhuda.mired.org> <7D6FA2B6-E7CE-4025-9867-DD80C279AEA9@masklinn.net> Message-ID: On 12/12/2011 4:00 AM, Masklinn wrote: > On 2011-12-12, at 09:55 , Terry Reedy wrote: >> On 12/12/2011 12:59 AM, Mike Meyer wrote: >>> On Sun, 11 Dec 2011 19:34:07 -0500 >>> Ned Batchelder wrote: >>>> Richard, I don't think I can provide you with a "why" for dynamic >>>> typing. >>> >>> And this is the wrong place to ask. Dynamic typing and naming objects >>> dates back to the precursors to LISP in the mid 50s. You should be >>> asking the people who made that decision. >> List was designed for writing algorithms. > I think you meant "lisp" here ;) Definitely. >>> By the same token, have you asked anyone why C/Java/etc. have static >>> typing and name locations? It's an equally valid question. >> C was designed for writing a computing machine operating system with mutable sequential memory slots numbered from 0 to 2**n - 1. > On the other hand, there are very few typed assemblies. The question of typing is somewhat orthogonal to that of naming value objects versus storage locations. There are strong stactic typed named value languages like ML. They even require different operators for int and float arithmetic, just like assembler. Assemblers also type data versus address registers and may have signed versus unsigned int operations. But most typing has to be done by the programmer, just as C requires the programmer to do garbage collection. -- Terry Jan Reedy From masklinn at masklinn.net Mon Dec 12 17:49:10 2011 From: masklinn at masklinn.net (Masklinn) Date: Mon, 12 Dec 2011 17:49:10 +0100 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: <4EE54BFF.3050603@nedbatchelder.com> <20111211215954.79f33232@bhuda.mired.org> <7D6FA2B6-E7CE-4025-9867-DD80C279AEA9@masklinn.net> Message-ID: <7FBD2B97-5965-4D47-9256-DE22A5BEA7AC@masklinn.net> On 2011-12-12, at 17:40 , Terry Reedy wrote: >>>> By the same token, have you asked anyone why C/Java/etc. have static >>>> typing and name locations? It's an equally valid question. >>> C was designed for writing a computing machine operating system with mutable sequential memory slots numbered from 0 to 2**n - 1. >> On the other hand, there are very few typed assemblies. > The question of typing is somewhat orthogonal to that of naming value objects versus storage locations. There are strong stactic typed named value languages like ML. I know, hence my quoting the part about typing, not the part about variable semantics. > They even require different operators for int and float arithmetic, just like assembler. That's got nothing to do with the typing or binding disciplines though, it's an issue of ML's type system. Haskell does not have that issue and is arguably more statically typed than MLs. > Assemblers also type data versus address registers and may have signed versus unsigned int operations. But most typing has to be done by the programmer Which is the point I was trying to make, a point which kind-of goes against the justification used for C being statically typed, as assemblies are also designed for driving systems with mutable sequential memory slots. From tjreedy at udel.edu Mon Dec 12 18:21:49 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Dec 2011 12:21:49 -0500 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: <7FBD2B97-5965-4D47-9256-DE22A5BEA7AC@masklinn.net> References: <4EE54BFF.3050603@nedbatchelder.com> <20111211215954.79f33232@bhuda.mired.org> <7D6FA2B6-E7CE-4025-9867-DD80C279AEA9@masklinn.net> <7FBD2B97-5965-4D47-9256-DE22A5BEA7AC@masklinn.net> Message-ID: On 12/12/2011 11:49 AM, Masklinn wrote: > Which is the point I was trying to make, a point which kind-of goes > against the justification used for C being statically typed, as > assemblies are also designed for driving systems with mutable > sequential memory slots. C was designed to remove some, but just some, of the burden of managing blocks of memory. One of the unfortunate ironies of C is that C programmers tend to forget that they *are* working with contiguous mutable blocks and not with isolated Python-like objects. Hence buffer-run exploits by malware writers who *do* remember and study the exact order of memory blocks in particular binaries. -- Terry Jan Reedy From mwm at mired.org Mon Dec 12 18:37:22 2011 From: mwm at mired.org (Mike Meyer) Date: Mon, 12 Dec 2011 09:37:22 -0800 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: References: <4EE54BFF.3050603@nedbatchelder.com> <20111211215954.79f33232@bhuda.mired.org> <7D6FA2B6-E7CE-4025-9867-DD80C279AEA9@masklinn.net> Message-ID: <20111212093722.36264852@mikmeyer-vm-fedora> On Mon, 12 Dec 2011 11:40:03 -0500 Terry Reedy wrote: > On 12/12/2011 4:00 AM, Masklinn wrote: > > On 2011-12-12, at 09:55 , Terry Reedy wrote: > >> On 12/12/2011 12:59 AM, Mike Meyer wrote: > >>> By the same token, have you asked anyone why C/Java/etc. have > >>> static typing and name locations? It's an equally valid question. > >> C was designed for writing a computing machine operating system > >> with mutable sequential memory slots numbered from 0 to 2**n - 1. > > On the other hand, there are very few typed assemblies. > The question of typing is somewhat orthogonal to that of naming value > objects versus storage locations. There are strong stactic typed > named value languages like ML. They even require different operators > for int and float arithmetic, just like assembler. Yup. You can also find dynamically typed named locations languages, like BCPL (though "untyped" might be more descriptive). It also has different operators for int and float arithmetic, because the locations don't have type information, so it has to come from the operator. References: <87pqhtafrz.fsf@vostro.rath.org> <20111019040214.GA5524@flay.puzzling.org> <87lishccj3.fsf@inspiron.ap.columbia.edu> <87fwipca9k.fsf@inspiron.ap.columbia.edu> <20111020001922.GA2191@chopin.edu.pl> <87mxcw1k2p.fsf@vostro.rath.org> <20111022002258.GB7138@chopin.edu.pl> Message-ID: <4EE64EE0.10103@rath.org> On 10/21/2011 08:22 PM, Jan Kaliszewski wrote: > An improved (and at the same time simplified) implementation (being also > a recipe for Python 2.x, though this list is about ideas for Py3.x): > > class CleanupManager(object): > > def __init__(self, initial_callbacks=()): > self.cleanup_callbacks = list(initial_callbacks) > > def register(self, callback, *args, **kwargs): > self.cleanup_callbacks.append((callback, args, kwargs)) > > def __enter__(self): > return self > > def __exit__(self, exc_type, exc, tb): > self._next_callback() > > def _next_callback(self): > if self.cleanup_callbacks: > callback, args, kwargs = self.cleanup_callbacks.pop() > try: > callback(*args, **kwargs) > finally: > # all cleanup callbacks to be used > # Py3.x: all errors to be reported > self._next_callback() > > I hope it implements well what you explained... I'm not sure if it is > worth to be added to the standard library (in the case of your primary > example I'd rather prefer that try-finally nested structure) -- though > in some cases it may become really useful: > > with CleanupMagager() as cm: > ... > cm.register(foo) > ... > if cond: > cm.register(bar) > else: > cm.register(spam) > for x in y: > cm.register(baz, x) I've opened a wishlist bug for this at http://bugs.python.org/issue13585. Best, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From greg.ewing at canterbury.ac.nz Mon Dec 12 22:13:22 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Dec 2011 10:13:22 +1300 Subject: [Python-ideas] Python Isn't Perfect: adding a 'gotchas' section to the tutorial In-Reply-To: <5E0C7E67-23DA-4F38-84ED-B3EB7137320F@masklinn.net> References: <4EE58D04.10403@canterbury.ac.nz> <5E0C7E67-23DA-4F38-84ED-B3EB7137320F@masklinn.net> Message-ID: <4EE66E72.5080803@canterbury.ac.nz> Masklinn wrote: > Of course technically that copy could very well be performed > "on write". This would significantly complexify the runtime as well. It could also lead to a lot of gratuitous inefficiency in programs, as people got into the habit of modifying anything passed to them in the knowledge that it wouldn't do any harm, but without considering the cost of all the implicit copying that it caused. -- Greg From greg.ewing at canterbury.ac.nz Mon Dec 12 22:21:11 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Dec 2011 10:21:11 +1300 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> Message-ID: <4EE67047.8050209@canterbury.ac.nz> Masklinn wrote: > FWIW, Haskell does not have a literal singleton (the standard defines > "unit" `()` and 2-tuple through 15-tuple) That's because, due to its static typing, there is no reason you would ever need to use a 1-tuple rather than a bare value. We're not that lucky in Python, though. -- Greg From ron3200 at gmail.com Mon Dec 12 23:07:13 2011 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 12 Dec 2011 16:07:13 -0600 Subject: [Python-ideas] Transportable indent level markers. /// ;;; \\\ In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> Message-ID: <1323727633.29879.181.camel@Gutsy> While typing this reply to the "Alternate Suite Delineation Syntax For Python?" thread. I realized that having things all on a single line isn't important or needed!!! Yes, it's not needed for templates, or code generation. So this is an alternate to the alternate. What is needed is a uniform way to represent indenting that is easy to transport to (ie paste into) code with different indent levels. INDENT LEVEL MARKERS: ---------------------- Offer a new way to spell , , and . To make that work, we need to use unique character sequences that aren't already used. After some thought, I came up with these indent-markers. /// # indent level +1 ;;; # indent level +0 \\\ # indent level -1 These are unique enough to be easy to see, and the tripled versions of them aren't really that bad as we already have triple quotes, and they would be less than the white space they are replacing in most cases. So this isn't brackets, or braces. It would be just an alternate spelling in already valid python. It should not effect the parser, grammar, or interpreter in any way if it is done in pythons tokenizer. :-) Reusing some previous examples. #----------------------- > def acc(n) {: s=n; return {: def (i) {: nonlocal s; s += i; return > s}}} #Standard python... def acc(n): s=n def add(i): nonlocal s s += i return s return add #Single line version... def acc(n): /// s=n ;;; def add(i): /// nonlocal s \ ;;; s+=i ;;; return s \\\ return add (The '\' character should still work.) #Transportable indent level version... This could also be valid... and could be transportable to any other indent level without having to edit it. (So would the above single line version by adding a ';;;' to the beginning.) ;;; def acc(n): # indent +0 /// s = n # indent +1 ;;; def add(i): # indent +0 /// nonlocal s # indent +1 ;;; s += i # indent +0 ;;; return s # indent +0 \\\ return add # indent -1 It has the nice properties of being uniform, and being able to be inserted easily into any existing code block without having to adjust the indent levels. I believe that these properties are more important than being able to put things on a single line and could be a very useful in code generating programs. ;-) It may be easy to implement. This could probably all be done in the tokinze.c get_tok() function, where it detects and converts white space to indention levels. Nothing else in python would need to be changed. The tokenizer already detects line continuations and different line endings, so this wouldn't be that out of place. Cheers, Ron From phd at phdru.name Mon Dec 12 23:36:38 2011 From: phd at phdru.name (Oleg Broytman) Date: Tue, 13 Dec 2011 02:36:38 +0400 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ In-Reply-To: <1323727633.29879.181.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: <20111212223638.GA32133@iskra.aviel.ru> On Mon, Dec 12, 2011 at 04:07:13PM -0600, Ron Adam wrote: > /// # indent level +1 > ;;; # indent level +0 > \\\ # indent level -1 [skip] > def acc(n): /// s=n ;;; def add(i): /// nonlocal s \ > ;;; s+=i ;;; return s \\\ return add from __future__ import brainfuck (I mean well-known esoteric programming language)? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From massimo.dipierro at gmail.com Mon Dec 12 23:40:31 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 12 Dec 2011 16:40:31 -0600 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ In-Reply-To: <1323727633.29879.181.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: <2B666E0F-8B3D-4E66-9E3A-D5025393D432@gmail.com> I do not like it and I am not convinced this belongs to Python Yet you may like this other solution: exec(reindent("def f(a):; x=0; for i in range(a):; x+=i; pass; return x; print f(5)")) here reindent is a modification of the web2py reindent function (used in templates). - ; is converted to \n - loops are closed with pass - ifs are also closed with pass - def can be closed with pass or return - class must also end with pass In web2py-ese templates we would not use it like this. we would write instead: {{def f(a):}}{{x=0}}{{for i in range(a):}}{{x+=i}}{{pass}}{{return x}}{{=f(5)}} or break it into {{ def f(a): x=0 for i in range(a): x+=i pass return x = f(5) }} (indentation is irrelevant even within the {{code}}) -------------- next part -------------- A non-text attachment was scrubbed... Name: reindent.py Type: text/x-python-script Size: 4046 bytes Desc: not available URL: -------------- next part -------------- On Dec 12, 2011, at 4:07 PM, Ron Adam wrote: > > While typing this reply to the "Alternate Suite Delineation Syntax For > Python?" thread. I realized that having things all on a single line > isn't important or needed!!! Yes, it's not needed for templates, or > code generation. So this is an alternate to the alternate. > > What is needed is a uniform way to represent indenting that is easy to > transport to (ie paste into) code with different indent levels. > > > INDENT LEVEL MARKERS: > ---------------------- > Offer a new way to spell , , and . > > To make that work, we need to use unique character sequences that aren't > already used. After some thought, I came up with these indent-markers. > > /// # indent level +1 > ;;; # indent level +0 > \\\ # indent level -1 > > These are unique enough to be easy to see, and the tripled versions of > them aren't really that bad as we already have triple quotes, and they > would be less than the white space they are replacing in most cases. > > So this isn't brackets, or braces. It would be just an alternate > spelling in already valid python. It should not effect the parser, > grammar, or interpreter in any way if it is done in pythons > tokenizer. :-) > > > Reusing some previous examples. > > #----------------------- > >> def acc(n) {: s=n; return {: def (i) {: nonlocal s; s += i; return >> s}}} > > #Standard python... > > def acc(n): > s=n > def add(i): > nonlocal s > s += i > return s > return add > > > #Single line version... > > def acc(n): /// s=n ;;; def add(i): /// nonlocal s \ > ;;; s+=i ;;; return s \\\ return add > > (The '\' character should still work.) > > > #Transportable indent level version... > > This could also be valid... and could be transportable to any other > indent level without having to edit it. > > (So would the above single line version by adding a ';;;' to the > beginning.) > > ;;; def acc(n): # indent +0 > /// s = n # indent +1 > ;;; def add(i): # indent +0 > /// nonlocal s # indent +1 > ;;; s += i # indent +0 > ;;; return s # indent +0 > \\\ return add # indent -1 > > It has the nice properties of being uniform, and being able to be > inserted easily into any existing code block without having to adjust > the indent levels. > > I believe that these properties are more important than being able to > put things on a single line and could be a very useful in code > generating programs. ;-) > > > It may be easy to implement. > > This could probably all be done in the tokinze.c get_tok() function, > where it detects and converts white space to indention levels. Nothing > else in python would need to be changed. The tokenizer already detects > line continuations and different line endings, so this wouldn't be that > out of place. > > Cheers, > Ron > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From anacrolix at gmail.com Mon Dec 12 23:45:56 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 13 Dec 2011 09:45:56 +1100 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ In-Reply-To: <2B666E0F-8B3D-4E66-9E3A-D5025393D432@gmail.com> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <2B666E0F-8B3D-4E66-9E3A-D5025393D432@gmail.com> Message-ID: ?_? On Tue, Dec 13, 2011 at 9:40 AM, Massimo Di Pierro wrote: > I do not like it and I am not convinced this belongs to Python > > Yet you may like this other solution: > > exec(reindent("def f(a):; x=0; for i in range(a):; x+=i; pass; return x; print f(5)")) > > here reindent is a modification of the web2py reindent function (used in templates). > > - ; is converted to \n > - loops are closed with pass > - ifs are also closed with pass > - def can be closed with pass or return > - class must also end with pass > > In web2py-ese templates we would not use it like this. we would write instead: > > {{def f(a):}}{{x=0}}{{for i in range(a):}}{{x+=i}}{{pass}}{{return x}}{{=f(5)}} > > or break it into > > {{ > def f(a): > ? x=0 > ? for i in range(a): > ? ? ?x+=i > ? ? ?pass > ? return x > = f(5) > }} (indentation is irrelevant even within the {{code}}) > > > > > > > > On Dec 12, 2011, at 4:07 PM, Ron Adam wrote: > >> >> While typing this reply to the "Alternate Suite Delineation Syntax For >> Python?" thread. ?I realized that having things ?all on a single line >> isn't important or needed!!! ?Yes, it's not needed for templates, or >> code generation. ?So this is an alternate to the alternate. >> >> What is needed is a uniform way to represent indenting that is easy to >> transport to (ie paste into) code with different indent levels. >> >> >> INDENT LEVEL MARKERS: >> ---------------------- >> Offer a new way to spell , , and . >> >> To make that work, we need to use unique character sequences that aren't >> already used. ?After some thought, I came up with these indent-markers. >> >> ? ?/// ? # indent level +1 >> ? ?;;; ? # indent level +0 >> ? ?\\\ ? # indent level -1 >> >> These are unique enough to be easy to see, and the tripled versions of >> them aren't really that bad as we already have triple quotes, and they >> would be less than the white space they are replacing in most cases. >> >> So this isn't brackets, or braces. It would be just an alternate >> spelling in already valid python. It should not effect the parser, >> grammar, or interpreter in any way if it is done in pythons >> tokenizer. ?:-) >> >> >> Reusing some previous examples. >> >> #----------------------- >> >>> ? def acc(n) {: s=n; return {: def (i) {: nonlocal s; s += i; return >>> s}}} >> >> #Standard python... >> >> def acc(n): >> ? ?s=n >> ? ?def add(i): >> ? ? ? ?nonlocal s >> ? ? ? ?s += i >> ? ? ? ?return s >> ? ?return add >> >> >> #Single line version... >> >> ?def acc(n): /// s=n ;;; def add(i): /// nonlocal s \ >> ?;;; s+=i ;;; return s \\\ return add >> >> (The '\' character should still work.) >> >> >> #Transportable indent level version... >> >> This could also be valid... and could be transportable to any other >> indent level without having to edit it. >> >> (So would the above single line version by adding a ';;;' to the >> beginning.) >> >> ;;; def acc(n): ? ? # indent +0 >> /// s = n ? ? ? ? ? # indent +1 >> ;;; def add(i): ? ? # indent +0 >> /// nonlocal s ? ? ?# indent +1 >> ;;; s += i ? ? ? ? ?# indent +0 >> ;;; return s ? ? ? ?# indent +0 >> \\\ return add ? ? ?# indent -1 >> >> It has the nice properties of being uniform, and being able to be >> inserted easily into any existing code block without having to adjust >> the indent levels. >> >> I believe that these properties are more important than being able to >> put things on a single line and could be a very useful in code >> generating programs. ;-) >> >> >> It may be easy to implement. >> >> This could probably all be done in the tokinze.c get_tok() function, >> where it detects and converts white space to indention levels. ?Nothing >> else in python would need to be changed. ?The tokenizer already detects >> line continuations and different line endings, so this wouldn't be that >> out of place. >> >> Cheers, >> ? Ron >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- ?_? From python at mrabarnett.plus.com Tue Dec 13 00:15:39 2011 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 12 Dec 2011 23:15:39 +0000 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ In-Reply-To: <1323727633.29879.181.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: <4EE68B1B.6020406@mrabarnett.plus.com> On 12/12/2011 22:07, Ron Adam wrote: > > While typing this reply to the "Alternate Suite Delineation Syntax For > Python?" thread. I realized that having things all on a single line > isn't important or needed!!! Yes, it's not needed for templates, or > code generation. So this is an alternate to the alternate. > > What is needed is a uniform way to represent indenting that is easy to > transport to (ie paste into) code with different indent levels. > > > INDENT LEVEL MARKERS: > ---------------------- > Offer a new way to spell,, and. > > To make that work, we need to use unique character sequences that aren't > already used. After some thought, I came up with these indent-markers. > > /// # indent level +1 > ;;; # indent level +0 > \\\ # indent level -1 > > These are unique enough to be easy to see, and the tripled versions of > them aren't really that bad as we already have triple quotes, and they > would be less than the white space they are replacing in most cases. > > So this isn't brackets, or braces. It would be just an alternate > spelling in already valid python. It should not effect the parser, > grammar, or interpreter in any way if it is done in pythons > tokenizer. :-) > [snip] -1 If the problem is with leading whitespace being stripped, then what about the non-breaking space ("\xA0")? We all use Unicode now, don't we? :-) The bonus is that it still looks like a space. From ron3200 at gmail.com Tue Dec 13 01:01:00 2011 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 12 Dec 2011 18:01:00 -0600 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ In-Reply-To: <4EE68B1B.6020406@mrabarnett.plus.com> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <4EE68B1B.6020406@mrabarnett.plus.com> Message-ID: <1323734460.29879.231.camel@Gutsy> On Mon, 2011-12-12 at 23:15 +0000, MRAB wrote: > On 12/12/2011 22:07, Ron Adam wrote: > > > > While typing this reply to the "Alternate Suite Delineation Syntax For > > Python?" thread. I realized that having things all on a single line > > isn't important or needed!!! Yes, it's not needed for templates, or > > code generation. So this is an alternate to the alternate. > > > > What is needed is a uniform way to represent indenting that is easy to > > transport to (ie paste into) code with different indent levels. > > > > > > INDENT LEVEL MARKERS: > > ---------------------- > > Offer a new way to spell,, and. > > > > To make that work, we need to use unique character sequences that aren't > > already used. After some thought, I came up with these indent-markers. > > > > /// # indent level +1 > > ;;; # indent level +0 > > \\\ # indent level -1 > > > > These are unique enough to be easy to see, and the tripled versions of > > them aren't really that bad as we already have triple quotes, and they > > would be less than the white space they are replacing in most cases. > > > > So this isn't brackets, or braces. It would be just an alternate > > spelling in already valid python. It should not effect the parser, > > grammar, or interpreter in any way if it is done in pythons > > tokenizer. :-) > > > [snip] > -1 > > If the problem is with leading whitespace being stripped, then what > about the non-breaking space ("\xA0")? We all use Unicode now, don't > we? :-) Think bigger! ie... easy automated program content and construction. You have this string... foo_bit.py """ def foo(*args): result = return result x = foo(a, b, c) """ And you want it to be inserted in this string automatically with the least amount of processing. (with another program you have.) bar_bit.py """ ... def bar(a, b, c): ... ... return x ... """ (where ... means more lines of python code.) Now if you do a quick simple substitution you get. """ ... def bar(a, b, c): ... def foo(*args): result = return result x = foo(a, b, c) ... return x ... """ Which won't work and is why template programs need to do a lot of pre and post processing to get things right. If foo_bit.py was like this... """ ;;; def foo(*args): ;;; result = ;;; return result ;;; x = foo(a, b, c) """ Then you would get... """ ... def bar(a, b, c): ... ;;; def foo(*args): ;;; result = ;;; return result ;;; x = foo(a, b, c) ... return x ... """ And the tokenizer in python could read that directly as ... ... def bar(a, b, c): ... def foo(*args): result = return result x = foo(a, b, c) ... return x ... Which would make doing things like this really easy instead of really tricky. The indent level of bar.py won't matter because the indent marker ';;;' says to use the current level of indent before this line. Does that make more sense now? Cheers, Ron From python at mrabarnett.plus.com Tue Dec 13 01:40:44 2011 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 13 Dec 2011 00:40:44 +0000 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ In-Reply-To: <1323734460.29879.231.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <4EE68B1B.6020406@mrabarnett.plus.com> <1323734460.29879.231.camel@Gutsy> Message-ID: <4EE69F0C.5060501@mrabarnett.plus.com> On 13/12/2011 00:01, Ron Adam wrote: > On Mon, 2011-12-12 at 23:15 +0000, MRAB wrote: >> On 12/12/2011 22:07, Ron Adam wrote: >>> >>> While typing this reply to the "Alternate Suite Delineation Syntax For >>> Python?" thread. I realized that having things all on a single line >>> isn't important or needed!!! Yes, it's not needed for templates, or >>> code generation. So this is an alternate to the alternate. >>> >>> What is needed is a uniform way to represent indenting that is easy to >>> transport to (ie paste into) code with different indent levels. >>> >>> >>> INDENT LEVEL MARKERS: >>> ---------------------- >>> Offer a new way to spell,, and. >>> >>> To make that work, we need to use unique character sequences that aren't >>> already used. After some thought, I came up with these indent-markers. >>> >>> /// # indent level +1 >>> ;;; # indent level +0 >>> \\\ # indent level -1 >>> >>> These are unique enough to be easy to see, and the tripled versions of >>> them aren't really that bad as we already have triple quotes, and they >>> would be less than the white space they are replacing in most cases. >>> >>> So this isn't brackets, or braces. It would be just an alternate >>> spelling in already valid python. It should not effect the parser, >>> grammar, or interpreter in any way if it is done in pythons >>> tokenizer. :-) >>> >> [snip] >> -1 >> >> If the problem is with leading whitespace being stripped, then what >> about the non-breaking space ("\xA0")? We all use Unicode now, don't >> we? :-) > > Think bigger! ie... easy automated program content and construction. > > You have this string... > > foo_bit.py > """ > def foo(*args): > result = > return result > x = foo(a, b, c) > """ > > And you want it to be inserted in this string automatically with the > least amount of processing. (with another program you have.) > > bar_bit.py > """ > ... > def bar(a, b, c): > ... > > ... > return x > ... > """ > (where ... means more lines of python code.) > > > Now if you do a quick simple substitution you get. > > """ > ... > def bar(a, b, c): > ... > def foo(*args): > result = > return result > x = foo(a, b, c) > ... > return x > ... > """ > OK, let's suppose that bar_bit.py is a template file with an insertion point: bar_bit.py """ ... def bar(a, b, c): ... ... return x ... """ I would expect the inserted code to be indented by however much the "" is indented: """ ... def bar(a, b, c): ... def foo(*args): result = return result x = foo(a, b, c) ... return x ... """ > Which won't work and is why template programs need to > do a lot of pre and post processing to get things right. > > If foo_bit.py was like this... > > """ > ;;; def foo(*args): > ;;; result = > ;;; return result > ;;; x = foo(a, b, c) > """ > > Then you would get... > > """ > ... > def bar(a, b, c): > ... > ;;; def foo(*args): > ;;; result = > ;;; return result > ;;; x = foo(a, b, c) > ... > return x > ... > """ > > And the tokenizer in python could read that directly as ... > > ... > def bar(a, b, c): > ... > def foo(*args): > result = > return result > x = foo(a, b, c) > ... > return x > ... > > Which would make doing things like this really easy instead of really > tricky. > > The indent level of bar.py won't matter because the indent marker ';;;' > says to use the current level of indent before this line. > > Does that make more sense now? > I still think it looks messy and that it's the responsibility of the templater to fix the indentation. From stephen at xemacs.org Tue Dec 13 03:44:29 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 13 Dec 2011 11:44:29 +0900 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EE67047.8050209@canterbury.ac.nz> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> Message-ID: <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> Greg Ewing writes: > Masklinn wrote: > > > FWIW, Haskell does not have a literal singleton (the standard defines > > "unit" `()` and 2-tuple through 15-tuple) > > That's because, due to its static typing, there is no > reason you would ever need to use a 1-tuple rather than > a bare value. We're not that lucky in Python, though. I think you have misstated your point? That's not due to static typing, that's because you may *always* identify 1-factor products with the only factor, and Haskell made a deliberate decision to consistently represent the isomorphism class by the factor rather than the product. Eg, Python uses the same strategy for characters and strings (characters as a category are isomorphic to one-element strings), but chose a different representative (the one-element string rather than the character). As for Python's relative lack of luck, I think that's unlikely to be the reason why things are as they are. I'm pretty sure that Python's omission of character objects was deliberate. I think it was the right choice, given that (1) despite category theory, I believe we think of tuples as composite objects (at least I do), but strings seem to be more "monolithic" in some sense, and (2) pretty much all languages have chosen to leave identifiers unmarked (I guess Perl and many template languages should be considered exceptions), and mark strings and characters (often both with quotation marks, sometimes characters with a prefix) -- may as well go with the familiar if there's no good reason otherwise. From tjreedy at udel.edu Tue Dec 13 06:24:23 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 13 Dec 2011 00:24:23 -0500 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: <1323727633.29879.181.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: On 12/12/2011 5:07 PM, Ron Adam wrote: > To make that work, we need to use unique character sequences that aren't > already used. After some thought, I came up with these indent-markers. > > /// # indent level +1 > ;;; # indent level +0 > \\\ # indent level -1 >>> === <<< would work even better ;-). -- Terry Jan Reedy From guido at python.org Tue Dec 13 16:53:35 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Dec 2011 07:53:35 -0800 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: On Mon, Dec 12, 2011 at 9:24 PM, Terry Reedy wrote: > On 12/12/2011 5:07 PM, Ron Adam wrote: > >> To make that work, we need to use unique character sequences that aren't >> already used. ?After some thought, I came up with these indent-markers. >> >> ? ? /// ? # indent level +1 >> ? ? ;;; ? # indent level +0 >> ? ? \\\ ? # indent level -1 > > > >>> > === > <<< > would work even better ;-). No they wouldn't, these are too similar to the markers left behind by merge tools for unmergable sections. Those markers were chosen because they are unlikely to occur in regular code. Let's keep it that way. :-) -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Thu Dec 15 00:53:55 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Dec 2011 18:53:55 -0500 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: On 12/13/2011 10:53 AM, Guido van Rossum wrote: > On Mon, Dec 12, 2011 at 9:24 PM, Terry Reedy wrote: >> On 12/12/2011 5:07 PM, Ron Adam wrote: >> >>> To make that work, we need to use unique character sequences that aren't >>> already used. After some thought, I came up with these indent-markers. >>> >>> /// # indent level +1 >>> ;;; # indent level +0 >>> \\\ # indent level -1 My objection to these is that the interpretation is arbitrary; I happen to read them the wrong way. >> >> >>>>> >> === >> <<< >> would work even better ;-). > > No they wouldn't, these are too similar to the markers left behind by > merge tools for unmergable sections. Those markers were chosen because > they are unlikely to occur in regular code. Let's keep it that way. > :-) Funny. I choose them for the same reason. >>> is also bad because some mail agents, like Thunderbird not only interpret initial runs of > as message quotes but replace them with various colored bars. So if this were a serious idea, I would propose something else like ->| -|- |<- But a moot point unless we actually do want a standardized transport encoding for indents and dedents. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Thu Dec 15 01:18:10 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Dec 2011 13:18:10 +1300 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: <4EE93CC2.1070200@canterbury.ac.nz> On 15/12/11 12:53, Terry Reedy wrote: > Funny. I choose them for the same reason. >>> is also bad because some mail > agents, like Thunderbird not only interpret initial runs of > as message > quotes but replace them with various colored bars. Does anyone know a way of turning that off, by the way? It drives me batty when I'm looking at Python interactive session transcripts. -- Greg From ron3200 at gmail.com Thu Dec 15 02:08:19 2011 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Dec 2011 19:08:19 -0600 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: <1323911299.9574.44.camel@Gutsy> On Wed, 2011-12-14 at 18:53 -0500, Terry Reedy wrote: > On 12/13/2011 10:53 AM, Guido van Rossum wrote: > > On Mon, Dec 12, 2011 at 9:24 PM, Terry Reedy wrote: > >> On 12/12/2011 5:07 PM, Ron Adam wrote: > >> > >>> To make that work, we need to use unique character sequences that aren't > >>> already used. After some thought, I came up with these indent-markers. > >>> > >>> /// # indent level +1 > >>> ;;; # indent level +0 > >>> \\\ # indent level -1 > > My objection to these is that the interpretation is arbitrary; I happen > to read them the wrong way. > > >> > >> > >>>>> > >> === > >> <<< > >> would work even better ;-). > > > > No they wouldn't, these are too similar to the markers left behind by > > merge tools for unmergable sections. Those markers were chosen because > > they are unlikely to occur in regular code. Let's keep it that way. > > :-) > > Funny. I choose them for the same reason. >>> is also bad because some > mail agents, like Thunderbird not only interpret initial runs of > as > message quotes but replace them with various colored bars. So if this > were a serious idea, I would propose something else like > ->| > -|- > |<- > But a moot point unless we actually do want a standardized transport > encoding for indents and dedents. I think if there is enough interest in it, it could be done. It only changes the pre-tokenized representation of the language. To do braces correctly, it would require much deeper changes. My choices are based on both easy visibility and also what be nice for single line input. The ';;;' is consistent with the ';' in places where it is used. Triple characters are easier to type than odd sequences. I suppose I'd switch the slashes though so they follow the top down flow better. ;;; def acc(n): # indent +0 \\\ s = n # indent +1 ;;; def add(i): # indent +0 \\\ nonlocal s # indent +1 ;;; s += i # indent +0 ;;; return s # indent +0 /// return add # indent -1 cheers, Ron From andre.roberge at gmail.com Thu Dec 15 02:13:34 2011 From: andre.roberge at gmail.com (Andre Roberge) Date: Wed, 14 Dec 2011 21:13:34 -0400 Subject: [Python-ideas] Fwd: Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> Message-ID: On Wed, Dec 14, 2011 at 9:08 PM, Ron Adam wrote: > On Wed, 2011-12-14 at 18:53 -0500, Terry Reedy wrote: > > On 12/13/2011 10:53 AM, Guido van Rossum wrote: > > > On Mon, Dec 12, 2011 at 9:24 PM, Terry Reedy wrote: > > >> On 12/12/2011 5:07 PM, Ron Adam wrote: > > >> > > >>> To make that work, we need to use unique character sequences that > aren't > > >>> already used. After some thought, I came up with these > indent-markers. > > >>> > > >>> /// # indent level +1 > > >>> ;;; # indent level +0 > > >>> \\\ # indent level -1 > Why not use INDENT, ;, and DEDENT (or <;> and ) ? ;-) Andr? -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Dec 15 02:14:04 2011 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 15 Dec 2011 01:14:04 +0000 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: <4EE93CC2.1070200@canterbury.ac.nz> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <4EE93CC2.1070200@canterbury.ac.nz> Message-ID: <4EE949DC.90200@mrabarnett.plus.com> On 15/12/2011 00:18, Greg Ewing wrote: > On 15/12/11 12:53, Terry Reedy wrote: > >> Funny. I choose them for the same reason.>>> is also bad because some mail >> agents, like Thunderbird not only interpret initial runs of> as message >> quotes but replace them with various colored bars. > > Does anyone know a way of turning that off, by the way? > It drives me batty when I'm looking at Python interactive > session transcripts. > It annoys me too, so I went looking: Quote bars http://kb.mozillazine.org/Quote_bars On my PC the prefs.js file was in a subfolder of: C:\Documents and Settings\Administrator\Application Data\Thunderbird\Profiles and it now shows the ">" as ">". From ncoghlan at gmail.com Thu Dec 15 02:21:34 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Dec 2011 11:21:34 +1000 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: On Thu, Dec 15, 2011 at 9:53 AM, Terry Reedy wrote: > But a moot point unless we actually do want a standardized transport > encoding for indents and dedents. And if we *were* going to do that, it's far more sensible to use a scheme that is at least *theoretically* open to the possibility of eventually allowing suite expressions. For the record, if anyone really wants to play with this, I'd start with the tokenize module, and figure out what tweaks are needed to create the following mappings between the source code and the token stream: "{:" <-> token.OP(":"), token.NEWLINE, token.INDENT ":}" <-> token.NEWLINE, token.DEDENT Then you can do source-source transforms by taking the output of the tweaked tokenizer and feeding it to the standard one and vice-versa. That's should be all you really need to create a delimited syntax variant. Things only get complex if you try to turn that delimited syntax variant into multi-line lambda support. P.S. See http://bugs.python.org/issue2134 in regard to why you can't use token.COLON for this -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Dec 15 02:29:48 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Dec 2011 11:29:48 +1000 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: <1323911299.9574.44.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> Message-ID: On Thu, Dec 15, 2011 at 11:08 AM, Ron Adam wrote: > It only changes the pre-tokenized representation of the language. ?To do > braces correctly, it would require much deeper changes. This comment makes me think you may be working off a misunderstanding of the way Python's tokenisation works. Suites in Python's token stream are *already* explicitly delimited: "INDENT" and "DEDENT" tokens respectively mark the beginning and end of each suite. There's no such concept in the token stream as "don't change the indent level" - that's the assumed behaviour. So an explicitly delimited syntax that just offers an alternative way to get INDENT and DEDENT tokens into the stream would be fairly straightforward. You'd probably also want an explicit ";;" token to force a token.NEWLINE into the token stream. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ron3200 at gmail.com Thu Dec 15 04:19:08 2011 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Dec 2011 21:19:08 -0600 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> Message-ID: <1323919148.10290.59.camel@Gutsy> On Thu, 2011-12-15 at 11:29 +1000, Nick Coghlan wrote: > On Thu, Dec 15, 2011 at 11:08 AM, Ron Adam wrote: > > It only changes the pre-tokenized representation of the language. To do > > braces correctly, it would require much deeper changes. > > This comment makes me think you may be working off a misunderstanding > of the way Python's tokenisation works. In tokenizer.c there is the function tok_get() at line 1292. The first part of it counts spaces and/or tabs and checks for other things and then returns either DEDENT, INDENT, or possibly an ERRORTOKEN. Since this would not add any new tokens, I referred to it as a pretokenized representation. The way it would work is to detects these characters and use it to determine wheather to return DEDENT, or INDENT tokens in place of any surrounding white space immediately before or after it. If these are found in the middle of a line, it would backup with tok_backup(), to just before the token, and then ruturn a NEWLINE. On the next tok_get() call, it would repeat. It's would not be hard to do at all. > Suites in Python's token stream are *already* explicitly delimited: > "INDENT" and "DEDENT" tokens respectively mark the beginning and end > of each suite. There's no such concept in the token stream as "don't > change the indent level" - that's the assumed behaviour. You are thinking of later in the chain I think. We doing the change before that is decided. Think of them as hints for the tokenizer. The "don't change the indent level" hint, ';;;' is just a hint to ignore the white space here and accept the previous level. Because of that, the text source is able to be white space insensitive as far as indent levels is concerned. > So an explicitly delimited syntax that just offers an alternative way > to get INDENT and DEDENT tokens into the stream would be fairly > straightforward. Right. :-) > You'd probably also want an explicit ";;" token to force a > token.NEWLINE into the token stream. That isn't needed. Any of these in the middle of a line will add a new line and back up, so the next call to tok_get() will find it, and so on. ;;; x = 3 ;;; y = 2 ;;; def add(x,y): \\\ return x + y So this would be converted as the tokenizer goes over it to.. x = 3 y = 2 def add(x,y): return x + y Line continuations '\' are handled in pretty much the same way, they just eat the next newline, and continue. There is no token for a line continuation. These would work on the same level, Cheers, Ron From ncoghlan at gmail.com Thu Dec 15 04:40:13 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Dec 2011 13:40:13 +1000 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: <1323919148.10290.59.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> <1323919148.10290.59.camel@Gutsy> Message-ID: On Thu, Dec 15, 2011 at 1:19 PM, Ron Adam wrote: >> You'd probably also want an explicit ";;" token to force a >> token.NEWLINE into the token stream. > > That isn't needed. ?Any of these in the middle of a line will add a new > line and back up, so the next call to tok_get() will find it, and so on. OK, take the way you're thinking (indent +1, indent 0, indent -1) and instead think in terms of starting a suite, terminating a statement and terminating a suite: /// -> {: ;;; -> ;; \\\ -> :} Now do you see why I'm saying you're needlessly complicating things? Suite delimiters and statement terminators (or separators) are the way full whitespace insensitivity is normally handled when designing a language syntax. There's no reason to get creative here when the standard terminology and conventions would work just fine. For example, it shouldn't be difficult to create a variant of the tokenize module's tokeniser that adds the following rules: {: -> emits OP(':'), NEWLINE, INDENT and increments the parenlevel ;; -> emits NEWLINE :} -> emits NEWLINE, DEDENT and decrements the parenlevel and a variant of the untokenizer() that looks ahead and *emits* those character sequences when applicable. That should be enough to let you use Python code in whitespace insensitive environments without changing the semantics: Encoding for transport: tokenize() normally, untokenize() with suite delimiters Decoding from transport: tokenize() with suite delimiter support, untokenize() normally Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ron3200 at gmail.com Thu Dec 15 06:09:17 2011 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Dec 2011 23:09:17 -0600 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> <1323919148.10290.59.camel@Gutsy> Message-ID: <1323925757.11678.59.camel@Gutsy> On Thu, 2011-12-15 at 13:40 +1000, Nick Coghlan wrote: > On Thu, Dec 15, 2011 at 1:19 PM, Ron Adam wrote: > >> You'd probably also want an explicit ";;" token to force a > >> token.NEWLINE into the token stream. > > > > That isn't needed. Any of these in the middle of a line will add a new > > line and back up, so the next call to tok_get() will find it, and so on. > > OK, take the way you're thinking (indent +1, indent 0, indent -1) and > instead think in terms of starting a suite, terminating a statement > and terminating a suite: > > /// -> {: > ;;; -> ;; > \\\ -> :} > > Now do you see why I'm saying you're needlessly complicating things? The complicated part is in getting across a new idea. ;-) > Suite delimiters and statement terminators (or separators) are the way > full whitespace insensitivity is normally handled when designing a > language syntax. There's no reason to get creative here when the > standard terminology and conventions would work just fine. > > For example, it shouldn't be difficult to create a variant of the > tokenize module's tokeniser that adds the following rules: I'm not referring to a whole new variant, Just a small tweak to the current one. > {: -> emits OP(':'), NEWLINE, INDENT and increments the parenlevel > ;; -> emits NEWLINE > :} -> emits NEWLINE, DEDENT and decrements the parenlevel This is the same idea with different spelling in this case. I think yours would be a bit harder to implement, but not all that hard to do. The version I proposed would be even easier to do. Unless you intend to enforce matching '{:' and ':}''s, then it become a bit harder still. I'm not sure that is needed, but it may seem strange to users if you don't require them to match. At what point should it be a part of the grammar? Ok how about some examples... how would you notate this code bit so it can be pasted into an already existing file without having to re-edit it? ;;; for x in items: \\\ sum = op(x) ;;; if sum > MAX: \\\ break Lets say you are going to past in several locations in a file and those locations may have different indent levels. How would you write that using your suggestion so you don't have to change it in anyway for it to work. With my suggestion, that would just work no matter the source file indent level. (*) All you need is the line numbers to insert it at. I think with your idea, you would also need a place holder of some type. * Assuming the required names are present. ;-) > and a variant of the untokenizer() that looks ahead and *emits* those > character sequences when applicable. That shouldn't be a problem in either case. > That should be enough to let you use Python code in whitespace > insensitive environments without changing the semantics: Right, except I'd be happier with something that doesn't alter the visible characters, the colon included. > Encoding for transport: tokenize() normally, untokenize() with suite delimiters > Decoding from transport: tokenize() with suite delimiter support, > untokenize() normally I think one of the differences is I'm proposing a mode that doesn't require the whole file to be in matching form. So the "tokenize() with suite delimiter support" would just be the same tokenize() that is normally used. And the "untokenize() with suite delimiters" would probably just be a file that you run on the source to get a delimited version. I'm not proposing any untokenize() support. But I'm not against it either. I also think indent markers are more consistent with how people think when they are programming python. It matches how the blocks are defined. If this was a language where we already had braces, then I'd definitely be thinking the other way. :-) Cheers, Ron From ncoghlan at gmail.com Thu Dec 15 06:25:33 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Dec 2011 15:25:33 +1000 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: <1323925757.11678.59.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> <1323919148.10290.59.camel@Gutsy> <1323925757.11678.59.camel@Gutsy> Message-ID: On Thu, Dec 15, 2011 at 3:09 PM, Ron Adam wrote: > Lets say you are going to past in several locations in a file and those > locations may have different indent levels. ?How would you write that > using your suggestion so you don't have to change it in anyway for it to > work. > > With my suggestion, that would just work no matter the source file > indent level. (*) ?All you need is the line numbers to insert it at. ?I > think with your idea, you would also need a place holder of some type. OK, I at least understand what you're trying to do now. However, I don't think you have a valid use case. What would a templating language based on your notation look like? How would you extend it to implement multi-line lambdas? If you can't cleanly do those two things, who cares if it lets you copy and paste snippets at arbitrary points in a Python file? What's the *functional* pay-off for the ugliness? (And I freely admit that my suggestion is also quite ugly. However, the status quo, with everyone inventing assorted templating languages for use with Python that are largely independent of Python itself already sucks) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Dec 15 06:31:15 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Dec 2011 15:31:15 +1000 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: <1323925757.11678.59.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> <1323919148.10290.59.camel@Gutsy> <1323925757.11678.59.camel@Gutsy> Message-ID: On Thu, Dec 15, 2011 at 3:09 PM, Ron Adam wrote: > Lets say you are going to past in several locations in a file and those > locations may have different indent levels. ?How would you write that > using your suggestion so you don't have to change it in anyway for it to > work. Forget to answer this bit: I wouldn't. I'd use a templating language, that's what they're for. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ron3200 at gmail.com Thu Dec 15 08:06:21 2011 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 15 Dec 2011 01:06:21 -0600 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> <1323919148.10290.59.camel@Gutsy> <1323925757.11678.59.camel@Gutsy> Message-ID: <1323932781.11678.131.camel@Gutsy> On Thu, 2011-12-15 at 15:25 +1000, Nick Coghlan wrote: > On Thu, Dec 15, 2011 at 3:09 PM, Ron Adam wrote: > > Lets say you are going to past in several locations in a file and those > > locations may have different indent levels. How would you write that > > using your suggestion so you don't have to change it in anyway for it to > > work. > > > > With my suggestion, that would just work no matter the source file > > indent level. (*) All you need is the line numbers to insert it at. I > > think with your idea, you would also need a place holder of some type. > > OK, I at least understand what you're trying to do now. However, I > don't think you have a valid use case. What would a templating > language based on your notation look like? Any program that can patch text files together. As to how it does it, it probably would use a script of some sort. This would just make it easier to by offering a way around indenting issues. Another use case is it makes transmitting python compatible code easier in a web page or email. The braces version would do that also. I think it could make python a bit more attractive for things like that. Where a app is transmitted and run at the destination. There are still the security issues to work out, but this would knock down one domino in that direction. For example they don't have to event ways to avoid the white space munging. Python could be represented as code blocks that might look like... MAX: \\\ break"> Except much longer. In this case there isn't a need for it to be pretty. But there is a need for it to work correctly. I was thinking it might be possible to have the python shell echo the indent markes in the left side as it scroll lines up. Replacing the '>>>' and '...' with the respective markes as you enter the lines. And it could also take these if they are pasted in. Cutting and pasting from one shell to another could work in that case. As far as unglyness goes... is it really that bad? you can leave the white space in. The indent hints are used not the spaces in this case. ;;; for x in items: \\\ sum = op(x) ;;; if sum > MAX: \\\ break And keep in mind that this isn't something you will see in normal use. It's just available for when its needed. But if you run across code like this it will still work. Nothing special would need to be done to make it work. > How would you extend it to > implement multi-line lambdas? As I said, it doesn't change the language at all. So extending lambda is not in the scope of this topic. That would be something new and should be done separately in any case. > If you can't cleanly do those two > things, who cares if it lets you copy and paste snippets at arbitrary > points in a Python file? What's the *functional* pay-off for the > ugliness? ? See above. > (And I freely admit that my suggestion is also quite ugly. However, > the status quo, with everyone inventing assorted templating languages > for use with Python that are largely independent of Python itself > already sucks) If it was a big patch to many files, I would say it isn't worth it. But it would be a very small patch to only one file and wouldn't change the grammer, ast, or anything else. It's almost free. I think it may have a nice benefit for very little effort. But I think we should just wait a bit and see if there is more interest first. ;-) Cheers, Ron From ethan at stoneleaf.us Thu Dec 15 07:51:01 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 14 Dec 2011 22:51:01 -0800 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: <4EE949DC.90200@mrabarnett.plus.com> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <4EE93CC2.1070200@canterbury.ac.nz> <4EE949DC.90200@mrabarnett.plus.com> Message-ID: <4EE998D5.7070902@stoneleaf.us> MRAB wrote: > On 15/12/2011 00:18, Greg Ewing wrote: >> On 15/12/11 12:53, Terry Reedy wrote: >> >>> Funny. I choose them for the same reason.>>> is also bad because >>> some mail >>> agents, like Thunderbird not only interpret initial runs of> as >>> message >>> quotes but replace them with various colored bars. >> >> Does anyone know a way of turning that off, by the way? >> It drives me batty when I'm looking at Python interactive >> session transcripts. >> > It annoys me too, so I went looking: > > Quote bars > http://kb.mozillazine.org/Quote_bars > > On my PC the prefs.js file was in a subfolder of: > > C:\Documents and Settings\Administrator\Application > Data\Thunderbird\Profiles > > and it now shows the ">" as ">". I went the other way and changed my python prompt to '-->'. ~Ethan~ From ncoghlan at gmail.com Thu Dec 15 08:57:47 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Dec 2011 17:57:47 +1000 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: <1323932781.11678.131.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> <1323919148.10290.59.camel@Gutsy> <1323925757.11678.59.camel@Gutsy> <1323932781.11678.131.camel@Gutsy> Message-ID: There's no way *any* variant of this idea (regardless of spelling) is getting anywhere near the stdlib without a long history on PyPI as a popular templating and code embedding tool. -- Nick Coghlan (via Gmail on Android, so likely to be more terse than usual) On Dec 15, 2011 5:06 PM, "Ron Adam" wrote: > On Thu, 2011-12-15 at 15:25 +1000, Nick Coghlan wrote: > > On Thu, Dec 15, 2011 at 3:09 PM, Ron Adam wrote: > > > Lets say you are going to past in several locations in a file and those > > > locations may have different indent levels. How would you write that > > > using your suggestion so you don't have to change it in anyway for it > to > > > work. > > > > > > With my suggestion, that would just work no matter the source file > > > indent level. (*) All you need is the line numbers to insert it at. I > > > think with your idea, you would also need a place holder of some type. > > > > OK, I at least understand what you're trying to do now. However, I > > don't think you have a valid use case. What would a templating > > language based on your notation look like? > > Any program that can patch text files together. As to how it does it, > it probably would use a script of some sort. This would just make it > easier to by offering a way around indenting issues. > > Another use case is it makes transmitting python compatible code easier > in a web page or email. The braces version would do that also. I think > it could make python a bit more attractive for things like that. Where > a app is transmitted and run at the destination. There are still the > security issues to work out, but this would knock down one domino in > that direction. > > For example they don't have to event ways to avoid the white space > munging. Python could be represented as code blocks that might look > like... > > MAX: > \\\ break"> > > Except much longer. In this case there isn't a need for it to be > pretty. But there is a need for it to work correctly. > > > I was thinking it might be possible to have the python shell echo the > indent markes in the left side as it scroll lines up. Replacing the > '>>>' and '...' with the respective markes as you enter the lines. And > it could also take these if they are pasted in. Cutting and pasting > from one shell to another could work in that case. > > As far as unglyness goes... is it really that bad? you can leave the > white space in. The indent hints are used not the spaces in this case. > > ;;; for x in items: > \\\ sum = op(x) > ;;; if sum > MAX: > \\\ break > > And keep in mind that this isn't something you will see in normal use. > It's just available for when its needed. But if you run across code > like this it will still work. Nothing special would need to be done to > make it work. > > > How would you extend it to > > implement multi-line lambdas? > > As I said, it doesn't change the language at all. So extending lambda > is not in the scope of this topic. That would be something new and > should be done separately in any case. > > > If you can't cleanly do those two > > things, who cares if it lets you copy and paste snippets at arbitrary > > points in a Python file? What's the *functional* pay-off for the > > ugliness? > > ? See above. > > > (And I freely admit that my suggestion is also quite ugly. However, > > the status quo, with everyone inventing assorted templating languages > > for use with Python that are largely independent of Python itself > > already sucks) > > If it was a big patch to many files, I would say it isn't worth it. > But it would be a very small patch to only one file and wouldn't change > the grammer, ast, or anything else. It's almost free. > > I think it may have a nice benefit for very little effort. But I think > we should just wait a bit and see if there is more interest first. > ;-) > > Cheers, > Ron > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy at jeremysanders.net Thu Dec 15 11:11:01 2011 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Thu, 15 Dec 2011 10:11:01 +0000 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: Ron Adam wrote: > > ;;; def acc(n): # indent +0 > /// s = n # indent +1 > ;;; def add(i): # indent +0 > /// nonlocal s # indent +1 > ;;; s += i # indent +0 > ;;; return s # indent +0 > \\\ return add # indent -1 > To allow easy use of python in environments which aren't space-sensitive, wouldn't another solution be to have some sort of non-space character which could act as leading spaces, e.g. def foo(): ;;a = 0 ;;for i in range(10): ;;;;a += i ;;return a Of course this wouldn't work for enviroments which lose or cannot contain line breaks, however. Jeremy From ubershmekel at gmail.com Thu Dec 15 11:18:17 2011 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Thu, 15 Dec 2011 12:18:17 +0200 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: On Thu, Dec 15, 2011 at 12:11 PM, Jeremy Sanders wrote: > Ron Adam wrote:To allow easy use of python in environments which aren't > space-sensitive, > wouldn't another solution be to have some sort of non-space character which > could act as leading spaces, e.g. > > Another solution - the templating language for bottle.py requires every block opened to be closed with an "end". Eg: if name: print name else: input("your name please:") __end__ Or we could use a special character for that... http://bottlepy.org/docs/dev/stpl.html#embedded-python-code --Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Thu Dec 15 11:20:38 2011 From: phd at phdru.name (Oleg Broytman) Date: Thu, 15 Dec 2011 14:20:38 +0400 Subject: [Python-ideas] Transportable indent level markers In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: <20111215102038.GA31413@iskra.aviel.ru> On Thu, Dec 15, 2011 at 10:11:01AM +0000, Jeremy Sanders wrote: > To allow easy use of python in environments which aren't space-sensitive, > wouldn't another solution be to have some sort of non-space character which > could act as leading spaces, e.g. > > def foo(): > ;;a = 0 > ;;for i in range(10): > ;;;;a += i > ;;return a > > Of course this wouldn't work for enviroments which lose or cannot contain > line breaks, however. def foo(): a = 0 for i in range(10): a += i return a ;-) Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From jeremy at jeremysanders.net Thu Dec 15 11:31:36 2011 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Thu, 15 Dec 2011 10:31:36 +0000 Subject: [Python-ideas] Transportable indent level markers References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <20111215102038.GA31413@iskra.aviel.ru> Message-ID: Oleg Broytman wrote: > > def foo(): > a = 0 > for i in range(10): > a += i > return a > > > ;-) Nice. I'd like the option of JSON too, please, though the brackets might be a bit confusing :-) Jeremy From ncoghlan at gmail.com Thu Dec 15 12:41:04 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Dec 2011 21:41:04 +1000 Subject: [Python-ideas] Inconsistent script/console behaviour In-Reply-To: References: Message-ID: (moved thread to python-ideas, where it belongs. Messing with 20 years of precedent is *not* a topic for python-dev) On Thu, Dec 15, 2011 at 6:58 PM, anatoly techtonik wrote: >>?You can't fix this without completely changing the way the interactive >> console treats blank lines. > > But the fix doesn't require changing the way interactive console treats > blank lines at all. It only requires to finish current block when a dedented > line is encountered and not throwing obviously confusing SyntaxError. At the > very least it should not say it is SyntaxError, because the code is pretty > valid Python code. If it appears to be invalid "Python Console code" - the > error message should say that explicitly. That would be a correct > user-friendly fix for this UX issue, but I'd still like the behavior to be > fixed - i.e. "allow dedented lines end current block in console without > SyntaxError". Right now I don't see the reasons why it is not possible. > > Please speak code when replying about use cases/examples that will be broken > - I didn't quite get the problem with "global scope if" statements. There are two steps to what the interactive console does: Step 1: accept a block of code. Step 2: compile and execute it in single statement mode (in order to get the result of any expressions for display) It is the *first* step that is terminated by a blank line. That step understands almost nothing about Python syntax - just enough to figure out whether or not it should be asking for a continuation line. (IIRC, it pretty much just does bracket matching and "if I have seen a ':' character, ask for continuation lines until one of them is blank". So yes, fixing example 1 *does* require significant changes to the way the interactive interpreter works. The interactive interpreter currently operates on individual statements. For compound statements, those statements are terminated by a blank line. If you don't terminate them correctly, then it will attempt to interpret two statements as one, and hence you will get UnexpectedDedentError. To fix example 1, the interactive interpreter would have to be redesigned to use exec mode instead, so it can accept multiple statements at once. That's not impossible, but is a lot of effort to address a relatively minor inconvenience (the inability to tolerate blank lines *within* a suite significantly more annoying in my experience, but even harder to fix than example 1 would be). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jimjjewett at gmail.com Thu Dec 15 16:33:08 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 15 Dec 2011 10:33:08 -0500 Subject: [Python-ideas] Transportable indent level markers. >>>===<<< In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <1323911299.9574.44.camel@Gutsy> <1323919148.10290.59.camel@Gutsy> <1323925757.11678.59.camel@Gutsy> Message-ID: On Thu, Dec 15, 2011 at 12:25 AM, Nick Coghlan wrote: > (And I freely admit that my suggestion is also quite ugly. However, > the status quo, with everyone inventing assorted templating languages > for use with Python that are largely independent of Python itself > already sucks) Guido actually endorsed ugly if it is just used for transport. That said, for embedding in a template, I would prefer to avoid ugly. But why does it have to be all punctuation? :indent 4 if is_frobbable(user, request): :indent 5 frob(request) :indent 4 else: :indent 5 spit_error("Can't frob here, mate!") The grammar would be that at the start of a (line, string, whatever record marker the embedding environment has), the literal ":indent" followed by whitespace followed by a decimal number followed by whitespace followed by a line of python code would be equivalent to that same line of python code indented times. Ideally, the actual leading whitespace would also demonstrate proper indenting, but it wouldn't *need* to, so the code would be robust against transmission. -jJ From jimjjewett at gmail.com Thu Dec 15 16:39:56 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 15 Dec 2011 10:39:56 -0500 Subject: [Python-ideas] Transportable indent level markers In-Reply-To: References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> <20111215102038.GA31413@iskra.aviel.ru> Message-ID: On Thu, Dec 15, 2011 at 5:31 AM, Jeremy Sanders wrote: > Oleg Broytman wrote: > >> >> def foo(): >> a = 0 >> for i in range(10): >> a += i >> return a >> This would be awkward when the code is supposed to produce an attribute value, so that entire elements are inside an element-open tag. > Nice. I'd like the option of JSON too, please, though the brackets > might be a bit confusing :-) If you're willing to wrap each line in a string, then leading whitespace will already be preserved. If you're not willing to wrap each line in a string, then it isn't clear how JSON would represent the code part, regardless of how indention is conveyed. -jJ From python at 2sn.net Thu Dec 15 17:26:50 2011 From: python at 2sn.net (Alexander Heger) Date: Thu, 15 Dec 2011 10:26:50 -0600 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA110C.4050507@2sn.net> References: <4EEA110C.4050507@2sn.net> Message-ID: <4EEA1FCA.4080000@2sn.net> Hi, in IDL/GDL, if at have an array x = [1,2,3] ?I can simply construct a new array by y = [0,x] which would give [0,1,2,3] I know you can do this case with x[0:0] = [0] but obviously more complicated cases are conceivable, common for me, in fact x=[1,2,3] y=[5,6,7] z=[0,x,4,y,8] result [0,1,2,3,4,5,6,7,8] or maybe even z=[0,x[0:2],5] and so forth. On the other hand, if you have a function you can pass the list of arguments to another function def f(*args): ? g(*args) which expands the list args. ?So, would it be possible - not even reasonable to have something similar for list and arrays, maybe dictionaries, in comprehensions as well? x = [1,2,3] y = [0,*x] to obtain [0,1,2,3] or similar for lists x = (1,2,3) y = (0,*x) to obtain (0,1,2,3) (for dictionaries, which are not sorted, the extend function seems fine) Or is there a way of doing this that in a similarly compact and obvious way I did not yet discover? -Alexander From masklinn at masklinn.net Thu Dec 15 18:27:40 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 15 Dec 2011 18:27:40 +0100 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA1FCA.4080000@2sn.net> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> Message-ID: On 2011-12-15, at 17:26 , Alexander Heger wrote: > > Or is there a way of doing this that in a similarly compact and > obvious way I did not yet discover? If the list is uniform, you can flatten a single level by using `itertools.chain`: >>> import itertools >>> x = [1,2,3] >>> y = itertools.chain.from_iterable([[0], x]) >>> list(y) [0, 1, 2, 3] >>> # alternatively ... y = list(itertools.chain([0], x)) >>> y [0, 1, 2, 3] >>> I know of no "better" way to do it at the moment, apart from using slice-assignment with a *stop* bound of 0: >>> y = [0, 0] >>> y[1:0] = x >>> y [0, 1, 2, 3, 0] From ubershmekel at gmail.com Thu Dec 15 18:30:57 2011 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Thu, 15 Dec 2011 19:30:57 +0200 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA1FCA.4080000@2sn.net> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> Message-ID: On Thu, Dec 15, 2011 at 6:26 PM, Alexander Heger wrote: > Hi, > > in IDL/GDL, if at have an array > x = [1,2,3] > ?I can simply construct a new array by > y = [0,x] > which would give > [0,1,2,3] > > I know you can do this case with > x[0:0] = [0] > but obviously more complicated cases are conceivable, common for me, in > fact > x=[1,2,3] > y=[5,6,7] > z=[0,x,4,y,8] > result > [0,1,2,3,4,5,6,7,8] > > or maybe even > z=[0,x[0:2],5] > and so forth. > > On the other hand, if you have a function you can pass the list of > arguments to another function > def f(*args): > ? g(*args) > which expands the list args. ?So, would it be possible - not even > reasonable to have something similar for list and arrays, maybe > dictionaries, in comprehensions as well? > > x = [1,2,3] > y = [0,*x] > to obtain [0,1,2,3] > > or similar for lists > x = (1,2,3) > y = (0,*x) > to obtain (0,1,2,3) > > (for dictionaries, which are not sorted, the extend function seems fine) > > Or is there a way of doing this that in a similarly compact and > obvious way I did not yet discover? > > -Alexander > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > This is the wrong list for such a question. Python-ideas is a place to discuss ideas for the python language itself. You can use stackoverflow.comin the future. In any case, is this good enough? >>> x = [1,2,3] >>> y = [0] + x >>> y [0, 1, 2, 3] Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Thu Dec 15 18:33:11 2011 From: phd at phdru.name (Oleg Broytman) Date: Thu, 15 Dec 2011 21:33:11 +0400 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA1FCA.4080000@2sn.net> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> Message-ID: <20111215173311.GA15847@iskra.aviel.ru> On Thu, Dec 15, 2011 at 10:26:50AM -0600, Alexander Heger wrote: > x = [1,2,3] > y = [0,*x] > to obtain [0,1,2,3] y = [0] + x > or similar for lists > x = (1,2,3) > y = (0,*x) > to obtain (0,1,2,3) y = (0,) + x Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From python at 2sn.net Thu Dec 15 18:35:12 2011 From: python at 2sn.net (Alexander Heger) Date: Thu, 15 Dec 2011 11:35:12 -0600 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> Message-ID: <4EEA2FD0.6020504@2sn.net> Dear Masklinn, thanks for your suggested solution. I know all of these, but 1) it is not as elegant or short 2) why does unpacking not work syntactically the same as for the function parameters? It seems a natural extension that appears not to have a syntactic conflict. If it is not even a necessity for consistency. So, the point is not that something like [0,*x,0,*y,0] can't be done in other ways, but that it can't be done in a neat way. -Alexander On 12/15/2011 11:27 AM, Masklinn wrote: > On 2011-12-15, at 17:26 , Alexander Heger wrote: >> >> Or is there a way of doing this that in a similarly compact and >> obvious way I did not yet discover? > > If the list is uniform, you can flatten a single level by using `itertools.chain`: > > >>> import itertools > >>> x = [1,2,3] > >>> y = itertools.chain.from_iterable([[0], x]) > >>> list(y) > [0, 1, 2, 3] > >>> # alternatively > ... y = list(itertools.chain([0], x)) > >>> y > [0, 1, 2, 3] > >>> > > I know of no "better" way to do it at the moment, apart from using slice-assignment with a *stop* bound of 0: > > >>> y = [0, 0] > >>> y[1:0] = x > >>> y > [0, 1, 2, 3, 0] > > From python at 2sn.net Thu Dec 15 18:41:09 2011 From: python at 2sn.net (Alexander Heger) Date: Thu, 15 Dec 2011 11:41:09 -0600 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <20111215173311.GA15847@iskra.aviel.ru> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <20111215173311.GA15847@iskra.aviel.ru> Message-ID: <4EEA3135.20107@2sn.net> Dear Oleg, OK, these work for the examples I listed, I had not tried that. Thanks! I still think the element extraction would be nice, though. what does not work your way is x = [1,2,3] y = (0,) + x but y = (0,*x) could do ... ? -Alexander On 12/15/2011 11:33 AM, Oleg Broytman wrote: > On Thu, Dec 15, 2011 at 10:26:50AM -0600, Alexander Heger wrote: >> x = [1,2,3] >> y = [0,*x] >> to obtain [0,1,2,3] > > y = [0] + x > >> or similar for lists >> x = (1,2,3) >> y = (0,*x) >> to obtain (0,1,2,3) > > y = (0,) + x > > Oleg. From andre.roberge at gmail.com Thu Dec 15 18:47:32 2011 From: andre.roberge at gmail.com (Andre Roberge) Date: Thu, 15 Dec 2011 13:47:32 -0400 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA3135.20107@2sn.net> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <20111215173311.GA15847@iskra.aviel.ru> <4EEA3135.20107@2sn.net> Message-ID: On Thu, Dec 15, 2011 at 1:41 PM, Alexander Heger wrote: > Dear Oleg, > > OK, these work for the examples I listed, I had not tried that. Thanks! > > I still think the element extraction would be nice, though. > what does not work your way is > x = [1,2,3] > y = (0,) + x > > The following works --> x = [1,2,3] --> y = (0,) + tuple(x) --> y (0, 1, 2, 3) Explicit (i.e. tuple conversion) is better than implicit (The Zen of Python) Andr? > but > y = (0,*x) > could do ... ? > > -Alexander > > On 12/15/2011 11:33 AM, Oleg Broytman wrote: > >> On Thu, Dec 15, 2011 at 10:26:50AM -0600, Alexander Heger wrote: >> >>> x = [1,2,3] >>> y = [0,*x] >>> to obtain [0,1,2,3] >>> >> >> y = [0] + x >> >> or similar for lists >>> x = (1,2,3) >>> y = (0,*x) >>> to obtain (0,1,2,3) >>> >> >> y = (0,) + x >> >> Oleg. >> > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at gmail.com Thu Dec 15 18:48:28 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 15 Dec 2011 17:48:28 +0000 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA2FD0.6020504@2sn.net> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <4EEA2FD0.6020504@2sn.net> Message-ID: On 15 December 2011 17:35, Alexander Heger wrote: > Dear Masklinn, > > thanks for your suggested solution. > > I know all of these, but > 1) it is not as elegant or short > 2) why does unpacking not work syntactically the same as for the function > parameters? > It seems a natural extension that appears not to have a syntactic > conflict. If it is not even a necessity for consistency. > > So, the point is not that something like > [0,*x,0,*y,0] > can't be done in other ways, but that it can't be done in a neat way. > I quite like that (suggested) syntax. Michael > > -Alexander > > > On 12/15/2011 11:27 AM, Masklinn wrote: > >> On 2011-12-15, at 17:26 , Alexander Heger wrote: >> >>> >>> Or is there a way of doing this that in a similarly compact and >>> obvious way I did not yet discover? >>> >> >> If the list is uniform, you can flatten a single level by using >> `itertools.chain`: >> >> >>> import itertools >> >>> x = [1,2,3] >> >>> y = itertools.chain.from_iterable(**[[0], x]) >> >>> list(y) >> [0, 1, 2, 3] >> >>> # alternatively >> ... y = list(itertools.chain([0], x)) >> >>> y >> [0, 1, 2, 3] >> >>> >> >> I know of no "better" way to do it at the moment, apart from using >> slice-assignment with a *stop* bound of 0: >> >> >>> y = [0, 0] >> >>> y[1:0] = x >> >>> y >> [0, 1, 2, 3, 0] >> >> >> ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Dec 15 18:53:02 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Dec 2011 04:53:02 +1100 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <4EEA2FD0.6020504@2sn.net> Message-ID: <4EEA33FE.8060200@pearwood.info> Michael Foord wrote: > On 15 December 2011 17:35, Alexander Heger wrote: > >> Dear Masklinn, >> >> thanks for your suggested solution. >> >> I know all of these, but >> 1) it is not as elegant or short >> 2) why does unpacking not work syntactically the same as for the function >> parameters? >> It seems a natural extension that appears not to have a syntactic >> conflict. If it is not even a necessity for consistency. >> >> So, the point is not that something like >> [0,*x,0,*y,0] >> can't be done in other ways, but that it can't be done in a neat way. >> > > > I quite like that (suggested) syntax. So do I. But I question the need for it. +0. -- Steven From phd at phdru.name Thu Dec 15 18:53:41 2011 From: phd at phdru.name (Oleg Broytman) Date: Thu, 15 Dec 2011 21:53:41 +0400 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> Message-ID: <20111215175341.GA16314@iskra.aviel.ru> On Thu, Dec 15, 2011 at 07:30:57PM +0200, Yuval Greenfield wrote: > This is the wrong list for such a question. Python-ideas is a place to > discuss ideas for the python language itself. IMO it's ok because the OP proposed a new feature: > On Thu, Dec 15, 2011 at 6:26 PM, Alexander Heger wrote: > > y = (0,*x) and we're going to discuss why it will never happens. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From jeanpierreda at gmail.com Thu Dec 15 18:53:33 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 15 Dec 2011 12:53:33 -0500 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA2FD0.6020504@2sn.net> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <4EEA2FD0.6020504@2sn.net> Message-ID: This would behave differently than function calls! I can't call foo(a, b, *x, c) I'm actually +1 on unpacking in literals, though. -- Devin On Thu, Dec 15, 2011 at 12:35 PM, Alexander Heger wrote: > Dear Masklinn, > > thanks for your suggested solution. > > I know all of these, but > 1) it is not as elegant or short > 2) why does unpacking not work syntactically the same as for the function > parameters? > It seems a natural extension that appears not to have a syntactic conflict. > ?If it is not even a necessity for consistency. > > So, the point is not that something like > [0,*x,0,*y,0] > can't be done in other ways, but that it can't be done in a neat way. > > -Alexander > > > On 12/15/2011 11:27 AM, Masklinn wrote: >> >> On 2011-12-15, at 17:26 , Alexander Heger wrote: >>> >>> >>> Or is there a way of doing this that in a similarly compact and >>> obvious way I did not yet discover? >> >> >> If the list is uniform, you can flatten a single level by using >> `itertools.chain`: >> >> ? ? >>> ?import itertools >> ? ? >>> ?x = [1,2,3] >> ? ? >>> ?y = itertools.chain.from_iterable([[0], x]) >> ? ? >>> ?list(y) >> ? ? [0, 1, 2, 3] >> ? ? >>> ?# alternatively >> ? ? ... y = list(itertools.chain([0], x)) >> ? ? >>> ?y >> ? ? [0, 1, 2, 3] >> ? ? >>> >> >> I know of no "better" way to do it at the moment, apart from using >> slice-assignment with a *stop* bound of 0: >> >> ? ? >>> ?y = [0, 0] >> ? ? >>> ?y[1:0] = x >> ? ? >>> ?y >> ? ? [0, 1, 2, 3, 0] >> >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From fuzzyman at gmail.com Thu Dec 15 19:02:24 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 15 Dec 2011 18:02:24 +0000 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA33FE.8060200@pearwood.info> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <4EEA2FD0.6020504@2sn.net> <4EEA33FE.8060200@pearwood.info> Message-ID: On 15 December 2011 17:53, Steven D'Aprano wrote: > Michael Foord wrote: > >> On 15 December 2011 17:35, Alexander Heger wrote: >> >> Dear Masklinn, >>> >>> thanks for your suggested solution. >>> >>> I know all of these, but >>> 1) it is not as elegant or short >>> 2) why does unpacking not work syntactically the same as for the function >>> parameters? >>> It seems a natural extension that appears not to have a syntactic >>> conflict. If it is not even a necessity for consistency. >>> >>> So, the point is not that something like >>> [0,*x,0,*y,0] >>> can't be done in other ways, but that it can't be done in a neat way. >>> >>> >> >> I quite like that (suggested) syntax. >> > > So do I. But I question the need for it. > > +0. > > Well, the new tuple unpacking syntax makes it easy to work with the head and tail of a sequence: head, *rest = some_sequence *rest, tail = some_sequence This syntax provides a corollary: some_sequence = (head, *rest) some_sequence = (*rest, tail) Tuple unpacking like this being the equivalent of car and cdr, with the corollary being the equivalent of cons. (Given my limited understanding of these operations.) Michael > > -- > Steven > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn at masklinn.net Thu Dec 15 19:25:41 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 15 Dec 2011 19:25:41 +0100 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA2FD0.6020504@2sn.net> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <4EEA2FD0.6020504@2sn.net> Message-ID: <82DCC16A-AF24-488D-AECB-9B4AD23FFF9E@masklinn.net> On 2011-12-15, at 18:35 , Alexander Heger wrote: > > I know all of these, but > 1) it is not as elegant or short Sure, I was just answering the rest of of the question > > 2) why does unpacking not work syntactically the same as for the function parameters? Because nobody's thought of it so far. > So, the point is not that something like > [0,*x,0,*y,0] > can't be done in other ways, but that it can't be done in a neat way. Well, you *did* ask: > Or is there a way of doing this that in a similarly compact and > obvious way I did not yet discover? From masklinn at masklinn.net Thu Dec 15 19:28:51 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 15 Dec 2011 19:28:51 +0100 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <4EEA2FD0.6020504@2sn.net> <4EEA33FE.8060200@pearwood.info> Message-ID: On 2011-12-15, at 19:02 , Michael Foord wrote: > Well, the new tuple unpacking syntax makes it easy to work with the head > and tail of a sequence: > > head, *rest = some_sequence > *rest, tail = some_sequence > > This syntax provides a corollary: > > some_sequence = (head, *rest) > some_sequence = (*rest, tail) > > Tuple unpacking like this being the equivalent of car and cdr, with the > corollary being the equivalent of cons. (Given my limited understanding of > these operations.) Even more so with Py3's unpacking supporting more than just head and tail: >>> a, b, *c, d, e = range(15) >>> a 0 >>> b 1 >>> d 13 >>> e 14 >>> c [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] Alexander's syntax would be nicely parallel to this. On the other hand, much as Steven I'm not sure how useful it is, especially for tuples. From ethan at stoneleaf.us Thu Dec 15 18:44:01 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 15 Dec 2011 09:44:01 -0800 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> Message-ID: <4EEA31E1.3040104@stoneleaf.us> Yuval Greenfield wrote: > This is the wrong list for such a question. Python-ideas is a place to > discuss ideas for the python language itself. python-ideas is exactly the place to ask about syntax changes to Python. ~Ethan~ From guido at python.org Thu Dec 15 20:07:07 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Dec 2011 11:07:07 -0800 Subject: [Python-ideas] Inconsistent script/console behaviour In-Reply-To: References: Message-ID: On Thu, Dec 15, 2011 at 3:41 AM, Nick Coghlan wrote: > There are two steps to what the interactive console does: > > Step 1: accept a block of code. > Step 2: compile and execute it in single statement mode (in order to > get the result of any expressions for display) > > It is the *first* step that is terminated by a blank line. That step > understands almost nothing about Python syntax - just enough to figure > out whether or not it should be asking for a continuation line. (IIRC, > it pretty much just does bracket matching and "if I have seen a ':' > character, ask for continuation lines until one of them is blank". > Actually, in CPython at least, the first step uses a special feature of the parser that can tell whether what has been entered so far is complete or not. Try entering a subtle syntax error: >>> def foo(): ... x = 1 ... y x File "", line 3 y x ^ SyntaxError: invalid syntax >>> It does treat an entirely blank line (with no comment or whitespace) as a dedent all the way back to zero. The problem with allowing what Anatoly asks for is that the added line could introduce another indented block instead of just a statement. Almost all languages with an interactive mode have some kind of syntactic awkwardness or differences. E.g. I recently had problems with Haskell's interactive mode, which won't let you define functions directly -- you must use 'let'; and sqlite3 reads anything until you enter a semicolon. Trying to mess with Python's approach is likely to make things worse in some other way. If you think otherwise, show us the patch. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Thu Dec 15 21:24:27 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Dec 2011 21:24:27 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 12/13/2011 03:44 AM, Stephen J. Turnbull wrote: > Greg Ewing writes: > > Masklinn wrote: > > > > > FWIW, Haskell does not have a literal singleton (the standard defines > > > "unit" `()` and 2-tuple through 15-tuple) > > > > That's because, due to its static typing, there is no > > reason you would ever need to use a 1-tuple rather than > > a bare value. We're not that lucky in Python, though. > > I think you have misstated your point? That's not due to static > typing, that's because you may *always* identify 1-factor products > with the only factor, and Haskell made a deliberate decision to > consistently represent the isomorphism class by the factor rather than > the product. Well, I would say the reason is that the type "tuple of any length" does not exist in Haskell. So there's no way you will have to pass a 1-tuple to a function that operates on tuples only. But of course, if we all used tuples as tuples only, we wouldn't have to do that either. It's only because we use tuples as sequences every so often. Georg From ned at nedbatchelder.com Thu Dec 15 21:42:42 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Dec 2011 15:42:42 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4EEA5BC2.4070402@nedbatchelder.com> On 12/15/2011 3:24 PM, Georg Brandl wrote: > On 12/13/2011 03:44 AM, Stephen J. Turnbull wrote: >> Greg Ewing writes: >> > Masklinn wrote: >> > >> > > FWIW, Haskell does not have a literal singleton (the standard defines >> > > "unit" `()` and 2-tuple through 15-tuple) >> > >> > That's because, due to its static typing, there is no >> > reason you would ever need to use a 1-tuple rather than >> > a bare value. We're not that lucky in Python, though. >> >> I think you have misstated your point? That's not due to static >> typing, that's because you may *always* identify 1-factor products >> with the only factor, and Haskell made a deliberate decision to >> consistently represent the isomorphism class by the factor rather than >> the product. > Well, I would say the reason is that the type "tuple of any length" does > not exist in Haskell. So there's no way you will have to pass a 1-tuple > to a function that operates on tuples only. > > But of course, if we all used tuples as tuples only, we wouldn't have to do > that either. It's only because we use tuples as sequences every so often. This is another place where Python is inconsistent. We're told, "lists are for homogenous sequences of varying length, like a C array; tuples are for heterogenous aggregations of known length, like a C struct." Then we define a function foo(*args), and Python gives us a tuple! :-( --Ned. > Georg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From g.brandl at gmx.net Thu Dec 15 21:51:01 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Dec 2011 21:51:01 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA5BC2.4070402@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: On 12/15/2011 09:42 PM, Ned Batchelder wrote: > On 12/15/2011 3:24 PM, Georg Brandl wrote: >> On 12/13/2011 03:44 AM, Stephen J. Turnbull wrote: >>> Greg Ewing writes: >>> > Masklinn wrote: >>> > >>> > > FWIW, Haskell does not have a literal singleton (the standard defines >>> > > "unit" `()` and 2-tuple through 15-tuple) >>> > >>> > That's because, due to its static typing, there is no >>> > reason you would ever need to use a 1-tuple rather than >>> > a bare value. We're not that lucky in Python, though. >>> >>> I think you have misstated your point? That's not due to static >>> typing, that's because you may *always* identify 1-factor products >>> with the only factor, and Haskell made a deliberate decision to >>> consistently represent the isomorphism class by the factor rather than >>> the product. >> Well, I would say the reason is that the type "tuple of any length" does >> not exist in Haskell. So there's no way you will have to pass a 1-tuple >> to a function that operates on tuples only. >> >> But of course, if we all used tuples as tuples only, we wouldn't have to do >> that either. It's only because we use tuples as sequences every so often. > This is another place where Python is inconsistent. We're told, "lists > are for homogenous sequences of varying length, like a C array; tuples > are for heterogenous aggregations of known length, like a C struct." > Then we define a function foo(*args), and Python gives us a tuple! :-( Yep. To be consistent, we'd need an "immutable list" type... another thing that Haskell has no need for :) Georg From solipsis at pitrou.net Thu Dec 15 21:51:47 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Dec 2011 21:51:47 +0100 Subject: [Python-ideas] Python Users Aren't Perfect References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: <20111215215147.22f41c80@pitrou.net> On Thu, 15 Dec 2011 15:42:42 -0500 Ned Batchelder wrote: > On 12/15/2011 3:24 PM, Georg Brandl wrote: > > On 12/13/2011 03:44 AM, Stephen J. Turnbull wrote: > >> Greg Ewing writes: > >> > Masklinn wrote: > >> > > >> > > FWIW, Haskell does not have a literal singleton (the standard defines > >> > > "unit" `()` and 2-tuple through 15-tuple) > >> > > >> > That's because, due to its static typing, there is no > >> > reason you would ever need to use a 1-tuple rather than > >> > a bare value. We're not that lucky in Python, though. > >> > >> I think you have misstated your point? That's not due to static > >> typing, that's because you may *always* identify 1-factor products > >> with the only factor, and Haskell made a deliberate decision to > >> consistently represent the isomorphism class by the factor rather than > >> the product. > > Well, I would say the reason is that the type "tuple of any length" does > > not exist in Haskell. So there's no way you will have to pass a 1-tuple > > to a function that operates on tuples only. > > > > But of course, if we all used tuples as tuples only, we wouldn't have to do > > that either. It's only because we use tuples as sequences every so often. > This is another place where Python is inconsistent. We're told, "lists > are for homogenous sequences of varying length, like a C array; tuples > are for heterogenous aggregations of known length, like a C struct." > Then we define a function foo(*args), and Python gives us a tuple! :-( How is it inconsistent? Function signatures generally have a fixed (or mostly fixed) number of heterogenous arguments. Regards Antoine. From timothy.c.delaney at gmail.com Thu Dec 15 21:59:26 2011 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Fri, 16 Dec 2011 07:59:26 +1100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA5BC2.4070402@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: On 16 December 2011 07:42, Ned Batchelder wrote: > This is another place where Python is inconsistent. We're told, "lists > are for homogenous sequences of varying length, like a C array; tuples are > for heterogenous aggregations of known length, like a C struct." Then we > define a function foo(*args), and Python gives us a tuple! :-( > How is that inconsistent? At the point where the tuple is constructed, it has a known length. And it's definitely a heterogenous aggregation. I think where you're getting confused is that you're thinking of a *single* struct definition for every tuple. But the concept you should have is that each tuple has its own struct definition. And with functions, the structure is defined at function call time. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Thu Dec 15 22:00:03 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Dec 2011 16:00:03 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <20111215215147.22f41c80@pitrou.net> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <20111215215147.22f41c80@pitrou.net> Message-ID: <4EEA5FD3.2070603@nedbatchelder.com> On 12/15/2011 3:51 PM, Antoine Pitrou wrote: > On Thu, 15 Dec 2011 15:42:42 -0500 > Ned Batchelder > wrote: >> On 12/15/2011 3:24 PM, Georg Brandl wrote: >>> On 12/13/2011 03:44 AM, Stephen J. Turnbull wrote: >>>> Greg Ewing writes: >>>> > Masklinn wrote: >>>> > >>>> > > FWIW, Haskell does not have a literal singleton (the standard defines >>>> > > "unit" `()` and 2-tuple through 15-tuple) >>>> > >>>> > That's because, due to its static typing, there is no >>>> > reason you would ever need to use a 1-tuple rather than >>>> > a bare value. We're not that lucky in Python, though. >>>> >>>> I think you have misstated your point? That's not due to static >>>> typing, that's because you may *always* identify 1-factor products >>>> with the only factor, and Haskell made a deliberate decision to >>>> consistently represent the isomorphism class by the factor rather than >>>> the product. >>> Well, I would say the reason is that the type "tuple of any length" does >>> not exist in Haskell. So there's no way you will have to pass a 1-tuple >>> to a function that operates on tuples only. >>> >>> But of course, if we all used tuples as tuples only, we wouldn't have to do >>> that either. It's only because we use tuples as sequences every so often. >> This is another place where Python is inconsistent. We're told, "lists >> are for homogenous sequences of varying length, like a C array; tuples >> are for heterogenous aggregations of known length, like a C struct." >> Then we define a function foo(*args), and Python gives us a tuple! :-( > How is it inconsistent? Function signatures generally have a fixed (or > mostly fixed) number of heterogenous arguments. What? The whole point of the "def foo(*args)" syntax is so a function can take an unknown-length list of arguments, which will be treated uniformly. --Ned. > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From masklinn at masklinn.net Thu Dec 15 22:05:26 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 15 Dec 2011 22:05:26 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <20111215215147.22f41c80@pitrou.net> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <20111215215147.22f41c80@pitrou.net> Message-ID: <354A8605-6CDF-40C5-AEA9-BFF7FF0DBB05@masklinn.net> On 2011-12-15, at 21:51 , Antoine Pitrou wrote: > On Thu, 15 Dec 2011 15:42:42 -0500 > Ned Batchelder > wrote: >> This is another place where Python is inconsistent. We're told, "lists >> are for homogenous sequences of varying length, like a C array; tuples >> are for heterogenous aggregations of known length, like a C struct." >> Then we define a function foo(*args), and Python gives us a tuple! :-( > > How is it inconsistent? Function signatures generally have a fixed (or > mostly fixed) number of heterogenous arguments. But *args is mostly dedicated to an arbitrary (non-fixed) number of homogenous arguments. In statically typed languages with varargs, they're generally represented as an array-type collection. It's either that or a "rest of that crap" catchall used to forward arguments without caring much for what they are. From masklinn at masklinn.net Thu Dec 15 22:06:43 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 15 Dec 2011 22:06:43 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA5FD3.2070603@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <20111215215147.22f41c80@pitrou.net> <4EEA5FD3.2070603@nedbatchelder.com> Message-ID: On 2011-12-15, at 22:00 , Ned Batchelder wrote: > > What? The whole point of the "def foo(*args)" syntax is so a function can take an unknown-length list of arguments, which will be treated uniformly. That's not *entirely* true, it's also used for proxy functions, in order to just pass a bunch of arguments (positional or keywords) along to the next level without having to unpack and repack them one by one. From ned at nedbatchelder.com Thu Dec 15 22:06:59 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Dec 2011 16:06:59 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA5FD3.1020907@stoneleaf.us> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEA5FD3.1020907@stoneleaf.us> Message-ID: <4EEA6173.9040402@nedbatchelder.com> On 12/15/2011 4:00 PM, Ethan Furman wrote: > Ned Batchelder wrote: >> This is another place where Python is inconsistent. We're told, >> "lists are for homogenous sequences of varying length, like a C >> array; tuples are for heterogenous aggregations of known length, like >> a C struct." Then we define a function foo(*args), and Python gives >> us a tuple! :-( > > Where is that in the docs? Sounds like a patch is needed: > > "lists are for sequences where items need to be added/removed; tuples > are for sequences/aggregations where items will not be added/removed > once the tuple is created" > I don't know if it appears in the docs, but I hear it all the time, and Guido has said it (http://mail.python.org/pipermail/python-dev/2003-March/033964.html): Tuples are for heterogeneous data, list are for homogeneous data. Tuples are *not* read-only lists. I don't want to get too far off the original point, which was: Python isn't as simple as we'd like to thing, and even smart beginners can be tripped up by things we've provided to them. --Ned. > ~Ethan~ > > PS > As Antoine noted, a tuple for 'args' is appropriate, as once args is > created at function call time, we won't be adding or removing from it. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Thu Dec 15 22:09:31 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Dec 2011 16:09:31 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: <4EEA620B.1000300@nedbatchelder.com> On 12/15/2011 3:59 PM, Tim Delaney wrote: > On 16 December 2011 07:42, Ned Batchelder > wrote: > > This is another place where Python is inconsistent. We're told, > "lists are for homogenous sequences of varying length, like a C > array; tuples are for heterogenous aggregations of known length, > like a C struct." Then we define a function foo(*args), and > Python gives us a tuple! :-( > > > How is that inconsistent? At the point where the tuple is constructed, > it has a known length. And it's definitely a heterogenous aggregation. > > I think where you're getting confused is that you're thinking of a > *single* struct definition for every tuple. But the concept you should > have is that each tuple has its own struct definition. And with > functions, the structure is defined at function call time. > Tim, this seems misguided to me. Finish that foo function definition: it will *have* to have "for a in args:" Since I don't know the length of args when I write the function, I have to treat it as an unknown length. What good is a "structure" that changes length and definition with every instance? I think you're trying too hard to fit the reality into the theory. --Ned. > Tim Delaney > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Thu Dec 15 22:11:10 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Dec 2011 16:11:10 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <20111215215147.22f41c80@pitrou.net> <4EEA5FD3.2070603@nedbatchelder.com> Message-ID: <4EEA626E.8040101@nedbatchelder.com> On 12/15/2011 4:06 PM, Masklinn wrote: > On 2011-12-15, at 22:00 , Ned Batchelder wrote: >> What? The whole point of the "def foo(*args)" syntax is so a function can take an unknown-length list of arguments, which will be treated uniformly. > That's not *entirely* true, it's also used for proxy functions, in order to just pass a bunch of arguments (positional or keywords) along to the next level without having to unpack and repack them one by one. We are in agreement: I don't know the length, and I will treat them uniformly (pass them all through to the next function). --Ned. From ethan at stoneleaf.us Thu Dec 15 22:00:03 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 15 Dec 2011 13:00:03 -0800 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA5BC2.4070402@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: <4EEA5FD3.1020907@stoneleaf.us> Ned Batchelder wrote: > This is another place where Python is inconsistent. We're told, "lists > are for homogenous sequences of varying length, like a C array; tuples > are for heterogenous aggregations of known length, like a C struct." > Then we define a function foo(*args), and Python gives us a tuple! :-( Where is that in the docs? Sounds like a patch is needed: "lists are for sequences where items need to be added/removed; tuples are for sequences/aggregations where items will not be added/removed once the tuple is created" ~Ethan~ PS As Antoine noted, a tuple for 'args' is appropriate, as once args is created at function call time, we won't be adding or removing from it. From ncoghlan at gmail.com Thu Dec 15 22:52:04 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2011 07:52:04 +1000 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA31E1.3040104@stoneleaf.us> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <4EEA31E1.3040104@stoneleaf.us> Message-ID: This won't happen because it is a lot of work for no real gain over the more explicit "tuple(x)" -- Nick Coghlan (via Gmail on Android, so likely to be more terse than usual) On Dec 16, 2011 4:45 AM, "Ethan Furman" wrote: > Yuval Greenfield wrote: > >> This is the wrong list for such a question. Python-ideas is a place to >> discuss ideas for the python language itself. >> > > python-ideas is exactly the place to ask about syntax changes to Python. > > ~Ethan~ > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Dec 15 22:24:26 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 15 Dec 2011 13:24:26 -0800 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA620B.1000300@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEA620B.1000300@nedbatchelder.com> Message-ID: <4EEA658A.3010509@stoneleaf.us> Ned Batchelder wrote: > On 12/15/2011 3:59 PM, Tim Delaney wrote: >> On 16 December 2011 07:42, Ned Batchelder wrote: >> >> This is another place where Python is inconsistent. We're told, >> "lists are for homogenous sequences of varying length, like a C >> array; tuples are for heterogenous aggregations of known length, >> like a C struct." Then we define a function foo(*args), and >> Python gives us a tuple! :-( >> >> >> How is that inconsistent? At the point where the tuple is constructed, >> it has a known length. And it's definitely a heterogenous aggregation. >> >> I think where you're getting confused is that you're thinking of a >> *single* struct definition for every tuple. But the concept you should >> have is that each tuple has its own struct definition. And with >> functions, the structure is defined at function call time. >> > Tim, this seems misguided to me. Finish that foo function definition: > it will *have* to have "for a in args:" Since I don't know the length > of args when I write the function, I have to treat it as an unknown > length. What good is a "structure" that changes length and definition > with every instance? I think you're trying too hard to fit the reality > into the theory. Python is a dynamic language -- why can't it have dynamic structs? ~Ethan~ From fuzzyman at gmail.com Thu Dec 15 22:53:56 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 15 Dec 2011 21:53:56 +0000 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: <4EEA3135.20107@2sn.net> References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <20111215173311.GA15847@iskra.aviel.ru> <4EEA3135.20107@2sn.net> Message-ID: On 15 December 2011 17:41, Alexander Heger wrote: > Dear Oleg, > > OK, these work for the examples I listed, I had not tried that. Thanks! > > I still think the element extraction would be nice, though. > what does not work your way is > x = [1,2,3] > y = (0,) + x > > but > y = (0,*x) > could do ... ? > In fact there's already an issue for this, with patch: http://bugs.python.org/issue2292 Actually it goes a lot further than just the syntax you suggest... It extends tuple unpacking into a few more places as well. All the best, Michael Foord > > -Alexander > > > On 12/15/2011 11:33 AM, Oleg Broytman wrote: > >> On Thu, Dec 15, 2011 at 10:26:50AM -0600, Alexander Heger wrote: >> >>> x = [1,2,3] >>> y = [0,*x] >>> to obtain [0,1,2,3] >>> >> >> y = [0] + x >> >> or similar for lists >>> x = (1,2,3) >>> y = (0,*x) >>> to obtain (0,1,2,3) >>> >> >> y = (0,) + x >> >> Oleg. >> > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Thu Dec 15 22:54:16 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Dec 2011 16:54:16 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA658A.3010509@stoneleaf.us> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEA620B.1000300@nedbatchelder.com> <4EEA658A.3010509@stoneleaf.us> Message-ID: <4EEA6C88.1030203@nedbatchelder.com> On 12/15/2011 4:24 PM, Ethan Furman wrote: > Ned Batchelder wrote: >> On 12/15/2011 3:59 PM, Tim Delaney wrote: >>> On 16 December 2011 07:42, Ned Batchelder wrote: >>> >>> This is another place where Python is inconsistent. We're told, >>> "lists are for homogenous sequences of varying length, like a C >>> array; tuples are for heterogenous aggregations of known length, >>> like a C struct." Then we define a function foo(*args), and >>> Python gives us a tuple! :-( >>> >>> >>> How is that inconsistent? At the point where the tuple is >>> constructed, it has a known length. And it's definitely a >>> heterogenous aggregation. >>> >>> I think where you're getting confused is that you're thinking of a >>> *single* struct definition for every tuple. But the concept you >>> should have is that each tuple has its own struct definition. And >>> with functions, the structure is defined at function call time. >>> >> Tim, this seems misguided to me. Finish that foo function >> definition: it will *have* to have "for a in args:" Since I don't >> know the length of args when I write the function, I have to treat it >> as an unknown length. What good is a "structure" that changes length >> and definition with every instance? I think you're trying too hard >> to fit the reality into the theory. > > Python is a dynamic language -- why can't it have dynamic structs? > I honestly don't know how to interpret this. In what way is it a "struct" if I have no idea how many fields it has, or how they might differ from each other? I think we've lost the utility of this discussion. --Ned. > ~Ethan~ > From masklinn at masklinn.net Thu Dec 15 22:55:54 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 15 Dec 2011 22:55:54 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA626E.8040101@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <20111215215147.22f41c80@pitrou.net> <4EEA5FD3.2070603@nedbatchelder.com> <4EEA626E.8040101@nedbatchelder.com> Message-ID: <72413090-E197-40A6-A49E-D9D56B38AFAA@masklinn.net> On 15 d?c. 2011, at 22:11, Ned Batchelder wrote: > On 12/15/2011 4:06 PM, Masklinn wrote: >> On 2011-12-15, at 22:00 , Ned Batchelder wrote: >>> What? The whole point of the "def foo(*args)" syntax is so a function can take an unknown-length list of arguments, which will be treated uniformly. >> That's not *entirely* true, it's also used for proxy functions, in order to just pass a bunch of arguments (positional or keywords) along to the next level without having to unpack and repack them one by one. > We are in agreement: I don't know the length, and I will treat them uniformly (pass them all through to the next function). Ah yes, you're right, I had not considered this use case under that angle. From grosser.meister.morti at gmx.net Thu Dec 15 23:14:45 2011 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Thu, 15 Dec 2011 23:14:45 +0100 Subject: [Python-ideas] Transportable indent level markers. /// ; ; ; \\\ In-Reply-To: <1323727633.29879.181.camel@Gutsy> References: <20111210161608.13313643@bhuda.mired.org> <4EE40DA7.4020702@pearwood.info> <4EE58F56.10507@canterbury.ac.nz> <1323727633.29879.181.camel@Gutsy> Message-ID: <4EEA7155.5090405@gmx.net> Need a new why of indentation? (V) (;,,;) (V) Why not Zoidberg?? Seriously though, I don't think something like this will be ever accepted. It totally goes against pythons idea behind indentation. On 12/12/2011 11:07 PM, Ron Adam wrote: > > While typing this reply to the "Alternate Suite Delineation Syntax For > Python?" thread. I realized that having things all on a single line > isn't important or needed!!! Yes, it's not needed for templates, or > code generation. So this is an alternate to the alternate. > > What is needed is a uniform way to represent indenting that is easy to > transport to (ie paste into) code with different indent levels. > > > INDENT LEVEL MARKERS: > ---------------------- > Offer a new way to spell,, and. > > To make that work, we need to use unique character sequences that aren't > already used. After some thought, I came up with these indent-markers. > > /// # indent level +1 > ;;; # indent level +0 > \\\ # indent level -1 > > These are unique enough to be easy to see, and the tripled versions of > them aren't really that bad as we already have triple quotes, and they > would be less than the white space they are replacing in most cases. > > So this isn't brackets, or braces. It would be just an alternate > spelling in already valid python. It should not effect the parser, > grammar, or interpreter in any way if it is done in pythons > tokenizer. :-) > > > Reusing some previous examples. > > #----------------------- > >> def acc(n) {: s=n; return {: def (i) {: nonlocal s; s += i; return >> s}}} > > #Standard python... > > def acc(n): > s=n > def add(i): > nonlocal s > s += i > return s > return add > > > #Single line version... > > def acc(n): /// s=n ;;; def add(i): /// nonlocal s \ > ;;; s+=i ;;; return s \\\ return add > > (The '\' character should still work.) > > > #Transportable indent level version... > > This could also be valid... and could be transportable to any other > indent level without having to edit it. > > (So would the above single line version by adding a ';;;' to the > beginning.) > > ;;; def acc(n): # indent +0 > /// s = n # indent +1 > ;;; def add(i): # indent +0 > /// nonlocal s # indent +1 > ;;; s += i # indent +0 > ;;; return s # indent +0 > \\\ return add # indent -1 > > It has the nice properties of being uniform, and being able to be > inserted easily into any existing code block without having to adjust > the indent levels. > > I believe that these properties are more important than being able to > put things on a single line and could be a very useful in code > generating programs. ;-) > > > It may be easy to implement. > > This could probably all be done in the tokinze.c get_tok() function, > where it detects and converts white space to indention levels. Nothing > else in python would need to be changed. The tokenizer already detects > line continuations and different line endings, so this wouldn't be that > out of place. > > Cheers, > Ron > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ncoghlan at gmail.com Fri Dec 16 00:06:01 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2011 09:06:01 +1000 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA6173.9040402@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEA5FD3.1020907@stoneleaf.us> <4EEA6173.9040402@nedbatchelder.com> Message-ID: On Fri, Dec 16, 2011 at 7:06 AM, Ned Batchelder wrote: > Where is that in the docs?? Sounds like a patch is needed: > > "lists are for sequences where items need to be added/removed; tuples are > for sequences/aggregations where items will not be added/removed once the > tuple is created" > > I don't know if it appears in the docs, but I hear it all the time, and > Guido has said it > (http://mail.python.org/pipermail/python-dev/2003-March/033964.html): > > Tuples are for heterogeneous data, list are for homogeneous data. > Tuples are *not* read-only lists. > > I don't want to get too far off the original point, which was: Python isn't > as simple as we'd like to thing, and even smart beginners can be tripped up > by things we've provided to them. Guido has softened his stance on that point over the years (IIRC, the question came up explicitly in the discussion over making tuple() fully conform to the Sequence ABC - you can guess the outcome from the fact that tuple these days in fact *does* fully conform to that ABC, including the previously missing index() and count() methods). So tuples have two valid use cases: as read-only arbitrary-length sequences of homogeneous data and as fixed-length sequences of heterogeneous data. These days, the latter use case is often better served by creating a collections.namedtuple() definition rather than using a bare tuple directly. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Fri Dec 16 00:13:01 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 Dec 2011 23:13:01 +0000 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <20111215173311.GA15847@iskra.aviel.ru> <4EEA3135.20107@2sn.net> Message-ID: On 15 December 2011 21:53, Michael Foord wrote: > In fact there's already an issue for this, with patch: > > http://bugs.python.org/issue2292 > > Actually it goes a lot further than just the syntax you suggest... It > extends tuple unpacking into a few more places as well. Nice. It seemed to have general support, plus BDFL approval. What's needed to get it moving for 3.3? The issue mentions a PEP, but I don't know if that was just when it was in the "Python 3000" timeframe. Paul. From fuzzyman at gmail.com Fri Dec 16 00:28:37 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 15 Dec 2011 23:28:37 +0000 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <20111215173311.GA15847@iskra.aviel.ru> <4EEA3135.20107@2sn.net> Message-ID: On 15 December 2011 23:13, Paul Moore wrote: > On 15 December 2011 21:53, Michael Foord wrote: > > In fact there's already an issue for this, with patch: > > > > http://bugs.python.org/issue2292 > > > > Actually it goes a lot further than just the syntax you suggest... It > > extends tuple unpacking into a few more places as well. > > Nice. It seemed to have general support, plus BDFL approval. What's > needed to get it moving for 3.3? The issue mentions a PEP, but I don't > know if that was just when it was in the "Python 3000" timeframe. > Guido was -0 in the email linked to from the issue. Not sure if that counts as approval... Michael > > Paul. > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Dec 16 00:33:24 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Dec 2011 15:33:24 -0800 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <20111215173311.GA15847@iskra.aviel.ru> <4EEA3135.20107@2sn.net> Message-ID: On Thu, Dec 15, 2011 at 3:28 PM, Michael Foord wrote: > > > On 15 December 2011 23:13, Paul Moore wrote: > >> On 15 December 2011 21:53, Michael Foord wrote: >> > In fact there's already an issue for this, with patch: >> > >> > http://bugs.python.org/issue2292 >> > >> > Actually it goes a lot further than just the syntax you suggest... It >> > extends tuple unpacking into a few more places as well. >> >> Nice. It seemed to have general support, plus BDFL approval. What's >> needed to get it moving for 3.3? The issue mentions a PEP, but I don't >> know if that was just when it was in the "Python 3000" timeframe. >> > > > Guido was -0 in the email linked to from the issue. Not sure if that > counts as approval... > > Michael > I think that -0 was contextual (too many moving parts for the original Py3k release). Today I am +1. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 16 00:48:43 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2011 09:48:43 +1000 Subject: [Python-ideas] list / array comprehensions extension In-Reply-To: References: <4EEA110C.4050507@2sn.net> <4EEA1FCA.4080000@2sn.net> <20111215173311.GA15847@iskra.aviel.ru> <4EEA3135.20107@2sn.net> Message-ID: On Fri, Dec 16, 2011 at 9:33 AM, Guido van Rossum wrote: > I think that -0 was contextual (too many moving parts for the original Py3k > release). Today I am +1. I stand corrected :) (And after skimming #2292, I'm also +1 on the course of action outlined there - the idea is sound in principle, but needs a PEP to thrash out all the niggling little details) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Fri Dec 16 02:16:40 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 15 Dec 2011 20:16:40 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA5BC2.4070402@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: On 12/15/2011 3:42 PM, Ned Batchelder wrote: > This is another place where Python is inconsistent. We're told, "lists > are for homogenous sequences of varying length, like a C array; tuples > are for heterogenous aggregations of known length, like a C struct." I have not been told that for several years, and I am pretty sure you will not find any such thing in the current docs. I consider it pretty much obsolete, as the differences that flowed from that idea are gone. In Python 3, tuples have all the non-mutating sequence methods that list does. The situation was much different in 1.4. -- Terry Jan Reedy From guido at python.org Fri Dec 16 02:39:04 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Dec 2011 17:39:04 -0800 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: On Thu, Dec 15, 2011 at 5:16 PM, Terry Reedy wrote: > On 12/15/2011 3:42 PM, Ned Batchelder wrote: > > This is another place where Python is inconsistent. We're told, "lists >> are for homogenous sequences of varying length, like a C array; tuples >> are for heterogenous aggregations of known length, like a C struct." >> > > I have not been told that for several years, and I am pretty sure you will > not find any such thing in the current docs. I consider it pretty much > obsolete, as the differences that flowed from that idea are gone. In Python > 3, tuples have all the non-mutating sequence methods that list does. The > situation was much different in 1.4. > I strongly disagree. Being immutable sequences (i.e. homogeneous) is a minor secondary role for tuples. Their primary role remains to hold a small bunch of heterogeneous values -- like namedtuple, but without needing forethought. A good example are dictionary items -- these are (key, value) pairs where for a given dict, the keys are all of the same type (or of a small set of related types) and ditto for the values, but the key type and the value types are unrelated. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Fri Dec 16 03:23:17 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 16 Dec 2011 13:23:17 +1100 Subject: [Python-ideas] Python Users Aren't Perfect References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: <87sjklz23u.fsf@benfinney.id.au> Ned Batchelder writes: > This is another place where Python is inconsistent. We're told, "lists > are for homogenous sequences of varying length, like a C array; tuples > are for heterogenous aggregations of known length, like a C struct." I think that's a poor rendition of the distinction. Rather, what I advise is: if the *meaning* of an item depends on its position in the sequence (like with a C struct), use a tuple. If the meaning of an item is unaffected by its position in the sequence, use a list. > Then we define a function foo(*args), and Python gives us a tuple! :-( Yes, exactly: the positional arguments to the function are *not* a homogeneous sequence, so a list doesn't connote the right thing. The position is important ? we call them ?positional arguments? ? so a tuple makes sense. -- \ ?I am the product of millions of generations of individuals who | `\ each fought against a hostile universe and won, and I aim to | _o__) maintain the tradition.? ?Paul Z. Myers, 2009-09-12 | Ben Finney From ned at nedbatchelder.com Fri Dec 16 03:55:16 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Dec 2011 21:55:16 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <87sjklz23u.fsf@benfinney.id.au> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <87sjklz23u.fsf@benfinney.id.au> Message-ID: <4EEAB314.1030207@nedbatchelder.com> On 12/15/2011 9:23 PM, Ben Finney wrote: > Ned Batchelder > writes: > >> Then we define a function foo(*args), and Python gives us a tuple! :-( > Yes, exactly: the positional arguments to the function are *not* a > homogeneous sequence, so a list doesn't connote the right thing. The > position is important ? we call them ?positional arguments? ? so a tuple > makes sense. In general the positional arguments to a function are not homogenous, but the foo(*args) syntax is for precisely when you don't know how many args you will get, and you will of necessity treat them homogenously, no? We've been through this in another branch of this thread... --Ned. From ned at nedbatchelder.com Fri Dec 16 03:56:27 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Dec 2011 21:56:27 -0500 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> Message-ID: <4EEAB35B.907@nedbatchelder.com> On 12/15/2011 8:39 PM, Guido van Rossum wrote: > On Thu, Dec 15, 2011 at 5:16 PM, Terry Reedy > wrote: > > On 12/15/2011 3:42 PM, Ned Batchelder wrote: > > This is another place where Python is inconsistent. We're > told, "lists > are for homogenous sequences of varying length, like a C > array; tuples > are for heterogenous aggregations of known length, like a C > struct." > > > I have not been told that for several years, and I am pretty sure > you will not find any such thing in the current docs. I consider > it pretty much obsolete, as the differences that flowed from that > idea are gone. In Python 3, tuples have all the non-mutating > sequence methods that list does. The situation was much different > in 1.4. > > > I strongly disagree. Being immutable sequences (i.e. homogeneous) is a > minor secondary role for tuples. Their primary role remains to hold a > small bunch of heterogeneous values -- like namedtuple, but without > needing forethought. A good example are dictionary items -- these are > (key, value) pairs where for a given dict, the keys are all of the > same type (or of a small set of related types) and ditto for the > values, but the key type and the value types are unrelated. Could you explain why the foo(*args) syntax creates args as a tuple rather than a list? --Ned. > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Dec 16 04:08:55 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 16 Dec 2011 12:08:55 +0900 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA626E.8040101@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <20111215215147.22f41c80@pitrou.net> <4EEA5FD3.2070603@nedbatchelder.com> <4EEA626E.8040101@nedbatchelder.com> Message-ID: <87fwglxlfc.fsf@uwakimon.sk.tsukuba.ac.jp> Ned Batchelder writes: > On 12/15/2011 4:06 PM, Masklinn wrote: > > On 2011-12-15, at 22:00 , Ned Batchelder wrote: > >> What? The whole point of the "def foo(*args)" syntax is so a function can take an unknown-length list of arguments, which will be treated uniformly. > > That's not *entirely* true, it's also used for proxy functions, in order to just pass a bunch of arguments (positional or keywords) along to the next level without having to unpack and repack them one by one. > We are in agreement: I don't know the length, and I will treat them > uniformly (pass them all through to the next function). That's way too high a level of abstraction. The discussion I found persuasive is the varargs analogy. /* "Hello Ned" in C */ #include char *s = "string"; int i = 1; double x = 3.1415926; int main (int argc, char *argv[]) { printf ("string %s, int %d, double %f\nHello Ned! Are we homogenized yet?"); exit(-1); } Of course it still demands "correct perception" on the part of the user, but I don't see this as "inconsistent". From guido at python.org Fri Dec 16 04:35:21 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Dec 2011 19:35:21 -0800 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEAB35B.907@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEAB35B.907@nedbatchelder.com> Message-ID: On Thu, Dec 15, 2011 at 6:56 PM, Ned Batchelder wrote: > On 12/15/2011 8:39 PM, Guido van Rossum wrote: > > On Thu, Dec 15, 2011 at 5:16 PM, Terry Reedy wrote: > >> On 12/15/2011 3:42 PM, Ned Batchelder wrote: >> >> This is another place where Python is inconsistent. We're told, "lists >>> are for homogenous sequences of varying length, like a C array; tuples >>> are for heterogenous aggregations of known length, like a C struct." >>> >> >> I have not been told that for several years, and I am pretty sure you >> will not find any such thing in the current docs. I consider it pretty much >> obsolete, as the differences that flowed from that idea are gone. In Python >> 3, tuples have all the non-mutating sequence methods that list does. The >> situation was much different in 1.4. >> > > I strongly disagree. Being immutable sequences (i.e. homogeneous) is a > minor secondary role for tuples. Their primary role remains to hold a small > bunch of heterogeneous values -- like namedtuple, but without needing > forethought. A good example are dictionary items -- these are (key, value) > pairs where for a given dict, the keys are all of the same type (or of a > small set of related types) and ditto for the values, but the key type and > the value types are unrelated. > > Could you explain why the foo(*args) syntax creates args as a tuple rather > than a list? > It's a historical accident. In ABC, functions didn't have multiple arguments -- instead there was a single argument that was a tuple. If you go back way, way in Python's early history (before 1.0 I think) you'll find that arguments kind of worked the same way. Then an idiom developed to accept a variable number of arguments, in order to support default argument values (because Python's tuples, unlike ABC's, were sequences even then). This turned out to be awkward if you wanted 1 or 2 arguments (the 1-arg case had to be dealt with special because you'd receive a plain value instead of a tuple) and then the *args and arg=value syntax was invented. But because previously varargs had been tuples, we kept them tuples. (I think this was also in the days that you couldn't unpack a list -- but I'm not actually sure of that. I do know that tuple unpacking was always in the language.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From grosser.meister.morti at gmx.net Fri Dec 16 05:21:52 2011 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Fri, 16 Dec 2011 05:21:52 +0100 Subject: [Python-ideas] contextlib.maybe In-Reply-To: <20111211050723.GA12197@air> References: <20111211050723.GA12197@air> Message-ID: <4EEAC760.1000109@gmx.net> On 12/11/2011 06:07 AM, Julian Berman wrote: > Twice recently I've found myself wanting to write the following code: > > def fn(a_file=None): > responsible_for_closing = False > > if a_file is None: > a_file = open(a_default_location) > responsible_for_closing = True > > do_stuff(a_file) > > if responsible_for_closing: > a_file.close() > What about this? def fn(a_file=None): if a_file is None: with open(a_default_location) as a_file: do_stuff(a_file) else: do_stuff(a_file) That's how I do it when this comes up. > which can be written slightly shorter I know, but it's still a tiny bit > messy and repetitive. What I'd prefer to write is something more like: > > def fn(a_file=None): > with contextlib.maybe(a_file, open, default) as a_file: > do_stuff(a_file) > > where `maybe` takes an object and conditionally runs a context manager > if a check fails. Implementation would be: > > > @contextlib.contextmanager > def maybe(got, contextfactory, *args, checkif=bool, **kwargs): > if checkif(got): > yield got > else: > with contextfactory(*args, **kwargs) as got: > yield got > > > It's hard to gauge utility for such simple functions (though contextlib > already has closing(), so I figured it'd be worth asking at least). > Would this be useful to others? Or perhaps I'm completely missing > something and you've got suggestions on how to better have an API where > an argument can be fetched if not provided but a context manager would > preferably need to be run to do so. From mwm at mired.org Fri Dec 16 05:34:28 2011 From: mwm at mired.org (Mike Meyer) Date: Thu, 15 Dec 2011 20:34:28 -0800 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA6C88.1030203@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEA620B.1000300@nedbatchelder.com> <4EEA658A.3010509@stoneleaf.us> <4EEA6C88.1030203@nedbatchelder.com> Message-ID: <20111215203428.77c0c821@bhuda.mired.org> On Thu, 15 Dec 2011 16:54:16 -0500 Ned Batchelder wrote: > On 12/15/2011 4:24 PM, Ethan Furman wrote: > > Python is a dynamic language -- why can't it have dynamic structs? > I honestly don't know how to interpret this. I do. Ethan is incorrectly trying to turn a similarity - that both Python tuples and C structs are collections of heterogeneous data - into an analogy. They aren't analogs, as that's pretty much the only similarity. If you really want a dynamic tuple - a dynamic collection of heterogeneous data accessed by an index - use a list. That was standard practice in LISP-like languages for a long time. But Python (and most modern LISPs) has better tools for such. For instance, the proper analog for a C struct in Python is the class. You provide definitions of them both along with names for their heterogeneous components. You create instances of them both that can assign different values to those components. The components are properly accessed by name. There are probably other similarities. There are a number of differences as well. Most notably, instances of classes are dynamic. You can change, and even add and delete, components dynamically. So Python already has a dynamic struct. You just have to get the analogies right. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From greg.ewing at canterbury.ac.nz Fri Dec 16 06:50:39 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Dec 2011 18:50:39 +1300 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEA620B.1000300@nedbatchelder.com> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEA620B.1000300@nedbatchelder.com> Message-ID: <4EEADC2F.9050009@canterbury.ac.nz> Ned Batchelder wrote: > Finish that foo function definition: > it will *have* to have "for a in args:" Not necessarily. It might be something like this: def foo(*args): if len(args) == 2: x, y = args fizzle2d(x, y) else: x, y, z = args fizzle3d(x, y, z) In other words, it might be what is effectively an overloaded function with a number of different possible calling signatures. Once you've decided which signature is being invoked, each member of the argument tuple takes on a specific role. -- Greg From greg.ewing at canterbury.ac.nz Fri Dec 16 07:06:43 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Dec 2011 19:06:43 +1300 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEAB35B.907@nedbatchelder.com> Message-ID: <4EEADFF3.3060105@canterbury.ac.nz> Guido van Rossum wrote: > (I think this was also in the days that you > couldn't unpack a list -- but I'm not actually sure of that. I do know > that tuple unpacking was always in the language.) I can definitely remember a time when you had to use (...) to unpack tuples and [...] to unpack lists. My Python memory doesn't go back further than that, though. -- Greg From ethan at stoneleaf.us Fri Dec 16 08:50:11 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 15 Dec 2011 23:50:11 -0800 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEADC2F.9050009@canterbury.ac.nz> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEA620B.1000300@nedbatchelder.com> <4EEADC2F.9050009@canterbury.ac.nz> Message-ID: <4EEAF833.1080206@stoneleaf.us> Greg Ewing wrote: > Ned Batchelder wrote: >> Finish that foo function definition: it will *have* to have "for a in >> args:" > > Not necessarily. It might be something like this: > > def foo(*args): > if len(args) == 2: > x, y = args > fizzle2d(x, y) > else: > x, y, z = args > fizzle3d(x, y, z) > > In other words, it might be what is effectively an > overloaded function with a number of different possible > calling signatures. Once you've decided which signature > is being invoked, each member of the argument tuple > takes on a specific role. > Thanks, Greg -- this is pretty much what I meant about dynamic structs. :) ~Ethan~ From ncoghlan at gmail.com Fri Dec 16 09:34:54 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2011 18:34:54 +1000 Subject: [Python-ideas] contextlib.maybe In-Reply-To: <4EEAC760.1000109@gmx.net> References: <20111211050723.GA12197@air> <4EEAC760.1000109@gmx.net> Message-ID: On Fri, Dec 16, 2011 at 2:21 PM, Mathias Panzenb?ck wrote: > On 12/11/2011 06:07 AM, Julian Berman wrote: >> >> Twice recently I've found myself wanting to write the following code: >> >> ? ? def fn(a_file=None): >> ? ? ? ? responsible_for_closing = False >> >> ? ? ? ? if a_file is None: >> ? ? ? ? ? ? a_file = open(a_default_location) >> ? ? ? ? ? ? responsible_for_closing = True >> >> ? ? ? ? do_stuff(a_file) >> >> ? ? ? ? if responsible_for_closing: >> ? ? ? ? ? ? a_file.close() >> > > What about this? > > ? ? ? ?def fn(a_file=None): > ? ? ? ? ? ? ? ?if a_file is None: > ? ? ? ? ? ? ? ? ? ? ? ?with open(a_default_location) as a_file: > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?do_stuff(a_file) > ? ? ? ? ? ? ? ?else: > ? ? ? ? ? ? ? ? ? ? ? ?do_stuff(a_file) > > > That's how I do it when this comes up. With contextlib2.ContextStack [1] the problem of conditional release of resources can be handled as follows: def fn(a_file=None): with ContextStack() as stack: if a_file is None: # The stack will release the file when we're done a_file = stack.enter_context(open(a_default_location)) do_stuff(a_file) If you're interested in seeing this (or something like it) in the standard library for 3.3, grab the module of PyPI, play around with it and send me feedback on the BitBucket issue tracker [2]. [1] http://contextlib2.readthedocs.org/en/latest/index.html#contextlib2.ContextStack [2] https://bitbucket.org/ncoghlan/contextlib2/issues?status=new&status=open Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From masklinn at masklinn.net Fri Dec 16 09:37:11 2011 From: masklinn at masklinn.net (Masklinn) Date: Fri, 16 Dec 2011 09:37:11 +0100 Subject: [Python-ideas] Python Users Aren't Perfect In-Reply-To: <4EEADC2F.9050009@canterbury.ac.nz> References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <4EEA620B.1000300@nedbatchelder.com> <4EEADC2F.9050009@canterbury.ac.nz> Message-ID: On 2011-12-16, at 06:50 , Greg Ewing wrote: > Ned Batchelder wrote: >> Finish that foo function definition: it will *have* to have "for a in args:" > > Not necessarily. It might be something like this: > > def foo(*args): > if len(args) == 2: > x, y = args > fizzle2d(x, y) > else: > x, y, z = args > fizzle3d(x, y, z) > > In other words, it might be what is effectively an > overloaded function with a number of different possible > calling signatures. Once you've decided which signature > is being invoked, each member of the argument tuple > takes on a specific role. Is that really used instead of optional arguments with a placeholder value e.g. def foo(x, y, z=None): if z is None: fizzle2d(x, y) else: fizzle3d(x, y, z) ? From ben+python at benfinney.id.au Fri Dec 16 10:08:17 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 16 Dec 2011 20:08:17 +1100 Subject: [Python-ideas] Python Users Aren't Perfect References: <20111210144230.GA16801@iskra.aviel.ru> <4EE5FAEC.1080907@nedbatchelder.com> <20111212140758.301e7431@pitrou.net> <4EE5FE81.3060304@nedbatchelder.com> <062FD665-B80A-401E-A3C2-75CA2ABE4424@masklinn.net> <4EE67047.8050209@canterbury.ac.nz> <87d3btyyuq.fsf@uwakimon.sk.tsukuba.ac.jp> <4EEA5BC2.4070402@nedbatchelder.com> <87sjklz23u.fsf@benfinney.id.au> <4EEAB314.1030207@nedbatchelder.com> Message-ID: <87obv8zxxa.fsf@benfinney.id.au> Ned Batchelder writes: > On 12/15/2011 9:23 PM, Ben Finney wrote: > > the positional arguments to the function are *not* a homogeneous > > sequence, so a list doesn't connote the right thing. The position is > > important ? we call them ?positional arguments? ? so a tuple makes > > sense. > In general the positional arguments to a function are not homogenous, > but the foo(*args) syntax is for precisely when you don't know how > many args you will get It's valuable for the case where the arguments need to be passed verbatim to a superclass. At some point, something in the call chain should know the meaning of each positional argument. > and you will of necessity treat them homogenously, no? If the function consumes the positional arguments as a homogeneous list, I'd say that's a poor design (the function should instead specify a single argument which is a homogeneous sequence). -- \ ?If you define cowardice as running away at the first sign of | `\ danger, screaming and tripping and begging for mercy, then yes, | _o__) Mr. Brave man, I guess I'm a coward.? ?Jack Handey | Ben Finney From alon at horev.net Sat Dec 17 02:00:09 2011 From: alon at horev.net (Alon Horev) Date: Sat, 17 Dec 2011 03:00:09 +0200 Subject: [Python-ideas] context managers used to allocate resources can be abused - a solution Message-ID: Hello, while I was implementing a connection pool i've noticed a pitfall of our beloved with statements: with pool.get_connection() as conn: ....conn.execute(...) conn.execute(...) # the connection has been returned to the pool and does not belong to the user! a proposed solution: with protected_context(pool.get_connection()) as conn: ....conn.execute(...) conn.execute(...) # raises OutOfContextError() with protected_context(file("/tmp/bla.txt", "w")) as f: ....file.write("blo") file.write("blu") # raises OutOfContextError() the solution is to basically proxy all methods to the real object until the context ends and then the proxy expires. what do you think about adding it to contextlib? Thanks, Alon Horev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 17 03:07:55 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Dec 2011 12:07:55 +1000 Subject: [Python-ideas] context managers used to allocate resources can be abused - a solution In-Reply-To: References: Message-ID: On Sat, Dec 17, 2011 at 11:00 AM, Alon Horev wrote: > Hello, > > while I was implementing a connection pool?i've noticed a pitfall of our > beloved with statements: > > with pool.get_connection() as conn: > ....conn.execute(...) > conn.execute(...) # the connection has been returned to the pool and?does > not belong to the user! That's a bug in the connection pool implementation. If the underlying connection has been returned to the pool, the proxy returned by the context manager shouldn't work any more. Try the above code structure with a file object and it will give you "IO operation on closed file". Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Sun Dec 18 04:23:13 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 17 Dec 2011 22:23:13 -0500 Subject: [Python-ideas] context managers used to allocate resources can be abused - a solution In-Reply-To: References: Message-ID: On 12/16/2011 9:07 PM, Nick Coghlan wrote: > On Sat, Dec 17, 2011 at 11:00 AM, Alon Horev wrote: >> Hello, >> >> while I was implementing a connection pool i've noticed a pitfall of our >> beloved with statements: >> >> with pool.get_connection() as conn: >> ....conn.execute(...) >> conn.execute(...) # the connection has been returned to the pool and does >> not belong to the user! Good catch! > That's a bug in the connection pool implementation. If the underlying > connection has been returned to the pool, the proxy returned by the > context manager shouldn't work any more. I believe Nick is saying that conn.__exit__ should close the connection and release the resource, that being the point of with statements and context managers. Can you open a tracker issue? > Try the above code structure with a file object and it will give you > "IO operation on closed file". Because file_object.__exit__ closes the connection to the OS resource. -- Terry Jan Reedy From ncoghlan at gmail.com Sun Dec 18 07:41:26 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Dec 2011 16:41:26 +1000 Subject: [Python-ideas] context managers used to allocate resources can be abused - a solution In-Reply-To: References: Message-ID: On Sun, Dec 18, 2011 at 1:23 PM, Terry Reedy wrote: >> That's a bug in the connection pool implementation. If the underlying >> connection has been returned to the pool, the proxy returned by the >> context manager shouldn't work any more. > > > I believe Nick is saying that conn.__exit__ should close the connection and > release the resource, that being the point of with statements and context > managers. Can you open a tracker issue? No, that's not what I'm saying at all - the whole purpose of using a connection pool is that you *don't* release the underlying OS resource when you return a resource to the pool. What I'm saying is that if a pool supports the context management protocol for *temporary* access to one of the pool resources, then *that* context manager is responsible for ensuring that the resource reference returned is correctly invalidated in __exit__ (just as closing a file object prevents further IO operations). I'm not aware of any such pools in the stdlib (multiprocessing has some pool APIs, but I don't believe they support the context management protocol for pooled resources), so I'm suggesting that the only problem here is in the design of the OP's own pool API. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From nathan.alexander.rice at gmail.com Mon Dec 19 00:28:27 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Sun, 18 Dec 2011 18:28:27 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python Message-ID: I believe it would be a good idea in instances where it is known that a collection of a single type is going to be returned, to return a subclass with type information and type specific methods "mixed in". You could provide member methods as collection methods that operate in a vectorized manner, returning a new collection or iterator with the results much like the mathematical functions in NumPy. This would also give people a reliable method to make functions operate on both scalar and vector values. I believe this could be implemented without needing subclasses for everything under the sun with a generic collection "type contract" mix-in. If a developer wanted to provide additional type specific collection/iterator methods they would of course need to subclass that. To avoid handcuffing people with types (which is definitely un-pythonic) and maintain backwards compatibility, the standard collection modification methods could be hooked so that if an object of an incorrect type is added, a warning is raised and the collection gracefully degrades by removing mixed-in type information and methods. Additionally, a method could be provided that lets the user "terminate the contract" causing the collection to degrade without a warning. I have several motivations for this: -- Performing a series of operations using comprehensions or map tends to be highly verbose in an uninformative way. Compare the current method with what would be possible using "typed" collections: L2 = [X(e) for e in L1] L3 = [Y(e) for e in L2] vs L2 = X(L1) # assuming X has been updated to work in both vector/scalar L3 = Y(L2) # context... L2 = [Z(Y(X(e))) for e in L1] vs L2 = Z(Y(X(L1))) L2 = [e.X().Y().Z() for e in L1] vs L2 = L1.X().Y().Z() # assuming vectorized versions of member methods #are folded into the collection via the mixin. -- Because collections are type agnostic, it is not possible to place methods on them that are type specific. This leads to a lot of cases where python forces you to read inside out or a the syntax gets very disjoint in general. A good example of this is: "\n".join(l.capitalize() for l in my_string.split("\n")) which could reduce to something far more readable, such as: my_string.split("\n").capitalize().join_items("\n") Besides the benefits to basic language usability (in my opinion) there are tool and development benefits: -- The additional type information would simplify static analysis and provide cues for optimization (I'm looking at pypy here; their list strategies play to this perfectly) -- The warning on "violating the contract" and without first terminating it would be a helpful tool in catching and debugging errors. I have some thoughts on syntax and specifics that I think would work well, however I wanted to solicit more feedback before I go too far down that path. Nathan From ncoghlan at gmail.com Mon Dec 19 00:45:08 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Dec 2011 09:45:08 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On Mon, Dec 19, 2011 at 9:28 AM, Nathan Rice wrote: > -- Performing a series of operations using comprehensions or map > tends to be highly verbose in an uninformative way. ?Compare the > current method with what would be possible using "typed" collections: > > L2 = [X(e) for e in L1] > L3 = [Y(e) for e in L2] > vs > L2 = X(L1) # assuming X has been updated to work in both vector/scalar > L3 = Y(L2) # context... This use case is why map() remains a builtin, even in Python 3: L2 = map(X, L1) L3 = map(Y, L2) Short, but explicit (no under-the-hood guessing about whether or not something should be treated as a scalar or vector value - in the general case, this distinction isn't as clear as you might think, just look at strings). > L2 = [Z(Y(X(e))) for e in L1] > vs > L2 = Z(Y(X(L1))) def XYZ(arg): """Look, I can document what this means!""" return Z(Y(X(arg))) L2 = map(XYZ, L1) > L2 = [e.X().Y().Z() for e in L1] > vs > L2 = L1.X().Y().Z() # assuming vectorized versions of member methods > #are folded into the collection via the mixin. def XYZ_methods(arg): """I can also document what *this* means""" return arg.X().Y().Z() L2 = map(XYZ_methods, L1) > -- ?Because collections are type agnostic, it is not possible to place > methods on them that are type specific. ?This leads to a lot of cases > where python forces you to read inside out or a the syntax gets > very disjoint in general. ?A good example of this is: > > "\n".join(l.capitalize() for l in my_string.split("\n")) > > which could reduce to something far more readable, such as: > > my_string.split("\n").capitalize().join_items("\n") Another bad example, since that's just a really verbose way of writing my_string.capitalize(). Short answer: what advantage does your proposal really offer over simply extracting the repetitive operation out to a use case specific function, and making effective use of the existing vectorisation utilities (i.e. map() and itertools)? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Mon Dec 19 01:18:02 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 18 Dec 2011 19:18:02 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On 12/18/2011 6:28 PM, Nathan Rice wrote: > I believe it would be a good idea in instances where it is known that a > collection of a single type is going to be returned, to return a subclass > with type information and type specific methods "mixed in". Are you familiar with numpy, which has typed multi-dimensional arrays and array operations? And the option to add user functions to work with such? -- Terry Jan Reedy From nathan.alexander.rice at gmail.com Mon Dec 19 03:02:38 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Sun, 18 Dec 2011 21:02:38 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: >> I believe it would be a good idea in instances where it is known that a >> collection of a single type is going to be returned, to return a subclass >> with type information and type specific methods "mixed in". > > Are you familiar with numpy, which has typed multi-dimensional arrays and > array operations? And the option to add user functions to work with such? I am, and vectorized functions/typed arrays from numpy are a source of inspiration for me in this proposal. The problem with having typed arrays/etc as an add-on is that you lose the benefits as soon as you step into code that isn't meant to interact with with the specific library. This isn't such an issue with NumPy since it has a healthy ecosystem, but there are lots of other instances where having type aware collections would be nice. From nathan.alexander.rice at gmail.com Mon Dec 19 03:24:12 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Sun, 18 Dec 2011 21:24:12 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On Sun, Dec 18, 2011 at 6:45 PM, Nick Coghlan wrote: > On Mon, Dec 19, 2011 at 9:28 AM, Nathan Rice > wrote: >> -- Performing a series of operations using comprehensions or map >> tends to be highly verbose in an uninformative way. ?Compare the >> current method with what would be possible using "typed" collections: >> >> L2 = [X(e) for e in L1] >> L3 = [Y(e) for e in L2] >> vs >> L2 = X(L1) # assuming X has been updated to work in both vector/scalar >> L3 = Y(L2) # context... > > This use case is why map() remains a builtin, even in Python 3: Yes, but map(lambda x: getattr(x, "method")(), thing) is ugly, and map(lambda x: x.method_2(param), map(lambda x: x.method(param), thing)) is really ugly. On top of that, it is asking more of code analysis tools to verify that code, and IDEs aren't going to be able to tell you the methods on the ambiguous x in the lambda. Sure if the only argument is self, you could call class.method, but I don't think that is the majority use case. > L2 = map(X, L1) > L3 = map(Y, L2) > > Short, but explicit (no under-the-hood guessing about whether or not > something should be treated as a scalar or vector value - in the > general case, this distinction isn't as clear as you might think, just > look at strings). Yes, I love that feature of strings, it is a source of lots of bugs, but I digress. The reason this partially solves that problem is that instead of having to do a bunch of guesswork on an iterable to see if you should do the vectorized version of the function, you just check to see if it is an instance of a certain TypedCollectionContract. If so, vectorize. No "isinstance(foo, Iterable) and not isinstance(foo, basestr)" silliness here. >> L2 = [Z(Y(X(e))) for e in L1] >> vs >> L2 = Z(Y(X(L1))) > > def XYZ(arg): > ? """Look, I can document what this means!""" > ? return Z(Y(X(arg))) > > L2 = map(XYZ, L1) What about WXY, and XZ, and WYZ, and YZ, and... >> L2 = [e.X().Y().Z() for e in L1] >> vs >> L2 = L1.X().Y().Z() # assuming vectorized versions of member methods >> #are folded into the collection via the mixin. > > def XYZ_methods(arg): > ? """I can also document what *this* means""" > ? return arg.X().Y().Z() > > L2 = map(XYZ_methods, L1) > >> -- ?Because collections are type agnostic, it is not possible to place >> methods on them that are type specific. ?This leads to a lot of cases >> where python forces you to read inside out or a the syntax gets >> very disjoint in general. ?A good example of this is: >> >> "\n".join(l.capitalize() for l in my_string.split("\n")) >> >> which could reduce to something far more readable, such as: >> >> my_string.split("\n").capitalize().join_items("\n") > > Another bad example, since that's just a really verbose way of writing > my_string.capitalize(). The python interpreter says otherwise... >>> foo = "line 1\nline 2\nline 3" >>> foo.capitalize() 'Line 1\nline 2\nline 3' >>> "\n".join(s.capitalize() for s in foo.split("\n")) 'Line 1\nLine 2\nLine 3' >>> > Short answer: what advantage does your proposal really offer over > simply extracting the repetitive operation out to a use case specific > function, and making effective use of the existing vectorisation > utilities (i.e. map() and itertools)? IDEs can provide context hints, interpreters can use the contract to change how they treat the collection to improve performance, lint style code analysis will be easier and it is across the board less verbose than the currently available options, while reading left to right with fewer non letter characters I.E. more understandable. Nathan From wuwei23 at gmail.com Mon Dec 19 04:39:19 2011 From: wuwei23 at gmail.com (alex23) Date: Sun, 18 Dec 2011 19:39:19 -0800 (PST) Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> On Dec 19, 12:24?pm, Nathan Rice wrote: > Yes, but map(lambda x: getattr(x, "method")(), thing) is ugly from operator import methodcaller method = methodcaller('method') result = map(method, thing) > map(lambda x: x.method_2(param), map(lambda x: x.method(param), > thing)) is really ugly. method = methodcaller('method', param) method2 = methodcaller('method_2', param) result = map(method2, map(method, thing)) If your code is ugly, stop writing ugly code. :) From wuwei23 at gmail.com Mon Dec 19 05:14:28 2011 From: wuwei23 at gmail.com (alex23) Date: Sun, 18 Dec 2011 20:14:28 -0800 (PST) Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> Message-ID: <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> Received via email, this isn't a private discussion: > Nathan Rice wrote: > I fail to see how the clunky act of importing a function most python > users probably don't know exists then creating another new function is > better than chaining one or two method calls. If you consider importing functionality 'clunky', then we're already at an impasse over what we consider to be 'elegant'. I also don't believe that user ignorance of standard library features mandates the inclusion of _everything_ into the base langauge. > Just because you CAN do something already doesn't mean you can do it in > an elegant way. And just because YOU find it inelegant doesn't make it so. I find using operator & functools _far_ clearer in intent than using lambda, _and it works right now_, which was the point I was trying to make here. You wrote unreadable code and then tried to use that as an argument for your idea. Breaking down complex statements into clearer parts isn't a radical notion; cramming more and more functionality onto a single line isn't something to which Python needs to aspire. > If the only metric was "can we do this?" we would all still be using Fortran. And if it's "I don't want to have to code to deal with this, let the language change to do it", you end up with PHP. From ncoghlan at gmail.com Mon Dec 19 05:29:59 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Dec 2011 14:29:59 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On Mon, Dec 19, 2011 at 12:24 PM, Nathan Rice wrote: > On Sun, Dec 18, 2011 at 6:45 PM, Nick Coghlan wrote: >> Another bad example, since that's just a really verbose way of writing >> my_string.capitalize(). > > The python interpreter says otherwise... > >>>> foo = "line 1\nline 2\nline 3" >>>> foo.capitalize() > 'Line 1\nline 2\nline 3' >>>> "\n".join(s.capitalize() for s in foo.split("\n")) > 'Line 1\nLine 2\nLine 3' Ah, my mistake, I was thinking of title() rather than capitalize(). Still: def capitalize_lines(s): return "\n".join(s.capitalize() for line in s.split("\n")) There comes a time when the contortions people go to to avoid naming a frequently repeated operation just get silly. If you do something a lot, pull it out into a function and name it. The named function can even be a local closure if the usage is sufficiently specific to one operation - then it can still reference local variables without requiring a lot of additional parameters. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From nathan.alexander.rice at gmail.com Mon Dec 19 05:31:31 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Sun, 18 Dec 2011 23:31:31 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> Message-ID: My apologies for the direct email. Gmail is somewhat retarded when it comes to the python lists, and I don't always remember to change the to:... >> Nathan Rice wrote: >> I fail to see how the clunky act of importing a function most python >> users probably don't know exists then creating another new function is >> better than chaining one or two method calls. > > If you consider importing functionality 'clunky', then we're already > at an impasse over what we consider to be 'elegant'. I also don't > believe that user ignorance of standard library features mandates the > inclusion of _everything_ into the base langauge. There are a lot of warts in the standard library, (thus Py3). I'm all for importing things that deserve to be in their own namespace with other, similar things. I'm not for having to import something that is needlessly more verbose than solutions that are available without an import (see comprehensions). If you were just trying to map a single argument function I'd say "yeah, go for it" but from my perspective, that code is square peg, meet round hole. >> Just because you CAN do something already doesn't mean you can do it in >> an elegant way. > > And just because YOU find it inelegant doesn't make it so. I find > using operator & functools _far_ clearer in intent than using lambda, > _and it works right now_, which was the point I was trying to make > here. You wrote unreadable code and then tried to use that as an > argument for your idea. Breaking down complex statements into clearer > parts isn't a radical notion; cramming more and more functionality > onto a single line isn't something to which Python needs to aspire. What makes a solution elegant? I'd say having access to a cross cutting paradigm that is backwards compatible, reduces code while making it more readable and provides opportunities to improve the toolchain is more elegant. At any rate, I could care less about the lambda, that is a strawman. >> If the only metric was "can we do this?" we would all still be using Fortran. > > And if it's "I don't want to have to code to deal with this, let the > language change to do it", you end up with PHP. Well, that would be true if you also added "I don't care if the language is internally consistent in any way" and a few others, but I'm not here to bash PHP. From ncoghlan at gmail.com Mon Dec 19 05:33:17 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Dec 2011 14:33:17 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On Mon, Dec 19, 2011 at 12:24 PM, Nathan Rice wrote: > Yes, I love that feature of strings, it is a source of lots of bugs, > but I digress. ?The reason this partially solves that problem is that > instead of having to do a bunch of guesswork on an iterable to see if > you should do the vectorized version of the function, you just check > to see if it is an instance of a certain TypedCollectionContract. ?If > so, vectorize. ?No "isinstance(foo, Iterable) and not isinstance(foo, > basestr)" silliness here. If you plan to introduce a new ABC to drive this, then I have a simple proposal: 1. Write a module that implements your "broadcast API for collections" 2. Publish it on PyPI 3. When it has popular uptake to indicate widespread user demand, then come back to us Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From nathan.alexander.rice at gmail.com Mon Dec 19 05:47:04 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Sun, 18 Dec 2011 23:47:04 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: > Ah, my mistake, I was thinking of title() rather than capitalize().>> Still:>> def capitalize_lines(s):> ? ?return "\n".join(s.capitalize() for line in s.split("\n"))>> There comes a time when the contortions people go to to avoid naming a> frequently repeated operation just get silly. If you do something a> lot, pull it out into a function and name it. The named function can> even be a local closure if the usage is sufficiently specific to one> operation - then it can still reference local variables without> requiring a lot of additional parameters. Oh, I agree that it is best just to make it a function, and I have (several times!). ?So have a lot of other people. >> Yes, I love that feature of strings, it is a source of lots of bugs, >> but I digress. ?The reason this partially solves that problem is that >> instead of having to do a bunch of guesswork on an iterable to see if >> you should do the vectorized version of the function, you just check >> to see if it is an instance of a certain TypedCollectionContract. ?If >> so, vectorize. ?No "isinstance(foo, Iterable) and not isinstance(foo, >> basestr)" silliness here. > > If you plan to introduce a new ABC to drive this, then I have a simple proposal: > > 1. Write a module that implements your "broadcast API for collections" > 2. Publish it on PyPI > 3. When it has popular uptake to indicate widespread user demand, then > come back to us I certainly plan to make an implementation for example purposes. And if this was Ruby and not Python, that module would actually be universally useful, because they can rewire their internals better than we can. Unfortunately, in Python land it is very difficult to hook into list/iterator/etc creation on a global scale. As a result, a user would have to be very careful about calling into other people's code, and the type information would still get blown away pretty frequently. Kind of like if you subclass string; all the string methods still return strings, so the language is working against you there. Regardless, I'm sure everything that gets added to the language was first a hugely popular PyPI project, that is clearly the way the language evolves. From ncoghlan at gmail.com Mon Dec 19 06:44:52 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Dec 2011 15:44:52 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On Mon, Dec 19, 2011 at 2:47 PM, Nathan Rice wrote: > Regardless, I'm sure everything that gets added to the > language was first a hugely popular PyPI project, that is clearly the > way the language evolves. Not everything, no. The real criteria is to have solid use cases where a proposal clearly improves the language. Sometimes that's just obvious (e.g. supporting a popular new compression protocol), sometimes a PEP is enough to explain, other times real world experience on PyPI is the best option. You don't have *any* of those at this point (just some vague hand-waving), so publishing a PyPI package is the most obvious way to start acquiring more data (and to prove that there's even a version of the idea that can be taken beyond the hand-waving stage). What you seem to be asking for is a general purpose typed container factory along the following lines: def typed_container(container_type, data_type): class TypedContainer(container_type): def __getattr__(self, attr): data_type_attr = getattribute(data_type, attr) if callable(data_type_attr): _result_type = type(self) def _broadcast(*args, **kwds): _result_type(data_type_attr(x, *args, **kwds) for x in self) return _broadcast return data_type_attr return TypedContainer I think it will have a lot of problems in practice (note that NumPy doesn't try to solve the broadcasting problem in general, just for a single specific data type), but, if the concept has any merit at all, that's certainly something that can be demonstrated quite adequately on PyPI. To get a better idea of the level of evidence you're trying to reach if your suggestion is ever going to get anywhere, try taking a look at http://www.boredomandlaziness.org/2011/02/justifying-python-language-changes.html and http://www.boredomandlaziness.org/2011/02/status-quo-wins-stalemate.html. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Mon Dec 19 07:48:32 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 19 Dec 2011 19:48:32 +1300 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: <4EEEDE40.9090402@canterbury.ac.nz> Nick Coghlan wrote: > What you seem to be asking for is a general purpose typed container > factory along the following lines: > > def typed_container(container_type, data_type): > class TypedContainer(container_type): > def __getattr__(self, attr): > data_type_attr = getattribute(data_type, attr) > if callable(data_type_attr): > _result_type = type(self) > def _broadcast(*args, **kwds): > _result_type(data_type_attr(x, *args, **kwds) for x in self) > return _broadcast > return data_type_attr > return TypedContainer Seems to me this would be better described as a "broadcasting container" than a "typed container". The only use it makes of the element type is to speed things up a bit by extracting bound methods. So the typed-ness is not an essential feature of its functionality, just something required to support an optimisation. Extended to handle the various operator methods, this might be a useful thing to have around. In conjunction with array.array, it could provide a kind of "numpy lite" for when depending on full-blown numpy would seem like overkill. -- Greg From ncoghlan at gmail.com Mon Dec 19 09:08:10 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Dec 2011 18:08:10 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: <4EEEDE40.9090402@canterbury.ac.nz> References: <4EEEDE40.9090402@canterbury.ac.nz> Message-ID: On Mon, Dec 19, 2011 at 4:48 PM, Greg Ewing wrote: > Extended to handle the various operator methods, this might be a useful > thing to have around. In conjunction with array.array, it could provide > a kind of "numpy lite" for when depending on full-blown numpy would seem > like overkill. Agreed, but that "might" is the killer - hence why PyPI is the appropriate place for this idea. Perhaps when it's been refined for a couple of years, a case might be made for standard lib inclusion. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From storchaka at gmail.com Mon Dec 19 09:33:40 2011 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 19 Dec 2011 10:33:40 +0200 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: 19.12.11 01:28, Nathan Rice ???????(??): > L2 = [X(e) for e in L1] > L3 = [Y(e) for e in L2] > vs > L2 = X(L1) # assuming X has been updated to work in both vector/scalar > L3 = Y(L2) # context... L = ['a', 'bc', ['ada', 'a']] What is len(L)? 3 or [1, 2, 2] or [1, 2, [3, 1]]? > L2 = [Z(Y(X(e))) for e in L1] > vs > L2 = Z(Y(X(L1))) > > L2 = [e.X().Y().Z() for e in L1] > vs > L2 = L1.X().Y().Z() # assuming vectorized versions of member methods > #are folded into the collection via the mixin. What is L.count('a')? 1 or [1, 0, 1] or [1, 0, [2, 1]]? From storchaka at gmail.com Mon Dec 19 09:39:08 2011 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 19 Dec 2011 10:39:08 +0200 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> Message-ID: 19.12.11 06:14, alex23 ???????(??): > I find > using operator& functools _far_ clearer in intent than using lambda, > _and it works right now_, which was the point I was trying to make > here. I find using list comprehensions and generator expressions even more clearer. From nathan.alexander.rice at gmail.com Mon Dec 19 14:52:33 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Mon, 19 Dec 2011 08:52:33 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> Message-ID: >> L2 = [X(e) for e in L1]>> L3 = [Y(e) for e in L2]>> vs>> L2 = X(L1) # assuming X has been updated to work in both vector/scalar>> L3 = Y(L2) # context...>>> L = ['a', 'bc', ['ada', 'a']]>> What is len(L)? 3 or [1, 2, 2] or [1, 2, [3, 1]]?>>>> L2 = [Z(Y(X(e))) for e in L1]>> vs>> L2 = Z(Y(X(L1)))>>>> L2 = [e.X().Y().Z() for e in L1]>> vs>> L2 = L1.X().Y().Z() # assuming vectorized versions of member methods>> #are folded into the collection via the mixin.>>> What is L.count('a')? 1 or [1, 0, 1] or [1, 0, [2, 1]]? A fair concern; if the vectorized version of the child method were given the same name as the child method, I agree that this could result in ambiguity. There are multiple ways that member methods could be made available on the collection, including a proxy attribute, renaming, etc. ... >> I find >> using operator& functools _far_ clearer in intent than using lambda, >> >> _and it works right now_, which was the point I was trying to make >> here. > > > I find using list comprehensions and generator expressions even more > clearer. In general I think comprehensions are superior to map because they are more intuitive. Based on my experience reading code in the community I think that is well supported. Sadly, even though I have issues with the way they are used in many cases, lambdas are superior to a lot of other options because they are simple and cover many use cases. When used inline with simple expressions they provide a lot of mileage. From nathan.alexander.rice at gmail.com Mon Dec 19 15:30:37 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Mon, 19 Dec 2011 09:30:37 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: > Not everything, no. The real criteria is to have solid use cases where > a proposal clearly improves the language. Sometimes that's just > obvious (e.g. supporting a popular new compression protocol), > sometimes a PEP is enough to explain, other times real world > experience on PyPI is the best option. You don't have *any* of those > at this point (just some vague hand-waving), so publishing a PyPI > package is the most obvious way to start acquiring more data (and to > prove that there's even a version of the idea that can be taken beyond > the hand-waving stage). Couple things. 1. The "broadcasting" that people seemed to have latched on to is only part of what I put forward, and I agree it is something that would have to be done *correctly* to be beneficial. I would have no issues with providing a userspace lib to do this if type decorations were included in homogeneous collections/iterables, as long as the implementation of the decoration didn't suffer from some form of "string failure" (string subclasses are basically worthless as methods return strings, not an instance of the class). 2. A "type decorator" on homogeneous collections and iterables has a lot of nice little benefits throughout the toolchain. 3. Being able to place methods on a "type decorator" is useful, it solves issues like "foo".join() which really wants to be a method on string collections. 4. I wanted to gauge people's feelings before I went through the steps involved in writing a PEP. I believe that is the right thing to do, so I don't feel the "hand waving" comment is warranted. I've already learned people view collections that provide child object methods in vector form as a very big change even if it is backwards compatible; that is fine, I'm willing to shelve that if consensus is that people aren't comfortable with it. > What you seem to be asking for is a general purpose typed container > factory along the following lines: > ? ?def typed_container(container_type, data_type): > ? ? ? ?class TypedContainer(container_type): > ? ? ? ? ? ?def __getattr__(self, attr): > ? ? ? ? ? ? ? ?data_type_attr = getattribute(data_type, attr) > ? ? ? ? ? ? ? ?if callable(data_type_attr): > ? ? ? ? ? ? ? ? ? ?_result_type = type(self) > ? ? ? ? ? ? ? ? ? ?def _broadcast(*args, **kwds): > ? ? ? ? ? ? ? ? ? ? ? ?_result_type(data_type_attr(x, *args, **kwds) > for x in self) > ? ? ? ? ? ? ? ? ? ?return _broadcast > ? ? ? ? ? ? ? ?return data_type_attr > ? ? ? ?return TypedContainer Something along those lines. Again I feel people have latched on to one element of what I proposed here, to the detriment of the proposal as a whole. > I think it will have a lot of problems in practice (note that NumPy > doesn't try to solve the broadcasting problem in general, just for a > single specific data type), but, if the concept has any merit at all, > that's certainly something that can be demonstrated quite adequately > on PyPI. This may be the case, thus my request for input. I agree that a thoughtful approach is prudent, however as I have stated, as it currently stands the language would not be very supportive of an add-on module that does this. With PyPy, I could probably get my hooks in deeply enough that I could make something really useful, however given CPython is the current flavor du jour, I doubt it would get much traction. Kind of hard to sell someone on syntactic sugar when they have to constantly wrap return values. > To get a better idea of the level of evidence you're trying to reach > if your suggestion is ever going to get anywhere, try taking a look at > http://www.boredomandlaziness.org/2011/02/justifying-python-language-changes.html > and http://www.boredomandlaziness.org/2011/02/status-quo-wins-stalemate.html. I agree that changes to syntax and commonly used modules that impact how people interface with them should be carefully vetted. Type decorations on homogeneous collections/iterators are effectively invisible in that perspective though; the main problem with them as I see it is that it involves touching a lot of code to implement, even if the actual implementation would be simple. I appreciate your feedback, Nathan From ethan at stoneleaf.us Mon Dec 19 15:10:09 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Dec 2011 06:10:09 -0800 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: <4EEF45C1.7030201@stoneleaf.us> Nathan Rice wrote: > To avoid handcuffing people with types (which is definitely un-pythonic) > and maintain backwards compatibility, the standard collection > modification methods could be hooked so that if an object of an > incorrect type is added, a warning is raised and the collection > gracefully degrades by removing mixed-in type information and > methods. -1 to automatic downgrading. Imagine in step C that the collection is auto-downgraded and later in step Q a method is called that no longer exists -- Exception raised. (I find a Warning less than useful -- either you aren't going to use one of the special methods, in which case you didn't need the special container and the warning is just noise, or you do need the special methods and you'll get an exception further on when you try to use it and it's no longer there.) > Additionally, a method could be provided that lets the user > "terminate the contract" causing the collection to degrade without a > warning. +1 Either it was not needed to begin with, or the need has been satisfied and now we need to add in objects with a different type Do you have some examples of functions where you pass in a container object and get the same type of object back, but it's a different object? ~Ethan~ From nathan.alexander.rice at gmail.com Mon Dec 19 16:47:05 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Mon, 19 Dec 2011 10:47:05 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: <4EEF45C1.7030201@stoneleaf.us> References: <4EEF45C1.7030201@stoneleaf.us> Message-ID: >> To avoid handcuffing people with types (which is definitely un-pythonic) >> and maintain backwards compatibility, the standard collection >> modification methods could be hooked so that if an object of an >> incorrect type is added, a warning is raised and the collection >> gracefully degrades by removing mixed-in type information and >> methods. > > > -1 to automatic downgrading. ?Imagine in step C that the collection is > auto-downgraded and later in step Q a method is called that no longer exists > -- Exception raised. ?(I find a Warning less than useful -- either you > aren't going to use one of the special methods, in which case you didn't > need the special container and the warning is just noise, or you do need the > special methods and you'll get an exception further on when you try to use > it and it's no longer there.) Personally, I like warnings. When my code passes tests, and I see a warning, often times it clues me in to an additional edge case or subtle bug. What are the other options? Throw an exception immediately, or don't downgrade at all, then throw an exception when someone tries to use a method that doesn't exist on a child? Throwing an exception immediately forces people to be explicit which is good, but it isn't backwards compatible. Not downgrading at all seems like it could be a source of annoyingly subtle bugs, since it is possible that you could put something in the list that fulfills part of the contract, but not all of it, or fulfills the contract in a subtly incorrect way. >> Additionally, a method could be provided that lets the user >> "terminate the contract" causing the collection to degrade without a >> warning. > > > +1 ?Either it was not needed to begin with, or the need has been satisfied > and now we need to add in objects with a different type > > > > Do you have some examples of functions where you pass in a container object > and get the same type of object back, but it's a different object? The restriction on being the same type is not needed. If a list is homogeneous, an iterator or set derived from the list would be homogeneous, and keeping that information in tact would probably be a good thing. With the slight adjustment, list, tuple, set, filter, iter (+most of itertools), sorted, reversed and heapq off the top of my head. Looking at the list, you could give type decorations legs just by having iterables of decorated objects retain the decoration, and having the basic collection types inherit type decorations at construction. From storchaka at gmail.com Mon Dec 19 18:12:03 2011 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 19 Dec 2011 19:12:03 +0200 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> Message-ID: 19.12.11 15:52, Nathan Rice ???????(??): >>> L2 = [X(e) for e in L1]>> L3 = [Y(e) for e in L2]>> vs>> L2 = X(L1) # assuming X has been updated to work in both vector/scalar>> L3 = Y(L2) # context...>>> L = ['a', 'bc', ['ada', 'a']]>> What is len(L)? 3 or [1, 2, 2] or [1, 2, [3, 1]]?>>>> L2 = [Z(Y(X(e))) for e in L1]>> vs>> L2 = Z(Y(X(L1)))>>>> L2 = [e.X().Y().Z() for e in L1]>> vs>> L2 = L1.X().Y().Z() # assuming vectorized versions of member methods>> #are folded into the collection via the mixin.>>> What is L.count('a')? 1 or [1, 0, 1] or [1, 0, [2, 1]]? > > A fair concern; if the vectorized version of the child method were > given the same name as the child method, I agree that this could > result in ambiguity. > > There are multiple ways that member methods could be made available on > the collection, including a proxy attribute, renaming, etc. len is not method, it is function (although it uses method __len__). There are many functions applicable to list and to its elements (in particular when members of list are lists). From nathan.alexander.rice at gmail.com Mon Dec 19 18:47:55 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Mon, 19 Dec 2011 12:47:55 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> Message-ID: >>>> L2 = [X(e) for e in L1]>> ?L3 = [Y(e) for e in L2]>> ?vs>> ?L2 = X(L1) # >>>> assuming X has been updated to work in both vector/scalar>> ?L3 = Y(L2) # >>>> context...>>> ?L = ['a', 'bc', ['ada', 'a']]>> ?What is len(L)? 3 or [1, 2, >>>> 2] or [1, 2, [3, 1]]?>>>> ?L2 = [Z(Y(X(e))) for e in L1]>> ?vs>> ?L2 = >>>> Z(Y(X(L1)))>>>> ?L2 = [e.X().Y().Z() for e in L1]>> ?vs>> ?L2 = >>>> L1.X().Y().Z() # assuming vectorized versions of member methods>> ?#are >>>> folded into the collection via the mixin.>>> ?What is L.count('a')? 1 or [1, >>>> 0, 1] or [1, 0, [2, 1]]? >> >> >> A fair concern; if the vectorized version of the child method were >> given the same name as the child method, I agree that this could >> result in ambiguity. >> >> There are multiple ways that member methods could be made available on >> the collection, including a proxy attribute, renaming, etc. > > > len is not method, it is function (although it uses method __len__). There > are many functions applicable to list and to its elements (in particular > when members of list are lists). I am aware of this. I was talking about the "collection inside a collection" example after that. Additionally, now I realize my initial reading of your example was too hasty. You violate the contract in the mixed list example with strings and lists, and that should rightfully not provide vectorized methods; providing type decorators for very abstract notions such as "iterable" is a bad idea imho. Of course, [[1, 2], [3, 4], [5, 6]] could serve as an alternate example of why you can't directly map child methods with the same names. From ethan at stoneleaf.us Mon Dec 19 19:01:06 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Dec 2011 10:01:06 -0800 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> Message-ID: <4EEF7BE2.4030203@stoneleaf.us> Nathan Rice wrote: > [[1, 2], [3, 4], [5, 6]] could serve as an alternate > example of why you can't directly map child methods with the same > names. If you have to change the names, doesn't that negate the ability of "a reliable method to make functions operate on both scalar and vector values"? ~Ethan~ From nathan.alexander.rice at gmail.com Mon Dec 19 20:26:34 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Mon, 19 Dec 2011 14:26:34 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: <4EEF7BE2.4030203@stoneleaf.us> References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> <4EEF7BE2.4030203@stoneleaf.us> Message-ID: On Mon, Dec 19, 2011 at 1:01 PM, Ethan Furman wrote: > Nathan Rice wrote: >> >> [[1, 2], [3, 4], [5, 6]] could serve as an alternate >> example of why you can't directly map child methods with the same >> names. > > > If you have to change the names, doesn't that negate the ability of "a > reliable method to make functions operate on both scalar and > vector values"? To clarify... Functions could reliably support both vector and scalar context, by checking that a type contract exists. Because of the name clash between parent and elementwise member methods when dealing with collections of collections (and some other classes), if the type contract provided "broadcast" versions of child methods, they would have to be provided with under an alias, perhaps X -> elementwise_X. I do not think people should be able to be oblivious of when they are calling collection and member methods, that encourages sloppy programming. I like the broadcast feature because it makes it easier to write code in a clear left to right narrative style, and that seems very pythonic to me compared with an "inside-out" style. From ethan at stoneleaf.us Mon Dec 19 21:09:25 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Dec 2011 12:09:25 -0800 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> <4EEF7BE2.4030203@stoneleaf.us> Message-ID: <4EEF99F5.9050300@stoneleaf.us> Nathan Rice wrote: > On Mon, Dec 19, 2011 at 1:01 PM, Ethan Furman wrote: >> Nathan Rice wrote: >>> [[1, 2], [3, 4], [5, 6]] could serve as an alternate >>> example of why you can't directly map child methods with the same >>> names. >> >> If you have to change the names, doesn't that negate the ability of "a >> reliable method to make functions operate on both scalar and >> vector values"? > > To clarify... > > Functions could reliably support both vector and scalar context, by > checking that a type contract exists. That sounds like type checking, which is not Pythonic. Duck-typing, on the other hand, *is*. def capitalize_me(some_obj): some_obj = some_obj.capitalize() --> capitalize_me("a simple string") "A simple string" --> capitalize_me(["a", "list", "of", "strings"]) ["A", "List", "Of", "Strings"] > Because of the name clash between parent and elementwise member > methods when dealing with collections of collections (and some other > classes), if the type contract provided "broadcast" versions of child > methods, they would have to be provided with under an alias, perhaps X > -> elementwise_X. I do not think people should be able to be > oblivious of when they are calling collection and member methods, that > encourages sloppy programming. I like the broadcast feature because > it makes it easier to write code in a clear left to right narrative > style, and that seems very pythonic to me compared with an > "inside-out" style. Duck-typing is neither oblivious, nor sloppy -- at least, not inherently. Is my sample code above not what you had in mind? ~Ethan~ From joshua.landau.ws at gmail.com Mon Dec 19 21:37:58 2011 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 19 Dec 2011 20:37:58 +0000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> <4EEF7BE2.4030203@stoneleaf.us> Message-ID: On 19 December 2011 19:26, Nathan Rice wrote: > On Mon, Dec 19, 2011 at 1:01 PM, Ethan Furman wrote: > > Nathan Rice wrote: > >> > >> [[1, 2], [3, 4], [5, 6]] could serve as an alternate > >> example of why you can't directly map child methods with the same > >> names. > > > > > > If you have to change the names, doesn't that negate the ability of "a > > reliable method to make functions operate on both scalar and > > vector values"? > > To clarify... > > Functions could reliably support both vector and scalar context, by > checking that a type contract exists. > > Because of the name clash between parent and elementwise member > methods when dealing with collections of collections (and some other > classes), if the type contract provided "broadcast" versions of child > methods, they would have to be provided with under an alias, perhaps X > -> elementwise_X. I do not think people should be able to be > oblivious of when they are calling collection and member methods, that > encourages sloppy programming. I like the broadcast feature because > it makes it easier to write code in a clear left to right narrative > style, and that seems very pythonic to me compared with an > "inside-out" style. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > ...or we could just extend Pep 225 (deferred) with "~." so we have "['a', 'b', 'c']~.upper()" [syntax debatable]. You don't get the type-checking but that seemed more of a problem to me, as I'm not a real duck. Your example: my_string.split("\n").capitalize().join_items("\n") to "\n".join(my_string.split("\n")~.capitalize()) And personally I like "string.join(lst)" over "lst.join_with(string)"... -- Joshua, pointlessly interjecting once again -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan.alexander.rice at gmail.com Mon Dec 19 23:28:09 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Mon, 19 Dec 2011 17:28:09 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> <4EEF7BE2.4030203@stoneleaf.us> Message-ID: >> Functions could reliably support both vector and scalar context, by>> checking that a type contract exists.>>> That sounds like type checking, which is not Pythonic. ?Duck-typing, on the> other hand, *is*. While you'll hear no argument from me on duck typing, in this instance that collides head-on with having clear names for elementwise versions of member methods. I personally would feel better erring on the side of clearer names at the cost of some kind of *very cursory* examination of what you are being passed. Exactly what form that examination might take is an open question that deserves further thought. > def capitalize_me(some_obj):> ? ?some_obj = some_obj.capitalize()>> --> capitalize_me("a simple string")> "A simple string">> --> capitalize_me(["a", "list", "of", "strings"])> ["A", "List", "Of", "Strings"] Though I don't want to comment on the definition of capitalize_me, in terms of function behavior across the string and list, that is pretty much what I am talking about. >> Because of the name clash between parent and elementwise member>> methods when dealing with collections of collections (and some other>> classes), if the type contract provided "broadcast" versions of child>> methods, they would have to be provided with under an alias, perhaps X>> -> elementwise_X. ?I do not think people should be able to be>> oblivious of when they are calling collection and member methods, that>> encourages sloppy programming. ?I like the broadcast feature because>> it makes it easier to write code in a clear left to right narrative>> style, and that seems very pythonic to me compared with an>> "inside-out" style.>>> Duck-typing is neither oblivious, nor sloppy -- at least, not inherently.> ?Is my sample code above not what you had in mind? I don't think duck typing is sloppy, with the caveat that the thing you expect to behave like a duck is some kind of animal, reasonably crafted animatronic robot, or operated puppet :). I feel like when you move from a single object to a collection that is basically like having a second function, which has been integrated in with the first function for the purposes of API simplicity. Then, it becomes an outer function that has the job of returning the correct inner function and executing it with the given input. --- On Mon, Dec 19, 2011 at 3:37 PM, Joshua Landau wrote: > ...or we could just extend Pep 225?(deferred)?with "~." so we have "['a', > 'b', 'c']~.upper()" [syntax debatable]. You don't get the type-checking but > that seemed more of a problem to me, as I'm not a real duck. That does run into some of the same problems w.r.t duck typing, and solutions to problems with overloaded operators are hard to google for. If the method calls were aliased, google searches would be easier and less experienced users wouldn't have to worry about wrestling with operator overloading. That being said, I think elementwise operators are a great idea in a numeric/scientific context. There are of course both pros and cons to having type declarations. I know that people do not like to be limited by a lack of foresight in their predecessors; I have run into unreasonable interface/type specifications and annoyances with private/protected variables in Java and I would never want to see that infiltrate Python. I think what I am trying to sell here is more akin to metadata than static types. People should be able to basically keep doing what they are already doing, but have a reasonable mechanism to provide additional object metadata in a standard way that is easily accessible by downstream consumers. That gives you a lot of the upsides of type declarations while staying Pythonic. Maybe a better option than having a "type contract" base for homogeneous collections is to give objects a metadata namespace with a loose schema, and declare it there? As long as you could provide the metadata at object creation time you would still get all the same benefits, and it would avoid potentially overworking the class/mixin concept. > Your example: > my_string.split("\n").capitalize().join_items("\n") > to > "\n".join(my_string.split("\n")~.capitalize()) > > And personally I like "string.join(lst)" over "lst.join_with(string)"... Of course, everyone has their own preferences; having a very small set of internally consistent ways to do something rather than just one way is good for this reason. Nathan From ncoghlan at gmail.com Tue Dec 20 00:07:55 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 Dec 2011 09:07:55 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On Tue, Dec 20, 2011 at 12:30 AM, Nathan Rice wrote: > 1. The "broadcasting" that people seemed to have latched on to is only > part of what I put forward, and I agree it is something that would > have to be done *correctly* to be beneficial. ?I would have no issues > with providing a userspace lib to do this if type decorations were > included in homogeneous collections/iterables, as long as the > implementation of the decoration didn't suffer from some form of > "string failure" (string subclasses are basically worthless as methods > return strings, not an instance of the class). I deliberately ignored everything else, because the broadcasting aspect is the only part that doesn't reek of "Python, why u no use static typing?" and hence the only part I find particularly interesting. If you reframe the typing ideas in terms of Abstract Base Classes and generation of typed proxies, the rest of it may become interesting. > 4. I wanted to gauge people's feelings before I went through the steps > involved in writing a PEP. ?I believe that is the right thing to do, > so I don't feel the "hand waving" comment is warranted. ?I've already > learned people view collections that provide child object methods in > vector form as a very big change even if it is backwards compatible; > that is fine, I'm willing to shelve that if consensus is that people > aren't comfortable with it. The "Hand waving" comment is absolutely warranted, because without code, we don't really know what you mean. > This may be the case, thus my request for input. ?I agree that a > thoughtful approach is prudent, however as I have stated, as it > currently stands the language would not be very supportive of an > add-on module that does this. ?With PyPy, I could probably get my > hooks in deeply enough that I could make something really useful, > however given CPython is the current flavor du jour, I doubt it would > get much traction. ?Kind of hard to sell someone on syntactic sugar > when they have to constantly wrap return values. Everything the numpy folks have done, they have done with the type system as it currently stands. It would be perfectly possible to create proxies for objects that implement the container ABCs that provide the type constraints that you describe, all without touching the language core or standard library. If you take full advantage of the power that descriptors and metaclasses offer, then you can do a *lot* with this concept as a PyPI module. As a specific suggestion, I'd advise exploring a "container.broadcast" descriptor on a mixin type that let you do things like: L2 = L1.broadcast(X, *args, **kwds) L3 = L2.broadcast(Y, *args, **kwds) L2 = L1.broadcast(X).broadcast(Y).broadcast(Z) L2 = L1.broadcast_chain(X)(Y)(Z).apply() L2 = L1.broadcast.X().broadcast.Y().broadcast.Z() L2 = L1.broadcast_chain.X().Y().Z().apply() my_string.split("\n").broadcast.capitalize().join_items("\n") > I agree that changes to syntax and commonly used modules that impact > how people interface with them should be carefully vetted. ?Type > decorations on homogeneous collections/iterators are effectively > invisible in that perspective though; the main problem with them as I > see it is that it involves touching a lot of code to implement, even > if the actual implementation would be simple. No, they're not invisible at all - they're a massive conceptual addition, on a similar scale to Abstract Base Classes themselves. Syntax matters for readability purposes, but it's the *semantics* that is at issue here (and also in many other PEPs). We're talking years of exploration and debate here, not something that can be resolved in a few weeks or months. There's zero chance of anything like this making it into 3.3., so think 3.4 at the earliest, and more likely 3.5 (and that's only if coded explorations of the concept prove useful in practice - it's entirely possible that the real outcome will be "never"). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Tue Dec 20 00:10:55 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 Dec 2011 18:10:55 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On 12/18/2011 9:24 PM, Nathan Rice wrote: >> Another bad example, since that's just a really verbose way of writing >> my_string.capitalize(). > > The python interpreter says otherwise... > >>>> foo = "line 1\nline 2\nline 3" >>>> foo.capitalize() > 'Line 1\nline 2\nline 3' >>>> "\n".join(s.capitalize() for s in foo.split("\n")) > 'Line 1\nLine 2\nLine 3' Using the right method (as Nick corrected himself): >>> foo.title() 'Line 1\nLine 2\nLine 3' -- Terry Jan Reedy From ncoghlan at gmail.com Tue Dec 20 00:26:49 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 Dec 2011 09:26:49 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On Tue, Dec 20, 2011 at 9:10 AM, Terry Reedy wrote: > On 12/18/2011 9:24 PM, Nathan Rice wrote: > >>> Another bad example, since that's just a really verbose way of writing >>> my_string.capitalize(). >> >> >> The python interpreter says otherwise... >> >>>>> foo = "line 1\nline 2\nline 3" >>>>> foo.capitalize() >> >> 'Line 1\nline 2\nline 3' >>>>> >>>>> "\n".join(s.capitalize() for s in foo.split("\n")) >> >> 'Line 1\nLine 2\nLine 3' > > > Using the right method (as Nick corrected himself): >>>> foo.title() > > 'Line 1\nLine 2\nLine 3' Nope, foo.title() isn't exactly the same - Nathan's variant capitalises the first letter of each line, foo.title() would capitalise the first letter of each word. Not a common operation, but split + capitalise + join is likely the simplest way to spell it. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From joshua.landau.ws at gmail.com Tue Dec 20 02:05:14 2011 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 20 Dec 2011 01:05:14 +0000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: <1ec1a14b-f521-4e01-b3f5-f60e1ed0bab9@q11g2000vbq.googlegroups.com> <5e760100-547c-4208-b47e-26277ec94943@z17g2000vbe.googlegroups.com> <4EEF7BE2.4030203@stoneleaf.us> Message-ID: On 19 December 2011 22:28, Nathan Rice wrote: > On Mon, Dec 19, 2011 at 3:37 PM, Joshua Landau > wrote: > > ...or we could just extend Pep 225 (deferred) with "~." so we have "['a', > > 'b', 'c']~.upper()" [syntax debatable]. You don't get the type-checking > but > > that seemed more of a problem to me, as I'm not a real duck. > > That does run into some of the same problems w.r.t duck typing, and > solutions to problems with overloaded operators are hard to google > for. If the method calls were aliased, google searches would be > easier and less experienced users wouldn't have to worry about > wrestling with operator overloading. The thing is, "listofstrings.method()" is less obviously elementwise than "listofstrings~.method()", *especially* if "~" becomes the default elementwise character. So I have to disagree. Additionally, "listofstrings~.method()" is easy to paste into an interpreter, and should quickly yield enlightenment. I don't see that as a valid argument. > That being said, I think > elementwise operators are a great idea in a numeric/scientific > context. > But not always in the same case as your broadcasting idea? The elementwise operator is far more general, so I'd say it has a more general use-case in response. Only a fraction of my code is of the "numeric/scientific" type, and I'd lap up elementwise operators. > There are of course both pros and cons to having type declarations. I > know that people do not like to be limited by a lack of foresight in > their predecessors; I have run into unreasonable interface/type > specifications and annoyances with private/protected variables in > Java and I would never want to see that infiltrate Python. I think > what I am trying to sell here is more akin to metadata than static > types. People should be able to basically keep doing what they are > already doing, but have a reasonable mechanism to provide additional > object metadata in a standard way that is easily accessible by > downstream consumers. That gives you a lot of the upsides of type > declarations while staying Pythonic. > The thing is... what? This isn't a criticism. I'm genuinely lost as to what you mean. I lost you on the "metadata" part. In what way is this metadata, and in turn not typing? Can you do: foo = "a b".split() foo[1] = 1 If you can, then this whole idea should really be generalised more, preferably to the elementwise operator level, as it's got no real type constraints. If you can't, then how is it not typing? Again, this is a genuine question. I'm home-taught, so please don't expect me to know the lingo :P. I actually love the idea of typing when done well. I've just started Haskell (quite a shock to my Python'd brain) and I've thoroughly enjoyed its type system. I just think that it has no place in python. It only works in Haskell because Haskell is built on it, and types can't go a-changin' (nor can values, for that matter). In python, this sort of system will mean that typed code/arrays/iterators will severely grind with untyped code/values/arrays/iterators. Someone defines a 'typed collection or iterator', to paraphrase the title. Someone else makes a goose-that-looks-like-a-duck, but doesn't have a beak. With an untyped system - such as the current method or an elementwise operator - that's fine as you just want it swim, but with your typed container something has to go wrong! Either it crashed, which is anti-duck-typing, or it falls back down, in which case you end up reverting everything to the old method anyway. Haskell gets around this by having both types and 'things that implement things' and so could do this because it knows what everything implements. But in python we don't know, and won't until all our dusty code is rewritten. I apologise for using a language I've just started to learn as a reference, but that's the best reference I've had. *I really should check this, but it's late and I'm tired. If I've said the same thing twice or rambled on about hexagonal motorised paintings for too long, blame my cat...* Maybe a better option than having a "type contract" base for > homogeneous collections is to give objects a metadata namespace with a > loose schema, and declare it there? As long as you could provide the > metadata at object creation time you would still get all the same > benefits, and it would avoid potentially overworking the class/mixin > concept. > > > Your example: > > my_string.split("\n").capitalize().join_items("\n") > > to > > "\n".join(my_string.split("\n")~.capitalize()) > > > > And personally I like "string.join(lst)" over "lst.join_with(string)"... > > Of course, everyone has their own preferences; having a very small set > of internally consistent ways to do something rather than just one way > is good for this reason. > > > Nathan > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > Before anyone gets too many ideas about what is and isn't: >>> str.capitalize("ABCDEFGH") 'Abcdefgh' Remember that these are predominantly lowercasing actions :P -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Dec 20 02:14:49 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 Dec 2011 20:14:49 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On 12/19/2011 9:30 AM, Nathan Rice wrote: > Couple things. > > 1. The "broadcasting" that people seemed to have latched on to is only > part of what I put forward, Perhaps because it is the most understandable. > and I agree it is something that would > have to be done *correctly* to be beneficial. I would have no issues > with providing a userspace lib to do this if type decorations were > included in homogeneous collections/iterables, The meaning of 'homogeneous' depends on the context -- the purpose and use of the collection. For some purposes -- str(o), len(c), o in c, c.index(o), and others, -- all objects, collections, or seqeuences *are* 'homogeneous' as instances or subclasses of 'object'. On the other hand, even [-1, 0, 1] is heterogeneous with respect to both sqrt and log, with the divide different for each. So I do not consider 'homogeneous' to be a property of collections as such. Python's current restricted-type mutable sequence factory is array.array. The types do not even have to be Python types, just machine storage types. The typecode is part of the object and exposed as an attribute. Such sequences cannot be 'degraded' because type-checking is done with all operations. It would not be difficult to make a TypedList class that did the same, either subclassing or wrapping list. What you have noticed is that iter(array(tc,init)) does not get the typecode information, so potentially useful information is lost. Your first concrete proposal might be that the information be kept and that arrayiterators get a type attribute corresponding to the Python type that the produced values are converted to. Also, array could expose the mapping to typecodes to Python types. These changes would allow experiments that would show the value of your basic idea. > as long as the > implementation of the decoration didn't suffer from some form of > "string failure" (string subclasses are basically worthless as methods > return strings, not an instance of the class). This problem is generic to subclassing built-in classes. List would be a better example here since strings already are specialized sequences. > 2. A "type decorator" on homogeneous collections and iterables has a > lot of nice little benefits throughout the toolchain. That is what you need to demonstrate, because it does not seem clear yet. What would you do with an arrayiterator with a type attribute. By the way, a 'decorator' in Python is a specific category of callable used in a specific way. Perhaps you mean 'type attribute'? > 3. Being able to place methods on a "type decorator" is useful, 'Placing methods' on an attribute or even a callable does not mean much. You can only concretely add methods to concrete classes, not abstract categories. > it solves issues like "foo".join() which really wants to be a method on > string collections. No it does not. 'String collection' is a category, not a class. Nor can it be a class without drastically revising Python. It is a category that cuts across all generic collection classes. So .join has to be a method of the joiner class. > 4. I wanted to gauge people's feelings before I went through the steps > involved in writing a PEP. I believe that is the right thing to do, > so I don't feel the "hand waving" comment is warranted. To the extent one does not understand what you say, and to the extent that it seems disconnected from concrete reality, it is easy to see it as hand waving. That you perhaps did not understand why .join is a string method points in that direction. > I've already > learned people view collections that provide child object methods in > vector form as a very big change Because we understand that non-method functions have virtues, and Python already has collection functions. > even if it is backwards compatible; that is fine. Backwards compatible duplication needs justification. ... > I agree that changes to syntax and commonly used modules that impact > how people interface with them should be carefully vetted. Type > decorations on homogeneous collections/iterators are effectively I am still not sure what you are really proposing. You may have the germ of a useful idea, but I think it needs clarification and a demonstration. > invisible in that perspective though; Slowdowns are not invisible. Requiring a type check on every addition to every built-in collection might result in such. > the main problem with them as I > see it is that it involves touching a lot of code to implement, even > if the actual implementation would be simple. Changes that touch a lot of code are fairly rare and require major benefits. One was the switch to new-style classes started in 2.2 and ended in 3.0. Several people contributed patches. They must have thought that unifying types and classes into one system was worth it. In 3.3, the two unicode implementations (one per build) are effectively combined with a third with a new C-level API. Adding and tweaking the new API (which continues today) and converting the entire C core and stdlib codebase to the new API has required something on the order of 50 patches over 3 months, so far. But it improves performance (overall) and removed the inherent bugs in representing 3-bytes chars with 2 2-byte chars and in having different Python builds respond differently to the same code. Note that the PEP concretely lays out the new C structures and API and that there was a prototype implementation showing benefits before it was approved. -- Terry Jan Reedy From pytom at yahoo.cn Tue Dec 20 04:07:20 2011 From: pytom at yahoo.cn (Tom Zhou) Date: Tue, 20 Dec 2011 11:07:20 +0800 (CST) Subject: [Python-ideas] Columnize in module "cmd" In-Reply-To: <1324349593.51908.YahooMailNeo@web92411.mail.cnh.yahoo.com> References: <1324349593.51908.YahooMailNeo@web92411.mail.cnh.yahoo.com> Message-ID: <1324350440.30993.YahooMailNeo@web92413.mail.cnh.yahoo.com> Hi~ alls recently, i focus on the module "cmd", and find some confused things-- the function named "columnize". Why we need a multiloop as "for nrows .. ? ? ? ? for col .. ? ? ? ? ? ? ? ? ? ?for row.." ?? i think we can make a easier method, for example, first, find out the maxlen str in list, and use its length as the standard size to format the list. Ok, maybe i ignore something, so please give me some hints. --- thanks tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Dec 20 10:25:41 2011 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 20 Dec 2011 11:25:41 +0200 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: 20.12.11 01:07, Nick Coghlan ???????(??): > L2 = L1.broadcast(X, *args, **kwds) > L3 = L2.broadcast(Y, *args, **kwds) > > L2 = L1.broadcast(X).broadcast(Y).broadcast(Z) > L2 = L1.broadcast_chain(X)(Y)(Z).apply() > > L2 = L1.broadcast.X().broadcast.Y().broadcast.Z() > L2 = L1.broadcast_chain.X().Y().Z().apply() > > my_string.split("\n").broadcast.capitalize().join_items("\n") from idontlikecomprehensions import broadcast, broadcast_chain L2 = broadcast(L1).X(*args, **kwds) L2 = broadcast_chain(L1).X().Y().Z().apply() From guido at python.org Tue Dec 20 16:14:38 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Dec 2011 08:14:38 -0700 Subject: [Python-ideas] Columnize in module "cmd" In-Reply-To: <1324350440.30993.YahooMailNeo@web92413.mail.cnh.yahoo.com> References: <1324349593.51908.YahooMailNeo@web92411.mail.cnh.yahoo.com> <1324350440.30993.YahooMailNeo@web92413.mail.cnh.yahoo.com> Message-ID: That code is trying to find an optimal solution where the columns may be of different width. Good luck! --Guido On Mon, Dec 19, 2011 at 8:07 PM, Tom Zhou wrote: > > Hi~ alls > recently, i focus on the module "cmd", and find some confused things-- > the function named "columnize". Why we need a multiloop as > "for nrows .. > for col .. > for row.." > ?? i think we can make a easier method, for example, first, find out the > maxlen str in list, and use its length as the standard size to format the > list. > Ok, maybe i ignore something, so please give me some hints. > > > --- > thanks > tom > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan.alexander.rice at gmail.com Tue Dec 20 19:50:23 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Tue, 20 Dec 2011 13:50:23 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: Nick: My main issue with anything you've said is that I felt your original statement about going off, creating a library, evangelizing it vigorously until it is popular and coming back in several years as a prerequisite to even engaging in a thoughtful discussion on the subject was overly dismissive and phrased in a combative manner. I do appreciate the strong challenges you've presented though. Terry: My apologies for being unclear. I do not take enough care in using terminology in the standard way. In my case, I think code is a better communication mechanism, so that is where I will be focusing most of my efforts moving forward. ***Broadcasting/Elementwise operations*** I agree code speaks more clearly than words: I put a module called "elementwise" on pypi (http://pypi.python.org/pypi/elementwise/0.111220) that implements my idea of what a nice broadcast proxy should do. The proxy itself could be leaner, I started out trying to make it in a very surgical manner, but 1.) There are serious dragons in the way python handles lookup of operator overloading special methods 2.) there are serious dragons in how python handles complex inheritance graphs that result in "object.__new__() takes no parameters", despite not having any builtin bases and having no base class overriding __new__ or __init__ 3.) Proxying EVERYTHING is pythonic in some sense. I am aware that there are probably issues around having some combination of special methods defined that will confuse the interpreter. The syntax is really simple: your_iterable_elementwise = ElementwiseProxy(your_iterable) your_iterable_elementwise.method_1().method_2().method_3() (your_iterable_elementwise + 1) * 2 + 10 The __iter__ method is how you break the cycle, so when you are done working in an elementwise manner, just call {list|set|tuple|...}(your_iterable_elementwise). In place operator modification should work as well. All operations on an ElementwiseProxy return another ElementwiseProxy with a parent attribute that you can use to backtrack in the operation history, if for some reason you were brought into the loop partway through. I also included an ElementwiseProxyMixin that gives you a ruby-esque class.each property for iterables, which is basically just returns a "ElementwiseProxy(self)". I think the syntax is nicer. ***I can haz moar Metadata (was "typed" collections)*** Because "typed" is sort of a dirty word in the python community, and some people have had issues with other terms I have used, I am going to settle on addressing things in term of the problem. The problem is metadata - languages with declared types have a lot of it, python has a lot less. Metadata is useful, and we should try to pack as much if it in as we can, because it lets us do things in a intelligent ways on a case by case basis. I understand annotations are designed for this, but they are absolutely pointless without some sort of schema or strong conventions. I see from the archived discussions that people wanted to avoid a design by committee situation, but people working in the tool-chain aren't going to mess with annotations unless there is one metadata standard that is pretty much ubiquitous (at which point it should be included in the stdlib anyhow). I think it is a decent idea to have it incubate outside the stdlib, but I don't see any progress on that front at all. With that being said, I would like to reopen the discussion about an metadata annotation schema, so we can get an incubation implementation going and begin evangelism. From tjreedy at udel.edu Wed Dec 21 00:01:26 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Dec 2011 18:01:26 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On 12/20/2011 1:50 PM, Nathan Rice wrote: > I put a module called "elementwise" on pypi > (http://pypi.python.org/pypi/elementwise/0.111220) that implements my > idea of what a nice broadcast proxy should do. I downloaded and took a brief look. I hope to get back to it later. 2.) there are serious dragons in > how python handles complex inheritance graphs that result in > "object.__new__() takes no parameters", despite not having any builtin > bases and having no base class overriding __new__ or __init__ Best not to use object as the apex of multiple inheritance. > Because "typed" is sort of a dirty word in the python community, Not exactly true, and unnecessarily combative. More true is that careless use of 'typed' has gotten tiresome. Python is strongly dynamically typed. But people occasionally post -- again the same day you posted to python list -- that Python is weakly typed. I am tired of explaining that 'typed' is not synonymous with 'statically typed'. Or consider your subject line. Python collections are typed both as to collection object and the contents. Python has narrow-content typed sequences. So just saying you want 'typed collections' does not say anything. We already have them. We even have propagation of narrow-content typing for operations on bytes and strings. But we do not have anything similar for numbers or user classes. And that might be worthwhile. So your subject seems more like 'adding generic narrowly typed sequences to Python'. -- Terry Jan Reedy From steve at pearwood.info Wed Dec 21 01:09:36 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 21 Dec 2011 11:09:36 +1100 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: <4EF123C0.2010002@pearwood.info> Terry Reedy wrote: > 2.) there are serious dragons in >> how python handles complex inheritance graphs that result in >> "object.__new__() takes no parameters", despite not having any builtin >> bases and having no base class overriding __new__ or __init__ > > Best not to use object as the apex of multiple inheritance. Is that possible in Python 3? >> Because "typed" is sort of a dirty word in the python community, > > Not exactly true, and unnecessarily combative. More true is that > careless use of 'typed' has gotten tiresome. Python is strongly > dynamically typed. But people occasionally post -- again the same day > you posted to python list -- that Python is weakly typed. I am tired of > explaining that 'typed' is not synonymous with 'statically typed'. +1 -- Steven From jeanpierreda at gmail.com Wed Dec 21 01:51:43 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 20 Dec 2011 19:51:43 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: > Not exactly true, and unnecessarily combative. More true is that careless > use of 'typed' has gotten tiresome. Python is strongly dynamically typed. > But people occasionally post -- again the same day you posted to python list > -- that Python is weakly typed. I am tired of explaining that 'typed' is not > synonymous with 'statically typed'. I don't find this much less careless. How do you differentiate between the "strong typing" of Python and the "strong typing" of Agda? It isn't a binary quantity. Perhaps, instead, we should stop claiming things are "strong" or "weak". If I said that, relatively speaking, Python is weakly typed, people would get offended -- not because I made any technically incorrect statement (on the spectrum, Python is far closer to assembly than Agda), but because to call it "weak" is insulting. -- Devin On Tue, Dec 20, 2011 at 6:01 PM, Terry Reedy wrote: > On 12/20/2011 1:50 PM, Nathan Rice wrote: > >> I put a module called "elementwise" on pypi >> (http://pypi.python.org/pypi/elementwise/0.111220) that implements my >> idea of what a nice broadcast proxy should do. > > > I downloaded and took a brief look. I hope to get back to it later. > > > 2.) there are serious dragons in >> >> how python handles complex inheritance graphs that result in >> "object.__new__() takes no parameters", despite not having any builtin >> bases and having no base class overriding __new__ or __init__ > > > Best not to use object as the apex of multiple inheritance. > > >> Because "typed" is sort of a dirty word in the python community, > > > Not exactly true, and unnecessarily combative. More true is that careless > use of 'typed' has gotten tiresome. Python is strongly dynamically typed. > But people occasionally post -- again the same day you posted to python list > -- that Python is weakly typed. I am tired of explaining that 'typed' is not > synonymous with 'statically typed'. > > Or consider your subject line. Python collections are typed both as to > collection object and the contents. Python has narrow-content typed > sequences. So just saying you want 'typed collections' does not say > anything. We already have them. We even have propagation of narrow-content > typing for operations on bytes and strings. > > But we do not have anything similar for numbers or user classes. And that > might be worthwhile. So your subject seems more like 'adding generic > narrowly typed sequences to Python'. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From tjreedy at udel.edu Wed Dec 21 01:59:29 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Dec 2011 19:59:29 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On 12/20/2011 7:51 PM, Devin Jeanpierre wrote: >> Not exactly true, and unnecessarily combative. More true is that careless >> use of 'typed' has gotten tiresome. Python is strongly dynamically typed. >> But people occasionally post -- again the same day you posted to python list >> -- that Python is weakly typed. I am tired of explaining that 'typed' is not >> synonymous with 'statically typed'. > > I don't find this much less careless. How do you differentiate between > the "strong typing" of Python and the "strong typing" of Agda? It > isn't a binary quantity. > > Perhaps, instead, we should stop claiming things are "strong" or > "weak". If I said that, relatively speaking, Python is weakly typed, > people would get offended -- not because I made any technically > incorrect statement (on the spectrum, Python is far closer to assembly > than Agda), but because to call it "weak" is insulting. If you are going to use term idiosyncratically, then consider giving you definition along with it. See https://en.wikipedia.org/wiki/Strongly_typed for a common usage, by which Python is strongly typed. -- Terry Jan Reedy From tjreedy at udel.edu Wed Dec 21 02:09:12 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Dec 2011 20:09:12 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: <4EF123C0.2010002@pearwood.info> References: <4EF123C0.2010002@pearwood.info> Message-ID: On 12/20/2011 7:09 PM, Steven D'Aprano wrote: > Terry Reedy wrote: > >> 2.) there are serious dragons in >>> how python handles complex inheritance graphs that result in >>> "object.__new__() takes no parameters", despite not having any builtin >>> bases and having no base class overriding __new__ or __init__ >> >> Best not to use object as the apex of multiple inheritance. > > Is that possible in Python 3? I believe the recommendation I have seen is something like this >>> class C(): def __init__(self, *arg, **kwds): pass >>> C(1) <__main__.C object at 0x00000000034C0F98> whereas >>> object(1) Traceback (most recent call last): File "", line 1, in object(1) TypeError: object.__new__() takes no parameters Now, so others have posted, define A and B with super, inheriting from C instead of object, and D with super inheriting from A and B, and all should go well instead of crashing. But I have not tested this myself. Same for ordinary single inheritance chain. -- Terry Jan Reedy From ncoghlan at gmail.com Wed Dec 21 02:14:52 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Dec 2011 11:14:52 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: On Wed, Dec 21, 2011 at 10:51 AM, Devin Jeanpierre wrote: > Perhaps, instead, we should stop claiming things are "strong" or > "weak". If I said that, relatively speaking, Python is weakly typed, > people would get offended -- not because I made any technically > incorrect statement (on the spectrum, Python is far closer to assembly > than Agda), but because to call it "weak" is insulting. When you can mutate a str object into an int object (or vice-versa), then you can claim Python is weakly typed without being technically incorrect. Weak typing has a very specific meaning: objects can change their type without changing their identity (e.g. via pointer casting in C and C++). Python lets objects *lie* about their types to some degree (by altering __class__), but type(obj) will always reveal the true underling type (indeed, "obj.__class__ != type(obj)" is one of the ways to detect when you've been given a proxy object, if the distinction matters for a particular use case). (For CPython, extension module authors can actually use C code to get around the strong typing if they really try, but the authors of such code get no sympathy when it inevitably blows up in obscure and hard to debug ways. Old-style classes in 2.x can also be legitimately described as weakly typed, since *all* instances of such classes share a single underlying type, and __class__ is the true determinant of their behaviour) Weak vs strong typing and dynamic vs static typing are well-defined concepts - it's just all too common that folks that initially learn to program with a static language confuse the two spectra and think that "static typing" and "strong typing" are the same thing. They're not only not the same, they're actually completely orthogonal. CPython, for example, uses the weak static typing of C to implement Python's strong dynamic typing mechanisms. IronPython and Jython have strong typing at both levels, but retain the static vs dynamic split. I believe PyPy uses strong dynamic typing throughout (although RPython has a type inference mechanism and a few other tricks to support translation to machine code) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Dec 21 02:38:33 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Dec 2011 11:38:33 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: I realised I gave a quick definition of strong vs weak typing, but not dynamic vs static. Here's explanations for all four: Strong typing: an object's type is immutable. To change the type, you must change the object's identity (i.e. create a new object). vs Weak typing: an object's type (and hence its behaviour) can be changed while leaving its identity untouched. Python allows weak typing at the __class__ level (to support proxy objects and similar metaprogramming tools), but the underlying object model of the language is strongly typed. Static typing: types are assigned not only to objects, but also to labels that refer to objects. The type of the label and the type of the object must match. Accordingly, variables must be explicitly associated with a type via variable declarations. vs Dynamic typing: types are assigned only to objects, and labels themselves are untyped. Accordingly, variables can be defined implicitly just by assigning a value to them. (Some otherwise statically typed languages include explicit support for dynamically typed references) (An interest hybrid variant for static vs dynamic is an approach where labels *are* typed, but they acquire their type from the first value assigned to them. Automatic type inferencing can make static typing significantly less painful to work with, while still picking up most type errors at compile time. C++11 has gone that way with its introduction of "auto" type declarations for initialised types, where you explicitly tell the compiler "this variable is of the same type as the result of the initialiser"). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jeanpierreda at gmail.com Wed Dec 21 02:41:24 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 20 Dec 2011 20:41:24 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: > If you are going to use term idiosyncratically, then consider giving you > definition along with it. See > https://en.wikipedia.org/wiki/Strongly_typed > for a common usage, by which Python is strongly typed. The list of "strongly typed" languages is prefixed with the following warning: > Note that some of these definitions are contradictory, others are merely > orthogonal, and still others are special cases (with additional constraints) > of other, more "liberal" (less strong) definitions. Because of the wide > divergence among these definitions, it is possible to defend claims about > most programming languages that they are either strongly or weakly typed. That is the point I was trying to make. -- Devin On Tue, Dec 20, 2011 at 7:59 PM, Terry Reedy wrote: > On 12/20/2011 7:51 PM, Devin Jeanpierre wrote: >>> >>> Not exactly true, and unnecessarily combative. More true is that careless >>> use of 'typed' has gotten tiresome. Python is strongly dynamically typed. >>> But people occasionally post -- again the same day you posted to python >>> list >>> -- that Python is weakly typed. I am tired of explaining that 'typed' is >>> not >>> synonymous with 'statically typed'. >> >> >> I don't find this much less careless. How do you differentiate between >> the "strong typing" of Python and the "strong typing" of Agda? It >> isn't a binary quantity. >> >> Perhaps, instead, we should stop claiming things are "strong" or >> "weak". If I said that, relatively speaking, Python is weakly typed, >> people would get offended -- not because I made any technically >> incorrect statement (on the spectrum, Python is far closer to assembly >> than Agda), but because to call it "weak" is insulting. > > > If you are going to use term idiosyncratically, then consider giving you > definition along with it. See > https://en.wikipedia.org/wiki/Strongly_typed > for a common usage, by which Python is strongly typed. > > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From jeanpierreda at gmail.com Wed Dec 21 02:46:43 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 20 Dec 2011 20:46:43 -0500 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: Hm, I made a lie. That isn't the point I was trying to make. Or rather, I made two points at once originally, and the second one contradicted the first one :) (I stated that strong vs weak is dumb, and then I said Python was weakly typed. I meant to say "in the sense that Agda is strong", but I guess I screwed that one up) *sigh*. It's perfectly possible to say that Python is strongly typed, with a self-consistent definition of "strongly typed". It's also possible to say that it's weakly typed, with another self-consistent definition. There is no _standard_ definition. The definition that Agda uses is not the definition that Python uses. I think people tend to choose the one that is least likely to call their favorite language "weak". Devin On Tue, Dec 20, 2011 at 8:41 PM, Devin Jeanpierre wrote: >> If you are going to use term idiosyncratically, then consider giving you >> definition along with it. See >> https://en.wikipedia.org/wiki/Strongly_typed >> for a common usage, by which Python is strongly typed. > > The list of "strongly typed" languages is prefixed with the following warning: > >> Note that some of these definitions are contradictory, others are merely >> orthogonal, and still others are special cases (with additional constraints) >> of other, more "liberal" (less strong) definitions. Because of the wide >> divergence among these definitions, it is possible to defend claims about >> most programming languages that they are either strongly or weakly typed. > > That is the point I was trying to make. > > -- Devin > > On Tue, Dec 20, 2011 at 7:59 PM, Terry Reedy wrote: >> On 12/20/2011 7:51 PM, Devin Jeanpierre wrote: >>>> >>>> Not exactly true, and unnecessarily combative. More true is that careless >>>> use of 'typed' has gotten tiresome. Python is strongly dynamically typed. >>>> But people occasionally post -- again the same day you posted to python >>>> list >>>> -- that Python is weakly typed. I am tired of explaining that 'typed' is >>>> not >>>> synonymous with 'statically typed'. >>> >>> >>> I don't find this much less careless. How do you differentiate between >>> the "strong typing" of Python and the "strong typing" of Agda? It >>> isn't a binary quantity. >>> >>> Perhaps, instead, we should stop claiming things are "strong" or >>> "weak". If I said that, relatively speaking, Python is weakly typed, >>> people would get offended -- not because I made any technically >>> incorrect statement (on the spectrum, Python is far closer to assembly >>> than Agda), but because to call it "weak" is insulting. >> >> >> If you are going to use term idiosyncratically, then consider giving you >> definition along with it. See >> https://en.wikipedia.org/wiki/Strongly_typed >> for a common usage, by which Python is strongly typed. >> >> >> -- >> Terry Jan Reedy >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas From pytom at yahoo.cn Wed Dec 21 04:28:54 2011 From: pytom at yahoo.cn (Tom Zhou) Date: Wed, 21 Dec 2011 11:28:54 +0800 (CST) Subject: [Python-ideas] =?utf-8?b?5Zue5aSN77yaICBDb2x1bW5pemUgaW4gbW9kdWxl?= =?utf-8?b?ICJjbWQi?= In-Reply-To: References: <1324349593.51908.YahooMailNeo@web92411.mail.cnh.yahoo.com> <1324350440.30993.YahooMailNeo@web92413.mail.cnh.yahoo.com> Message-ID: <1324438134.11446.YahooMailNeo@web92408.mail.cnh.yahoo.com> Thanks ?a lot for your reply!? I've reservations, ?maybe code looks like:? """ colwidth = max(map(len, list)) ncols = displaywidth/(colwidth+2)+1 nrows = size/ncols+1 for nrows ? ? ?for ncols ? ? ? ? ? texts = list[...] ? ? ? ? ? output texts """ is clear, and just waste some space of stdout. --- Best Regards tom ________________________________ ???? Guido van Rossum ???? Tom Zhou ??? "python-ideas at python.org" ????? 2011?12?20?, ???, ?? 7:14 ??: Re: [Python-ideas] Columnize in module "cmd" That code is trying to find an optimal solution where the columns may be of different width. Good luck! --Guido On Mon, Dec 19, 2011 at 8:07 PM, Tom Zhou wrote: > >Hi~ alls > >recently, i focus on the module "cmd", and find some confused things-- the function named "columnize". Why we need a multiloop as >"for nrows .. >? ? ? ? for col .. >? ? ? ? ? ? ? ? ? ?for row.." >?? i think we can make a easier method, for example, first, find out the maxlen str in list, and use its length as the standard size to format the list. >Ok, maybe i ignore something, so please give me some hints. > > > > >--- >thankstom > > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >http://mail.python.org/mailman/listinfo/python-ideas > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Dec 21 12:16:19 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 21 Dec 2011 22:16:19 +1100 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: References: Message-ID: <4EF1C003.4000208@pearwood.info> Nick Coghlan wrote: > Weak vs strong typing and dynamic vs static typing are well-defined > concepts - it's just all too common that folks that initially learn to > program with a static language confuse the two spectra and think that > "static typing" and "strong typing" are the same thing. [...] While I like the definitions Nick has given, I think he's a tad optimistic to claim that the various foo-typing are "well-defined". I think that weak and strong typing aren't dichotomies, but extremes in a continuum. Most languages include elements of both weak and strong typing, particularly coercion of ints to floats. Chris Smith's influential article "What To Know Before Debating Type Systems" goes further, suggesting that weak and strong typing are meaningless terms. I don't go that far, but you should read his article: http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ See also http://en.wikipedia.org/wiki/Type_system -- Steven From ncoghlan at gmail.com Wed Dec 21 15:04:02 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Dec 2011 00:04:02 +1000 Subject: [Python-ideas] Adding "Typed" collections/iterators to Python In-Reply-To: <4EF1C003.4000208@pearwood.info> References: <4EF1C003.4000208@pearwood.info> Message-ID: On Wed, Dec 21, 2011 at 9:16 PM, Steven D'Aprano wrote: > I think that weak and strong typing aren't dichotomies, but extremes in a > continuum. Most languages include elements of both weak and strong typing, > particularly coercion of ints to floats. > > Chris Smith's influential article "What To Know Before Debating Type > Systems" goes further, suggesting that weak and strong typing are > meaningless terms. I don't go that far, but you should read his article: Yeah, I'd only ever encountered weak/strong under the meanings I gave, but the Wikipedia article Terry linked was eye-opening (and does indeed suggest that if you're going to use weak/strong as terms, it's necessary to define them in situ). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jxo6948 at rit.edu Thu Dec 22 00:06:51 2011 From: jxo6948 at rit.edu (John O'Connor) Date: Wed, 21 Dec 2011 18:06:51 -0500 Subject: [Python-ideas] defaultattrgetter Message-ID: Pythons: What are your thoughts on the concept of a `defaultattrgetter`? It would be to operator.attrgetter what getattr(foo, x, default) is to getattr(foo, x). I dont like that attrgetter requires the attribute to exist or else the getter be wrapped in a try catch with respect to how defaultdict solves roughly the same problem for dictionary keys. The semantics could be something such as: from operator import defaultattrgetter _x = defaultattrgetter({'x': 0}) _y = defaultattrgetter({'y': 1}) - or - _xy = defaultattrgetter({'x': 0, 'y': 1}) One use case I am thinking of is functions that may be decorated with attributes x and/or y. Obviously a python implementation of defaultattrgetter would be trivial to implement but one of the benefits of these functions is the speed. It also seems like it would fit in well with the rest of the operator module. Generally speaking, would anyone else have a use for this? - John From steve at pearwood.info Thu Dec 22 00:33:00 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 22 Dec 2011 10:33:00 +1100 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: Message-ID: <4EF26CAC.2030602@pearwood.info> John O'Connor wrote: > Pythons: > > What are your thoughts on the concept of a `defaultattrgetter`? Why create a new function for it? Why not just give attrgetter a keyword only argument default? -- Steven From jxo6948 at rit.edu Thu Dec 22 08:05:01 2011 From: jxo6948 at rit.edu (John O'Connor) Date: Thu, 22 Dec 2011 02:05:01 -0500 Subject: [Python-ideas] defaultattrgetter In-Reply-To: <4EF26CAC.2030602@pearwood.info> References: <4EF26CAC.2030602@pearwood.info> Message-ID: On Wed, Dec 21, 2011 at 6:33 PM, Steven D'Aprano wrote: > Why create a new function for it? Why not just give attrgetter a keyword > only argument default? I need to revise my example. It should be: _x = defaultattrgetter(('x', 0)) _xy = defaultattrgetter(('x', 0), ('y', 1)) Which is what I had originally but I was too quick to change it in the course of writing thinking the dict notation looked cleaner. But, since the argument order matters a dict wont work. The same applies to using keyword arguments. I'm not sure if there is a clean way to add this type of functionally to attrgetter without strings being a special case. From stefan_ml at behnel.de Thu Dec 22 08:23:03 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 22 Dec 2011 08:23:03 +0100 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: John O'Connor, 22.12.2011 08:05: > On Wed, Dec 21, 2011 at 6:33 PM, Steven D'Aprano wrote: >> Why create a new function for it? Why not just give attrgetter a keyword >> only argument default? > > I need to revise my example. It should be: > _x = defaultattrgetter(('x', 0)) > _xy = defaultattrgetter(('x', 0), ('y', 1)) > > Which is what I had originally but I was too quick to change it in the > course of writing thinking the dict notation looked cleaner. But, > since the argument order matters a dict wont work. The same applies to > using keyword arguments. I'm not sure if there is a clean way to add > this type of functionally to attrgetter without strings being a > special case. I don't consider it a major use case to be able to use different default return values for different steps in the lookup process. If you want that, write your own lookup function, that's trivial enough. If such a feature gets added (which would be for Python 3.3 or later), I second Steven's proposal of making it a keyword argument, i.e. lookup_a_b_c = operator.attrgetter('a', 'b', 'c', default=123) Stefan From masklinn at masklinn.net Thu Dec 22 08:57:30 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 22 Dec 2011 08:57:30 +0100 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: On 2011-12-22, at 08:23 , Stefan Behnel wrote: > John O'Connor, 22.12.2011 08:05: >> On Wed, Dec 21, 2011 at 6:33 PM, Steven D'Aprano wrote: >>> Why create a new function for it? Why not just give attrgetter a keyword >>> only argument default? >> >> I need to revise my example. It should be: >> _x = defaultattrgetter(('x', 0)) >> _xy = defaultattrgetter(('x', 0), ('y', 1)) >> >> Which is what I had originally but I was too quick to change it in the >> course of writing thinking the dict notation looked cleaner. But, >> since the argument order matters a dict wont work. The same applies to >> using keyword arguments. I'm not sure if there is a clean way to add >> this type of functionally to attrgetter without strings being a >> special case. > > I don't consider it a major use case to be able to use different default return values for different steps in the lookup process. If you want that, write your own lookup function, that's trivial enough. > > If such a feature gets added (which would be for Python 3.3 or later), I second Steven's proposal of making it a keyword argument, i.e. > > lookup_a_b_c = operator.attrgetter('a', 'b', 'c', default=123) The problem with that is ? does the default value apply to all three attributes? What if you need a default value for one but not the others, or different values for all three? Does this behavior really make sense? Maybe a dict of default values, at least when extracting more than one attribute? From stefan_ml at behnel.de Thu Dec 22 09:09:28 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 22 Dec 2011 09:09:28 +0100 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: Masklinn, 22.12.2011 08:57: > On 2011-12-22, at 08:23 , Stefan Behnel wrote: >> John O'Connor, 22.12.2011 08:05: >>> On Wed, Dec 21, 2011 at 6:33 PM, Steven D'Aprano wrote: >>>> Why create a new function for it? Why not just give attrgetter a keyword >>>> only argument default? >>> >>> I need to revise my example. It should be: >>> _x = defaultattrgetter(('x', 0)) >>> _xy = defaultattrgetter(('x', 0), ('y', 1)) >>> >>> Which is what I had originally but I was too quick to change it in the >>> course of writing thinking the dict notation looked cleaner. But, >>> since the argument order matters a dict wont work. The same applies to >>> using keyword arguments. I'm not sure if there is a clean way to add >>> this type of functionally to attrgetter without strings being a >>> special case. >> >> I don't consider it a major use case to be able to use different default return values for different steps in the lookup process. If you want that, write your own lookup function, that's trivial enough. >> >> If such a feature gets added (which would be for Python 3.3 or later), I second Steven's proposal of making it a keyword argument, i.e. >> >> lookup_a_b_c = operator.attrgetter('a', 'b', 'c', default=123) > The problem with that is ? does the default value apply to all three attributes? That's what I meant: it should hit the most common case and leave the rest to the user to handle. > What if you need a default value for one but not the others, or different values for all three? Does this behavior really make sense? What would be the use case? Would you want the lookup process to continue on the default argument if an intermediate attribute is not found? And if a subsequent lookup fails? Take the corresponding default value again and keep looking up on it? That feels rather unpredictable from a users' point of view. I think the idea is that you ask for an attribute, which (in the less common cases) happens to be a multi-step lookup, and if the attribute is not found, you want it to return a default value *for the attribute you requested*, i.e. not a different value for any of the intermediate attributes, only a specific value that corresponds to the last attribute in the lookup chain. In the 99.9% case, that will be something like None or a "keep going, I don't care" value, not something that depends on the lookup path in any way. I don't think the 0.1% case where you want more than that is worth a substantially more complicated API. Stefan From masklinn at masklinn.net Thu Dec 22 09:25:05 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 22 Dec 2011 09:25:05 +0100 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: On 2011-12-22, at 09:09 , Stefan Behnel wrote: > >> What if you need a default value for one but not the others, or different values for all three? Does this behavior really make sense? > > What would be the use case? Would you want the lookup process to continue on the default argument if an intermediate attribute is not found? And if a subsequent lookup fails? Take the corresponding default value again and keep looking up on it? That feels rather unpredictable from a users' point of view. A multiple-*args attrgetter fetches different arguments, for a "lookup process" you need a dotted path (attrgetter('foo.bar.baz')). For that one a default value returned after any lookup failure makes sense, but it's not what attrgetter('foo', 'bar', 'baz') does: >>> from collections import namedtuple >>> from operator import attrgetter >>> r = namedtuple('T', 'a b c d e')(1, 2, 3, 4, 5) >>> attrgetter('a.__index__')(r) >>> attrgetter('a', 'b', 'e')(r) (1, 2, 5) And for the second case, you'd need to be able to specify a default value for each of a, b and e. > I think the idea is that you ask for an attribute, which (in the less common cases) happens to be a multi-step lookup, and if the attribute is not found, you want it to return a default value *for the attribute you requested*, i.e. not a different value for any of the intermediate attributes Right, we're talking about different things and as noted above I think you're figuring the behavior of attrgetter(arg0, arg1, ?) wrong. > I don't think the 0.1% case where you want more than that is worth a substantially more complicated API. Thing is, I don't know about others but my main use of attrgetter *by far* is to fetch multiple attributes simultanously. When fetching a single attribute (even with multiple steps), it's often (if not always) shorter to use a lambda with an attribute lookup. From stefan_ml at behnel.de Thu Dec 22 09:35:42 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 22 Dec 2011 09:35:42 +0100 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: Masklinn, 22.12.2011 09:25: > On 2011-12-22, at 09:09 , Stefan Behnel wrote: >> >>> What if you need a default value for one but not the others, or different values for all three? Does this behavior really make sense? >> >> What would be the use case? Would you want the lookup process to continue on the default argument if an intermediate attribute is not found? And if a subsequent lookup fails? Take the corresponding default value again and keep looking up on it? That feels rather unpredictable from a users' point of view. > A multiple-*args attrgetter fetches different arguments, for a "lookup process" you need a dotted path (attrgetter('foo.bar.baz')). Ah, thanks, I remembered that incorrectly then. In that case, I vote -1 for adding the feature in the first place, because it would complicate an otherwise simple API too much and the current syntax proposals are too verbose and too hard to read. Stefan From __peter__ at web.de Thu Dec 22 10:27:49 2011 From: __peter__ at web.de (Peter Otten) Date: Thu, 22 Dec 2011 10:27:49 +0100 Subject: [Python-ideas] defaultattrgetter References: <4EF26CAC.2030602@pearwood.info> Message-ID: Stefan Behnel wrote: > Masklinn, 22.12.2011 09:25: >> On 2011-12-22, at 09:09 , Stefan Behnel wrote: >>> >>>> What if you need a default value for one but not the others, or >>>> different values for all three? Does this behavior really make sense? >>> >>> What would be the use case? Would you want the lookup process to >>> continue on the default argument if an intermediate attribute is not >>> found? And if a subsequent lookup fails? Take the corresponding default >>> value again and keep looking up on it? That feels rather unpredictable >>> from a users' point of view. >> A multiple-*args attrgetter fetches different arguments, for a "lookup >> process" you need a dotted path (attrgetter('foo.bar.baz')). > > Ah, thanks, I remembered that incorrectly then. > > In that case, I vote -1 for adding the feature in the first place, because > it would complicate an otherwise simple API too much and the current > syntax proposals are too verbose and too hard to read. I still like the addition, I think >>> attrgetter("last_name", default="unknown")(None) 'unknown' would be the most common use case by far and >>> location = None >>> attrgetter("x", "y", "z", default=0)(location) (0, 0, 0) >>> attrgetter("x.unit", default=None)(location) None # don't know if there's no location.x or no location.x.unit are useful and easy enough to grasp. From jxo6948 at rit.edu Thu Dec 22 10:55:20 2011 From: jxo6948 at rit.edu (John O'Connor) Date: Thu, 22 Dec 2011 04:55:20 -0500 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: > I think the idea is that you ask for an attribute, which (in the less common > cases) happens to be a multi-step lookup, and if the attribute is not found, > you want it to return a default value *for the attribute you requested*, > i.e. not a different value for any of the intermediate attributes, only a > specific value that corresponds to the last attribute in the lookup chain. > In the 99.9% case, that will be something like None or a "keep going, I > don't care" value, not something that depends on the lookup path in any way. > > I don't think the 0.1% case where you want more than that is worth a > substantially more complicated API. > I could settle for the default= but I think it is too simple and thus incomplete. I think it is extreme to say that one default argument covers 99.9% use case. I do think you are right about the ignored path but that path may use more than one condition. _xy = defaultattrgetter(('x', True), ('y', False)) x, y = _xy(foo) # usually do x unless told otherwise if x: ... # usually dont do y if y: ... I guess one alternative could be: _xy = attrgetter('x', 'y', defaults={'x': True, 'y': False}) but that just looks like DRY without the D. From arnodel at gmail.com Thu Dec 22 11:21:46 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Thu, 22 Dec 2011 10:21:46 +0000 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: On Dec 22, 2011 9:56 AM, "John O'Connor" wrote: > > > I think the idea is that you ask for an attribute, which (in the less common > > cases) happens to be a multi-step lookup, and if the attribute is not found, > > you want it to return a default value *for the attribute you requested*, > > i.e. not a different value for any of the intermediate attributes, only a > > specific value that corresponds to the last attribute in the lookup chain. > > In the 99.9% case, that will be something like None or a "keep going, I > > don't care" value, not something that depends on the lookup path in any way. > > > > I don't think the 0.1% case where you want more than that is worth a > > substantially more complicated API. > > > > I could settle for the default= but I think it is too simple and thus > incomplete. I think it is extreme to say that one default argument > covers 99.9% use case. I do think you are right about the ignored path > but that path may use more than one condition. > > _xy = defaultattrgetter(('x', True), ('y', False)) > x, y = _xy(foo) > # usually do x unless told otherwise > if x: ... > # usually dont do y > if y: ... > > > I guess one alternative could be: > _xy = attrgetter('x', 'y', defaults={'x': True, 'y': False}) > > but that just looks like DRY without the D. You could have attrgetter('x', 'y', defaults=(True, False)) to put the D back in. Arnaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Dec 22 11:32:17 2011 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 22 Dec 2011 12:32:17 +0200 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: 22.12.11 11:55, John O'Connor ???????(??): > I do think you are right about the ignored path > but that path may use more than one condition. > > _xy = defaultattrgetter(('x', True), ('y', False)) > x, y = _xy(foo) > # usually do x unless told otherwise > if x: ... > # usually dont do y > if y: ... > > > I guess one alternative could be: > _xy = attrgetter('x', 'y', defaults={'x': True, 'y': False}) Or _xy = attrgetter('x', 'y', defaults=(True, False)) Sometimes, however, will be better to specify not value, but factory (when value is list, dictionary, or complex object), as for defauldict. _xyz = defaultattrgetter('x', ('y', int, 123), ('z', list)) x has not default value, y has default value int(123), z has default value list(). From ncoghlan at gmail.com Thu Dec 22 13:14:12 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Dec 2011 22:14:12 +1000 Subject: [Python-ideas] defaultattrgetter In-Reply-To: References: <4EF26CAC.2030602@pearwood.info> Message-ID: Once again, please don't get so enamoured of higher order functions that you miss the obvious solution: *just write a new function that does exactly what you want*. "Swiss Army APIs" are not a good thing. Sometimes they're fairly unavoidable because they're exposing a complex underlying operation with a lot of moving parts (e.g. subprocess.Popen), but other times they're useless cruft that is so hard to remember that most people never bother with them, as just writing the custom function is significantly easier. So, for the 3 examples given: def _x(arg): return getattr(arg, 'x', 0) def _y(arg): return getattr(arg, 'y', 1) def _xy(arg): return _x(arg), _y(arg) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.rodola at gmail.com Tue Dec 27 18:57:29 2011 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Tue, 27 Dec 2011 18:57:29 +0100 Subject: [Python-ideas] Should stdlib provide something to properly handle SIGINT/SIGTERM signals? Message-ID: Since this is a problem which occurs very often, I'd like to hear your opinion as to whether something like this might find some place in the python stdlib (signal module maybe?). Please read the discussion included in this recipe: http://code.activestate.com/recipes/577997/ It should provide a description of the problem and the general use case. Thanks in advance for your comments, --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ From techtonik at gmail.com Tue Dec 27 18:01:50 2011 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 27 Dec 2011 09:01:50 -0800 (PST) Subject: [Python-ideas] Idea: Google Groups web interface for Ideas Message-ID: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> As you may know, the python-ideas list is opened only to subscribers. This is inconvenient, because: 1. it requires three step subscription process 2. it is impossible to post a reply to existing thread/idea There is a web-interface in Google Groups at https://groups.google.com/forum/#!forum/python-ideas that can solve problems above and provide some more nifty features such as embedded search. But there comes another problem that messages posted through the group doesn't end in list, because list requires subscription. I've already tried to find a solution, but run out of time, so I summarized the proposal at http://wiki.python.org/moin/MailmanWithGoogleGroups I may or may not be able to publish outcomes of my research, so it would be nice to get some help in investigating the problem and publishing a solution on aforementioned wiki page. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Dec 27 22:52:49 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Dec 2011 16:52:49 -0500 Subject: [Python-ideas] Idea: Google Groups web interface for Ideas In-Reply-To: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> Message-ID: On 12/27/2011 12:01 PM, anatoly techtonik wrote: > As you may know, the python-ideas list is opened only to subscribers. > This is inconvenient, because: > 1. it requires three step subscription process > 2. it is impossible to post a reply to existing thread/idea > > There is a web-interface in Google Groups at > https://groups.google.com/forum/#!forum/python-ideas > that can solve > problems above and provide some more nifty features such as embedded > search. But there comes another problem that messages posted through the > group doesn't end in list, because list requires subscription. I've > already tried to find a solution, but run out of time, so I summarized > the proposal at http://wiki.python.org/moin/MailmanWithGoogleGroups I interact with this list as newsgroup gmane.comp.python.ideas via news.gmane.org. The first post, which can be (and usually should be) a reply to an existing post, requires that one respond to an email to the from address. This is the miminum necessary to protect from spam. No big deal. Google mail and google groups are main gateways for spam on python list. We do not need that here. -- Terry Jan Reedy From tjreedy at udel.edu Tue Dec 27 22:58:31 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Dec 2011 16:58:31 -0500 Subject: [Python-ideas] Should stdlib provide something to properly handle SIGINT/SIGTERM signals? In-Reply-To: References: Message-ID: On 12/27/2011 12:57 PM, Giampaolo Rodol? wrote: > Since this is a problem which occurs very often, I'd like to hear your > opinion as to whether something like this might find some place in the > python stdlib (signal module maybe?). > Please read the discussion included in this recipe: > http://code.activestate.com/recipes/577997/ > It should provide a description of the problem and the general use case. It seems to me that your problem is this "From http://docs.python.org/library/atexit.html: the functions registered via this module are not called when the program is killed by a signal not handled by Python " So it seems to me that the solution should be an option to change that, or rather, change the handling of exit signals. -- Terry Jan Reedy From ethan at stoneleaf.us Wed Dec 28 06:40:23 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Dec 2011 21:40:23 -0800 Subject: [Python-ideas] Should stdlib provide something to properly handle SIGINT/SIGTERM signals? In-Reply-To: References: Message-ID: <4EFAABC7.6090203@stoneleaf.us> Giampaolo Rodol? wrote: > The behavior I would expect when facing such a problem is the > following: > > * I want my exit function to always be called, no matter what > o I'm aware that it won't be called in case of SIGKILL or > os._exit() though > * I do not want to override previously registered signal handlers > (if any) > o Instead I want the old handler to be called fist and my > exit function to be called last Aren't nested handlers usually called in a last-in/first-out manner? ~Ethan~ From techtonik at gmail.com Wed Dec 28 11:00:55 2011 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 28 Dec 2011 02:00:55 -0800 (PST) Subject: [Python-ideas] Idea: Google Groups web interface for Ideas In-Reply-To: References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> Message-ID: <30503727.947.1325066456024.JavaMail.geo-discussion-forums@yqai28> On Tuesday, December 27, 2011 10:53:56 PM UTC+3, RunThePun wrote: > > On Tue, Dec 27, 2011 at 7:01 PM, anatoly techtonik wrote: > >> As you may know, the python-ideas list is opened only to subscribers. >> This is inconvenient, because: >> 1. it requires three step subscription process >> 2. it is impossible to post a reply to existing thread/idea >> >> There is a web-interface in Google Groups at https://groups.google.com/** >> forum/#!forum/python-ideas that >> can solve problems above and provide some more nifty features such as >> embedded search. But there comes another problem that messages posted >> through the group doesn't end in list, because list requires subscription. >> I've already tried to find a solution, but run out of time, so I >> summarized the proposal at >> http://wiki.python.org/moin/MailmanWithGoogleGroups >> >> I may or may not be able to publish outcomes of my research, so it would >> be nice to get some help in investigating the problem and publishing a >> solution on aforementioned wiki page. Thanks. >> >> >> > > Concerning the search problem I've used google queries as such: > > list comprehensions site: > http://mail.python.org/pipermail/python-ideas/ > > I agree that having a "nosey" or "star" feature like issue trackers could > be nice, though I'm not sure Google Groups is the most modern > infrastructure to solve all our problems. I remember hearing that open > google groups get a lot of spam for example. > Over last six months I found only 3 spam messages sent from https://groups.google.com/forum/#!forum/python-ideas and for some reason I think that my mailbox filter would be smart enough to put them into appropriate folder even they came from Mailman. Maybe mailman can be improved? > eg would it help if the PyPI login cookie allowed you to post on > mailman? If mailman allowed starring threads? > Certainly. But the threads should be stacked in different order that just by month, because some threads can span over several months. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Wed Dec 28 11:11:03 2011 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 28 Dec 2011 02:11:03 -0800 (PST) Subject: [Python-ideas] Idea: Google Groups web interface for Ideas In-Reply-To: References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> Message-ID: <14270238.69.1325067063613.JavaMail.geo-discussion-forums@yqip20> On Wednesday, December 28, 2011 12:52:49 AM UTC+3, Terry Reedy wrote: > > On 12/27/2011 12:01 PM, anatoly techtonik wrote: > > As you may know, the python-ideas list is opened only to subscribers. > > This is inconvenient, because: > > 1. it requires three step subscription process > > 2. it is impossible to post a reply to existing thread/idea > > > > There is a web-interface in Google Groups at > > https://groups.google.com/forum/#!forum/python-ideas > > that can solve > > problems above and provide some more nifty features such as embedded > > search. But there comes another problem that messages posted through the > > group doesn't end in list, because list requires subscription. I've > > already tried to find a solution, but run out of time, so I summarized > > the proposal at http://wiki.python.org/moin/MailmanWithGoogleGroups > > I interact with this list as newsgroup gmane.comp.python.ideas via > news.gmane.org. The first post, which can be (and usually should be) a > reply to an existing post, requires that one respond to an email to the > from address. This is the miminum necessary to protect from spam. No big > deal. > > Google mail and google groups are main gateways for spam on python list. > We do not need that here. > Does Mailman support confirmation for messages with high SPAM ratio? There is already an Groups interface at https://groups.google.com/forum/#!topic/python-ideas/ and if a person is not subscribed in Mailman, the message appears in Group, but doesn't appear in Mailman even after the person is finally subscribed (if that persons notices and understands Mailman bounce message instructing him to subscribe). Somebody needs to put a notice about obligatory subscription to Mailman to https://groups.google.com/forum/#!topic/python-ideas/ if nobody wants more tight groups integration. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Wed Dec 28 11:15:24 2011 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 28 Dec 2011 02:15:24 -0800 (PST) Subject: [Python-ideas] PEP x: Static module/package inspection In-Reply-To: <29228470.233.1324982829840.JavaMail.geo-discussion-forums@yqbl25> References: <29228470.233.1324982829840.JavaMail.geo-discussion-forums@yqbl25> Message-ID: <20544069.58.1325067324546.JavaMail.geo-discussion-forums@yqiz15> (reposting this from Google Group once more as the previous post missed Mailing List, because I was not subscribed in Mailman) *Static module/package inspection* Abstract: - static: without execution (as opposed to dynamic) - module/package: .py or __init__.py file - inspection: get an overview of the contents *What should this do?* * * The proposal to add a mechanism to Python interpreter to get an outline of module/package contents without importing or executing module/package. The outline includes names of classes, functions, variables. It also should contain values for variables that could be provided without sophisticated calculations (e.g. a string, integer, but probably not expressions as it may lead to security leaks). *Why?* * * *user story PEPx.001:* As a Python package maintainer, I find it bothersome to repeatedly write bolierplate code (e.g. setup.py) to package my single file module. The reason I should write setup.py is to provide version and description info. This info is already available in my module source code. So I need to either copy/paste the info from the module manually, or to import (and hence execute) my module during packaging and installation, which I don't want either, because modules are often installed with root privileges. With this PEP, packing tool will be able to extract meta information from my module without executing it or without me manually copying version fields into some 'package configuration file'. *user story PEPx.002:* As a Python Application developer, I find it really complicated to provide plugin extension subsystem for my users. Users need a mechanism to switch between different versions of the plugin, and this mechanism is usually provided by external tool such as setuptools to manage and install multiple versions of plugins in local Python package repository. It is rather hard to create an alternative approach, because you are forced to maintain external meta-data about your plugin modules even in case it is already available inside the module. With this PEP, Python Application will be able to inspect meta-data embedded inside of plugins before choosing which version to load. This will also provide a standard mechanism for applications to check modules returned by packaging tools without executing them. This will greatly simplify writing and debugging custom plugins loaders on different platforms. *Feedback goal* At this stage I'd like to a community response to two separate questions: 1. If everybody feels this functionality will be useful for Python 2. If the solution is technically feasible -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Wed Dec 28 15:32:40 2011 From: brian at python.org (Brian Curtin) Date: Wed, 28 Dec 2011 08:32:40 -0600 Subject: [Python-ideas] Idea: Google Groups web interface for Ideas In-Reply-To: <14270238.69.1325067063613.JavaMail.geo-discussion-forums@yqip20> References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> <14270238.69.1325067063613.JavaMail.geo-discussion-forums@yqip20> Message-ID: On Wed, Dec 28, 2011 at 04:11, anatoly techtonik wrote: > Somebody needs to put a notice about obligatory subscription to Mailman to > https://groups.google.com/forum/#!topic/python-ideas/ if nobody wants more > tight groups integration. Whoever created that can put up the notice. I've never even heard of this Google Groups mirror to know who would be the one to do that. I really do not think anything is going to happen here - it works fine for everyone else and the hundreds of other mailing lists on python.org as well as the thousands of mailing lists around the world. Responding to old threads seems to be your biggest point for this, but you can always start a new thread and say "bringing up this topic from 6 months ago", link the original thread, then restart discussion - that's acceptable. From fuzzyman at gmail.com Wed Dec 28 16:28:44 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Wed, 28 Dec 2011 15:28:44 +0000 Subject: [Python-ideas] PEP x: Static module/package inspection In-Reply-To: <20544069.58.1325067324546.JavaMail.geo-discussion-forums@yqiz15> References: <29228470.233.1324982829840.JavaMail.geo-discussion-forums@yqbl25> <20544069.58.1325067324546.JavaMail.geo-discussion-forums@yqiz15> Message-ID: On 28 December 2011 10:15, anatoly techtonik wrote: > (reposting this from Google Group once more as the previous post missed > Mailing List, because I was not subscribed in Mailman) > > *Static module/package inspection* > > Abstract: > - static: without execution (as opposed to dynamic) > - module/package: .py or __init__.py file > - inspection: get an overview of the contents > > *What should this do?* > * > * > The proposal to add a mechanism to Python interpreter to get an outline of > module/package contents without importing or executing module/package. The > outline includes names of classes, functions, variables. It also should > contain values for variables that could be provided without sophisticated > calculations (e.g. a string, integer, but probably not expressions as it > may lead to security leaks). > > *Why?* > * > * > *user story PEPx.001:* > As a Python package maintainer, I find it bothersome to repeatedly write > bolierplate code (e.g. setup.py) to package my single file module. The > reason I should write setup.py is to provide version and description info. > This info is already available in my module source code. So I need to > either copy/paste the info from the module manually, or to import (and > hence execute) my module during packaging and installation, which I don't > want either, because modules are often installed with root privileges. > > With this PEP, packing tool will be able to extract meta information from > my module without executing it or without me manually copying version > fields into some 'package configuration file'. > > *user story PEPx.002:* > As a Python Application developer, I find it really complicated to provide > plugin extension subsystem for my users. Users need a mechanism to switch > between different versions of the plugin, and this mechanism is usually > provided by external tool such as setuptools to manage and install multiple > versions of plugins in local Python package repository. It is rather hard > to create an alternative approach, because you are forced to maintain > external meta-data about your plugin modules even in case it is already > available inside the module. > > With this PEP, Python Application will be able to inspect > meta-data embedded inside of plugins before choosing which version to load. > This will also provide a standard mechanism for applications to check > modules returned by packaging tools without executing them. This will > greatly simplify writing and debugging custom plugins loaders on different > platforms. > > > *Feedback goal* > At this stage I'd like to a community response to two separate questions: > 1. If everybody feels this functionality will be useful for Python > 2. If the solution is technically feasible > > On a simple level, all of this is already "obtainable" by using the ast module that can parse Python code. I would love to see a "python-object" layer on top of this that will take an ast for a module (or other object) and return something that represents the same object as the ast. So all module level objects will have corresponding objects - where they are Python objects (builtin-literals) then they will represented exactly. For classes and functions you'll get an object back that has the same attributes plus some metadata (e.g. for functions / methods what arguments they take etc). That is certainly doable and would make introspecting-without-executing a lot simpler. I think your specific use cases are better served by adding functionality to the packaging (distutils2) package however. I'd particularly like to see plugin support in packaging (a cutdown version of setuptools entry points). All the best, Michael > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan.alexander.rice at gmail.com Wed Dec 28 22:18:38 2011 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Wed, 28 Dec 2011 16:18:38 -0500 Subject: [Python-ideas] PEP x: Static module/package inspection In-Reply-To: <20544069.58.1325067324546.JavaMail.geo-discussion-forums@yqiz15> References: <29228470.233.1324982829840.JavaMail.geo-discussion-forums@yqbl25> <20544069.58.1325067324546.JavaMail.geo-discussion-forums@yqiz15> Message-ID: On Wed, Dec 28, 2011 at 5:15 AM, anatoly techtonik wrote: > user story PEPx.001: > As a Python package maintainer, I find it bothersome to repeatedly write > bolierplate code (e.g. setup.py) to package my single file module. The > reason I should write setup.py is to provide version and description info. > This info is already available in my module source code. So I need to either > copy/paste the info from the module manually, or to import (and hence > execute) my module during packaging and installation, which I don't want > either, because modules are often installed with root privileges. I agree this is a pain. I also agree with Micheal that this is moreso a packager issue. Part of the problem is that I don't believe there is a strong enough convention around writing modules with an eye to being package tools accessible. If there was a PEP on module metadata for packaging tools to use for introspection, that might motivate package tool authors to support automated packaging :) *HINT HINT* Sphinx could also take advantage of some of it too. > With this PEP, packing tool will be able to extract meta information from my > module without executing it or without me manually copying version fields > into some 'package configuration file'. > > user story PEPx.002: > As a Python Application developer, I find it really complicated to provide > plugin extension subsystem for my users. Users need a mechanism to switch > between different versions of the plugin, and this mechanism is usually > provided by external tool such as setuptools to manage and install multiple > versions of plugins in local Python package repository. It is rather hard to > create an alternative approach, because you are forced to maintain external > meta-data about your plugin modules even in case it is already available > inside the module. See above. Maintaining the same information twice is definitely a bad thing, but we already have the ability to do everything required. What is missing is good, strong conventions on module metadata annotation that tool creators write to. > With this PEP, Python Application will be able to inspect meta-data embedded > inside of plugins before choosing which version to load. This will also > provide a standard mechanism for applications to check modules returned by > packaging tools without executing them. This will greatly simplify writing > and debugging custom plugins loaders on different platforms. Having more nuanced import behavior is something I can get behind. Sure, I can wrap an import in a try except, and check the __version__ if is defined (after determining if it is a string/tuple/etc, and possibly parsing it), but more nuanced behavior would certainly be nice. Being able to specify version in the import line (and have multiple versions installed), being able to get fine grained exception beyond ImportError (ParseError, anyone?), not having to worry that the same file is being imported twice, that sort of stuff. I'm +1 getting a module level metadata conventions PEP draft started. I'm also +1 on taking a look at import behavior (though that is tangential here). Nathan From guido at python.org Fri Dec 30 17:40:06 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Dec 2011 09:40:06 -0700 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: Hi Julien, Don't despair! I have tried to get people to warm up to dict addition too -- in fact it was my counter-proposal at the time when we were considering adding sets to the language. I will look at your proposal, but I have a point of order first: this should be discussed on python-ideas, not on python-dev. I have added python-ideas to the thread and moved python-dev to Bcc, so followups will hopefully all go to python-ideas. --Guido On Fri, Dec 30, 2011 at 7:26 AM, julien tayon wrote: > Hello, > Sorry to annoy the very busy core devs :) out of the blue > > I quite noticed people were > 1) wanting to have a new dict for Xmas > 2) strongly resenting dict addition. > > Even though I am not a good developper, I have come to a definition of > addition that would follow algebraic rules, and not something of a > dutch logic. (it is a jest, not a troll) > > I propose the following code to validate my point of view regarding > the dictionnatry addition as a proof of concept : > https://github.com/jul/ADictAdd_iction/blob/master/test.py > > It follows all my dusty math books regarding addition + it has the > amability to have rules of conservation. > > I pretty much see a real advantage in this behaviour in functional > programming (map/reduce). (see the demonstrate.py), and it has a sense > (if dict can be seen has vectors). > > I have been told to be a troll, but I am pretty serious. > > Since, I coded with luck, no internet, intuition, and a complete > ignorance of the real meaning of the magic methods most of the time, > thus the actual implementation of the addition surely needs a complete > refactoring. > > Sheers, > Bonne f?tes > Julien > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Dec 30 18:02:41 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Dec 2011 10:02:41 -0700 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: Hi Julien, Having now looked at your code, I realize we are not talking about the same concepts at all. What I meant is similar to set union on the keys, where if a key exists in both dicts, the value in the result is equal to one of the values in the operands (and if the value is the same for both operands, that value is also the result value). But looking at your code, you are more interested in *elementwise* addition, so that {'x': 1} + {'x': 2} results in {'x': 3}, and missing keys are considered to be zero. (For disjoint key sets, your result is the same as mine. :-) I think this would be a nice thing for some applications, but I'm doubtful that it's of enough interest to add to the standard library, let alone make it a standard feature of the built-in dict class. One problem is that the dict API is already very rich (dozens of methods and operations) Another problem is that your proposed class solves only part of the puzzle of elementwise operations -- if you have elementwise addition for dicts, why not for lists? But l1+l2 already has a meaning for list: concatenation. We've discussed this before. NumPy has got its own solution (a1+a2 does elementwise addition if a1 and a2 are NumPy arrays), but for the stdlib we've always decided that it's better to leave the choice up to the user, the app or the framework (like NumPy), instead of making a choice in the stdlib that causes backward incompatibility. (Had I known all this 22 years ago, maybe I could have chosen a different operator for list concatenation. But I don't think the issue is important enough to fix, even if we ever did a Python 4.) --Guido On Fri, Dec 30, 2011 at 9:40 AM, Guido van Rossum wrote: > Hi Julien, > > Don't despair! I have tried to get people to warm up to dict addition too > -- in fact it was my counter-proposal at the time when we were considering > adding sets to the language. I will look at your proposal, but I have a > point of order first: this should be discussed on python-ideas, not on > python-dev. I have added python-ideas to the thread and moved python-dev to > Bcc, so followups will hopefully all go to python-ideas. > > --Guido > > > On Fri, Dec 30, 2011 at 7:26 AM, julien tayon wrote: > >> Hello, >> Sorry to annoy the very busy core devs :) out of the blue >> >> I quite noticed people were >> 1) wanting to have a new dict for Xmas >> 2) strongly resenting dict addition. >> >> Even though I am not a good developper, I have come to a definition of >> addition that would follow algebraic rules, and not something of a >> dutch logic. (it is a jest, not a troll) >> >> I propose the following code to validate my point of view regarding >> the dictionnatry addition as a proof of concept : >> https://github.com/jul/ADictAdd_iction/blob/master/test.py >> >> It follows all my dusty math books regarding addition + it has the >> amability to have rules of conservation. >> >> I pretty much see a real advantage in this behaviour in functional >> programming (map/reduce). (see the demonstrate.py), and it has a sense >> (if dict can be seen has vectors). >> >> I have been told to be a troll, but I am pretty serious. >> >> Since, I coded with luck, no internet, intuition, and a complete >> ignorance of the real meaning of the magic methods most of the time, >> thus the actual implementation of the addition surely needs a complete >> refactoring. >> >> Sheers, >> Bonne f?tes >> Julien >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido ) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Dec 30 18:12:34 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 30 Dec 2011 10:12:34 -0700 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 10:02 AM, Guido van Rossum wrote: > What I meant is similar to set union on the keys, where if a key exists in > both dicts, the value in the result is equal to one of the values in the > operands (and if the value is the same for both operands, that value is also > the result value). +1 This is the one I was thinking of too. -eric From bruce at leapyear.org Fri Dec 30 20:37:21 2011 From: bruce at leapyear.org (Bruce Leban) Date: Fri, 30 Dec 2011 11:37:21 -0800 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 9:12 AM, Eric Snow wrote: > On Fri, Dec 30, 2011 at 10:02 AM, Guido van Rossum > wrote: > > What I meant is similar to set union on the keys, where if a key exists > in > > both dicts, the value in the result is equal to one of the values in the > > operands (and if the value is the same for both operands, that value is > also > > the result value). > > +1 > > This is the one I was thinking of too. > > -eric > My first thought on reading Julien's message was that {1:2} + {1:3} => {1:5} would be useful when doing reduction, There already is an operation that combines two dictionaries and does not combine values: update. Guido's version of + is a shorthand for: def __add__(self, other): result = self.copy() return result.update(other) Is saving one line of code be enough of a benefit? I think dicts are complex enough that if new functionality is added, it would be better to use named methods which can have meaningful names and are easier look up in the documentation. Furthermore, methods are more flexible as this example shows: def merge(self, other, merger=lambda x, y: x + y): """Merges another dictionary into this one, combining values.""" for k in other: if k in self: self[k] = merger(self[k], other[k]) else: self[k] = other[k] --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkbbwr at gmail.com Fri Dec 30 20:39:13 2011 From: jkbbwr at gmail.com (Jakob Bowyer) Date: Fri, 30 Dec 2011 19:39:13 +0000 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: <63E20FB6-D6CC-407E-92A0-AD8D55A625C4@gmail.com> Not that I'm smart enough to really contribute to these discussions but wouldn't it be an idea to have dictionary addition turn the values from d2 into d1 so that d1 = {1:1,2:2,3:3} d2 = {1:2,2:2,4:4} d3 = d1 + d2 print d3 # {1:2,2:2,3:3,4:4} # but d4 = d2 + d1 print d4 # {1:1,2:2,3:3,4:4} Meaning that replacement is conditional on which side the other dictionary is on with the + operator. This might be harder to understand. So it's not a great idea... Still throwing it in the ring is the only way to learn Sent from my iPad On 30 Dec 2011, at 17:12, Eric Snow wrote: > On Fri, Dec 30, 2011 at 10:02 AM, Guido van Rossum wrote: >> What I meant is similar to set union on the keys, where if a key exists in >> both dicts, the value in the result is equal to one of the values in the >> operands (and if the value is the same for both operands, that value is also >> the result value). > > +1 > > This is the one I was thinking of too. > > -eric > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From nathan at cmu.edu Fri Dec 30 21:01:54 2011 From: nathan at cmu.edu (Nathan Schneider) Date: Fri, 30 Dec 2011 15:01:54 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 2:37 PM, Bruce Leban wrote: > > On Fri, Dec 30, 2011 at 9:12 AM, Eric Snow > wrote: >> >> On Fri, Dec 30, 2011 at 10:02 AM, Guido van Rossum >> wrote: >> > What I meant is similar to set union on the keys, where if a key exists >> > in >> > both dicts, the value in the result is equal to one of the values in the >> > operands (and if the value is the same for both operands, that value is >> > also >> > the result value). >> >> +1 >> >> This is the one I was thinking of too. >> >> -eric > > > My first thought on reading Julien's message was that {1:2} + {1:3} => {1:5} > would be useful when doing reduction, ?There already is an operation that > combines two dictionaries and does not combine values: update. Guido's > version of + is a shorthand for: > > def __add__(self, other): > ? result = self.copy() > ? return result.update(other) The Counter class already supports this with the + operator. (It also supports element-wise max with the | operator.) Perhaps it would be useful to extend this support to dict, especially if there is a use case for element-wise string concatenation. As for a non-mutating alternative to update(): I have encountered a number of scenarios in which this is desirable. A previous suggestion [1] was to add a replace() method along these lines. But I think an operator would make it clearer which is the mutating vs. non-mutating one. Instead of +, I think << would be a better choice for this sort of modified concatenation/asymmetric union operation. For example, we can write: class Xdict(dict): def __lshift__(x, y): return Xdict(x.items() + y.items()) d = Xdict({'a': 9, 'b': 12}) d << {'a': 3, 'c': 8} # result: {'a': 3, 'b': 12, 'c': 8} d['a']==9 # True The rationale for <<: * __add__ (+) is quite common, and thus banning the addition of two dicts probably catches a lot of bugs. Moreover, when dict values are numeric, a user might expect element-wise addition (as in Counter). * __or__ (|) is classically a commutative operation, and is already overloaded by set and Counter. * Counter does not already overload <<, and thus could be extended to support the same behavior. * As far as I am aware, the only present use of << is for integer shifting, so this new idiom would not pose a conflict. Cheers, Nathan [1] http://groups.google.com/group/python-ideas/browse_thread/thread/275bbd1acfa8d433/6c87b56021ca5f44 > > Is saving one line of code be enough of a benefit? > > I think dicts are complex enough that if new functionality is added, it > would be better to use named methods which can have meaningful names and are > easier look up in the documentation. Furthermore, methods are more flexible > as this example shows: > > def merge(self, other, merger=lambda x, y: x + y): > ? """Merges another dictionary into this one, combining values.""" > ? for k in other: > ? ? if k in self: > ? ? ? self[k] = merger(self[k], other[k]) > ? ? else: > ? ? ? self[k] = other[k] > > > --- Bruce > Follow me:?http://www.twitter.com/Vroo?http://www.vroospeak.com > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From tjreedy at udel.edu Fri Dec 30 21:29:01 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 Dec 2011 15:29:01 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: > On Fri, Dec 30, 2011 at 7:26 AM, julien tayon > > wrote: > I quite noticed people were > 1) wanting to have a new dict for Xmas Someone has proposed a modified version of the current dict type. The questions are whether a) it will behave correctly in all situations; b) give better overall performance; and c) be as easily maintainable. > 2) strongly resenting dict addition. This strikes me as a nonsensical statement. As noted below, we already have a type of 'dict addition' -- and even 'dict subtraction'. Counterfactual negative statements like this lead to the impression that you are trolling, even if you do not intend to. > I propose the following code to validate my point of view regarding > the dictionnatry addition as a proof of concept : > https://github.com/jul/ADictAdd_iction/blob/master/test.py This link any gives the tests, not the accu_dict itself. So there is no way to know from the above exactly what you are proposing. From what Guido wrote in his next response, I am guessing that it is something like collections.Counter, which is an implementation of the multiset idea. >>> c = Counter(a=3, b=1) >>> d = Counter(a=1, b=2) >>> c + d # add two counters together: c[x] + d[x] Counter({'a': 4, 'b': 3}) >>> c - d # subtract (keeping only positive counts) Counter({'a': 2}) Tagging numbers by keys is a generalization of tagging them by position (index). So this is generalized vector arithmetic and that is what your tests test. ------------------------ You have mixed up some of the terminology. def test_add_permut(self): """( a + b ) + c = a + ( b + c )""" This is the associative law, not 'permutative'. def test_permut_scal_mul(self): """ ( an_int * other_scalar ) * a = an_int ( other_scalar * a )""" Again, associativity (of scalar multiplication). --- def test_other_mul_scal_commut(self): """ ( an_int + other_scalar ) a = an_int * a + other_scalar * a """ This is a distributive law. def test_scal_lin_combo(self): """ an_int ( a + b) = an_int *a + an_int * b """ So is this. --- def test_permutation(self): """a + b = b + a """ This is commutivity, which you had right earlier in the file: def test_commutativity(self): """ a + b = b + a """ --- def test_neutral(self): """ a + neutral = a """ Not exactly incorrect, but this is a test of the additive identify, which is usually called 'zero' rather than 'neutral'. def test_neutral_mul_scal(self): """ 1 * a = a """ while this tests the scalar multiplicative identify -- Terry Jan Reedy From tjreedy at udel.edu Fri Dec 30 21:33:43 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 Dec 2011 15:33:43 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 12/30/2011 12:02 PM, Guido van Rossum wrote: > But looking at your code, you are more interested in *elementwise* > addition, so that {'x': 1} + {'x': 2} results in {'x': 3}, and missing > keys are considered to be zero. This is part of what collections.Counter does. > (Had I known all this 22 years ago, maybe I could have chosen a > different operator for list concatenation. In base 1 notation, count addition amounts to sequence concatenation. So I think you intuitively made the right choice. -- Terry Jan Reedy From arnodel at gmail.com Fri Dec 30 21:45:50 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Fri, 30 Dec 2011 20:45:50 +0000 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 30 December 2011 20:29, Terry Reedy wrote: [...] > You have mixed up some of the terminology. [...] > --- > ? ?def test_permutation(self): > ? ? ? ?"""a + b = b + a """ > > This is commutivity, which you had right earlier in the file: You mean commutativity :) [...] > ? ?def test_neutral(self): > ? ? ? ?""" a + neutral = a """ > > Not exactly incorrect, but this is a test of the additive identify, which is > usually called 'zero' rather than 'neutral'. You mean additive identity (by the way, I think Julien used the word 'neutral' because in French we call it '?l?ment neutre') > ? ?def test_neutral_mul_scal(self): > ? ? ? ?""" 1 * a = a """ > > while this tests the scalar multiplicative identify You mean multiplicative identity :) Nit-picklingly yours, -- Arnaud From dreamingforward at gmail.com Fri Dec 30 23:00:14 2011 From: dreamingforward at gmail.com (Mark Janssen) Date: Fri, 30 Dec 2011 16:00:14 -0600 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 2:01 PM, Nathan Schneider wrote: > On Fri, Dec 30, 2011 at 2:37 PM, Bruce Leban wrote: >> >> On Fri, Dec 30, 2011 at 9:12 AM, Eric Snow >> wrote: >>> >>> On Fri, Dec 30, 2011 at 10:02 AM, Guido van Rossum >>> wrote: >>> > What I meant is similar to set union on the keys, where if a key exists >>> > in >>> > both dicts, the value in the result is equal to one of the values in the >>> > operands (and if the value is the same for both operands, that value is >>> > also >>> > the result value). >>> >>> +1 Far more powerful is to allow an optional "collision" function that is called *only* when a key collision occurs. Special case the default behavior so that little performance hit is made if no such function is given upon initialization. Add the functionality to defaultdict and see if it gets more traction. I think you'll find a lot of code can be refactored with this simple abstraction. It becomes trivial and canonical to create a graph class for example (at least in implementation if not in interface.) To add that feature to the interface, you need to refactor the set/dict relationship so that dict inherits from set. Then you'll really cooking with gas.... Winks, mark From guido at python.org Sat Dec 31 16:52:09 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 31 Dec 2011 08:52:09 -0700 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Sat, Dec 31, 2011 at 2:16 AM, julien tayon wrote: > Dear All :) > > 2011/12/30 Eric Snow : > > On Fri, Dec 30, 2011 at 10:02 AM, Guido van Rossum > wrote: > >> What I meant is similar to set union on the keys, where if a key exists > in > >> both dicts, the value in the result is equal to one of the values in the > >> operands (and if the value is the same for both operands, that value is > also > >> the result value). > > > > +1 > > > > This is the one I was thinking of too. > > > Well, since I have coded way too much in Perl, my altered sense of > reality has come to a concept I may be introducing too early which is > : algebrae. > > strings, lists, ... have a record algebrae. > ndarray, accudict have a linear algebrae > sets ... have sets algebrae. > > And much more algebrae exists wich all exists not only in my > imagination, but also in math (wich I quite dislike). (Abelian > stuff/Group/matrix/hilbert) > All of these algebrae are consistent as long as any object in the > chain of algebrae are following the same rules. > > And each of these are very legitimate (even though of course my dict > addition is the best without trying to be obnoxious). > > > I was kind of thinking of > 1) giving a property to object called .. __algebrae__, > 2) and through some magic being able to change the algebrae of an > object on the fly. > > My twisted sense of reality inherited from Perl (but a little less > than my math books) tells me There Is More Than One Way To > Consistently Add/Mul/Div/Sub It. > > > As a Proof of Concept I could deliver a monkeypatching of list() that > makes it behave like an numpy array. > Please don't present this in terms of modifications of existing functions/types/methods. Please use subclasses, new modules, new functions, etc. > But, at first I wish to concentrate on dict addition, since I can only > steal a few hours connectivity per day ... So I will try to answer to > everyone since I saw some spoilers of what I had hidden in my mind :) > > Cheers > Thanks > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at tayon.net Sat Dec 31 17:07:46 2011 From: julien at tayon.net (julien tayon) Date: Sat, 31 Dec 2011 17:07:46 +0100 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: Hello, I had tough time finding a connection to internet :) @Arnaud + Terry Thx, code corrected to have the right terminology. at least my code really test for the good rules :) But being precise matter, so that I changed all the terms to fit in. @nathan << is a symbol indicating an asymetry for left and right value, and I carefully tried to have a symetrical behaviour with left & right : associativity, distribution, commutativity. The whole point about using a symbol, is that it follow rules. That is the reason why consistent_addition class is all about testing linear algebraic rules (hopefully the right way, and now, I hope with the right terminology). But I have reason to think that there are more than way to do addition. Linear algebrae is the most used and intuitive one for non developer, and developer think of sets operations as more intuitive. I guess, it all come down to one point. Which is what is a dict as a mathematical object. If there is more than one choice, which has the most benefits. Should there be more than one definition ? @Guido & Eric In my book there are rules for sets. The question is a dict the same as a vector or as a set ? since sets are using logical operators for operations that are roughly sets operations why not use & ^ | for sets operation on dict ? it would be pretty consistent with sets operations. ( @jakkob ) In this case I would expect inconsistencies error when doing dict( a = 1 ) | dict( a = 2 , b = 2 ) (thus key-> value are seen like an atomic element of the set, unless a "value collision" operator is given, and this collision operator would recursively and naturely apply to the descendants) because I expect both a | b == b | a and a + b = b + a The problem would be indeed list + list vectorish behaviour is my prefered of course. but if we shift the record algebrae addition to another symbol (let's assume << ) we resolve the conflict. But then we have a problem of retro compatibility. We then may be tempted to use & | ^ (since it would be used for dict) and then we have a problem : should [ 1 , 2 ] & [ 3 , 2 ] mean set addition or applying & to all the element of the same rank ? I'm pretty stucked myself in trying to be consistent. Just for the record, why dit you not use "." (dot) for concatenation ? I know it is typographically unreadable on a messy screen, but is there a better reason ? @terry & Nathan Regarding Counter. on https://github.com/jul/ADictAdd_iction/blob/master/demonstrate.py#L120 you may notice that I do have all the property of counter, but counter does not as -this dict- aggregate values and (sub) totals at the same time. It is very convenient for map reduce since it does some re-reduce operations at reduce time. smart, no ? <:o) @terry Sorry for making assertive and carefree statements regarding strong resentment. My bad. @bruce, truth is aim at similarity cosinus for dict, and I imagine a map reduce on dict representing values of a model you want ideal = dict( blue = 1 , height = 180, weight = 70, wage = 500 ) in the filter before the map reduce you would want to filter all records close to this one by using similarity cosinus in this way : filter( lambda model : cos( ideal, model) > .7 , all_model ) http://en.wikipedia.org/wiki/Cosine_similarity of course I will try to advocate dot product, and metrics. As a result my real goal is to make people consider dict as vectors of path to value => value not sets of key => value To solve issues such as weighted decisions and non linear choices (based on trigger or a value belonging to a set) I can fairly easily concevie a projector (which would be a class transforming a vector in a vector, but not with a a matrix, but with computed rules ). @Mark Jansen +1 are you telepath ? Yes a key collision operator would be nice : given for instance an apache log, I will have segments of path, and I may want keys to be the accumulated graph of a user on a website. so my collision rule might be : (key being the referer value the page visited afterwards) dict( a = b, a = c ) + dict( b = e ) = dict( a = dict( b = e ), a = c) I thought of it, but I really loved the conservation rule. And I feared people would think of it as too complicated. Keep It Simple I learnt :) @all Wrapping everything to logicial sense. I was thinking of supersets of object (thus out of the stdlib) that would have different algebrae and could be used as casts on object to redifine + - * / cos & | ^ and for different objects would behave consistently ex : vector( dict ) vector( list ) vector( string ) would make dict & list & string behave like vectors thus mainly supporting elementwise operations sets( dict ) , sets( string ) ... would make dict string be sets of elements ... (with the original dict addition design of GvR et al) Each superset would have a unit test for the operation verifying that behaviour is consistent. (associativity, distributivity, ...) what about this solution ? -- Friendly jul From tjreedy at udel.edu Sat Dec 31 22:55:55 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 31 Dec 2011 16:55:55 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 12/31/2011 11:07 AM, julien tayon wrote: > The question is a dict the same as a vector or as a set ? A dict is a mapping, which is to say, a functional set of key,value pairs. 'Functional' means that the keys are unique. This is a key property (pun intended). A vector can be regarded as a functional set of count,value pairs. Some languages use dicts for vectors. Even in Python, people use dicts for sparse arrays of various dimensions, with unspecified pairs assumed to have a 0 value. So it makes the most sense to me for d1 + d2 to be interpreted as set union, with some resolution of key conflicts to restore the functional property. Your vector interpretation of adding values only works for values that can be added. > since sets are using logical operators for operations that are roughly > sets operations why not use & ^ | for sets operation on dict ? > it would be pretty consistent with sets operations. I agree, except that dicts are not just sets of pairs, but functional sets. If one wants to view a dict simply as a set of keys, values, or pairs, ignoring the functional property for the pairs, that is what the .keys(), .values(), and .items() set view methods are for. All the set operations are available on the resulting set view objects. -- Terry Jan Reedy From nathan at cmu.edu Sat Dec 31 23:57:02 2011 From: nathan at cmu.edu (Nathan Schneider) Date: Sat, 31 Dec 2011 17:57:02 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Sat, Dec 31, 2011 at 4:55 PM, Terry Reedy wrote: > On 12/31/2011 11:07 AM, julien tayon wrote: > >> The question is a dict the same as a vector or as a set ? > > > A dict is a mapping, which is to say, a functional set of key,value pairs. > 'Functional' means that the keys are unique. This is a key property (pun > intended). A vector can be regarded as a functional set of count,value > pairs. Some languages use dicts for vectors. Even in Python, people use > dicts for sparse arrays of various dimensions, with unspecified pairs > assumed to have a 0 value. > > So it makes the most sense to me for d1 + d2 to be interpreted as set union, > with some resolution of key conflicts to restore the functional property. > Your vector interpretation of adding values only works for values that can > be added. It occurs to me that any argument for a vector interpretation of dicts, with element-wise operations, should apply equally well to lists and tuples, which are perhaps even more natural as representations for vectors. But of course, the + operator for sequences is interpreted as concatenation, not element-wise addition; it would be strange if the reverse were true for dicts. The necessary resolution of key conflicts is what makes "+" feel less natural to me for dicts than it does for sequences, where the order of the operands is transparently reflected in the output. For dicts, in contrast, we are talking about a "lossy" union/combination where what is lost depends on the order of the operands?this is less transparent in the output. However it is spelled, though, I would be thrilled if such an operation were built in. :) Nathan > >> since sets are using logical operators for operations that are roughly >> sets operations why not use & ?^ | for sets operation on dict ? >> it would be pretty consistent with sets operations. > > > I agree, except that dicts are not just sets of pairs, but functional sets. > If one wants to view a dict simply as a set of keys, values, or pairs, > ignoring the functional property for the pairs, that is what the .keys(), > .values(), and .items() set view methods are for. All the set operations are > available on the resulting set view objects. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas