From castironpi at gmail.com Sun Aug 3 19:46:08 2014 From: castironpi at gmail.com (Aaron Brady) Date: Sun, 3 Aug 2014 12:46:08 -0500 Subject: [Python-ideas] Mutating while iterating In-Reply-To: References: Message-ID: On Sat, Jul 26, 2014 at 10:39 PM, Nick Coghlan wrote: > On 27 July 2014 06:51, Aaron Brady wrote: >> Hi, I asked about the inconsistency of the "RuntimeError" being raised when >> mutating a container while iterating over it here [1], "set and dict >> iteration" on Aug 16, 2012. > > Hi, > > This is clearly an issue of grave concern to you, but as Raymond > pointed out previously, you appear to have misunderstood the purpose > of those exceptions. They're there to prevent catastrophic failure of > the interpreter itself (i.e. segmentation faults), not to help find > bugs in user code. If users want to mutate containers while they're > iterating over them, they're generally free to do so. The only time > we'll actively disallow it is when such mutation will outright *break* > the iterator, rather than merely producing potentially surprising > results. > > I have closed the new issue and added a longer reply (with examples) > that will hopefully better explain why we have no intention of > changing this behaviour: http://bugs.python.org/issue22084#msg224100 Python is replete with examples of prohibiting structures which are likely bugs but aren't segfaults. There are also reciprocal-- though not necessarily inverse-- limitations of the unordered collections themselves ("set" and "dict"). The new behavior is transparent for the programmer; no possible programs can /rely/ on the existing behavior. The new behavior introduces no new objects, possibly except "IterationError", no new syntax, and no new costs. I propose we leave this discussion thread open for the time being. I also take the issue of /re/-assigning to keys during iteration to be settled as permitted. From ram.rachum at gmail.com Mon Aug 4 18:17:14 2014 From: ram.rachum at gmail.com (Ram Rachum) Date: Mon, 4 Aug 2014 09:17:14 -0700 (PDT) Subject: [Python-ideas] strptime without second argument as an inverse to __str__ Message-ID: What do you think about having `datetime.strptime`, when called without a `format` for the second argument, be a precise inverse of `datetime.__str__`? This is because I don't currently see an obvious way to get an inverse of `datetime.__str__`, and this seems like an okay place to put it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Aug 4 20:15:28 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 5 Aug 2014 04:15:28 +1000 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: Message-ID: <20140804181528.GP4525@ando> On Mon, Aug 04, 2014 at 09:17:14AM -0700, Ram Rachum wrote: > What do you think about having `datetime.strptime`, when called without a > `format` for the second argument, be a precise inverse of > `datetime.__str__`? This is because I don't currently see an obvious way to > get an inverse of `datetime.__str__`, and this seems like an okay place to > put it. Is str(datetime) guaranteed to use a specific format, or is that an implementation detail? -- Steven From alexander.belopolsky at gmail.com Mon Aug 4 20:40:57 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 4 Aug 2014 14:40:57 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <20140804181528.GP4525@ando> References: <20140804181528.GP4525@ando> Message-ID: On Mon, Aug 4, 2014 at 2:15 PM, Steven D'Aprano wrote: > > What do you think about having `datetime.strptime`, when called without a > > `format` for the second argument, be a precise inverse of > > `datetime.__str__`? This is because I don't currently see an obvious way > to > > get an inverse of `datetime.__str__`, and this seems like an okay place > to > > put it. > > Is str(datetime) guaranteed to use a specific format, or is that an > implementation detail? Why is this question relevant for Ram's proposal? As long as str(datetime) is guaranteed to be different for different datetimes, one should be able to implement an inverse. The inverse function should accept ISO format (with either ' ' or 'T' separator) and str(datetime) if it is different in the implementation. I agree that datetime type should provide a simple way to construct instances from well-formatted strings, but I don't think datetime.strptime() is a good choice of name. I would much rather have date(str), time(str) and datetime(str) constructors. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Mon Aug 4 21:00:17 2014 From: skip at pobox.com (Skip Montanaro) Date: Mon, 4 Aug 2014 14:00:17 -0500 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> Message-ID: On Mon, Aug 4, 2014 at 1:40 PM, Alexander Belopolsky wrote: > Why is this question relevant for Ram's proposal? It would seem to have some impact on how hard it is to create a general inverse. Will one format work for all platforms ("one and done"), or will the inverse implementation potentially have to be updated as new platforms come into (or go out of) existence? Also, would the creation of such an inverse lock the implementation into existing format(s)? For example, when fed a datetime object, the CSV module will stringify it for output. If I create a CSV file with one version of Python, then read it into another version of Python (or on a different platform), it's not unreasonable that I would expect one-argument strptime() to parse it. That would lock you into a specific format. If only one format exists today, no big deal, bless it and move on. Skip From alexander.belopolsky at gmail.com Mon Aug 4 21:23:08 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 4 Aug 2014 15:23:08 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> Message-ID: On Mon, Aug 4, 2014 at 3:00 PM, Skip Montanaro wrote: > > Why is this question relevant for Ram's proposal? > > It would seem to have some impact on how hard it is to create a > general inverse. Will one format work for all platforms ("one and > done"), or will the inverse implementation potentially have to be > updated as new platforms come into (or go out of) existence? I think str(datetime) format is an implementation detail to the same extent as str(int) or str(float) is. In the past, these variations did not prevent providing (sometimes imperfect) inverse. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Mon Aug 4 22:14:11 2014 From: skip at pobox.com (Skip Montanaro) Date: Mon, 4 Aug 2014 15:14:11 -0500 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> Message-ID: On Mon, Aug 4, 2014 at 2:23 PM, Alexander Belopolsky wrote: > > On Mon, Aug 4, 2014 at 3:00 PM, Skip Montanaro wrote: >> >> > Why is this question relevant for Ram's proposal? >> >> It would seem to have some impact on how hard it is to create a >> general inverse. Will one format work for all platforms ("one and >> done"), or will the inverse implementation potentially have to be >> updated as new platforms come into (or go out of) existence? > > > I think str(datetime) format is an implementation detail to the same extent > as str(int) or str(float) is. In the past, these variations did not prevent > providing (sometimes imperfect) inverse. I took a look at whatever version of CPython I have laying about (some variant of 2.7). str(datetime) seems to be well-defined as calling isoformat with " " as the separator. The only caveat is that if the microsecond field is zero, it's omitted. If that behavior holds true in 3.x, only two cases require consideration: %Y-%m-%d %H:%M:%S %Y-%m-%d %H:%M:%S.%f Skip From wolfgang.maier at biologie.uni-freiburg.de Mon Aug 4 22:56:56 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 04 Aug 2014 22:56:56 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> Message-ID: On 04.08.2014 22:14, Skip Montanaro wrote: > On Mon, Aug 4, 2014 at 2:23 PM, Alexander Belopolsky > wrote: >> >> On Mon, Aug 4, 2014 at 3:00 PM, Skip Montanaro wrote: >>> >>>> Why is this question relevant for Ram's proposal? >>> >>> It would seem to have some impact on how hard it is to create a >>> general inverse. Will one format work for all platforms ("one and >>> done"), or will the inverse implementation potentially have to be >>> updated as new platforms come into (or go out of) existence? >> >> >> I think str(datetime) format is an implementation detail to the same extent >> as str(int) or str(float) is. In the past, these variations did not prevent >> providing (sometimes imperfect) inverse. > > I took a look at whatever version of CPython I have laying about (some > variant of 2.7). str(datetime) seems to be well-defined as calling > isoformat with " " as the separator. The only caveat is that if the > microsecond field is zero, it's omitted. > > If that behavior holds true in 3.x, only two cases require consideration: > it does hold true in 3.x, but the documented behavior is slightly more complex (I assume also in 2.x): datetime.__str__() For a datetime instance d, str(d) is equivalent to d.isoformat(' '). datetime.isoformat(sep='T') Return a string representing the date and time in ISO 8601 format, YYYY-MM-DDTHH:MM:SS.mmmmmm or, if microsecond is 0, YYYY-MM-DDTHH:MM:SS If utcoffset() does not return None, a 6-character string is appended, giving the UTC offset in (signed) hours and minutes: YYYY-MM-DDTHH:MM:SS.mmmmmm+HH:MM or, if microsecond is 0 YYYY-MM-DDTHH:MM:SS+HH:MM The optional argument sep (default 'T') is a one-character separator, placed between the date and time portions of the result. > %Y-%m-%d %H:%M:%S > %Y-%m-%d %H:%M:%S.%f > => plus timezone versions of the above. Wolfgang From steve at pearwood.info Tue Aug 5 03:39:44 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 5 Aug 2014 11:39:44 +1000 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> Message-ID: <20140805013944.GQ4525@ando> On Mon, Aug 04, 2014 at 10:56:56PM +0200, Wolfgang Maier wrote: [...] > it does hold true in 3.x, but the documented behavior is slightly more > complex (I assume also in 2.x): > > datetime.__str__() > For a datetime instance d, str(d) is equivalent to d.isoformat(' '). Since str(d) is documented to use a well-defined format, then I agree that it makes sense to make the second argument to d.strptime optional, and default to that same format. The concern I had was the sort of scenario Skip suggested: I might write out a datetime object as a string on one machine, where the format is X, and read it back elsewhere, where the format is Y, leading to at best an exception and at worse incorrect data. +1 on the suggestion. -- Steven From wolfgang.maier at biologie.uni-freiburg.de Tue Aug 5 23:22:11 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 05 Aug 2014 23:22:11 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <20140805013944.GQ4525@ando> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: On 05.08.2014 03:39, Steven D'Aprano wrote: > > Since str(d) is documented to use a well-defined format, then I agree > that it makes sense to make the second argument to d.strptime optional, > and default to that same format. The concern I had was the sort of > scenario Skip suggested: I might write out a datetime object as a string > on one machine, where the format is X, and read it back elsewhere, where > the format is Y, leading to at best an exception and at worse incorrect > data. > > +1 on the suggestion. > After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons: 1) strptime already has a clear counterpart and that's strftime. 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents. In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them. That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right. I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ? With this you could even pass an extra-argument for the date-time separator just like with the current isoformat. This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str(). -1 on the specifics of the proposal, +1 on the general idea. From j.wielicki at sotecware.net Tue Aug 5 23:28:22 2014 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Tue, 05 Aug 2014 23:28:22 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: <53E14C76.5010603@sotecware.net> On 05.08.2014 23:22, Wolfgang Maier wrote: > On 05.08.2014 03:39, Steven D'Aprano wrote: >> >> Since str(d) is documented to use a well-defined format, then I agree >> that it makes sense to make the second argument to d.strptime optional, >> and default to that same format. The concern I had was the sort of >> scenario Skip suggested: I might write out a datetime object as a string >> on one machine, where the format is X, and read it back elsewhere, where >> the format is Y, leading to at best an exception and at worse incorrect >> data. >> >> +1 on the suggestion. >> > > After looking a bit into the code of the datetime module, I am not > convinced anymore that strptime() is the right place for the > functionality for the following reasons: > > 1) strptime already has a clear counterpart and that's strftime. > > 2) strftime/strptime use explicit format strings, not any more > sophisticated parsing (as would be required to parse the different > formats that datetime.__str__ can produce) and they try, intentionally, > to mimick the behavior of their C equivalents. > > In other words, strftime/strptime have a very clear underlying concept, > which IMO should not be given up just because we are trying to stuff > some extra-functionality into them. > > That said, I still think that the basic idea - being able to > reverse-parse the output of datetime.__str__ - is right. > > I would suggest that a better place for this is an additional > classmethod constructor (the datetime class already has quite a number > of them). Maybe fromisostring() could be a suitable name ? Maybe rather fromisoformat(), to stay analogous with the formatting method? > With this you could even pass an extra-argument for the date-time > separator just like with the current isoformat. > This constructor would then be more like a counterpart to > datetime.isoformat(), but it could simply be documented that calling it > with fromisostring(datestring, sep=" ") can be used to parse strings > written with datetime.str(). > > -1 on the specifics of the proposal, > +1 on the general idea. +1 for this rating. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Tue Aug 5 23:35:06 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 5 Aug 2014 14:35:06 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: On Aug 5, 2014, at 14:22, Wolfgang Maier wrote: > On 05.08.2014 03:39, Steven D'Aprano wrote: >> >> Since str(d) is documented to use a well-defined format, then I agree >> that it makes sense to make the second argument to d.strptime optional, >> and default to that same format. The concern I had was the sort of >> scenario Skip suggested: I might write out a datetime object as a string >> on one machine, where the format is X, and read it back elsewhere, where >> the format is Y, leading to at best an exception and at worse incorrect >> data. >> >> +1 on the suggestion. > > After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons: > > 1) strptime already has a clear counterpart and that's strftime. > > 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents. > > In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them. What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses. > That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right. > > I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ? > With this you could even pass an extra-argument for the date-time separator just like with the current isoformat. > This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str(). Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months? What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name. > > -1 on the specifics of the proposal, > +1 on the general idea. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From encukou at gmail.com Tue Aug 5 23:46:10 2014 From: encukou at gmail.com (Petr Viktorin) Date: Tue, 5 Aug 2014 23:46:10 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: On Tue, Aug 5, 2014 at 11:35 PM, Andrew Barnert wrote: > On Aug 5, 2014, at 14:22, Wolfgang Maier wrote: > >> On 05.08.2014 03:39, Steven D'Aprano wrote: >>> >>> Since str(d) is documented to use a well-defined format, then I agree >>> that it makes sense to make the second argument to d.strptime optional, >>> and default to that same format. The concern I had was the sort of >>> scenario Skip suggested: I might write out a datetime object as a string >>> on one machine, where the format is X, and read it back elsewhere, where >>> the format is Y, leading to at best an exception and at worse incorrect >>> data. >>> >>> +1 on the suggestion. >> >> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons: >> >> 1) strptime already has a clear counterpart and that's strftime. >> >> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents. >> >> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them. > > What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses. +1 > >> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right. >> >> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ? >> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat. >> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str(). > > Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months? > > What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name. Let me just note this, since it hasn't been said here yet: When people say "iso" in the context of datestimes, they usually mean RFC 3339. As Wikipedia can tell you, ISO 8601 is a big complicated non-public specification under which today can be written as: - 2014-08-05 - 2014-W32-2 - 2014-217 ... and by now I can see why there's no ISO 8601 parser in the stdlib. RFC 3339, on the other hand, specifies one specific variant of ISO 8601: the one we're all used to, and which datetime's isoformat and __str__ return. (Just about the only exception is that to be compatible with ISO 8601, it still specifies "T"/"t" for the separator and graciously lets people agree on space.) From tjreedy at udel.edu Wed Aug 6 01:12:47 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 05 Aug 2014 19:12:47 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: On 8/5/2014 5:35 PM, Andrew Barnert wrote: > On Aug 5, 2014, at 14:22, Wolfgang Maier wrote: > >> On 05.08.2014 03:39, Steven D'Aprano wrote: >>> >>> Since str(d) is documented to use a well-defined format, then I agree >>> that it makes sense to make the second argument to d.strptime optional, >>> and default to that same format. The concern I had was the sort of >>> scenario Skip suggested: I might write out a datetime object as a string >>> on one machine, where the format is X, and read it back elsewhere, where >>> the format is Y, leading to at best an exception and at worse incorrect >>> data. >>> >>> +1 on the suggestion. >> >> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons: >> >> 1) strptime already has a clear counterpart and that's strftime. >> >> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents. >> >> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them. > > What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses. > >> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right. >> >> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ? >> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat. >> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str(). > > Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months? Probably yes > What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name. Given that str(dti) (datetime instance) is conceptually dt.tostr(dit), name the inverse as dti = dt.fromstr(s). -- Terry Jan Reedy From wolfgang.maier at biologie.uni-freiburg.de Wed Aug 6 10:35:57 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Wed, 06 Aug 2014 10:35:57 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: <53E1E8ED.6040403@biologie.uni-freiburg.de> On 05.08.2014 23:35, Andrew Barnert wrote: > On Aug 5, 2014, at 14:22, Wolfgang Maier wrote: > >> On 05.08.2014 03:39, Steven D'Aprano wrote: >>> >>> Since str(d) is documented to use a well-defined format, then I agree >>> that it makes sense to make the second argument to d.strptime optional, >>> and default to that same format. The concern I had was the sort of >>> scenario Skip suggested: I might write out a datetime object as a string >>> on one machine, where the format is X, and read it back elsewhere, where >>> the format is Y, leading to at best an exception and at worse incorrect >>> data. >>> >>> +1 on the suggestion. >> >> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons: >> >> 1) strptime already has a clear counterpart and that's strftime. >> >> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents. >> >> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them. > > What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses. > Yes, but strftime without format string would then be completely redundant with __str__ and isoformat with " " separator, which is really quite against the one and only one way of doing things idea. Plus again, right now strftime takes an explicit format string and then generates a datetime string with exactly this and only this format. In the optional format string scenario, it would have to generate slightly differently formatted output depending on whether there is microseconds and/or timezone information. So, like for strptime, this would change the very clearly defined current behavior into a mix of things, unnecessarily. >> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right. >> >> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ? >> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat. >> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str(). > > Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months? > > What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name. > Fair enough, it was just the first half-reasonable thing that came to my mind :) Being able to parse any valid ISO string would be another nice feature, but it's really a different story. Wolfgang From ethan at stoneleaf.us Wed Aug 6 12:34:49 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 06 Aug 2014 03:34:49 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <53E1E8ED.6040403@biologie.uni-freiburg.de> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> Message-ID: <53E204C9.30307@stoneleaf.us> On 08/06/2014 01:35 AM, Wolfgang Maier wrote: > > [...] which is really quite against the one and only one way of doing things idea. It's "One Obvious Way" not "Only One Way". -- ~Ethan~ From wolfgang.maier at biologie.uni-freiburg.de Wed Aug 6 12:40:33 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Wed, 06 Aug 2014 12:40:33 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <53E204C9.30307@stoneleaf.us> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us> Message-ID: On 06.08.2014 12:34, Ethan Furman wrote: > On 08/06/2014 01:35 AM, Wolfgang Maier wrote: >> >> [...] which is really quite against the one and only one way of doing >> things idea. > > It's "One Obvious Way" not "Only One Way". > I wasn't quoting, just paraphrasing. From abarnert at yahoo.com Wed Aug 6 15:48:44 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 6 Aug 2014 06:48:44 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <53E1E8ED.6040403@biologie.uni-freiburg.de> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> Message-ID: On Aug 6, 2014, at 1:35, Wolfgang Maier wrote: > On 05.08.2014 23:35, Andrew Barnert wrote: >> On Aug 5, 2014, at 14:22, Wolfgang Maier wrote: >> >>> On 05.08.2014 03:39, Steven D'Aprano wrote: >>>> >>>> Since str(d) is documented to use a well-defined format, then I agree >>>> that it makes sense to make the second argument to d.strptime optional, >>>> and default to that same format. The concern I had was the sort of >>>> scenario Skip suggested: I might write out a datetime object as a string >>>> on one machine, where the format is X, and read it back elsewhere, where >>>> the format is Y, leading to at best an exception and at worse incorrect >>>> data. >>>> >>>> +1 on the suggestion. >>> >>> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons: >>> >>> 1) strptime already has a clear counterpart and that's strftime. >>> >>> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents. >>> >>> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them. >> >> What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses. > > Yes, but strftime without format string would then be completely redundant with __str__ and isoformat with " " separator, which is really quite against the one and only one way of doing things idea. They're not redundant. str provides gives you some human-readable, ideally but not necessarily parseable, representation. isoformat gives you a specific format that you know is parseable by many other libraries and languages, and sorts in date order. strftime lets you specify a format to be parsed by specific code or problem-specific human expectations. Is the fact that they happen to overlap (which is already true, since you can always specify the same format explicitly if you want) any worse than the fact that str(3) and format(3, 'd') give you the same result? > Plus again, right now strftime takes an explicit format string and then generates a datetime string with exactly this and only this format. > In the optional format string scenario, it would have to generate slightly differently formatted output depending on whether there is microseconds and/or timezone information. So, like for strptime, this would change the very clearly defined current behavior into a mix of things, unnecessarily. The purpose of strftime and strptime is to be inverses of each other--to generate and parse datetime strings in a specified way. If one of those ways is "the default Python string representation" for one function, it should be true for the other. (Doesn't gnu strf/ptime have an extension that gives you % codes for "default" date and time representations, which don't guarantee anything other than that they be reasonable for the locale and reversible?) >>> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right. >>> >>> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ? >>> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat. >>> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str(). >> >> Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months? >> >> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name. > > Fair enough, it was just the first half-reasonable thing that came to my mind :) > Being able to parse any valid ISO string would be another nice feature, but it's really a different story. > > Wolfgang > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Wed Aug 6 16:02:20 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 6 Aug 2014 07:02:20 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> On Aug 5, 2014, at 14:46, Petr Viktorin wrote: > On Tue, Aug 5, 2014 at 11:35 PM, Andrew Barnert > wrote: >> On Aug 5, 2014, at 14:22, Wolfgang Maier wrote: >> >>> On 05.08.2014 03:39, Steven D'Aprano wrote: >>>> >>>> Since str(d) is documented to use a well-defined format, then I agree >>>> that it makes sense to make the second argument to d.strptime optional, >>>> and default to that same format. The concern I had was the sort of >>>> scenario Skip suggested: I might write out a datetime object as a string >>>> on one machine, where the format is X, and read it back elsewhere, where >>>> the format is Y, leading to at best an exception and at worse incorrect >>>> data. >>>> >>>> +1 on the suggestion. >>> >>> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons: >>> >>> 1) strptime already has a clear counterpart and that's strftime. >>> >>> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents. >>> >>> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them. >> >> What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses. > > +1 > >> >>> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right. >>> >>> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ? >>> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat. >>> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str(). >> >> Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months? >> >> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name. > > Let me just note this, since it hasn't been said here yet: > > When people say "iso" in the context of datestimes, they usually mean RFC 3339. RFC 3339 is still more complicated than just reversing Python's str or isoformat. IIRC (it's hard to check on my phone), it mandates that parsers should accept 2-digit years (including 3-digit or semicolon-and-two-digit years), lowercase T and Z, missing "-", and other things that you shouldn't generate but some code might. That being said, it's still obviously easier to write an RFC 3339 parser than a full ISO 8601 parser, and as long as someone is willing to write it (with sufficient tests) I don't see any problem with the stdlib having one. But I don't know that it should be called "fromisostring". "fromisoformat" isn't quite as bad, since at least it implies that it's the inverse of the same type's "isoformat", but it still seems misleading. From ethan at stoneleaf.us Wed Aug 6 16:05:48 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 06 Aug 2014 07:05:48 -0700 Subject: [Python-ideas] oow vs. oow [was: strptime without second argument as an inverse to __str__] In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us> Message-ID: <53E2363C.50405@stoneleaf.us> On 08/06/2014 03:40 AM, Wolfgang Maier wrote: > On 06.08.2014 12:34, Ethan Furman wrote: >> On 08/06/2014 01:35 AM, Wolfgang Maier wrote: >>> >>> [...] which is really quite against the one and only one way of doing >>> things idea. >> >> It's "One Obvious Way" not "Only One Way". >> > > I wasn't quoting, just paraphrasing. It's a bad paraphrase as the two have nearly completely different meanings. If you wish to pursue this sub-thread further I'll have to turn my portion over to D'Aprano (assuming he's willing) as he is much better at long explanations than I am. -- ~Ethan~ From abarnert at yahoo.com Wed Aug 6 16:04:52 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 6 Aug 2014 07:04:52 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: <79E4DDD2-7F63-4297-8D7C-566496C1BF05@yahoo.com> On Aug 5, 2014, at 16:12, Terry Reedy wrote: > On 8/5/2014 5:35 PM, Andrew Barnert wrote: >> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name. > > Given that str(dti) (datetime instance) is conceptually dt.tostr(dit), name the inverse as dti = dt.fromstr(s). Wow, now I feel stupid for not thinking of this one. +00:00:01 From ethan at stoneleaf.us Wed Aug 6 16:14:29 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 06 Aug 2014 07:14:29 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <20140805013944.GQ4525@ando> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> Message-ID: <53E23845.3030100@stoneleaf.us> On 08/04/2014 06:39 PM, Steven D'Aprano wrote: > On Mon, Aug 04, 2014 at 10:56:56PM +0200, Wolfgang Maier wrote: > [...] >> it does hold true in 3.x, but the documented behavior is slightly more >> complex (I assume also in 2.x): >> >> datetime.__str__() >> For a datetime instance d, str(d) is equivalent to d.isoformat(' '). > > Since str(d) is documented to use a well-defined format, then I agree > that it makes sense to make the second argument to d.strptime optional, > and default to that same format. The concern I had was the sort of > scenario Skip suggested: I might write out a datetime object as a string > on one machine, where the format is X, and read it back elsewhere, where > the format is Y, leading to at best an exception and at worse incorrect > data. What are the downsides of: dt = datetime.datetime.now() # assuming this works ;) sdt = str(dt) ndt = datetime.datetime(std) print(dt == ndt) #True -- ~Ethan~ From wolfgang.maier at biologie.uni-freiburg.de Wed Aug 6 17:19:30 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Wed, 06 Aug 2014 17:19:30 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <53E23845.3030100@stoneleaf.us> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: On 06.08.2014 16:14, Ethan Furman wrote: > On 08/04/2014 06:39 PM, Steven D'Aprano wrote: >> On Mon, Aug 04, 2014 at 10:56:56PM +0200, Wolfgang Maier wrote: >> [...] >>> it does hold true in 3.x, but the documented behavior is slightly more >>> complex (I assume also in 2.x): >>> >>> datetime.__str__() >>> For a datetime instance d, str(d) is equivalent to d.isoformat(' >>> '). >> >> Since str(d) is documented to use a well-defined format, then I agree >> that it makes sense to make the second argument to d.strptime optional, >> and default to that same format. The concern I had was the sort of >> scenario Skip suggested: I might write out a datetime object as a string >> on one machine, where the format is X, and read it back elsewhere, where >> the format is Y, leading to at best an exception and at worse incorrect >> data. > > What are the downsides of: > > dt = datetime.datetime.now() # assuming this works ;) > sdt = str(dt) > ndt = datetime.datetime(std) > print(dt == ndt) > #True > I'll refrain from mentioning "explicit is better than implicit" ;) It's just that it seems to be a design pattern of the datetime class to provide alternative constructors as classmethods instead of doing magic things in __new__ . There are fromtimestamp and utcfromtimestamp already, and you can think of datetime.now() the same way. After all, you could decide to have this called when datetime() is called without an argument. I guess there are just too many different things that *could* make sense to pass to __new__ and to be treated implicitly. Wolfgang From wolfgang.maier at biologie.uni-freiburg.de Wed Aug 6 17:22:47 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Wed, 06 Aug 2014 17:22:47 +0200 Subject: [Python-ideas] oow vs. oow [was: strptime without second argument as an inverse to __str__] In-Reply-To: <53E2363C.50405@stoneleaf.us> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us> <53E2363C.50405@stoneleaf.us> Message-ID: On 06.08.2014 16:05, Ethan Furman wrote: > On 08/06/2014 03:40 AM, Wolfgang Maier wrote: >> On 06.08.2014 12:34, Ethan Furman wrote: >>> On 08/06/2014 01:35 AM, Wolfgang Maier wrote: >>>> >>>> [...] which is really quite against the one and only one way of doing >>>> things idea. >>> >>> It's "One Obvious Way" not "Only One Way". >>> >> >> I wasn't quoting, just paraphrasing. > > It's a bad paraphrase as the two have nearly completely different meanings. > > If you wish to pursue this sub-thread further I'll have to turn my > portion over to D'Aprano (assuming he's willing) as he is much better at > long explanations than I am. > I don't think that's necessary :) I like focused threads ! From alexander.belopolsky at gmail.com Wed Aug 6 17:42:55 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 6 Aug 2014 11:42:55 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: On Wed, Aug 6, 2014 at 11:19 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > What are the downsides of: >> >> dt = datetime.datetime.now() # assuming this works ;) >> sdt = str(dt) >> ndt = datetime.datetime(std) >> print(dt == ndt) >> #True >> >> > I'll refrain from mentioning "explicit is better than implicit" ;) > > It's just that it seems to be a design pattern of the datetime class to > provide alternative constructors as classmethods instead of doing magic > things in __new__ . I don't think this is a "design pattern". In Python 2, having date(str) constructor was blocked by some magic that is there to support unpickling: >>> from datetime import date >>> date('\x07\xd0\x01\x01') datetime.date(2000, 1, 1) This is no longer an issue in Python 3. Note that if we allow date('2000-01-01'), this may become a more readable and efficient alternative to date(2001, 1, 1). -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.maier at biologie.uni-freiburg.de Wed Aug 6 18:20:37 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Wed, 06 Aug 2014 18:20:37 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: On 06.08.2014 17:42, Alexander Belopolsky wrote: > > On Wed, Aug 6, 2014 at 11:19 AM, Wolfgang Maier > > > wrote: > > What are the downsides of: > > dt = datetime.datetime.now() # assuming this works ;) > sdt = str(dt) > ndt = datetime.datetime(std) > print(dt == ndt) > #True > > > I'll refrain from mentioning "explicit is better than implicit" ;) > > It's just that it seems to be a design pattern of the datetime class > to provide alternative constructors as classmethods instead of doing > magic things in __new__ . > > > I don't think this is a "design pattern". In Python 2, having date(str) > constructor was blocked by some magic that is there to support unpickling: > > >>> from datetime import date > >>> date('\x07\xd0\x01\x01') > datetime.date(2000, 1, 1) > I see. None of my examples (fromtimestamp, utcfromtimestamp and now) involves string parsing though. > This is no longer an issue in Python 3. > > Note that if we allow date('2000-01-01'), this may become a more > readable and efficient alternative to date(2001, 1, 1). > One problem with this is the very first concern raised by Steven and Skip in this thread: choosing a string format that __new__ can deal with *would* lock you into this format. If later, for example, full-blown ISO 8601 or even just RFC 3339 parsing makes it into the module, wouldn't you rather want this to be done by __new__ when it sees a string ? Implementing the current proposal as a classmethod with its own name (once a good one is accepted) is a much more cautious approach. BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable. From ethan at stoneleaf.us Wed Aug 6 18:45:11 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 06 Aug 2014 09:45:11 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: <53E25B97.4010004@stoneleaf.us> On 08/06/2014 09:20 AM, Wolfgang Maier wrote: > On 06.08.2014 17:42, Alexander Belopolsky wrote: >> >> On Wed, Aug 6, 2014 at 11:19 AM, Wolfgang Maier >> > > >> wrote: >> >> What are the downsides of: >> >> dt = datetime.datetime.now() # assuming this works ;) >> sdt = str(dt) >> ndt = datetime.datetime(std) >> print(dt == ndt) >> #True >> >> >> I'll refrain from mentioning "explicit is better than implicit" ;) >> >> It's just that it seems to be a design pattern of the datetime class >> to provide alternative constructors as classmethods instead of doing >> magic things in __new__ . >> >> >> I don't think this is a "design pattern". In Python 2, having date(str) >> constructor was blocked by some magic that is there to support unpickling: >> >> >>> from datetime import date >> >>> date('\x07\xd0\x01\x01') >> datetime.date(2000, 1, 1) >> > > I see. None of my examples (fromtimestamp, utcfromtimestamp and now) involves string parsing though. > >> This is no longer an issue in Python 3. >> >> Note that if we allow date('2000-01-01'), this may become a more >> readable and efficient alternative to date(2001, 1, 1). >> > > One problem with this is the very first concern raised by Steven and Skip in this thread: choosing a string format that > __new__ can deal with *would* lock you into this format. > If later, for example, full-blown ISO 8601 or even just RFC 3339 parsing makes it into the module, wouldn't you rather > want this to be done by __new__ when it sees a string ? > Implementing the current proposal as a classmethod with its own name (once a good one is accepted) is a much more > cautious approach. > > BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable. For my own classes I accept both year, month, day, ..., or a single string in __new__. But for the stdlib I agree that .fromstr() is the better approach. +1 for .fromstr() -- ~Ethan~ From alexander.belopolsky at gmail.com Wed Aug 6 18:47:04 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 6 Aug 2014 12:47:04 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: On Wed, Aug 6, 2014 at 12:20 PM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable. We don't have int.fromstr, float.fromstr, or in fact a .fromstr constructor for any other type. IMO, date(str) is "the obvious way to do it." -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Aug 6 18:53:22 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 6 Aug 2014 12:53:22 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <53E25B97.4010004@stoneleaf.us> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <53E25B97.4010004@stoneleaf.us> Message-ID: On Wed, Aug 6, 2014 at 12:45 PM, Ethan Furman wrote: > For my own classes I accept both year, month, day, ..., or a single string > in __new__. > > But for the stdlib I agree that .fromstr() is the better approach. > Can you explain why what is good for your own classes is not good for stdlib? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Aug 6 19:01:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 06 Aug 2014 10:01:45 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: <53E25F79.1070507@stoneleaf.us> On 08/06/2014 09:47 AM, Alexander Belopolsky wrote: > On Wed, Aug 6, 2014 at 12:20 PM, Wolfgang Maier wrote: >> >> BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable. > > We don't have int.fromstr, float.fromstr, or in fact a .fromstr > constructor for any other type. IMO, date(str) is "the obvious > way to do it." int, float, and, I suspect, all the core data types, have the same __str__ as __repr__, so there's really no difference. datetime objects, on the other hand, definitely have different __str__ and __repr__, and the desire is to be able no eval(obj.__repr__()), not of __str__. -- ~Ethan~ From ethan at stoneleaf.us Wed Aug 6 19:02:19 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 06 Aug 2014 10:02:19 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <53E25B97.4010004@stoneleaf.us> Message-ID: <53E25F9B.4080801@stoneleaf.us> On 08/06/2014 09:53 AM, Alexander Belopolsky wrote: > On Wed, Aug 6, 2014 at 12:45 PM, Ethan Furman wrote: >> >> For my own classes I accept both year, month, day, ..., or a single string in __new__. >> >> But for the stdlib I agree that .fromstr() is the better approach. > > Can you explain why what is good for your own classes is not good for stdlib? I tolerate a higher level of risk in my own work, but the stdlib should be more stable. -- ~Ethan~ From tjreedy at udel.edu Wed Aug 6 21:26:16 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 06 Aug 2014 15:26:16 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: On 8/6/2014 12:47 PM, Alexander Belopolsky wrote: > > On Wed, Aug 6, 2014 at 12:20 PM, Wolfgang Maier > > > wrote: > > BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable. > > > We don't have int.fromstr, float.fromstr, or in fact a .fromstr > constructor for any other type. IMO, date(str) is "the obvious way to > do it." There are two parts to my suggestion. The first is to focus on the original goal of the thread, 'an inverse to __str__', and only that, not on automatically parsing, without providing a format, larger classes of possible strings. The second is to suggest a better spelling for that original goal than the original proposal (strptime without second argument) or alternate proposals like .fromisostring based on an expanded goal. I think '.fromstr' is the best possible spelling among '.from...' choices. If instead using the standard constructor works, fine with me. I am not sure of the criterion for adding alternative to .__init__ versus an alterntive method. -- Terry Jan Reedy From alexander.belopolsky at gmail.com Wed Aug 6 22:11:24 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 6 Aug 2014 16:11:24 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: On Wed, Aug 6, 2014 at 3:26 PM, Terry Reedy wrote: > I think '.fromstr' is the best possible spelling among '.from...' choices. With this I agree. Since the path is already paved with float.fromhex, I am casting my +0 for date/datetime.fromstr. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Aug 7 01:55:25 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 6 Aug 2014 16:55:25 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> Message-ID: <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> On Aug 6, 2014, at 8:42, Alexander Belopolsky wrote: > On Wed, Aug 6, 2014 at 11:19 AM, Wolfgang Maier wrote: >>> What are the downsides of: >>> >>> dt = datetime.datetime.now() # assuming this works ;) >>> sdt = str(dt) >>> ndt = datetime.datetime(std) >>> print(dt == ndt) >>> #True >> >> I'll refrain from mentioning "explicit is better than implicit" ;) >> >> It's just that it seems to be a design pattern of the datetime class to provide alternative constructors as classmethods instead of doing magic things in __new__ . > > I don't think this is a "design pattern". In Python 2, having date(str) constructor was blocked by some magic that is there to support unpickling: > > >>> from datetime import date > >>> date('\x07\xd0\x01\x01') > datetime.date(2000, 1, 1) > > This is no longer an issue in Python 3. > > Note that if we allow date('2000-01-01'), this may become a more readable and efficient alternative to date(2001, 1, 1). More readable maybe, but more efficient? You're doing the same work, plus string parsing; you're eliminating two parameters (but only if you use *args) at the cost of three locals; by any measure it's less efficient. But more readable is the important part. In isolation it's readable, the question is whether the added complexity (in the docs and in people's heads) of an effectively-overloaded constructor is worth the cost. Given that, unlike all the obvious parallel cases (int, float, etc.) this constructor will not accept the repr, I'm not sure the answer comes out the same. But maybe it does. I'm -0 on this, +1 on Terry's fromstr, +0 on strptime and strftime both accepting no arguments, -1 on only strptime, -0.5 on fromisostring/fromisoformat, and -1 on remembering any of the other ideas in this thread well enough to comment. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Aug 7 02:10:36 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 7 Aug 2014 10:10:36 +1000 Subject: [Python-ideas] oow vs. oow [was: strptime without second argument as an inverse to __str__] In-Reply-To: <53E2363C.50405@stoneleaf.us> References: <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us> <53E2363C.50405@stoneleaf.us> Message-ID: <20140807001036.GU4525@ando> On Wed, Aug 06, 2014 at 07:05:48AM -0700, Ethan Furman wrote: > On 08/06/2014 03:40 AM, Wolfgang Maier wrote: > >On 06.08.2014 12:34, Ethan Furman wrote: > >>On 08/06/2014 01:35 AM, Wolfgang Maier wrote: > >>> > >>>[...] which is really quite against the one and only one way of doing > >>>things idea. > >> > >>It's "One Obvious Way" not "Only One Way". > >> > > > >I wasn't quoting, just paraphrasing. > > It's a bad paraphrase as the two have nearly completely different meanings. > > If you wish to pursue this sub-thread further I'll have to turn my portion > over to D'Aprano (assuming he's willing) as he is much better at long > explanations than I am. I can make it a short explanation :-) "Only One Way" implies: assert len(collection_of_ways) == 1 "One Obvious Way" implies: assert any(way.is_obvious() for way in collection_of_ways) Now I suppose I'll have to go back and read the rest of the thread to understand what this is about *wink* -- Steven From alexander.belopolsky at gmail.com Thu Aug 7 02:19:40 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 6 Aug 2014 20:19:40 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert wrote: > Given that, unlike all the obvious parallel cases (int, float, etc.) this > constructor will not accept the repr, I'm not sure the answer comes out the > same. The parallel is in accepting str, not repr. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at ryanhiebert.com Thu Aug 7 02:27:35 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Wed, 6 Aug 2014 19:27:35 -0500 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: <29D6F3B3-0297-4FB3-9315-B9C18F78A82F@ryanhiebert.com> > On Aug 6, 2014, at 7:19 PM, Alexander Belopolsky wrote: > > > On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert > wrote: > Given that, unlike all the obvious parallel cases (int, float, etc.) this constructor will not accept the repr, I'm not sure the answer comes out the same. > > The parallel is in accepting str, not repr. Indeed. The precedent for repr is that may be eval-able, not that the repr string can be passed into the constructor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Aug 7 02:29:12 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 6 Aug 2014 20:29:12 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert wrote: > More readable maybe, but more efficient? You're doing the same work, plus > string parsing; you're eliminating two parameters (but only if you use > *args) at the cost of three locals; by any measure it's less efficient. >>> dis(lambda: date(2001, 1, 1)) 1 0 LOAD_GLOBAL 0 (date) 3 LOAD_CONST 1 (2001) 6 LOAD_CONST 2 (1) 9 LOAD_CONST 2 (1) 12 CALL_FUNCTION 3 15 RETURN_VALUE >>> dis(lambda: date('2001-01-01')) 1 0 LOAD_GLOBAL 0 (date) 3 LOAD_CONST 1 ('2001-01-01') 6 CALL_FUNCTION 1 9 RETURN_VALUE Since parsing will be done in C, it's cost can be made negligible. In implementations other than CPython, YMMV. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Aug 7 02:32:28 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 7 Aug 2014 10:32:28 +1000 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> Message-ID: <20140807003228.GV4525@ando> On Wed, Aug 06, 2014 at 06:48:44AM -0700, Andrew Barnert wrote: > The purpose of strftime and strptime is to be inverses of each > other--to generate and parse datetime strings in a specified way. If > one of those ways is "the default Python string representation" for > one function, it should be true for the other. As of Python 3.3, neither strftime nor strptime take a default format. It's only __str__ which has an implicit default format. > (Doesn't gnu strf/ptime > have an extension that gives you % codes for "default" date and time > representations, which don't guarantee anything other than that they > be reasonable for the locale and reversible?) I worry about something like that. Unless the default is guaranteed to be a particular format, what counts as "reasonable" when the string is written out and when read back in may not be the same. -- Steven From skip at pobox.com Thu Aug 7 03:06:25 2014 From: skip at pobox.com (Skip Montanaro) Date: Wed, 6 Aug 2014 20:06:25 -0500 Subject: [Python-ideas] oow vs. oow [was: strptime without second argument as an inverse to __str__] In-Reply-To: <20140807001036.GU4525@ando> References: <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us> <53E2363C.50405@stoneleaf.us> <20140807001036.GU4525@ando> Message-ID: On Wed, Aug 6, 2014 at 7:10 PM, Steven D'Aprano wrote: > Now I suppose I'll have to go back and read the rest of the thread to > understand what this is about *wink* Might be more fun to recast the Zen of Python into Python itself. :-) Skip From abarnert at yahoo.com Thu Aug 7 04:10:49 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 6 Aug 2014 19:10:49 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <20140807003228.GV4525@ando> References: <20140805013944.GQ4525@ando> <53E1E8ED.6040403@biologie.uni-freiburg.de> <20140807003228.GV4525@ando> Message-ID: <595C7047-90AF-4538-9756-8679E84C9AC2@yahoo.com> On Aug 6, 2014, at 17:32, Steven D'Aprano wrote: > On Wed, Aug 06, 2014 at 06:48:44AM -0700, Andrew Barnert wrote: > >> The purpose of strftime and strptime is to be inverses of each >> other--to generate and parse datetime strings in a specified way. If >> one of those ways is "the default Python string representation" for >> one function, it should be true for the other. > > As of Python 3.3, neither strftime nor strptime take a default format. > It's only __str__ which has an implicit default format. Exactly my point. They're currently balanced. Adding a default format to strptime only would mean they can no longer be used as inverses of each other. One solution is to also add a default format to strftime. The other solution is to recognize that if the desire is an inverse for str, strptime is not the best way to write that. (I didn't have a good alternative suggestion, but Terry's later fromstr seems perfect to me.) >> (Doesn't gnu strf/ptime >> have an extension that gives you % codes for "default" date and time >> representations, which don't guarantee anything other than that they >> be reasonable for the locale and reversible?) > > I worry about something like that. Unless the default is guaranteed to > be a particular format, what counts as "reasonable" when the string is > written out and when read back in may not be the same. I share that worry. Is it guaranteed that str on any Python anywhere can be parsed back to the same value on a different Python somewhere else? The docs for str (and the presumed docs got the new function) can make that guarantee, but most of the people in this thread didn't know that, or weren't confident in it. That's why I don't really like the idea of the inverse functions being __str__ and the constructor--it isn't obvious or explicit to the reader, and it's not quite as easy to look up. (That wasn't the initial proposal, but it came up later in the thread, so it was worth responding to.) From abarnert at yahoo.com Thu Aug 7 04:24:33 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 6 Aug 2014 19:24:33 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: On Aug 6, 2014, at 17:19, Alexander Belopolsky wrote: > > On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert wrote: >> Given that, unlike all the obvious parallel cases (int, float, etc.) this constructor will not accept the repr, I'm not sure the answer comes out the same. > > The parallel is in accepting str, not repr. Is it? Sure, I'll accept that's the parallel you have in mind, but is it a good one? The only way I can think to distinguish is this: For bytes, str, tuple, etc., there is no constructor from either string representation. For int, the two representations are identical. For float, they're different--and it's float(repr(f)) that gives you back the same value you started with. (Of course in the other direction, neither one is guaranteed to do so.) Also note that int can accept hex, etc. Python literal values as strings, the same ones eval can. Is there some documentation that implies that the int, float, etc. constructors are meant to take "human-readable" strings rather than "Python-evaluable" strings? Or some other case I'm missing? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.maier at biologie.uni-freiburg.de Thu Aug 7 10:17:13 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 07 Aug 2014 10:17:13 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: On 07.08.2014 02:29, Alexander Belopolsky wrote: > > On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert > > wrote: > > More readable maybe, but more efficient? You're doing the same work, > plus string parsing; you're eliminating two parameters (but only if > you use *args) at the cost of three locals; by any measure it's less > efficient. > > Since parsing will be done in C, it's cost can be made negligible. In > implementations other than CPython, YMMV. > Why would parsing occur in C ? The datetime module is implemented in pure Python. From wolfgang.maier at biologie.uni-freiburg.de Thu Aug 7 11:57:47 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 07 Aug 2014 11:57:47 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: <53E34D9B.5040004@biologie.uni-freiburg.de> On 07.08.2014 01:55, Andrew Barnert wrote: > On Aug 6, 2014, at 8:42, Alexander Belopolsky > > > wrote: > >> Note that if we allow date('2000-01-01'), this may become a more >> readable and efficient alternative to date(2001, 1, 1). > > I'm -0 on this, +1 on Terry's fromstr, +0 on strptime and strftime both > accept ing no arguments, -1 on only strptime, -0.5 on > fromisostring/fromisoformat, and -1 on remembering any of the other > ideas in this thread well enough to comment. > So to summarize, the three currently discussed options (in order of current votes for) are: - datetime.fromstr(string) classmethod to be used as an alternative constructor - datetime (string) add limited string parsing to the default constructor - datetime.strptime (string) parse datetime.__str__ format when format argument is not specified (the OP's original proposal) A point that has not been discussed yet is that the first two options could easily be implemented for datetime.date and datetime.time objects as well providing counterparts for date.__str_ and time.__str__ . With strptime this is not possible since datetime.date and datetime.time don't have such a method. In addition, the first two options, in particular, raise a scope question. I can see the following options: - string has to be of exactly the format generated by datetime.__str__, i.e. YYYY-MM-DD HH:MM:SS with optional microseconds and timezone information (the original proposal) - string has to be of a format that can be generated by datetime.isoformat, of which the datetime.__str__ format is a subset. The difference is that with datetime.isoformat the separator between the date and time portions of the string can be specified, while with datetime.__str__ this is fixed to " ". Accordingly with this option, you would have either: datetime.fromstr(string, sep = " ") or datetime(string, sep = " ") to be able to pass an optional separator. When absent the datetime.__str__ format is expected. - (potentially, but I don't think anyone opted for it: more powerful parsing of a wider range of formats) Personally, I'm in favor of the second option here for the following reason: the string format returned by datetime.__str__ is not fully ISO 8601 compatible because it uses " " as the separator instead of "T", i.e. if an application has to produce ISO 8601 compliant output it has to use datetime.isoformat not __str__, even though the two formats differ by just a single character. Hence, allowing the separator to be specified makes the new functionality a lot more useful at the expense of only a moderate increase in complexity. Note that this is still fundamentally different from asking for any full parsing for ISO 8601 and that it would have no impact on potential date.fromstr and time.fromstr methods or their default constructor versions since they do not have to deal with a separator. So my preferred complete version would be something like: datetime.fromstr (string, sep = ' ') Return a datetime object from a string as generated by datetime.isoformat(sep). With the default value of sep this is also the format generated by datetime.__str__(). date.fromstr (string) Return a date object from a string as generated by date.__str__() and date.isoformat(). time.fromstr (string) Return a time object from a string as generated by time.__str__() and time.isoformat(). Wolfgang From encukou at gmail.com Thu Aug 7 12:30:37 2014 From: encukou at gmail.com (Petr Viktorin) Date: Thu, 7 Aug 2014 12:30:37 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <53E34D9B.5040004@biologie.uni-freiburg.de> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> <53E34D9B.5040004@biologie.uni-freiburg.de> Message-ID: On Thu, Aug 7, 2014 at 11:57 AM, Wolfgang Maier wrote: > On 07.08.2014 01:55, Andrew Barnert wrote: >> >> On Aug 6, 2014, at 8:42, Alexander Belopolsky >> > > >> >> wrote: >> >>> Note that if we allow date('2000-01-01'), this may become a more >>> readable and efficient alternative to date(2001, 1, 1). >> >> >> I'm -0 on this, +1 on Terry's fromstr, +0 on strptime and strftime both >> accept ing no arguments, -1 on only strptime, -0.5 on >> >> fromisostring/fromisoformat, and -1 on remembering any of the other >> ideas in this thread well enough to comment. >> > > So to summarize, the three currently discussed options (in order of current > votes for) are: > > - datetime.fromstr(string) > classmethod to be used as an alternative constructor > > - datetime (string) > add limited string parsing to the default constructor > > - datetime.strptime (string) > parse datetime.__str__ format when format argument is not specified > (the OP's original proposal) > > A point that has not been discussed yet is that the first two options could > easily be implemented for datetime.date and datetime.time objects as well > providing counterparts for date.__str_ and time.__str__ . > With strptime this is not possible since datetime.date and datetime.time > don't have such a method. > > > In addition, the first two options, in particular, raise a scope question. I > can see the following options: > > - string has to be of exactly the format generated by datetime.__str__, i.e. > YYYY-MM-DD HH:MM:SS with optional microseconds and timezone information (the > original proposal) > > - string has to be of a format that can be generated by datetime.isoformat, > of which the datetime.__str__ format is a subset. > The difference is that with datetime.isoformat the separator between the > date and time portions of the string can be specified, while with > datetime.__str__ this is fixed to " ". > Accordingly with this option, you would have either: > > datetime.fromstr(string, sep = " ") or > datetime(string, sep = " ") > > to be able to pass an optional separator. When absent the datetime.__str__ > format is expected. > > - (potentially, but I don't think anyone opted for it: > more powerful parsing of a wider range of formats) > > > Personally, I'm in favor of the second option here for the following reason: > the string format returned by datetime.__str__ is not fully ISO 8601 > compatible because it uses " " as the separator instead of "T", i.e. if an > application has to produce ISO 8601 compliant output it has to use > datetime.isoformat not __str__, even though the two formats differ by just a > single character. Hence, allowing the separator to be specified makes the > new functionality a lot more useful at the expense of only a moderate > increase in complexity. > Note that this is still fundamentally different from asking for any full > parsing for ISO 8601 and that it would have no impact on potential > date.fromstr and time.fromstr methods or their default constructor versions > since they do not have to deal with a separator. According to Wikipedia's quote of the standard: "the character [T] may be omitted in applications where there is no risk of confusing a date and time of day representation with others defined in this International Standard." If you give the string to a deatetime constructor, there is obviously no such confusion. I'm not sure if replacing by space counts as omitting, but it's the de-facto standard :) Anyway, i'd prefer not taking sep as an argument, but accepting both ' ' and 'T', since other separators aren't widely used. Also, note that we while a full ISO parser may not be feasible, we can (and IMO should) implement a complete RFC 3339 parser (with optional space separator), and document that both __str__ and fromstr (or whatever it ends up to be) is indeed is compatible with RFC 3339. From 4kir4.1i at gmail.com Thu Aug 7 14:35:09 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 07 Aug 2014 16:35:09 +0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> Message-ID: <87vbq4a35u.fsf@gmail.com> Andrew Barnert writes: > On Aug 5, 2014, at 14:46, Petr Viktorin > wrote: >> When people say "iso" in the context of datestimes, they usually mean RFC 3339. > > RFC 3339 is still more complicated than just reversing Python's str or > isoformat. IIRC (it's hard to check on my phone), it mandates that > parsers should accept 2-digit years (including 3-digit or > semicolon-and-two-digit years), lowercase T and Z, missing "-", and > other things that you shouldn't generate but some code might. Please, don't spread misinformation. Among the explicit rfc 3339 design goals are simplicity and human readability. Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is relatively short and readable). Here's full ABNF: date-fullyear = 4DIGIT date-month = 2DIGIT ; 01-12 date-mday = 2DIGIT ; 01-28, 01-29, 01-30, 01-31 based on ; month/year time-hour = 2DIGIT ; 00-23 time-minute = 2DIGIT ; 00-59 time-second = 2DIGIT ; 00-58, 00-59, 00-60 based on leap second ; rules time-secfrac = "." 1*DIGIT time-numoffset = ("+" / "-") time-hour ":" time-minute time-offset = "Z" / time-numoffset partial-time = time-hour ":" time-minute ":" time-second [time-secfrac] full-date = date-fullyear "-" date-month "-" date-mday full-time = partial-time time-offset date-time = full-date "T" full-time Example: 1937-01-01T12:00:27.87+00:20 The format is so simple that people just write adhoc parsers using strptime() without installing any formal rfc3339 module (if it even exists). > That being said, it's still obviously easier to write an RFC 3339 > parser than a full ISO 8601 parser, and as long as someone is willing > to write it (with sufficient tests) I don't see any problem with the > stdlib having one. But I don't know that it should be called > "fromisostring". > > "fromisoformat" isn't quite as bad, since at least it implies that > it's the inverse of the same type's "isoformat", but it still seems > misleading. -- Akira From abarnert at yahoo.com Thu Aug 7 16:52:03 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 7 Aug 2014 07:52:03 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <87vbq4a35u.fsf@gmail.com> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> <87vbq4a35u.fsf@gmail.com> Message-ID: On Aug 7, 2014, at 5:35, Akira Li <4kir4.1i at gmail.com> wrote: > Andrew Barnert > writes: > >> On Aug 5, 2014, at 14:46, Petr Viktorin >> wrote: >>> When people say "iso" in the context of datestimes, they usually mean RFC 3339. >> >> RFC 3339 is still more complicated than just reversing Python's str or >> isoformat. IIRC (it's hard to check on my phone), it mandates that >> parsers should accept 2-digit years (including 3-digit or >> semicolon-and-two-digit years), lowercase T and Z, missing "-", and >> other things that you shouldn't generate but some code might. > > Please, don't spread misinformation. > > Among the explicit rfc 3339 design goals are simplicity and human > readability. > > Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is > relatively short and readable). OK, I just read it. Among other things: > NOTE: Per [ABNF] and ISO8601, the "T" and "Z" characters in this syntax may alternatively be lower case "t" or "z" respectively. This date/time format may be used in some environments or contexts that distinguish between the upper- and lower-case letters 'A'-'Z' and 'a'-'z' (e.g. XML). Specifications that use this format in such environments MAY further limit the date/time syntax so that the letters 'T' and 'Z' used in the date/time syntax must always be upper case. Applications that generate this format SHOULD use upper case letters. NOTE: ISO 8601 defines date and time separated by "T". Applications using this syntax may choose, for the sake of readability, to specify a full-date and full-time separated by (say) a space character. Klyne, et. al. Standards Track [Page 8] (And there's also a whole section of interpreting "legacy"/"deprecated" 2-digit years and how you should handle them.) So, is the RFC "spreading misinformation" about itself? > The format is so simple that people just write adhoc parsers using strptime() without installing any formal rfc3339 module (if it even exists). Sure, and people also do that and call it an ISO parser. If it can't interoperable with everything compliant applications may generate (much less deprecated formats the standard doesn't allow you to generate but mandates how you parse and interpret), it's not accurate to call it an RFC 3339 parser (at least not in a general-purpose library). As I said before, it's still certainly much easier to write an RFC 3339 parser than an ISO 8601 parser, but it's not as trivial as you're implying. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Aug 7 19:27:06 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 7 Aug 2014 13:27:06 -0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: On Thu, Aug 7, 2014 at 4:17 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > >> Since parsing will be done in C, it's cost can be made negligible. In >> implementations other than CPython, YMMV. >> >> > Why would parsing occur in C ? The datetime module is implemented in pure > Python. No. In CPython, datetime module is implemented in C. http://hg.python.org/cpython/file/default/Modules/_datetimemodule.c -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Aug 7 19:48:58 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 8 Aug 2014 03:48:58 +1000 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: On 8 Aug 2014 03:27, "Alexander Belopolsky" wrote: > > > On Thu, Aug 7, 2014 at 4:17 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: >>> >>> >>> Since parsing will be done in C, it's cost can be made negligible. In >>> implementations other than CPython, YMMV. >>> >> >> Why would parsing occur in C ? The datetime module is implemented in pure Python. > > > No. In CPython, datetime module is implemented in C. > > http://hg.python.org/cpython/file/default/Modules/_datetimemodule.c Don't we have both these days? (C accelerator with pure Python fallback) Anyway, I'm +1 for Wolfgang's trio of "fromstr" alternative constructors, but the suggested variation that allows both " " and "T" as the separator, rather than accepting a parameter. Anyone wanting more flexibility can use strptime, or else switch to something like dateutil. Cheers, Nick. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.maier at biologie.uni-freiburg.de Thu Aug 7 22:40:56 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 07 Aug 2014 22:40:56 +0200 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <53E23845.3030100@stoneleaf.us> <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com> Message-ID: <53E3E458.3060302@biologie.uni-freiburg.de> On 07.08.2014 19:27, Alexander Belopolsky wrote: > > On Thu, Aug 7, 2014 at 4:17 AM, Wolfgang Maier > > > wrote: > > > Since parsing will be done in C, it's cost can be made > negligible. In > implementations other than CPython, YMMV. > > > Why would parsing occur in C ? The datetime module is implemented in > pure Python. > > > No. In CPython, datetime module is implemented in C. > > http://hg.python.org/cpython/file/default/Modules/_datetimemodule.c > Just to make sure I understood things right: new additions to the datetime module would normally be implemented in the Python version. But then I was forgetting that your suggestion was about changing the default constructor, which would have to happen in the C version. Correct ? From 4kir4.1i at gmail.com Fri Aug 8 07:43:32 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Fri, 08 Aug 2014 09:43:32 +0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> <87vbq4a35u.fsf@gmail.com> Message-ID: <87lhqza64b.fsf@gmail.com> Andrew Barnert writes: > On Aug 7, 2014, at 5:35, Akira Li <4kir4.1i at gmail.com> wrote: > >> Andrew Barnert >> writes: >> >>> On Aug 5, 2014, at 14:46, Petr Viktorin >>> wrote: >>>> When people say "iso" in the context of datestimes, they usually mean RFC 3339. >>> >>> RFC 3339 is still more complicated than just reversing Python's str or >>> isoformat. IIRC (it's hard to check on my phone), it mandates that >>> parsers should accept 2-digit years (including 3-digit or >>> semicolon-and-two-digit years), lowercase T and Z, missing "-", and >>> other things that you shouldn't generate but some code might. >> >> Please, don't spread misinformation. >> >> Among the explicit rfc 3339 design goals are simplicity and human >> readability. >> >> Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is >> relatively short and readable). ... > (And there's also a whole section of interpreting > "legacy"/"deprecated" 2-digit years and how you should handle them.) > > So, is the RFC "spreading misinformation" about itself? You are *obviously* wrong for the rfc 3339 Internet Date/Time Format itself (used by __str__, isoformat -- relevant to the current topic). http://tools.ietf.org/html/rfc3339 You can be *subtly* wrong for the rfc as a whole. The full ABNF from my previous message contains date-fullyear definition. "The following profile of ISO 8601 [ISO8601] dates SHOULD be used in new protocols on the Internet" [rfc3339]: date-fullyear = 4DIGIT It means that the year SHOULD be *exactly* 4 digits i.e., the rfc 3339 Internet Date/Time Format uses only 4-digit years. I don't know what "mandates that parsers should accept 2-digit years" [Andrew Barnert] means (does "mandates" mean SHOULD or MUST here?) but it seems inspired by: "Internet Protocols MUST generate four digit years in dates. The use of 2-digit years is deprecated. If a 2-digit year is received, it should be accepted ONLY if an incorrect interpretation will not cause a protocol or processing failure" [rfc3339] and the words MUST and SHOULD are well-defined [rfc2119]. Do you see that 2-digit year MUST or SHOULD be accepted? (I don't see it). 'missing "-"' [Andrew Barnert] seems like an obvious mistake. All punctuaction except optional time-secfrac (fraction of a second) is mandatory. "and other things that you shouldn't generate but some code might." [Andrew Barnert] What things? Could you be more specific? There is not much room in the format. Redundant information is not included. >> The format is so simple that people just write adhoc parsers using > strptime() without installing any formal rfc3339 module (if it even > exists). > > Sure, and people also do that and call it an ISO parser. > > If it can't interoperable with everything compliant applications may > generate (much less deprecated formats the standard doesn't allow you > to generate but mandates how you parse and interpret), it's not > accurate to call it an RFC 3339 parser (at least not in a > general-purpose library). > > As I said before, it's still certainly much easier to write an RFC > 3339 parser than an ISO 8601 parser, but it's not as trivial as you're > implying. Compared to the full ISO 8601 format (I don't know whether it can be parsed unambiguously), rfc 3339 (a conformant subset of the ISO 8601 extended format) is simple by design. It *is* accurate to call it (with only 4-digit year support) an rfc 3339 parser in a general purpose library: compliant software MUST generate 4-digit year, Internet Date/Time Format SHOULD contain ONLY 4-digit year (show quotes from the rfc that contain MUST, SHOULD, etc that say otherwise if you disagree). To be fair, the rfc *does not forbid* 2-digit year outright in *all possible cases*. Regardless the rfc language nuances, 2-digit year is harmful in practice -- different software may interpret it differently: software that is aware of rfc 3339 MUST generate 4-digit year, software that is not aware of rfc 3339 can interpret 2-digit year differently. If an error is possible when 2-digit year should not be used: "it should be accepted ONLY if an incorrect interpretation will not cause a protocol or processing failure" [rfc3339] P.S. ... > > OK, I just read it. Among other things: > >> NOTE: Per [ABNF] and ISO8601, the "T" and "Z" characters in this >> syntax may alternatively be lower case "t" or "z" respectively. This >> date/time format may be used in some environments or contexts that >> distinguish between the upper- and lower-case letters 'A'-'Z' and >> a'-'z' (e.g. XML). Specifications that use this format in such >> environments MAY further limit the date/time syntax so that the >> letters 'T' and 'Z' used in the date/time syntax must always be >> upper case. Applications that generate this format SHOULD use upper >> case letters. NOTE: ISO 8601 defines date and time separated by >> "T". Applications using this syntax may choose, for the sake of >> readability, to specify a full-date and full-time separated by (say) >> a space character. Klyne, et. al. Standards Track [Page 8] What is the point of the copy-paste? Does handling lowercase "t", "z", and a space complicates the parsing in a meaningful manner? Imagine a parser that supports sep='T' (default for isoformat()) and sep=' ' (space for __str__). How hard do you think to extend such parser to support 't' as a separator? -- Akira From abarnert at yahoo.com Fri Aug 8 08:10:57 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 7 Aug 2014 23:10:57 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <87lhqza64b.fsf@gmail.com> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> <87vbq4a35u.fsf@gmail.com> <87lhqza64b.fsf@gmail.com> Message-ID: <1407478257.58223.YahooMailNeo@web181006.mail.ne1.yahoo.com> Mixing up responses to an previous email you already responded to with the new one makes it harder to reply, but I'll try. On Thursday, August 7, 2014 10:43 PM, Akira Li <4kir4.1i at gmail.com> wrote: > > Andrew Barnert writes: > >> On Aug 7, 2014, at 5:35, Akira Li <4kir4.1i at gmail.com> wrote: >> >>> Please, don't spread misinformation. >>> >>> Among the explicit rfc 3339 design goals are simplicity and human >>> readability. >>> >>> Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is >>> relatively short and readable). > ... >> (And there's also a whole section of interpreting >> "legacy"/"deprecated" 2-digit years and how you should > handle them.) >> >> So, is the RFC "spreading misinformation" about itself? > > You are *obviously* wrong for the rfc 3339 Internet Date/Time Format > itself (used by __str__, isoformat -- relevant to the current topic). You accused me of "spreading misinformation" by saying that RFC 3339 is more complicated than what Python's str generates. You also suggested that people often parse RFC 3339 with a simple strptime. I pointed out that the RFC itself clearly defines something more complicated than what Python's str generates, and that it can't be fully parsed by a simple strptime, and then added a parenthetical remark about support for 2-year dates. You cut out everything but the parenthetical remark, and replied to that as if it was the whole point of my message. And you even responded to nothing but the parenthetical 2-year comment in replies to completely separate sections of the message. I have no idea how to respond to that except to say: read it again. The fact that a compliant parser should accept any of "T", "t", or " " as a separator already makes it impossible to parse with strptime, and more complicated than parsing Python's str output. I don't see how you can dispute that, or how it's "spreading misinformation" to point that out. > ? "and other things that you shouldn't generate but some code? > might." > ? [Andrew Barnert] > > What things? Could you be more specific? I was specific, and you chopped it out of my message, or moved it to a different part of the message. >>> The format is so simple that people just write adhoc parsers using >> strptime() without installing any formal rfc3339 module (if it even >> exists). >> >> Sure, and people also do that and call it an ISO parser. >> >> If it can't interoperable with everything compliant applications may >> generate (much less deprecated formats the standard doesn't allow you >> to generate but mandates how you parse and interpret), it's not >> accurate to call it an RFC 3339 parser (at least not in a >> general-purpose library). >> >> As I said before, it's still certainly much easier to write an RFC >> 3339 parser than an ISO 8601 parser, but it's not as trivial as > you're >> implying. > > Compared to the full ISO 8601 format (I don't know whether it can be > parsed unambiguously), rfc 3339 (a conformant subset of the ISO 8601 > extended format) is simple by design. Since you're arguing here exactly what I said, in the very paragraph you just quoted?"it's certainly much easier to write an RFC 3339 parser than an ISO 8601 parser"?I don't know why you think you have to convince me of that fact. But again, that doesn't mean that a trivial strptime call is sufficient for a general-purpose library that claims to be a compliant RFC 3339 parser. > Regardless the rfc language nuances, 2-digit year is harmful in practice Of course. So what? If my point actually were that RFC 3339 was complicated because of 2-digit years, and if that point were true, then this rebuttal might be relevant?but in that case it would only be proving the opposite point you're trying to make, that RFC 3339 parsing is _hard_. Fortunately, that isn't my point, and your statement is just irrelevant rather than contradictory to your whole argument. > P.S. > ... > What is the point of the copy-paste? Does handling lowercase "t", > "z", > and a space complicates the parsing in a meaningful manner? Sure. Even ignoring the fact that you're claiming that people can parse it with strptime, the very fact that many people have written code that claims to be able to parse RFC 3339, and yet doesn't handle lowercase "t" and "z", shows that there is something you can get wrong; therefore, it is not trivial. > Imagine a parser that supports sep='T' (default for isoformat()) and > sep=' ' (space for __str__). How hard do you think to extend such parser > to support 't' as a separator? One again: "it's certainly much easier to write an RFC 3339 parser than an ISO 8601 parser," but it's not completely trivial. You seem to think that this is a black-or-white issue: either RFC 3339 is a complete failure and it's actually as hard to parse as ISO 8601, or RFC 3339 is trivially parseable and no one should bother to write a parser for a subset of the language because it's just as easy to parse the whole thing. In reality, neither one of those is true. The latter may be closer to the truth than the former, but it's still wrong. In other words, it makes perfect sense to write a parser for exactly what Python generates, and not claim it as RFC 3339 compliant. From 4kir4.1i at gmail.com Fri Aug 8 17:17:36 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Fri, 08 Aug 2014 19:17:36 +0400 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> <87vbq4a35u.fsf@gmail.com> <87lhqza64b.fsf@gmail.com> <1407478257.58223.YahooMailNeo@web181006.mail.ne1.yahoo.com> Message-ID: <87a97f9fjj.fsf@gmail.com> Please, don't put words in my mouth. To avoid Straw-man, quote me directly in context. As I do with you in my messages. Andrew Barnert writes: > Mixing up responses to an previous email you already responded to with > the new one makes it harder to reply, but I'll try. > > > On Thursday, August 7, 2014 10:43 PM, Akira Li <4kir4.1i at gmail.com> wrote: >> > Andrew Barnert writes: >> >>> On Aug 7, 2014, at 5:35, Akira Li <4kir4.1i at gmail.com> wrote: >>> >>>> Please, don't spread misinformation. >>>> >>>> Among the explicit rfc 3339 design goals are simplicity and human >>>> readability. >>>> >>>> Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is >>>> relatively short and readable). >> ... >>> (And there's also a whole section of interpreting >>> "legacy"/"deprecated" 2-digit years and how you should >> handle them.) >>> >>> So, is the RFC "spreading misinformation" about itself? >> >> You are *obviously* wrong for the rfc 3339 Internet Date/Time Format >> itself (used by __str__, isoformat -- relevant to the current topic). > > > You accused me of "spreading misinformation" by saying that RFC 3339 > is more complicated than what Python's str generates. You also > suggested that people often parse RFC 3339 with a simple strptime. You said [1]: "RFC 3339 is still more complicated than just reversing Python's str or isoformat. IIRC (it's hard to check on my phone), it mandates that parsers should accept 2-digit years (including 3-digit or semicolon-and-two-digit years), lowercase T and Z, missing "-", and other things that you shouldn't generate but some code might." [1] https://mail.python.org/pipermail/python-ideas/2014-August/028509.html What I consider misinformation: "it mandates that parsers should accept 2-digit years (including 3-digit or semicolon-and-two-digit years)" [Andrew Barnert] '"missing "-"' [Andrew Barnert] "and other things that you shouldn't generate but some code might." [Andrew Barnert] I've already described it quote by quote in [2] with all the gory details including a careful usage of MUST, SHOULD words from [rfc 2119]. [2] https://mail.python.org/pipermail/python-ideas/2014-August/028541.html [rfc 2119] http://tools.ietf.org/html/rfc2119 > I pointed out that the RFC itself clearly defines something more > complicated than what Python's str generates, and that it can't be > fully parsed by a simple strptime, and then added a parenthetical > remark about support for 2-year dates. It *is* obvious that the rfc is more complex than the str method e.g., the rfc supports 'T', 'Z' (default str doesn't generate them as far as I know). But all specific details (except "lowercase T and Z") you provided in the quote from [1] are wrong as described in [2]. What conclusions can you draw about the whole statement after that? > You cut out everything but the parenthetical remark, and replied to > that as if it was the whole point of my message. And you even > responded to nothing but the parenthetical 2-year comment in replies > to completely separate sections of the message. I have no idea how to > respond to that except to say: read it again. I've read and reread [1]. I don't see what "other things" you are referring to. To avoid ambiguity, could you use a direct quote and a link (as I've demonstrated above)? -- Akira From abarnert at yahoo.com Fri Aug 8 18:46:40 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 8 Aug 2014 09:46:40 -0700 Subject: [Python-ideas] strptime without second argument as an inverse to __str__ In-Reply-To: <87a97f9fjj.fsf@gmail.com> References: <20140804181528.GP4525@ando> <20140805013944.GQ4525@ando> <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> <87vbq4a35u.fsf@gmail.com> <87lhqza64b.fsf@gmail.com> <1407478257.58223.YahooMailNeo@web181006.mail.ne1.yahoo.com> <87a97f9fjj.fsf@gmail.com> Message-ID: <2B496614-E2B8-48CD-8B95-537B5A589CCC@yahoo.com> On Aug 8, 2014, at 8:17, Akira Li <4kir4.1i at gmail.com> wrote: > It *is* obvious that the rfc is more complex than the str method e.g., > the rfc supports 'T', 'Z' (default str doesn't generate them as far as I > know). This is the only point that matters here to me. If you're no longer disagreeing with it, I have no interest in continuing to argue, and I doubt anyone else has any interest in reading it. So, let me summarize and see if you have anything substantive to disagree with: People want a function to reverse __str__, possibly also handling space as a separator. Such a function is trivial to write. If it's not a fully compliant RFC 3339 parser, that's fine, and it will be useful for it's intended goal, as long as it isn't called fromrfc3339string (which no one has suggested anyway, although someone did suggest fromisostring, which is what started the whole RFC 3339 side track--obviously that name shouldn't be used either). From ncoghlan at gmail.com Sun Aug 10 14:33:49 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Aug 2014 22:33:49 +1000 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) Message-ID: (switching from python-dev to python-ideas) FWIW, I don't consider str.join *or* sum with an empty string as the starting point to be particularly intuitive ways of joining iterables of strings. str.join was invented before we had keyword-only arguments as a common construct, and before print became an ordinary function that accepted a "sep" keyword-only argument. I'd be interested in seeing a concrete proposal for a "concat" builtin that accepted a "sep" keyword only argument. Even if such a PEP ends up being rejected, it would hopefully help cut short the *next* potentially interminable thread on the topic by gathering the arguments for and against in a more readily accessible place. Regards, Nick. On 10 Aug 2014 18:26, "Stephen J. Turnbull" wrote: > Alexander Belopolsky writes: > > On Sat, Aug 9, 2014 at 3:08 AM, Stephen J. Turnbull > > > wrote: > > > > > All the suggestions > > > I've seen so far are (IMHO, YMMV) just as ugly as the present > > > situation. > > > > > > > What is ugly about allowing strings? CPython certainly has a way to to > > make sum(x, '') > > sum(it, '') itself is ugly. As I say, YMMV, but in general last I > heard arguments that are usually constants drawn from a small set of > constants are considered un-Pythonic; a separate function to express > that case is preferred. I like the separate function style. > > And that's the current situation, except that in the case of strings > it turns out to be useful to allow for "sums" that have "glue" at the > joints, so it's spelled as a string method rather than a builtin: eg, > ", ".join(paramlist). > > Actually ... if I were a fan of the "".join() idiom, I'd seriously > propose 0.sum(numeric_iterable) as the RightThang{tm]. Then we could > deprecate "".join(string_iterable) in favor of "".sum(string_iterable) > (with the same efficient semantics). > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Aug 10 20:25:08 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 10 Aug 2014 14:25:08 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: Message-ID: On 8/10/2014 8:33 AM, Nick Coghlan wrote: > (switching from python-dev to python-ideas) > > FWIW, I don't consider str.join *or* sum with an empty string as the > starting point to be particularly intuitive ways of joining iterables of > strings. I think sum(iofs, '') is obvious and I do not understand the determination to not allow it and implement it as ''.join(iofs). Perhaps that is because I think '+' is the *right* choice for string concatenation. Addition of tallies (base 1 numbers) is concatenation. For tallies, or strings, r and s, len(r+s) == len(r) + len(s). (That this is not true of sets is a reason to not use '+ for set union, which Python doesn't.) (I also think sum(it_of_lists, somelist), which is allowed, should be implemented as somelist.extend(it_of_lists), but that is another issue.) > str.join was invented before we had keyword-only arguments as a common > construct, and before print became an ordinary function that accepted a > "sep" keyword-only argument. The reason sep in print has to be keyword-only is because print takes an indefinite of main arguments rather than an iterable of items to print. This is appropriate because we generally print a few items that are not already in a collection, because collections of strings can be .joined, and because collections can have, and builtin collections do have, methods to join the string representations of their members. Print, str and .__str__ methods, and str.join works together wonderfully. The start paramenter of sum does not need to be keyword only because sum takes one iterable of items to sum. This is appropriate because a few items to sum, not in a collection, can be handled by '+', and sum is needed for large collections of items already in a collection. > I'd be interested in seeing a concrete proposal for a "concat" builtin > that accepted a "sep" keyword only argument. Even if such a PEP ends up > being rejected, it would hopefully help cut short the *next* potentially > interminable thread on the topic by gathering the arguments for and > against in a more readily accessible place. For the against list: New builtins should add something more than a pure synonym; "concat(iofs, sep=joiner)" is harder (6 more letters), not easier, to write than "joiner.join(iofs)". -- Terry Jan Reedy From abarnert at yahoo.com Sun Aug 10 21:48:08 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 10 Aug 2014 12:48:08 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: Message-ID: <1407700088.46613.YahooMailNeo@web181002.mail.ne1.yahoo.com> On Sunday, August 10, 2014 11:26 AM, Terry Reedy wrote: ? > For the against list: New builtins should add something more than a pure > synonym;? "concat(iofs, sep=joiner)" is harder (6 more letters), not > easier, to write than "joiner.join(iofs)". If concat instead meant "joiner.join(map(type(joiner), iofs))", that might be more useful than just a synonym. I've seen many novices try to figure out how to join up their list of ints. Also, it could conceivably be faster for user string-like classes, especially mutable ones (because then, as long as they had slice replacement, everything else would happen in C, whereas if they have to write a custom join, it will be looping in Python). Also, being a new function, it might be reasonable for concat to handle iterators differently from sequences (exponential expansion rather than preallocating). I've seen a couple people asking why ''.join(genexpr) uses twice as much memory as they expected (although far fewer than the joining-up-ints questions). Of course any (or all) of those also might well be a bad idea in its own right. But a new function gives us a chance to think through how it should work?and then, if it turns out it should work exactly like join, the answer may be "just a synonym, reject it", but if not, it might be reasonable to add it and gradually phase out join. From toddrjen at gmail.com Sun Aug 10 22:43:02 2014 From: toddrjen at gmail.com (Todd) Date: Sun, 10 Aug 2014 22:43:02 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <1407700088.46613.YahooMailNeo@web181002.mail.ne1.yahoo.com> References: <1407700088.46613.YahooMailNeo@web181002.mail.ne1.yahoo.com> Message-ID: On Aug 10, 2014 9:51 PM, "Andrew Barnert" wrote: > > On Sunday, August 10, 2014 11:26 AM, Terry Reedy wrote: > > > > For the against list: New builtins should add something more than a pure > > synonym; "concat(iofs, sep=joiner)" is harder (6 more letters), not > > easier, to write than "joiner.join(iofs)". > > > If concat instead meant "joiner.join(map(type(joiner), iofs))", that might be more useful than just a synonym. I've seen many novices try to figure out how to join up their list of ints. Couldn't this also be handled with a keyword argument to join? For example "joiner.join(iofs, convert=True)". > Also, it could conceivably be faster for user string-like classes, especially mutable ones (because then, as long as they had slice replacement, everything else would happen in C, whereas if they have to write a custom join, it will be looping in Python). This seems like something an abstract base class could take care of. > Also, being a new function, it might be reasonable for concat to handle iterators differently from sequences (exponential expansion rather than preallocating). I've seen a couple people asking why ''.join(genexpr) uses twice as much memory as they expected (although far fewer than the joining-up-ints questions). Isn't this an implementation detail that could be fixed? -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Mon Aug 11 02:20:53 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 11 Aug 2014 09:20:53 +0900 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: Message-ID: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > Addition of tallies (base 1 numbers) is concatenation. Except that nobody actually tallies that way. In practice everybody I've ever seen count to ten by tallying does something special for fives (Americans tally four times then cross the four vertical strokes with a fifth, Japanese build the character sei (? == true)). So what you're actually arguing is circular: if you represent a tally as a string with a singleton alphabet, it behaves like a string. But in Japanese it's actually a 5 element alphabet, and your homomorphism: > For tallies, or strings, r and s, len(r+s) == len(r) + len(s). fails. I think that rather than focus on formal properties, we should look at how sum is used in Python. For example, numerical sum is in fact an attractive nuisance as long as floats are considered numbers. So unless and until we deprecate floats, "formal sum" will always be splintered into multiple functions. As far as concat goes, I don't really see what advantage it has over joiner.join, except in teaching. That might be enough for a builtin in this case, since concatenating iterables of strings is a pretty frequent operation in my experience. From wolfgang.maier at biologie.uni-freiburg.de Mon Aug 11 08:56:28 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 11 Aug 2014 08:56:28 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 08/11/2014 02:20 AM, Stephen J. Turnbull wrote: > > As far as concat goes, I don't really see what advantage it has over > joiner.join, except in teaching. That might be enough for a builtin > in this case, since concatenating iterables of strings is a pretty > frequent operation in my experience. > I am not so sure about the benefit for teaching. I am using Python for teaching programming to absolute beginners at university and, in my experience, joiner.join is never a big hurdle. You have to explain it once, of course, but after that people, especially people not biased by other languages, do not have problems with it. Wolfgang From alexander.belopolsky at gmail.com Mon Aug 11 15:56:38 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 11 Aug 2014 09:56:38 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > I am using Python for teaching programming to absolute beginners at > university and, in my experience, joiner.join is never a big hurdle. In my experience, it is the asymmetry between x.join(y) and x.split(y) which causes most of the confusion. In x.join(y), x is the separator and y is the data being joined, but in x.split(y), it is the other way around. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.maier at biologie.uni-freiburg.de Mon Aug 11 16:12:29 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 11 Aug 2014 16:12:29 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 08/11/2014 03:56 PM, Alexander Belopolsky wrote: > > On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier > > > wrote: > > I am using Python for teaching programming to absolute beginners at > university and, in my experience, joiner.join is never a big hurdle. > > > In my experience, it is the asymmetry between x.join(y) and x.split(y) > which causes most of the confusion. In x.join(y), x is the separator > and y is the data being joined, but in x.split(y), it is the other way > around. > Once you have explained the beauty of s == sep.join(s.split(sep)) to anyone, they will not be confused again, but find it perfectly logical. From bborcic at gmail.com Mon Aug 11 16:46:37 2014 From: bborcic at gmail.com (Boris Borcic) Date: Mon, 11 Aug 2014 16:46:37 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Wolfgang Maier wrote: > > s == sep.join(s.split(sep)) > > to anyone, they will not be confused again, but find it perfectly logical. You are kidding, right? What will prevent them from using logic to recall the invariant mistakenly as s == sep.join(sep.split(s)) ? If only because there are natural use cases for passing mysep.join around as a callable that will join with a fixed separator, and there would similarly exist use cases for mysep.split, while there are really none for passing around mystring.split to split a fixed string by varying separators. This whole corner of python is a most ugly system of inconsistent acne scars in the 2.x series (dunno about 3.x). What with sum(...) pedantically telling you to use str.join - if it knows what you want, it should just to it. --- Ce courrier ?lectronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com From breamoreboy at yahoo.co.uk Mon Aug 11 16:59:12 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 11 Aug 2014 15:59:12 +0100 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 11/08/2014 14:56, Alexander Belopolsky wrote: > > On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier > > > wrote: > > I am using Python for teaching programming to absolute beginners at > university and, in my experience, joiner.join is never a big hurdle. > > > In my experience, it is the asymmetry between x.join(y) and x.split(y) > which causes most of the confusion. In x.join(y), x is the separator > and y is the data being joined, but in x.split(y), it is the other way > around. > Could you try something which to my understanding is unprecedented in the world of computing, as in point them to the docs? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From alexander.belopolsky at gmail.com Mon Aug 11 17:47:29 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 11 Aug 2014 11:47:29 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 10:59 AM, Mark Lawrence wrote: > Could you try something which to my understanding is unprecedented in the > world of computing, as in point them to the docs? Sure - this is the universal remedy to any design mistake, but how many people would need to look at the docs to understand something like join(list, sep=',') or split(string, sep=',')? And how many of those seeing x.join(y) and x.split(y) for the first time will guess which argument is data and which is separator? Beautiful is better than ugly. Explicit is better than implicit. .. Readability counts. .. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. .. (The Zen of Python, by Tim Peters, AKA python -m this) -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Mon Aug 11 17:55:25 2014 From: toddrjen at gmail.com (Todd) Date: Mon, 11 Aug 2014 17:55:25 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 3:56 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier < > wolfgang.maier at biologie.uni-freiburg.de> wrote: > >> I am using Python for teaching programming to absolute beginners at >> university and, in my experience, joiner.join is never a big hurdle. > > > In my experience, it is the asymmetry between x.join(y) and x.split(y) > which causes most of the confusion. In x.join(y), x is the separator and y > is the data being joined, but in x.split(y), it is the other way around. > What would be the solution to this? Making join a list method, or reversing the behavior of split which means it no longer acts on x? -------------- next part -------------- An HTML attachment was scrubbed... URL: From me+python at ixokai.io Mon Aug 11 18:21:41 2014 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 11 Aug 2014 09:21:41 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 8:47 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Mon, Aug 11, 2014 at 10:59 AM, Mark Lawrence > wrote: > >> Could you try something which to my understanding is unprecedented in the >> world of computing, as in point them to the docs? > > > Sure - this is the universal remedy to any design mistake, but how many > people would need to look at the docs to understand something like > join(list, sep=',') or split(string, sep=',')? And how many of those > seeing x.join(y) and x.split(y) for the first time will guess which > argument is data and which is separator? > Two new builtins now? "x.split(y)" is ambiguous because its terrible code, not because of the order. Names matter and exist to provide clarity. Don't name variables 'x' and 'y'. If it were as simple as "x.split(sep)" or "x.split('\t')" then I bet almost no one will need to consult the documentation to know which is which; similarly with "sep.join(list_of_strings)". Now for someone wanting to know how to join a list of strings and doesn't know how, sep.join(strings) is not per se obviously discoverable, but it isn't that bad and people will have to go to the docs anyways. They are likely to look in the string section I think, and find it. They might look in the list section, and may look in the bulitins section, but its still not that bad. As for how many would need to go look it up with your proposed builtins, anyone who wanted to set the separator, that's how many. And considering "," is not even kind of a sensible default for either, that'd be a lot. I remember this discussion very clearly when sum went in, and again during the py3k period, and here the zombie rises again. Beautiful is better than ugly. > Explicit is better than implicit. > .. > Readability counts. > .. > In the face of ambiguity, refuse the temptation to guess. > There should be one-- and preferably only one --obvious way to do it. > .. > > (The Zen of Python, by Tim Peters, AKA python -m this) > The Zen are nice words and good pieces of advice that are not a tool to wield to win an debate, especially since most of them are fundamentally subjective and require you to think in Python for their full meaning to have an impact. They were also written after the fact by a long margin as a joke, even if like most jokes they have a measure of truth in them. Adding builtins defies 'Explicit is better then implicit', and though ",".join(stuff) is arguably less then beautiful, the axiom says *better* not *best*, and the order of things to almost any function at all could be called ambiguous until you look it up. Finally, at least ",".join(stuff) directly (and explicitly!) refuses any temptation to guess. Zen aside, I'm not sure if sum(list_of_strings) is that pancea of readability and beauty and who-needs-documentation-now that its sort of implied in this thread, that it doesn't work for strings is for reasons already specified (CPython's immutable strings implies it could lead to *BROKEN* behavior in a reasonable implementation, even if CPython currently has a limited optimization). I ever so much don't believe we need "join" and "split" builtins, I much rather think your arguments for forcing people to use str.join make more sense, but I hope we don't see that either. This isn't a situation that causes a huge amount of confusion. Teach the students that sep.join(list_of_strings) make sense because who would know how to combine strings, but a string itself? Of course list_of_strings wouldn't know this. Watch the awareness blossom. Sometimes I think people overstate how big a deal things are in teaching because we all care about making Python teachable. --S -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Mon Aug 11 18:22:59 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 11 Aug 2014 12:22:59 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 11:55 AM, Todd wrote: > In my experience, it is the asymmetry between x.join(y) and x.split(y) >> which causes most of the confusion. In x.join(y), x is the separator and y >> is the data being joined, but in x.split(y), it is the other way around. >> > > What would be the solution to this? > Allow sum(list_of_strings, '') and stop mocking people who prefer it to ''.join(..). This will not solve all the issues with join/split, but at least a simple task of concatenating a list of strings will have a more or less obvious solution. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daviesk24 at yahoo.com Mon Aug 11 18:25:31 2014 From: daviesk24 at yahoo.com (Kevin Davies) Date: Mon, 11 Aug 2014 06:25:31 -1000 Subject: [Python-ideas] float comparison in doctest Message-ID: <53E8EE7B.50408@yahoo.com> Erik Bray has created a nifty addition to doctest to compare floating point numbers using a `+FLOAT_CMP` flag. It is implemented within Astropy (http://www.astropy.org/) and the related issue is https://github.com/astropy/astropy/issues/2662. According to Erik, it may need some tweaks, but I think it would be a really useful feature to make available in the doctest package itself. Kevin From toddrjen at gmail.com Mon Aug 11 18:24:35 2014 From: toddrjen at gmail.com (Todd) Date: Mon, 11 Aug 2014 18:24:35 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 6:22 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Mon, Aug 11, 2014 at 11:55 AM, Todd wrote: > >> In my experience, it is the asymmetry between x.join(y) and x.split(y) >>> which causes most of the confusion. In x.join(y), x is the separator and y >>> is the data being joined, but in x.split(y), it is the other way around. >>> >> >> What would be the solution to this? >> > > Allow sum(list_of_strings, '') and stop mocking people who prefer it to > ''.join(..). This will not solve all the issues with join/split, but at > least a simple task of concatenating a list of strings will have a more or > less obvious solution. > I am confused, if it won't solve the problem, how is it relevant to my question? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Mon Aug 11 18:38:10 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 11 Aug 2014 12:38:10 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 12:21 PM, Stephen Hansen wrote: > Don't name variables 'x' and 'y'. If it were as simple as "x.split(sep)" > or "x.split('\t')" then I bet almost no one will need to consult the > documentation to know which is which Sure, but on the same token if someone writes sep.split(x), how likely will this error be caught on a quick review? This is not theoretical. I've seen people make this mistake and asking for help in debugging rather non-obvious behaviors. -------------- next part -------------- An HTML attachment was scrubbed... URL: From me+python at ixokai.io Mon Aug 11 18:37:39 2014 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 11 Aug 2014 09:37:39 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 9:22 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Mon, Aug 11, 2014 at 11:55 AM, Todd wrote: > >> In my experience, it is the asymmetry between x.join(y) and x.split(y) >>> which causes most of the confusion. In x.join(y), x is the separator and y >>> is the data being joined, but in x.split(y), it is the other way around. >>> >> >> What would be the solution to this? >> > > Allow sum(list_of_strings, '') and stop mocking people who prefer it to > ''.join(..). This will not solve all the issues with join/split, but at > least a simple task of concatenating a list of strings will have a more or > less obvious solution. > Then we'll have two obvious solutions since "".join() and the huge body of code using it won't go away. One which is only even vaguely workable because of an implementation-specific optimization that isn't a promise of the language, which makes it at best obvious* if not obvious-ish. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Mon Aug 11 18:46:43 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 11 Aug 2014 12:46:43 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 12:37 PM, Stephen Hansen wrote: > Then we'll have two obvious solutions since "".join() and the huge body of > code using it won't go away. ''.join() is not an obvious solution to concatenation. It is a fringe case of a solution to a completely different problem - building a delimited string. Buy the same token, we have "two obvious solutions" to negation: -x and 0-x. -------------- next part -------------- An HTML attachment was scrubbed... URL: From me+python at ixokai.io Mon Aug 11 18:47:09 2014 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 11 Aug 2014 09:47:09 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 9:38 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Mon, Aug 11, 2014 at 12:21 PM, Stephen Hansen > wrote: > >> Don't name variables 'x' and 'y'. If it were as simple as "x.split(sep)" >> or "x.split('\t')" then I bet almost no one will need to consult the >> documentation to know which is which > > > Sure, but on the same token if someone writes sep.split(x), how likely > will this error be caught on a quick review? This is not theoretical. > I've seen people make this mistake and asking for help in debugging rather > non-obvious behaviors. > I've seen people make innumerable mistakes in the past, I've seen certain classes of mistakes repeated -- this isn't an argument for change by itself. People make mistakes. Its going to happen. That said, I find the idea that "x.split(sep)" as non-obvious to be... weird, to say the least. But, obvious is subjective. Your obvious may not be my obvious nor most people's obvious. Yes, sep.join(list) is a bit of a weird construct, but its one thing to learn, and its not a hard one to teach at that. In fact, it makes very logical sense once you explain it and makes people think of things more Pythonically after. I say from experience, not in theory. But, string.split(sep) is very natural. You seem to think that they need to be in the same order to be obvious but I don't see why nor do I think any of the alternatives are not without problems that are bigger issues. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Mon Aug 11 19:14:25 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 11 Aug 2014 13:14:25 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 12:47 PM, Stephen Hansen wrote: > Yes, sep.join(list) is a bit of a weird construct, but its one thing to > learn, and its not a hard one to teach at that. In fact, it makes very > logical sense once you explain it and makes people think of things more > Pythonically after. I say from experience, not in theory. But, > string.split(sep) is very natural. You seem to think that they need to be > in the same order to be obvious but I don't see why nor do I think any of > the alternatives are not without problems that are bigger issues. I am not suggesting any changes to str.join or str.split methods. I am just arguing that sum(list_of_strings, '') should be allowed by the language and its performance is a matter of the quality of implementation. Once you learn that string addition is concatenation in Python, it is natural to "sum" lists of Python strings regardless of whether it makes sense in your native language. sep.join(list) is not such a weird construct when sep is non-empty - it is the sep='' case which is weird and non-obvious. (Note that someone in this thread suggested demonstrating s == sep.join(s.split(sep)) invariant as a teaching tool, but this invariant fails when sep is empty.) When you are tasked with finding s1 + s2 + ... + sN given [s1, s2, ..., sN], it is sum that first comes to mind, not join. The situation is different when you have a separator to begin with, but when you don't using an empty separator feels like a performance hack in the absence of an efficient natural solution. -------------- next part -------------- An HTML attachment was scrubbed... URL: From haoyi.sg at gmail.com Mon Aug 11 19:10:20 2014 From: haoyi.sg at gmail.com (Haoyi Li) Date: Mon, 11 Aug 2014 10:10:20 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: > In fact, it makes very logical sense once you explain it and makes people think of things more Pythonically after. I say from experience, not in theory. Could you elaborate about "making people think more pythonically after" bit? I can see how explaining the API makes people understand the API, but I'm curious how it makes people behave differently after. On Mon, Aug 11, 2014 at 9:38 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Mon, Aug 11, 2014 at 12:21 PM, Stephen Hansen > wrote: > >> Don't name variables 'x' and 'y'. If it were as simple as "x.split(sep)" >> or "x.split('\t')" then I bet almost no one will need to consult the >> documentation to know which is which > > > Sure, but on the same token if someone writes sep.split(x), how likely > will this error be caught on a quick review? This is not theoretical. > I've seen people make this mistake and asking for help in debugging rather > non-obvious behaviors. > I've seen people make innumerable mistakes in the past, I've seen certain classes of mistakes repeated -- this isn't an argument for change by itself. People make mistakes. Its going to happen. That said, I find the idea that "x.split(sep)" as non-obvious to be... weird, to say the least. But, obvious is subjective. Your obvious may not be my obvious nor most people's obvious. Yes, sep.join(list) is a bit of a weird construct, but its one thing to learn, and its not a hard one to teach at that. In fact, it makes very logical sense once you explain it and makes people think of things more Pythonically after. I say from experience, not in theory. But, string.split(sep) is very natural. You seem to think that they need to be in the same order to be obvious but I don't see why nor do I think any of the alternatives are not without problems that are bigger issues. _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From me+python at ixokai.io Mon Aug 11 19:45:34 2014 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 11 Aug 2014 10:45:34 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 10:10 AM, Haoyi Li wrote: > > In fact, it makes very logical sense once you explain it and makes > people think of things more Pythonically after. I say from experience, not > in theory. > > Could you elaborate about "making people think more pythonically after" > bit? I can see how explaining the API makes people understand the API, but > I'm curious how it makes people behave differently after. > Well, its an opinion of course, so it may be as useful as the thought that the sky is high. It doesn't make people behave differently, but it does lead them to writing idiomatic code. There's naive Python code, which ideally should be clear and easy still, and then there's idiomatic Python code which has a beauty in its expressiveness while also being (relatively) efficient. But, if the opposite of "string.split(sep)" were "list_of_strings.join(sep)" then that would mean that List would have to know about how to combine strings. That's not very Pythonic. Also, Strings are immutable, that means: str1 = str1 + str2 logically leads one to assume that it is constructing an entirely new string, copying the contents of str1 and str2 into it, and discarding the original str1. In fact, it did that for a long time. You can't change strings, after all. That is an fundamental tenet, and when people learn that they will get a new tool and think closer to idiomatic Python. Now, CPython has an optimization for this case which is the only reason the idea of sum(list_of_strings) is not largely fundamentally broken, but that's not a promise and not a feature of the language. IMHO. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Mon Aug 11 19:51:31 2014 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 11 Aug 2014 12:51:31 -0500 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: Message-ID: On 08/10/2014 07:33 AM, Nick Coghlan wrote: > FWIW, I don't consider str.join *or* sum with an empty string as the > starting point to be particularly intuitive ways of joining iterables of > strings. > > str.join was invented before we had keyword-only arguments as a common > construct, and before print became an ordinary function that accepted a > "sep" keyword-only argument. > > I'd be interested in seeing a concrete proposal for a "concat" builtin that > accepted a "sep" keyword only argument. Even if such a PEP ends up being > rejected, it would hopefully help cut short the *next* potentially > interminable thread on the topic by gathering the arguments for and against > in a more readily accessible place. I think the contrast between the built in function "sum", and the string method "join", is very interesting from a language design point of view, but I'm finding it hard to describe just why in only a few words. Others have pointed out some of the more detailed aspects of these issues, so here are some of the more general wider views I think come into play. * Generality / Speciality Their are advantages to both ends of this scale. A more general function is very convenient, while a more specialised function can offer a greater degree of control. * Complexity / Quantity A function with a more complex signature can be equivalent to several functions with simpler signatures. But as they become more complex, they also become more difficult to use. It's not obvious how the "sum" function fits into these scales, and I believe that may be why it tends to come up as something that needs to be fixed frequently. (how depends on the viewpoint of the fixer) If "sum" was a method on numbers, it would clearly be more specialised, or if it was defined to call a method of what ever objects it was given, it would clearly be more general. It is what it is, and I don't think there was any conscious consideration of these concepts when it was created. Probably practicality over purity was more of a factor at the time. I believe these concepts are more likely to be intuitively considered as the developers experiences increase instead of being consciously considered. So they aren't formally defined in any documentation. How a "concat" built in would relate to these concepts. Would "concat" be very general and delegate it's work to methods so it works on a variety of objects, or would it be limited to just strings? I'm also concerned we may add a new function just to compliment the "sum" function, so that both of them look better. I think any new function needs to fit into the language as a whole on it's own grounds with a clear propose and design. Food for thought, Ron From me+python at ixokai.io Mon Aug 11 19:52:31 2014 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 11 Aug 2014 10:52:31 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 10:14 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > it makes sense in your native language. > No, it makes no sense at all in my native language. sum is, fundamentally, the total of numbers in my native language. > When you are tasked with finding s1 + s2 + ... + sN given [s1, s2, ..., > sN], it is sum that first comes to mind, not join. > It would have never occurred to me in a million years to address such a problem with sum. The first thing that'd likely have come to my mind a million years ago when I first started learning Python would be to look for some concat function, but failing that I'd run across join in the docs. > The situation is different when you have a separator to begin with, but > when you don't using an empty separator feels like a performance hack in > the absence of an efficient natural solution. > I'm pretty much going to just bow out at this point because the idea that "".join(list_of_strings) is some weird and non-obvious thing but when it has a separator, its clear and obvious, doesn't make any sense at all to me. That seems to fly directly in the face of "there should be one...." which you previously quoted. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Aug 11 19:53:29 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 11 Aug 2014 18:53:29 +0100 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 11, 2014 at 5:22 PM, Alexander Belopolsky wrote: > > On Mon, Aug 11, 2014 at 11:55 AM, Todd wrote: >>> >>> In my experience, it is the asymmetry between x.join(y) and x.split(y) >>> which causes most of the confusion. In x.join(y), x is the separator and y >>> is the data being joined, but in x.split(y), it is the other way around. >> >> >> What would be the solution to this? > > Allow sum(list_of_strings, '') and stop mocking people who prefer it to > ''.join(..). This will not solve all the issues with join/split, but at > least a simple task of concatenating a list of strings will have a more or > less obvious solution. I don't have any data here, but I bet people who know about str.join (even for its natural use cases like ", ".join(...)) outnumber the people who know that sum() takes a second argument by a very large factor. Of course this also means that sum()'s special error message is probably pretty ineffective at reaching the people it's trying to educate -- to do that we'd need to warn on str += str or something, which is clearly not happening. So I can see the argument for just making sum(iterable_of_strings, "") fast. But practically speaking, how would this work? In general str.join and sum have different semantics. What happens if we descend deep into the iterable and then discover a non-string (that might nonetheless still have a + operator)? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From haoyi.sg at gmail.com Mon Aug 11 19:51:00 2014 From: haoyi.sg at gmail.com (Haoyi Li) Date: Mon, 11 Aug 2014 10:51:00 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: One reason I think this is confusing is that "list.join(sep)" is how basically everyone else does it, including: Ruby, Javascript, Scala, Guava-Java (normal java doesn't have anything). People who don't do it this way include C# (static method), Boost-C++ (static method), PHP (static method) Nobody does it the Python way AFAICT. On Mon, Aug 11, 2014 at 10:45 AM, Stephen Hansen wrote: > On Mon, Aug 11, 2014 at 10:10 AM, Haoyi Li wrote: > >> > In fact, it makes very logical sense once you explain it and makes >> people think of things more Pythonically after. I say from experience, not >> in theory. >> >> Could you elaborate about "making people think more pythonically after" >> bit? I can see how explaining the API makes people understand the API, but >> I'm curious how it makes people behave differently after. >> > > Well, its an opinion of course, so it may be as useful as the thought that > the sky is high. It doesn't make people behave differently, but it does > lead them to writing idiomatic code. There's naive Python code, which > ideally should be clear and easy still, and then there's idiomatic Python > code which has a beauty in its expressiveness while also being (relatively) > efficient. > > But, if the opposite of "string.split(sep)" were > "list_of_strings.join(sep)" then that would mean that List would have to > know about how to combine strings. That's not very Pythonic. Also, Strings > are immutable, that means: > > str1 = str1 + str2 > > logically leads one to assume that it is constructing an entirely new > string, copying the contents of str1 and str2 into it, and discarding the > original str1. In fact, it did that for a long time. You can't change > strings, after all. That is an fundamental tenet, and when people learn > that they will get a new tool and think closer to idiomatic Python. Now, > CPython has an optimization for this case which is the only reason the idea > of sum(list_of_strings) is not largely fundamentally broken, but that's not > a promise and not a feature of the language. > > IMHO. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Aug 11 20:15:08 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 11 Aug 2014 14:15:08 -0400 Subject: [Python-ideas] float comparison in doctest In-Reply-To: <53E8EE7B.50408@yahoo.com> References: <53E8EE7B.50408@yahoo.com> Message-ID: On 8/11/2014 12:25 PM, Kevin Davies wrote: > Erik Bray has created a nifty addition to doctest to compare floating > point numbers using a `+FLOAT_CMP` flag. The problem with a simple flag is that almost_equal should sometimes be absolute and sometimes relative and in both cases a parameter in needed. I think there have been vague proposals for a float method, but "a - b < delta" is shorter than "a.almost_equat(b, abs=delta)" and probably easier to understand. > It is implemented within > Astropy (http://www.astropy.org/) and the related issue is > https://github.com/astropy/astropy/issues/2662. The related issue is applying flags to a block of test with a Sphinx directive. > According to Erik, it > may need some tweaks, but I think it would be a really useful feature to > make available in the doctest package itself. Erik would have to offer his code. -- Terry Jan Reedy From breamoreboy at yahoo.co.uk Mon Aug 11 20:57:23 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 11 Aug 2014 19:57:23 +0100 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 11/08/2014 18:51, Haoyi Li wrote: > One reason I think this is confusing is that "list.join(sep)" is how > basically everyone else does it, including: Ruby, Javascript, Scala, > Guava-Java (normal java doesn't have anything). People who don't do it > this way include C# (static method), Boost-C++ (static method), PHP > (static method) > > Nobody does it the Python way AFAICT. > So one and only one language got it right and the rest got it wrong, what about it? Now is it possible to leave this vitally important issue and move on to something trivial, e.g. helping people port Python 2 code to Python 3? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From tjreedy at udel.edu Mon Aug 11 21:00:26 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 11 Aug 2014 15:00:26 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 8/11/2014 1:14 PM, Alexander Belopolsky wrote: > sep.join(list) is not such a weird construct when sep is non-empty - it > is the sep='' case which is weird and non-obvious. (Note that someone > in this thread suggested demonstrating s == sep.join(s.split(sep)) > invariant as a teaching tool, but this invariant fails when sep is > empty.) Because s.split('') raises "ValueError: empty separator". I expected the result should be the same as list(s), making the invariant above true. But perhaps Guido thought splitting on '' might be a bug. re.split('', s) returns s, which seems wrong. The doc talks about 'occurences of the pattern'. I *see* an occurance of '' at every slice point. In a sense, Python slicing does too. >>> 'abc'[1:1] '' When I first learned Python, I knew to test edge case behavior rather than depend on my interpretation of docs, or worse, my expectations from previous experience and knowledge. The interactive interpreter makes little tests like 'ab'split('') and re.split('', 'ab') trivial and faster than reading (or debugging). -- Terry Jan Reedy From abarnert at yahoo.com Mon Aug 11 21:13:16 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 11 Aug 2014 12:13:16 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <75E96E84-7406-44B6-9436-1069F27DAF6A@yahoo.com> On Aug 11, 2014, at 10:45, Stephen Hansen wrote: > ... if the opposite of "string.split(sep)" were "list_of_strings.join(sep)" then that would mean that List would have to know about how to combine strings. That's not very Pythonic. I think the real problem here is that almost every OO language does it this way. Beginning programmers don't even know to look for a join function, so it doesn't matter where we put it. But someone coming to Python looking for the equivalent of -[NSArray componentsJoinedBySeparator:] or Enumerable#join or Sequence.joinStrings or whatever expects to find it as list.join. The fact that all of these other languages agreed on a stupid solution and Python on a more sensible one (in most of those languages, just as in Python, the enumeration/iteration/etc. protocol is a universal thing that every type already has to understand, while concatenating strings is something specific to strings) doesn't change the fact that they all agreed. The question is whether this is more of a positive, as an opportunity to get people thinking Pythonically early instead of trying to write Ruby or C# code in Python, or a negative, as a stumbling block to them using the language. Of course the obvious solution is to spell it like C++: stringstream ss; copy(a.begin(), a.end(), ostream_iterator(ss)); s = ss.str(); Then nobody will find it at all, no matter what language they come from; problem solved. :) > Also, Strings are immutable, that means: > > str1 = str1 + str2 > > logically leads one to assume that it is constructing an entirely new string, copying the contents of str1 and str2 into it, and discarding the original str1. In fact, it did that for a long time. You can't change strings, after all. That is an fundamental tenet, and when people learn that they will get a new tool and think closer to idiomatic Python. Now, CPython has an optimization for this case which is the only reason the idea of sum(list_of_strings) is not largely fundamentally broken, but that's not a promise and not a feature of the language. > > IMHO. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From masklinn at masklinn.net Mon Aug 11 21:29:01 2014 From: masklinn at masklinn.net (Masklinn) Date: Mon, 11 Aug 2014 21:29:01 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 2014-08-11, at 19:51 , Haoyi Li wrote: > One reason I think this is confusing is that "list.join(sep)" is how basically everyone else does it, including: Ruby, Javascript, Scala, Guava-Java (normal java doesn't have anything). People who don't do it this way include C# (static method), Boost-C++ (static method), PHP (static method) > > Nobody does it the Python way AFAICT. One could easily enough argue that MLs and Haskell do: although they don't generally have objects (and thus no method), their signature is (function name may vary) join :: String -> [String] -> String join separator list which ? in haskell ? can be infixed to separator `intercalate` list and which in both languages nicely lends itself to partial application (much like Python's `sep.join` expression) From wolfgang.maier at biologie.uni-freiburg.de Mon Aug 11 21:38:51 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 11 Aug 2014 21:38:51 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 11.08.2014 19:14, Alexander Belopolsky wrote: > > sep.join(list) is not such a weird construct when sep is non-empty - it > is the sep='' case which is weird and non-obvious. (Note that someone > in this thread suggested demonstrating s == sep.join(s.split(sep)) > invariant as a teaching tool, but this invariant fails when sep is > empty.) For the record, this doesn't fail because of any weirdness about ''.join(s). It's just that s.split() does not take an empty string as separator. So, ok, I should have said: s == sep.join(s.split(sep)) for any allowed sep (which should be an obvious requirement) but this has nothing to do with the rest of the discussion other that it is a bit peculiar that join and sep do not act perfectly symmetrical. On the other hand, a builtin function sum and a string method split would be alot more asymmetric. > When you are tasked with finding s1 + s2 + ... + sN given [s1, > s2, ..., sN], it is sum that first comes to mind, not join. Not *my* first thought when it comes to strings, but if it is yours, then you try it once and you get an appropriate error message pointing you to the correct solution. Ok for me. > The situation is different when you have a separator to begin with, but when > you don't using an empty separator feels like a performance hack in the > absence of an efficient natural solution. > From tjreedy at udel.edu Mon Aug 11 21:55:35 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 11 Aug 2014 15:55:35 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 8/11/2014 9:56 AM, Alexander Belopolsky wrote: > > On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier > > > wrote: > > I am using Python for teaching programming to absolute beginners at > university and, in my experience, joiner.join is never a big hurdle. > > > In my experience, it is the asymmetry between x.join(y) and x.split(y) > which causes most of the confusion. In x.join(y), x is the separator Given that the two parameters of join are a concrete string and the abstraction 'iterable of strings', join can only be a method of the joiner. I would first teach ' '.join(it_of_strings) as the 'normal' case of joining 'words', along with print with the default sep = ' '. > and y is the data being joined, but in x.split(y), it is the other way > around. *If* sep is present, then sep.split(string) would be possible. But when sep is *not* not present, split cannot be a method of something that is not there. So I think I would teach s.split() first and then add .split(sep) and .splitlines(). I would also teach join and split together since they are, at their cores (excluding special cases), inverses. -- Terry Jan Reedy From wolfgang.maier at biologie.uni-freiburg.de Mon Aug 11 22:17:50 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 11 Aug 2014 22:17:50 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53E924EE.40704@biologie.uni-freiburg.de> On 11.08.2014 21:55, Terry Reedy wrote: > On 8/11/2014 9:56 AM, Alexander Belopolsky wrote: >> >> On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier >> > > >> >> wrote: >> >> I am using Python for teaching programming to absolute beginners at >> university and, in my experience, joiner.join is never a big hurdle. >> >> >> In my experience, it is the asymmetry between x.join(y) and x.split(y) >> which causes most of the confusion. In x.join(y), x is the separator > > Given that the two parameters of join are a concrete string and the > abstraction 'iterable of strings', join can only be a method of the joiner. > > I would first teach ' '.join(it_of_strings) as the 'normal' case of > joining 'words', along with print with the default sep = ' '. > > >> and y is the data being joined, but in x.split(y), it is the other way >> around. > > *If* sep is present, then sep.split(string) would be possible. But when > sep is *not* not present, split cannot be a method of something that is > not there. So I think I would teach s.split() first and then add > .split(sep) and .splitlines(). > > I would also teach join and split together since they are, at their > cores (excluding special cases), inverses. > I like to show students early on that with Python they can do things very quickly that would be very hard to achieve manually. Most of my students are biologists so they do not think initially about theoretical aspects of programming much, but want to know whether they can do something with it fast. For join/split, I typically use problems like: - you'd like to use program xy to work on your data, but it expects input elements to be separated by semicolons when all you have is a tab-delimited format or - you have input consisting of numbers with ',' as the decimal separator (the default in German-speaking countries), but downstream software expects '.' ,i.e., they can all be solved with the general pattern: s = new_sep.join(s.split(old_sep)) Seeing that, in Python, you can solve these problems (in principle) with one line of quite understandable code is a very convincing argument for starting to learn the language. Wolfgang From alexander.belopolsky at gmail.com Mon Aug 11 22:34:38 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 11 Aug 2014 16:34:38 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: Message-ID: Getting back to the topic of this thread ... On Sun, Aug 10, 2014 at 8:33 AM, Nick Coghlan wrote: > I'd be interested in seeing a concrete proposal for a "concat" builtin > that accepted a "sep" keyword only argument. I don't think concat() is the same operation as sep.join(). I view concat as an operation that is naturally defined on a pair of sequences and can be naturally (via reduce) extended to apply to an iterable of sequences: z = concat(x, y) <=> len(z) = len(x) + len(y), z[i] = x[i] if i < len(x), z[i] = y[i - len(x)] otherwise. (Note that the definition works for any abstract sequence.) I find it odd that although CPython defines concatenation as a part of the sequence protocol at the implementation level, the abstract base class Sequence does not. This omission may present an opportunity to design Sequence.__concat__ and builtin concat() so that concatenation of iterables of sequences can be implemented efficiently in concrete types. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Aug 11 22:48:23 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 11 Aug 2014 13:48:23 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1407790103.79044.YahooMailNeo@web181001.mail.ne1.yahoo.com> On Monday, August 11, 2014 12:56 PM, Terry Reedy wrote: > Given that the two parameters of join are a concrete string and the > abstraction 'iterable of strings', join can only be a method of the > joiner. In many languages (those with generic types), the notion of Sequence, or S:Sequence where S.ElementType==String, or similar, makes sense. The problem is that you can't define methods on specializations of types, only on the generic types. And Sequence.join makes no sense, because that implies the nonsensical Sequence.join. (Neither of those applies to C++, where you can add methods to specializations, or add methods to the template and they just don't appear on specializations where they make no sense because of SFINAE. But I don't know of any other language that does things that way.) But since most of those languages have leaky type systems despite most of them supposedly being statically strongly typed, it "works". It's been fun to watch people try to do the same thing in Swift, which really is strongly typed, and will not let you define a Sequence.join method unless it compiles for any ElementType. Of course there's no such problem defining a String.join method whose parameter is S:Sequence where S.ElementType== String, so String.join works just fine. So anyone coming to Swift from Python can add a join method without even thinking twice, while anyone coming from any other language will fight with the compiler for a while, then give up and either copy to an ObjC NSArray types to escape the type system, or copy and paste their join loop all over their code. But anyway, pointing out that every OO language gets this wrong may be entertaining, but it doesn't help the fact that everyone coming to Python from those other languages looks for it in the wrong place. From ethan at stoneleaf.us Mon Aug 11 23:10:31 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 11 Aug 2014 14:10:31 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53E93147.4050504@stoneleaf.us> On 08/11/2014 10:53 AM, Nathaniel Smith wrote: > > But practically speaking, how would this work? In general str.join and > sum have different semantics. What happens if we descend deep into the > iterable and then discover a non-string (that might nonetheless still > have a + operator)? The same thing that happens now if you pass a list to join with a non-string entry: --> ' '.join(['some', 'list', 'of', 'words', 'and', 10, 'as', 'a', 'number']) Traceback (most recent call last): File "", line 1, in TypeError: sequence item 5: expected string, int found -- ~Ethan~ From tjreedy at udel.edu Mon Aug 11 23:17:29 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 11 Aug 2014 17:17:29 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <75E96E84-7406-44B6-9436-1069F27DAF6A@yahoo.com> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <75E96E84-7406-44B6-9436-1069F27DAF6A@yahoo.com> Message-ID: On 8/11/2014 3:13 PM, Andrew Barnert wrote: > Beginning programmers don't even know to look for a join function, The rather complete index, where they will find it as a bytearray/bytes/str functions, as well as variations of thread.join. > But someone coming to > Python looking for the equivalent of -[NSArray > componentsJoinedBySeparator:] or Enumerable#join or > Sequence.joinStrings or whatever expects to find it as list.join. When they don't find it, there is the index. The first thing people should learn is that the Python doc set includes a tutorial for beginners, a Reference manual for syntax, a Library manual for objects, and a collective Index for all of these. The Index starts with a Symbol page for looking up thing like '*' or '|=' that do not work in search engines. -- Terry Jan Reedy From ethan at stoneleaf.us Mon Aug 11 23:32:10 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 11 Aug 2014 14:32:10 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <75E96E84-7406-44B6-9436-1069F27DAF6A@yahoo.com> Message-ID: <53E9365A.60207@stoneleaf.us> On 08/11/2014 02:17 PM, Terry Reedy wrote: > > The first thing people should learn is that the Python doc set includes a tutorial for beginners, a Reference manual for > syntax, a Library manual for objects, and a collective Index for all of these. The Index starts with a Symbol page for > looking up thing like '*' or '|=' that do not work in search engines. +1 Before using any language one should familiarize oneself with it, and what better way than the tutorial that comes with the language? (And no, I've never read it -- but I have read others, and several books, and a couple classes to fill in the chinks. ;) -- ~Ethan~ From ethan at stoneleaf.us Mon Aug 11 23:35:50 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 11 Aug 2014 14:35:50 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> Message-ID: <53E93736.8030105@stoneleaf.us> On 08/11/2014 02:25 PM, Nathaniel Smith wrote: > On Mon, Aug 11, 2014 at 10:10 PM, Ethan Furman wrote: >> On 08/11/2014 10:53 AM, Nathaniel Smith wrote: >>> >>> >>> But practically speaking, how would this work? In general str.join and >>> sum have different semantics. What happens if we descend deep into the >>> iterable and then discover a non-string (that might nonetheless still >>> have a + operator)? >> >> >> The same thing that happens now if you pass a list to join with a non-string >> entry: >> >> --> ' '.join(['some', 'list', 'of', 'words', 'and', 10, 'as', 'a', >> 'number']) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: sequence item 5: expected string, int found > > class Nasty: > def __radd__(self, other): > return other + "foo" > > "".join(["some", "strings", "and", "one", Nasty()]) > sum(["some", "strings", "and", "one", Nasty()], "") Quite frankly, I regard this as a point in sum's favor. We have, effectively, a string-subclass and join chokes on it. -- ~Ethan~ From njs at pobox.com Mon Aug 11 23:25:04 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 11 Aug 2014 22:25:04 +0100 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <53E93147.4050504@stoneleaf.us> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> Message-ID: On Mon, Aug 11, 2014 at 10:10 PM, Ethan Furman wrote: > On 08/11/2014 10:53 AM, Nathaniel Smith wrote: >> >> >> But practically speaking, how would this work? In general str.join and >> sum have different semantics. What happens if we descend deep into the >> iterable and then discover a non-string (that might nonetheless still >> have a + operator)? > > > The same thing that happens now if you pass a list to join with a non-string > entry: > > --> ' '.join(['some', 'list', 'of', 'words', 'and', 10, 'as', 'a', > 'number']) > Traceback (most recent call last): > File "", line 1, in > TypeError: sequence item 5: expected string, int found class Nasty: def __radd__(self, other): return other + "foo" "".join(["some", "strings", "and", "one", Nasty()]) sum(["some", "strings", "and", "one", Nasty()], "") -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From wolfgang.maier at biologie.uni-freiburg.de Tue Aug 12 00:12:50 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 12 Aug 2014 00:12:50 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <53E93736.8030105@stoneleaf.us> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> Message-ID: On 11.08.2014 23:35, Ethan Furman wrote: > >> class Nasty: >> def __radd__(self, other): >> return other + "foo" >> >> "".join(["some", "strings", "and", "one", Nasty()]) >> sum(["some", "strings", "and", "one", Nasty()], "") > Interesting. So a slight variation enables sum-based string concatenation *right now*: class ZeroAdditionProperty: def __add__(self, other): return other nullstr = ZeroAdditionProperty() sum(["some", "strings", "and", "one"], nullstr) Wolfgang From ethan at stoneleaf.us Tue Aug 12 00:13:40 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 11 Aug 2014 15:13:40 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> Message-ID: <53E94014.6090008@stoneleaf.us> On 08/11/2014 02:53 PM, Nathaniel Smith wrote: >> >> Quite frankly, I regard this as a point in sum's favor. We have, effectively, a string-subclass and join chokes on it. > > Yes, but the proposal I was responding to was to do something like > > def sum(it, start=0): > if start == "": > return "".join(it) > ... regular sum impl here ... > > And the point is that this is not trivially a transparent optimization. Ah, gotcha. Even without an optimization in sum, I'd still rather it did what it said: add things together. -- ~Ethan~ From tjreedy at udel.edu Tue Aug 12 00:15:18 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 11 Aug 2014 18:15:18 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <53E93736.8030105@stoneleaf.us> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> Message-ID: On 8/11/2014 5:35 PM, Ethan Furman wrote: > On 08/11/2014 02:25 PM, Nathaniel Smith wrote: >> class Nasty: >> def __radd__(self, other): >> return other + "foo" >> >> "".join(["some", "strings", "and", "one", Nasty()]) >> sum(["some", "strings", "and", "one", Nasty()], "") I don't understand the point of this. > Quite frankly, I regard this as a point in sum's favor. We have, > effectively, a string-subclass and join chokes on it. Nasty is a subclass of object, with no default value. Make it a real str subclass and join works fine. class Nasty(str): def __radd__(self, other): return other + "foo" print("".join(["some", "strings", "and", "one", Nasty()])) >>> somestringsandone -- Terry Jan Reedy From wolfgang.maier at biologie.uni-freiburg.de Tue Aug 12 00:23:11 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 12 Aug 2014 00:23:11 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> Message-ID: On 12.08.2014 00:15, Terry Reedy wrote: > On 8/11/2014 5:35 PM, Ethan Furman wrote: >> On 08/11/2014 02:25 PM, Nathaniel Smith wrote: > >>> class Nasty: >>> def __radd__(self, other): >>> return other + "foo" >>> >>> "".join(["some", "strings", "and", "one", Nasty()]) >>> sum(["some", "strings", "and", "one", Nasty()], "") > > I don't understand the point of this. > >> Quite frankly, I regard this as a point in sum's favor. We have, >> effectively, a string-subclass and join chokes on it. > > Nasty is a subclass of object, with no default value. Make it a real > str subclass and join works fine. > > class Nasty(str): > def __radd__(self, other): > return other + "foo" > > print("".join(["some", "strings", "and", "one", Nasty()])) > >>> > somestringsandone > No, it's not, at least not as intended or the result would be somestringsandonefoo The point about Ethan's example is that join only works with str and subclasses thereof, but not with proxy classes wrapping a str object. From njs at pobox.com Mon Aug 11 23:53:12 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 11 Aug 2014 22:53:12 +0100 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <53E93736.8030105@stoneleaf.us> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> Message-ID: On 11 Aug 2014 22:36, "Ethan Furman" wrote: > > On 08/11/2014 02:25 PM, Nathaniel Smith wrote: >> >> On Mon, Aug 11, 2014 at 10:10 PM, Ethan Furman wrote: >>> >>> On 08/11/2014 10:53 AM, Nathaniel Smith wrote: >>>> >>>> >>>> >>>> But practically speaking, how would this work? In general str.join and >>>> sum have different semantics. What happens if we descend deep into the >>>> iterable and then discover a non-string (that might nonetheless still >>>> have a + operator)? >>> >>> >>> >>> The same thing that happens now if you pass a list to join with a non-string >>> entry: >>> >>> --> ' '.join(['some', 'list', 'of', 'words', 'and', 10, 'as', 'a', >>> 'number']) >>> Traceback (most recent call last): >>> File "", line 1, in >>> TypeError: sequence item 5: expected string, int found >> >> >> class Nasty: >> def __radd__(self, other): >> return other + "foo" >> >> "".join(["some", "strings", "and", "one", Nasty()]) >> sum(["some", "strings", "and", "one", Nasty()], "") > > > Quite frankly, I regard this as a point in sum's favor. We have, effectively, a string-subclass and join chokes on it. Yes, but the proposal I was responding to was to do something like def sum(it, start=0): if start == "": return "".join(it) ... regular sum impl here ... And the point is that this is not trivially a transparent optimization. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at 2sn.net Tue Aug 12 00:56:27 2014 From: python at 2sn.net (Alexander Heger) Date: Tue, 12 Aug 2014 08:56:27 +1000 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: > You are kidding, right? What will prevent them from using logic to recall > the invariant mistakenly as > s == sep.join(sep.split(s)) ? If only because there are natural use cases > for passing mysep.join around as a callable that will join with a fixed > separator, and there would similarly exist use cases for mysep.split, while > there are really none for passing around mystring.split to split a fixed > string by varying separators. > > This whole corner of python is a most ugly system of inconsistent acne scars > in the 2.x series (dunno about 3.x). What with sum(...) pedantically telling > you to use str.join - if it knows what you want, it should just to it. I tend to agree... From python at 2sn.net Tue Aug 12 01:16:24 2014 From: python at 2sn.net (Alexander Heger) Date: Tue, 12 Aug 2014 09:16:24 +1000 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <53E924EE.40704@biologie.uni-freiburg.de> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E924EE.40704@biologie.uni-freiburg.de> Message-ID: > - you have input consisting of numbers with ',' as the decimal separator > (the default in German-speaking countries), but downstream software expects > '.' > > ,i.e., they can all be solved with the general pattern: > > s = new_sep.join(s.split(old_sep)) In [1]: 'a,b'.replace(',','.') Out[1]: 'a.b' From tjreedy at udel.edu Tue Aug 12 02:15:12 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 11 Aug 2014 20:15:12 -0400 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> Message-ID: On 8/11/2014 6:23 PM, Wolfgang Maier wrote: > On 12.08.2014 00:15, Terry Reedy wrote: >> Nasty is a subclass of object, with no default value. Make it a real >> str subclass and join works fine. >> >> class Nasty(str): >> def __radd__(self, other): >> return other + "foo" >> >> print("".join(["some", "strings", "and", "one", Nasty()])) If one runs a program with a print statement from an Idle editor, >> >>> >> somestringsandone the shell prints the >>> prompt and then the output. I copy the prompt as a separator. > No, it's not, I have been copy and pasting python code and output for over 17 years. You could have verified that I did so accurately here in less than a minute. I keep the Idle icon pinned to my task bar and usually have a scratch file for experiments in its recent files list, if not already open, to make this sort of thing easy. > at least not as intended I was talking about reality, not intentions that were not clear to me. -- Terry Jan Reedy From ethan at stoneleaf.us Tue Aug 12 05:03:21 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 11 Aug 2014 20:03:21 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> Message-ID: <53E983F9.1090802@stoneleaf.us> On 08/11/2014 05:15 PM, Terry Reedy wrote: > On 8/11/2014 6:23 PM, Wolfgang Maier wrote: >> On 12.08.2014 00:15, Terry Reedy wrote: > >>> Nasty is a subclass of object, with no default value. Make it a real >>> str subclass and join works fine. >>> >>> class Nasty(str): >>> def __radd__(self, other): >>> return other + "foo" >>> >>> print("".join(["some", "strings", "and", "one", Nasty()])) > > If one runs a program with a print statement from an Idle editor, > >>> >>> >>> somestringsandone > > the shell prints the >>> prompt and then the output. I copy the prompt as a separator. > >> No, it's not, > > I have been copy and pasting python code and output for over 17 years. You could have verified that I did so accurately > here in less than a minute. I keep the Idle icon pinned to my task bar and usually have a scratch file for experiments > in its recent files list, if not already open, to make this sort of thing easy. > >> at least not as intended > > I was talking about reality, not intentions that were not clear to me. His remark was pointed at the fact that your output is missing the final "foo". Remove the 'r' from __radd__, though, and you would have what you were trying to demonstrate. -- ~Ethan~ From stephen at xemacs.org Tue Aug 12 06:45:05 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 12 Aug 2014 13:45:05 +0900 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <53E94014.6090008@stoneleaf.us> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> <53E94014.6090008@stoneleaf.us> Message-ID: <87oavqiaem.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > Even without an optimization in sum, I'd still rather it did what > it said: add things together. But strings aren't additive, they're multiplicative: >>> "x" "y" 'xy' >>> This is a convention deeply embedded in mathematics notation (multiplicative operators are noncommutative and frequently omitted in notation, additive operators are commutative and explicit notation required). The point is not that it's *wrong* to think that "strings are additive, so sum *should* apply to iterables of strings". Rather, it's that it's *not wrong* to think that "strings are non-additive and therefore not sum()-able". (I'm somewhere in between.) Why choose '+' as the string multiplication operator, then? Because programming languages aren't as flexible as mathematics notation, compromises are often made. Among other things, '*' also has a meaning for strings (when the other operand is an integer it means repetition -- oddly enough, it's commutative!) I suppose Guido could have been a pure algebraist about it and chosen '*' and '^' for concatenation and repetition, respectively, but that looks odd to me. The lesson in the end is that although there are many functions that are convenient to express in Python using operator notation, the choice of operators should not be taken too seriously in deciding how they will compose. From stephen at xemacs.org Tue Aug 12 07:51:19 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 12 Aug 2014 14:51:19 +0900 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87mwbai7c8.fsf@uwakimon.sk.tsukuba.ac.jp> Nathaniel Smith writes: > I don't have any data here, but I bet people who know about str.join > (even for its natural use cases like ", ".join(...)) outnumber the > people who know that sum() takes a second argument by a very large > factor. This is easy to fix, it now occurs to me. Allow types with __add__ to provide an optional __sum__ method, and give the numeric ABC a default __sum__ implementation. (It would be nice if it could check for floats and restart with fsum if one is encountered. And of course, there may be other ABCs with __add__ that could get a default __sum__.) Then sum could be just def sum(itr, start=0): if start = 0: itr = iter(itr) start = next(itr) return start.__sum__(itr) and class str(...): def __sum__(self, itr): return self + ''.join(itr) # probably can be optimized Is it really it worth it, though? > But practically speaking, how would this work? In general str.join and > sum have different semantics. sum(iter_of_str) currently doesn't have semantics. The semantics proponents of sum() seem to expect is precisely ''.join(iter_of_str). Where's the problem? > What happens if we descend deep into the iterable and then discover > a non-string (that might nonetheless still have a + operator)? We lose, er, an exception is raised. Why is that a problem? I think most people who want a polymorphic sum() expect it to accept a homogeneous iterable as the first argument. I don't think they have expectations that sum will be equivalent to def new_sum(it, start=0): # compatible signature ;-) it = iter(it) result = result or next(it) for x in it: result = result + next(it) return result for heterogeneous iterables. Among other things, how do you decide the appropriate return type? start's? That of next(iter(it))? The "most important" of the types in it? Ask for a BDFL pronouncement at each invocation? I suppose you could ask that functions that operate on iterables be partially applicable in the sense that if they *do* raise on the "wrong" type, the exception should provide a partial result, the oddball operand, and an iterable containing the unconsumed operands as attributes. Then the __sum__ method could handle heterogeneous operands if it wants to. Note that partial_sum + oddball may have a different type from the expected one even if it works. This seems like a recipe for bugs to me. Are there use cases for such heterogenous sums? The only exception that might be pretty safe would be a case where you can coerce the oddball to the partial result's type. But in the salient case of str, pretty much every x has a str(x). I don't think that an optimized version of: def new_sum(iter, start): expected_type = type(start) result = start for x in iter: try: result = result + x except TypeError: result = result + expected_type(x) return result is really what we want when type(start) == str, so it probably shouldn't be default, and probably not when type(start) is numeric, either. From vito.detullio at gmail.com Tue Aug 12 08:15:26 2014 From: vito.detullio at gmail.com (Vito De Tullio) Date: Tue, 12 Aug 2014 08:15:26 +0200 Subject: [Python-ideas] float comparison in doctest References: <53E8EE7B.50408@yahoo.com> Message-ID: Terry Reedy wrote: > I think there have been vague proposals for a float method, but > "a - b < delta" is shorter than "a.almost_equat(b, abs=delta)" and > probably easier to understand. what if a < b? or if a or b (or both) are NaN/+Inf/-Inf? Or... There are many pitfalls comparing floats, isn't better to delegate these checks in a function? -- By ZeD From stephen at xemacs.org Tue Aug 12 08:29:20 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 12 Aug 2014 15:29:20 +0900 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <53E983F9.1090802@stoneleaf.us> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> <53E983F9.1090802@stoneleaf.us> Message-ID: <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > His remark was pointed at the fact that your output is missing the > final "foo". Remove the 'r' from __radd__, though, and you would > have what you were trying to demonstrate. Believe someone when he says he copy-pasted. ;-) Unless you've actually run the code and got a different result, and even then you should probably include version information etc. The presence of __radd__ (or __add__, for that matter, although it's reasonably difficult to create a str derivative without it) is irrelevant to why "".join works when Nasty is derived from str. You're confusing the actual semantics of str.join (repeated copying at appropriate offsets into a sufficiently large buffer) with the naive implementation of sum (an iterated application of '+'). From daviesk24 at yahoo.com Tue Aug 12 08:49:56 2014 From: daviesk24 at yahoo.com (Kevin Davies) Date: Mon, 11 Aug 2014 20:49:56 -1000 Subject: [Python-ideas] float comparison in doctest In-Reply-To: References: <53E8EE7B.50408@yahoo.com> Message-ID: <53E9B914.1000702@yahoo.com> An HTML attachment was scrubbed... URL: From wolfgang.maier at biologie.uni-freiburg.de Tue Aug 12 09:16:09 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 12 Aug 2014 09:16:09 +0200 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> <53E983F9.1090802@stoneleaf.us> <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53E9BF39.6060601@biologie.uni-freiburg.de> On 08/12/2014 08:29 AM, Stephen J. Turnbull wrote: > Ethan Furman writes: > > > His remark was pointed at the fact that your output is missing the > > final "foo". Remove the 'r' from __radd__, though, and you would > > have what you were trying to demonstrate. > > Believe someone when he says he copy-pasted. ;-) Unless you've > actually run the code and got a different result, and even then you > should probably include version information etc. > > The presence of __radd__ (or __add__, for that matter, although it's > reasonably difficult to create a str derivative without it) is > irrelevant to why "".join works when Nasty is derived from str. > You're confusing the actual semantics of str.join (repeated copying at > appropriate offsets into a sufficiently large buffer) with the naive > implementation of sum (an iterated application of '+'). > Exactly. So my point was that when you don't subclass str, but instead use a wrapper around it, you can give it a as str-like interface as you want so the thing looks and feels like a string to users, it will still not work as part of an iterable passed to .join (because .join is C code moving around arrays of chars and is completely ignorant of all the nice methods you added to your object). Sum on the other hand knows how to use .__add__ and .__radd__ . Not that I think this is an argument against str.join() - it is a specialized method to join strings efficiently and it's good at that (in fact, I think, that's why it makes complete sense to have it implemented as a str method because this and nothing else is the data type it can handle). I just found Ethan's Nasty() example interesting. Wolfgang From ncoghlan at gmail.com Tue Aug 12 09:35:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Aug 2014 17:35:04 +1000 Subject: [Python-ideas] float comparison in doctest In-Reply-To: References: <53E8EE7B.50408@yahoo.com> Message-ID: On 12 Aug 2014 04:16, "Terry Reedy" wrote: > > On 8/11/2014 12:25 PM, Kevin Davies wrote: >> >> Erik Bray has created a nifty addition to doctest to compare floating >> point numbers using a `+FLOAT_CMP` flag. > > > The problem with a simple flag is that almost_equal should sometimes be absolute and sometimes relative and in both cases a parameter in needed. I think there have been vague proposals for a float method, but > "a - b < delta" is shorter than "a.almost_equat(b, abs=delta)" and probably easier to understand. For more formal tests, we already have unittest.TestCase.assertAlmostEqual. This sounds like a reasonable flag for doctest's original purpose of checking that examples in docs are showing sensible answers. Cheers, Nick. > > > > It is implemented within >> >> Astropy (http://www.astropy.org/) and the related issue is >> https://github.com/astropy/astropy/issues/2662. > > > The related issue is applying flags to a block of test with a Sphinx directive. > > >> According to Erik, it >> may need some tweaks, but I think it would be a really useful feature to >> make available in the doctest package itself. > > > Erik would have to offer his code. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Aug 12 17:24:27 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Aug 2014 08:24:27 -0700 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> <53E983F9.1090802@stoneleaf.us> <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53EA31AB.7030200@stoneleaf.us> On 08/11/2014 11:29 PM, Stephen J. Turnbull wrote: > Ethan Furman writes: > > > His remark was pointed at the fact that your output is missing the > > final "foo". Remove the 'r' from __radd__, though, and you would > > have what you were trying to demonstrate. > > Believe someone when he says he copy-pasted. ;-) Unless you've > actually run the code and got a different result, and even then you > should probably include version information etc. I did believe -- I just also forgot that Python would use __radd__ in a subclass, and so thought that was a bug. But you're right, I should have tried it myself before posting -- my apologies. -- ~Ethan~ From stephen at xemacs.org Wed Aug 13 06:38:25 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 13 Aug 2014 13:38:25 +0900 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <53E9BF39.6060601@biologie.uni-freiburg.de> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> <53E983F9.1090802@stoneleaf.us> <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp> <53E9BF39.6060601@biologie.uni-freiburg.de> Message-ID: <87ha1hhum6.fsf@uwakimon.sk.tsukuba.ac.jp> Wolfgang Maier writes: > Exactly. So my point was that when you don't subclass str, but instead > use a wrapper around it, you can give it a as str-like interface as you > want so the thing looks and feels like a string to users, it will still > not work as part of an iterable passed to .join You mean this behavior? wideload:~ 12:42$ python3.2 >>> ... >>> class N: ... def __init__(self, s=''): ... self.s = s ... def __str__(self): ... return self.s ... >>> " ".join(['a', N('b')]) Traceback (most recent call last): File "", line 1, in TypeError: sequence item 1: expected str instance, N found >>> ' '.join(str(x) for x in ['a', N('b')]) 'a b' >>> Given the fact that every object is str-able, I don't think we want to give "str(x) for x in" semantics to str.join. So I think the answer is "if you want Nasty to automatically acquire all the behaviors of str, make it a subclass of str". I can't think of a use case where subclassing would be problematic. > Sum on the other hand knows how to use .__add__ and .__radd__ . It seems to me that that's a strong argument against "summing strings" with the current implementation of sum(), given the ease with which you can construct types where the "sum" of an iterable can be implemented efficiently and gives the same answer as the generic algorithm based on '+', but the generic algorithm is inefficient (just make it immutable). I suppose most Sequence types are arrays of pointers at the C level, or otherwise implement O(1) '+=', so either the join-style "just memmove the arrays into a sufficiently large buffer", or iterated '+=', does the trick for an efficient generic sum. This just guesswork, though. From ncoghlan at gmail.com Wed Aug 13 07:50:13 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Aug 2014 15:50:13 +1000 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: <87ha1hhum6.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> <53E983F9.1090802@stoneleaf.us> <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp> <53E9BF39.6060601@biologie.uni-freiburg.de> <87ha1hhum6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 13 August 2014 14:38, Stephen J. Turnbull wrote: > Wolfgang Maier writes: > > > Exactly. So my point was that when you don't subclass str, but instead > > use a wrapper around it, you can give it a as str-like interface as you > > want so the thing looks and feels like a string to users, it will still > > not work as part of an iterable passed to .join > > You mean this behavior? > > wideload:~ 12:42$ python3.2 >>>> > ... >>>> class N: > ... def __init__(self, s=''): > ... self.s = s > ... def __str__(self): > ... return self.s > ... >>>> " ".join(['a', N('b')]) > Traceback (most recent call last): > File "", line 1, in > TypeError: sequence item 1: expected str instance, N found >>>> ' '.join(str(x) for x in ['a', N('b')]) > 'a b' >>>> > > Given the fact that every object is str-able, I don't think we want to > give "str(x) for x in" semantics to str.join. So I think the answer > is "if you want Nasty to automatically acquire all the behaviors of > str, make it a subclass of str". Note that this is a general problem - it is quite common to use explicit type checks against str rather than relying on ducktyping. In theory, a suitable ABC could be defined (using collections.UserString as a starting point), but nobody has ever found it a pressing enough problem to take the time to do so - it's generally easier to just inherit from str. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Wed Aug 13 08:21:42 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 13 Aug 2014 15:21:42 +0900 Subject: [Python-ideas] [Python-Dev] sum(...) limitation In-Reply-To: References: <20140802203513.GA10447@k2> <20140804181013.GO4525@ando> <53E4055D.2040305@stoneleaf.us> <53E51269.5030209@stoneleaf.us> <7414D373-F598-4805-9DE8-F9779D08FEE8@gmail.com> <53E571B8.7030103@stoneleaf.us> <87fvh6jg1y.fsf@uwakimon.sk.tsukuba.ac.jp> <874mxkkb0f.fsf@uwakimon.sk.tsukuba.ac.jp> <53E7CBA4.40105@g.nevcal.com> <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp> <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> Redirecting to python-ideas, so trimming less than I might. Chris Barker writes: > On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull > wrote: > > > I'm referring to removing the unnecessary information that there's a > > better way to do it, and simply raising an error (as in Python 3.2, > > say) which is all a RealProgrammer[tm] should ever need! > > > > I can't imagine anyone is suggesting that -- disallow it, but don't tell > anyone why? As I said, it's a regression. That's exactly the behavior in Python 3.2. > The only thing that is remotely on the table here is: > > 1) remove the special case for strings -- buyer beware -- but consistent > and less "ugly" It's only consistent if you believe that Python has strict rules for use of various operators. It doesn't, except as far as they are constrained by precedence. For example, I have an application where I add bytestrings bytewise modulo N <= 256, and concatenate them. In fact I use function call syntax, but the obvious operator syntax is '+' for the bytewise addition, and '*' for the concatenation. It's not in the Zen, but I believe in the maxim "If it's worth doing, it's worth doing well." So for me, 1) is out anyway. > 2) add a special case for strings that is fast and efficient -- may be as > simple as calling "".join() under the hood --no more code than the > exception check. Sure, but what about all the other immutable containers with __add__ methods? What about mappings with key-wise __add__ methods whose values might be immutable but have __add__ methods? Where do you stop with the special-casing? I consider this far more complex and ugly than the simple "sum() is for numbers" rule (and even that is way too complex considering accuracy of summing floats). > And I doubt anyone really is pushing for anything but (2) I know that, but I think it's the wrong solution to the problem (which is genuine IMO). The right solution is something generic, possibly a __sum__ method. The question is whether that leads to too much work to be worth it (eg, "homogeneous_iterable"). > > Because obviously we'd want the attractive nuisance of "if you > > have __add__, there's a default definition of __sum__" > > now I'm confused -- isn't that exactly what we have now? Yes and my feeling (backed up by arguments that I admit may persuade nobody but myself) is that what we have now kinda sucks[tm]. It seemed like a good idea when I first saw it, but then, my apps don't scale to where the pain starts in my own usage. > > It's possible that Python could provide some kind of feature that > > would allow an optimized sum function for every type that has > > __add__, but I think this will take a lot of thinking. > > does it need to be every type? As it is the common ones work fine already > except for strings -- so if we add an optimized string sum() then we're > done. I didn't say provide an optimized sum(), I said provide a feature enabling people who want to optimize sum() to do so. So yes, it needs to be every type (the optional __sum__ method is a proof of concept, modulo it actually being implementable ;-). > > *Somebody* will do it (I don't think anybody is +1 on restricting > > sum() to a subset of types with __add__). > > uhm, that's exactly what we have now Exactly. Who's arguing that the sum() we have now is a ticket to Paradise? I'm just saying that there's probably somebody out there negative enough on the current situation to come up with an answer that I think is general enough (and I suspect that python-dev consensus is that demanding, too). > sum() can be used for any type that has an __add__ defined. I'd like to see that be mutable types with __iadd__. > What I fail to see is why it's better to raise an exception and > point users to a better way, than to simply provide an optimization > so that it's a mute issue. Because inefficient sum() is an attractive nuisance, easy to overlook, and likely to bite users other than the author. > The only justification offered here is that will teach people that summing > strings (and some other objects?) Summing tuples works (with appropriate start=tuple()). Haven't benchmarked, but I bet that's O(N^2). > is order(N^2) and a bad idea. But: > > a) Python's primary purpose is practical, not pedagogical (not that it > isn't great for that) My argument is that in practical use sum() is a bad idea, period, until you book up on the types and applications where it *does* work. N.B. It doesn't even work properly for numbers (inaccurate for floats). > b) I doubt any naive users learn anything other than "I can't use sum() for > strings, I should use "".join()". For people who think that special-casing strings is a good idea, I think this is about as much benefit as you can expect. Why go farther?<0.5 wink/> > I submit that no naive user is going to get any closer to a proper > understanding of algorithmic Order behavior from this small hint. Which > leaves no reason to prefer an Exception to an optimization. TOOWTDI. str.join is in pretty much every code base by now, and tutorials and FAQs recommending its user and severely deprecating sum for strings are legion. > One other point: perhaps this will lead a naive user into thinking -- > "sum() raises an exception if I try to use it inefficiently, so it must be > OK to use for anything that doesn't raise an exception" -- that would be a > bad lesson to mis-learn.... That assumes they know about the start argument. I think most naive users will just try to sum a bunch of tuples, and get the "can't add 0, tuple" Exception and write a loop. I suspect that many of the users who get the "use str.join" warning along with the Exception are unaware of the start argument, too. They expect sum(iter_of_str) to magically add the strings. Ie, when in 3.2 they got the uninformative "can't add 0, str" message, they did not immediately go "d'oh" and insert ", start=''" in the call to sum, they wrote a loop. > while we are at it, having the default sum() for floats be fsum() > would be nice How do you propose to implement that, given math.fsum is perfectly happy to sum integers? You can't just check one or a few leading elements for floatiness. I think you have to dispatch on type(start), but then sum(iter_of_floats) DTWT. So I would suggest changing the signature to sum(it, start=0.0). This would probably be acceptable to most users with iterables of ints, but does imply some performance hit. > This does turn sum() into a function that does type-based dispatch, > but isn't python full of those already? do something special for > the types you know about, call the generic dunder method for the > rest. AFAIK Python is moving in the opposite direction: if there's a common need for dispatching to type-specific implementations of a method, define a standard (not "generic") dunder for the purpose, and have the builtin (or operator, or whatever) look up (not "call") the appropriate instance in the usual way, then call it. If there's a useful generic implementation, define an ABC to inherit from that provides that generic implementation. From castironpi at gmail.com Wed Aug 13 14:21:16 2014 From: castironpi at gmail.com (Aaron Brady) Date: Wed, 13 Aug 2014 07:21:16 -0500 Subject: [Python-ideas] Mutating while iterating In-Reply-To: References: Message-ID: On Sun, Aug 3, 2014 at 12:46 PM, Aaron Brady wrote: > On Sat, Jul 26, 2014 at 10:39 PM, Nick Coghlan wrote: [snip] >> Hi, >> >> This is clearly an issue of grave concern to you, but as Raymond >> pointed out previously, you appear to have misunderstood the purpose >> of those exceptions. They're there to prevent catastrophic failure of >> the interpreter itself (i.e. segmentation faults), not to help find >> bugs in user code. If users want to mutate containers while they're >> iterating over them, they're generally free to do so. The only time >> we'll actively disallow it is when such mutation will outright *break* >> the iterator, rather than merely producing potentially surprising >> results. >> >> I have closed the new issue and added a longer reply (with examples) >> that will hopefully better explain why we have no intention of >> changing this behaviour: http://bugs.python.org/issue22084#msg224100 > > > Python is replete with examples of prohibiting structures which are > likely bugs but aren't segfaults. There are also reciprocal-- though > not necessarily inverse-- limitations of the unordered collections > themselves ("set" and "dict"). The new behavior is transparent for > the programmer; no possible programs can /rely/ on the existing > behavior. The new behavior introduces no new objects, possibly except > "IterationError", no new syntax, and no new costs. > > I propose we leave this discussion thread open for the time being. I > also take the issue of /re/-assigning to keys during iteration to be > settled as permitted. Nick, It works by comparing the state of the container, to its state when the iterator was opened. We're ensuring it will always have a unique state, up to comparison. A state can be reused once no iterators refer to it, hence needing the reference count. A full "object" is not needed for a memo, only the reference count, but the "object" is easier and only twice the size, as "PyBaseObject Type" is allocated anyway. I'll point out that among the additional costs that there aren't, garbage collection isn't any slower, as both "tp traverse" and "tp clear" are empty in the "PyBaseObject Type" definition, on line 3511-3512 in "typeobject.c" at time of writing [1]. [1] http://svn.python.org/view/python/trunk/Objects/typeobject.c?revision=81744&view=markup#l3511 From erik.m.bray at gmail.com Wed Aug 13 17:30:05 2014 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 13 Aug 2014 11:30:05 -0400 Subject: [Python-ideas] float comparison in doctest Message-ID: [Sorry for breaking the threading--turns out I didn't have a python-ideas subscription from this address] Just a few followup points I wanted to make to Kevin's post about a proposed +FLOAT_CMP flag for doctest. A better link for the implementation is this PR in Astropy: https://github.com/astropy/astropy/pull/2087 I can't take full credit either--most of the existing code was borrowed (with some small improvements) from the SymPy project. I had started on a similar feature independently, but then borrowed their implementation upon seeing that it was further along than mine. On 12 Aug 2014 04:16, "Terry Reedy" wrote: > The problem with a simple flag is that almost_equal should sometimes be > absolute and sometimes relative and in both cases a parameter in needed. I > think there have been vague proposals for a float method, but "a - b < delta" > is shorter than "a.almost_equat(b, abs=delta)" and probably easier to > understand. Indeed, that's what I had in mind when I told Kevin that there were still issues with this. There's no obvious way (I can think of at least) to parameterize doctest flags. However, the original immediate purpose of this feature was to handle very small differences in representation of the same value between different platforms, and in that respect it has worked very well. Astropy has a lot of doctests, and many of which have floating point outputs. This +FLOAT_CMP flag enabled removing tons of ellipses from the test outputs, and restoring the full outputs which certainly read better in the docs. For more complete unit tests of course we use assert_almost_equal type functions. That said, if anyone has any ideas for allowing tweaking the tolerances for a doctest flag that would be great. If this otherwise seems like a good idea in general to include in the doctest module I will offer a patch. Erik From steve at pearwood.info Wed Aug 13 18:37:30 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 14 Aug 2014 02:37:30 +1000 Subject: [Python-ideas] sum(...) limitation In-Reply-To: <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53E7CBA4.40105@g.nevcal.com> <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp> <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140813163729.GI4525@ando> I'm going to cut straight to the chase here because this thread, and its related ones, on Python-Dev are giving me a headache and overloading my inbox. So I'm going to make a probably futile :-( attempt to cut off yet another huge thread before it starts by explaining why I think sum() ought to stay exactly as it is. Built-in sum() is already quite complex. It has a fast path and a slow path, and that's just for numbers. While it's tempting to imagine sum() being even more clever and able to handle more cases, that increases the complexity and makes it more likely to end up slower rather than faster, or buggy, or both. Better to let the caller choose a specialist function (like numpy.array.sum, or math.fsum, or str.join) that handles the caller's specialist needs, than to try to make sum() master of everything. The more special cases sum() has, the more the pressure to add even more. In the statistics module, I have a private _sum() function which tried really hard to deal with high-accuracy sums of mixed arbitrary numeric types without compromising too badly on speed, and it's much harder than it seems. Trying to handle non-numeric types too increases the complexity significantly. If you're smarter than me (I expect that many of you probably are) and believe that you can write a version of sum() which: (1) is fast (2) has better than O(N**2) performance (3) is correct (4) is accurate (5) handles INFs and NANs (for those types which have them) (6) handles mixed types (for those types which allow mixing) (7) honours subclasses with custom __add__ and __radd__ methods (8) and keeps the expected semantics that sum() is like repeated addition (or concatenation) and does so for *both* numeric and non-numeric cases (like strings, bytes, tuples, lists), then PLEASE write some code and publish it. I for one would love to see it or use it for the statistics module. But until you have tried writing such a thing, whether in C or Python, I think you're probably underestimating how hard it is and how fragile the result will be. So, a plea: please stop trying to overloaded poor ol' built-in sum. sum() is *not* the One Obvious Way to add arbitrary objects in every domain. sum() is intended for simple cases of adding numbers, it is not intended as a specialist summation function for everything under the sun that can be added or concatenated. A bit of history, as I remember it: sum() exists because for half of Python's lifetime, people were regularly defining this: def sum(numbers): return reduce(lambda a, b: a+b, numbers) so they could easy add up a bunch of numbers: num_pages = sum([ch.pages() for ch in self.chapters]) sort of thing. Since this was a common need, it was decided to add it to the built-ins. But sum() was never intended to replace str.join or list.extend, let alone even more exotic cases. Built-in sum is aimed at sequences of numbers, not strings, lists, tuples, or Widgets for that matter. Perhaps giving it a start parameter was a mistake, but it is there and backwards compatibility says it isn't going to be removed. But that doesn't mean that the use of sum() on arbitrary types ought to be *encouraged*, even if it is *allowed*. Conceptually, sum() is intended to behave like: for value in sequence: start = start + value That means calling custom __add__ or __radd__ methods if they exist. It also means that sum() cannot delegate to (say) str.join() without changing the semantics. Given: class Special(str): def __radd__(self, other): print("I'm special!") return other + str(self) s = Special('y') the sum 'x' + s is *not* the same as ''.join(['x', s]). A similar argument applies to list.extend, and the source code in bltinmodule.c already makes that point. Replying to a couple of points from Stephen: On Wed, Aug 13, 2014 at 03:21:42PM +0900, Stephen J. Turnbull wrote: > > sum() can be used for any type that has an __add__ defined. > > I'd like to see that be mutable types with __iadd__. Surely you don't mean that. That would mean that sum([1, 2, 3]) would no longer work, since ints are not mutable types with __iadd__. [...] > Summing tuples works (with appropriate start=tuple()). Haven't > benchmarked, but I bet that's O(N^2). Correct: increasing the number of tuples being added by a factor of 10 requires almost a factor of 100 more time: py> t = tuple([(i,) for i in range(1000)]) py> with Stopwatch(): ... _ = sum(t, ()) ... time taken: 0.003805 seconds py> t *= 10 py> with Stopwatch(): ... _ = sum(t, ()) ... time taken: 0.230225 seconds py> t *= 10 py> with Stopwatch(): ... _ = sum(t, ()) ... time taken: 32.206471 seconds > My argument is that in practical use sum() is a bad idea, period, > until you book up on the types and applications where it *does* work. > N.B. It doesn't even work properly for numbers (inaccurate for floats). sum() works fine for its intended uses, especially: - summing ints exactly - "low precision" sums of floats I put "low precision" in scare quotes because, for many purposes, that precision is plenty high enough. For the use-case of adding together a dozen or a hundred positive floats of similar magnitude, sum() is fine. It's only advanced and high-precision uses where it falls short. To put it another way: if you want to add the mass of Jupiter to the mass of a flea, you probably want math.fsum(). If you want to add the weight of an apple to the weight of a banana, both measured on a typical kitchen scale, sum() will do the job perfectly adequately. > > while we are at it, having the default sum() for floats be fsum() > > would be nice > > How do you propose to implement that, given math.fsum is perfectly > happy to sum integers? And you really don't want sum(integers) to convert to float by default: py> from math import fsum py> fsum([10**30, 1234]) 1e+30 py> sum([10**30, 1234]) 1000000000000000000000000001234 Insisting that there ought to be one and only one way to sum up is, in my opinion, foolhardy, no matter how attractive it might seem. I believe that the right way is what Python already has: a simple sum() for simple cases, a few specialist sums (including ''.join) for the most common or important specialist cases, and leave the rest to third-party libraries or code you write yourself. That way the caller can then decide exactly what trade-offs between time, memory, convenience, accuracy and behaviour they wish, instead of invariably being surprised or disappointed by whatever trade-offs the one ?ber-sum() made. -- Steven From ckaynor at zindagigames.com Wed Aug 13 18:59:52 2014 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Wed, 13 Aug 2014 09:59:52 -0700 Subject: [Python-ideas] sum(...) limitation In-Reply-To: <20140813163729.GI4525@ando> References: <53E7CBA4.40105@g.nevcal.com> <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp> <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> Message-ID: On Wed, Aug 13, 2014 at 9:37 AM, Steven D'Aprano wrote: > In the statistics module, I have a private _sum() function which tried > really hard to deal with high-accuracy sums of mixed arbitrary numeric > types without compromising too badly on speed, and it's much harder than > it seems. Trying to handle non-numeric types too increases the > complexity significantly. If you're smarter than me (I expect that many > of you probably are) and believe that you can write a version of sum() > which: > > (1) is fast > (2) has better than O(N**2) performance > (3) is correct > (4) is accurate > (5) handles INFs and NANs (for those types which have them) > (6) handles mixed types (for those types which allow mixing) > (7) honours subclasses with custom __add__ and __radd__ methods > (8) and keeps the expected semantics that sum() is like repeated > addition (or concatenation) > > and does so for *both* numeric and non-numeric cases (like strings, > bytes, tuples, lists), then PLEASE write some code and publish it. I for > one would love to see it or use it for the statistics module. But until > you have tried writing such a thing, whether in C or Python, I think > you're probably underestimating how hard it is and how fragile the > result will be. > The basic form I would use for an updated sum, which would often, but not always, perform better would be something like (preferably, written in C, not pure Python): def sum(items, start=0): value = copy.copy(start) # Make sure that start is not mutated. for item in items: value += item return value To avoid needing to access copy.copy, something like the following would also work, but with a bit more local complexity: def sum(items, start=0): count = len(items) if count > 0: value = start + items[0] # Make sure not to mutate start. else return start for i in range(1, count): value += items[i] return value Either of these would likely perform better when summing items which support an optimized __iadd__, which allows many types to perform better. For those which do not support __iadd__, it would fallback to the slower __add__ but still function. Note that the first version may be slower than existing some in some cases, namely if copy.copy(start) is very slow for the type. The second case should never be slower than the existing sum, presuming __iadd__ is not implemented in a slower manner than __add__. Due to the string add optimization CPython, this should make strings sum better than O(N**2), many custom types will also sum faster, and, for all well-behaved types, should produce the same result as the existing sum. Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Aug 13 19:46:30 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 14 Aug 2014 03:46:30 +1000 Subject: [Python-ideas] sum(...) limitation In-Reply-To: References: <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> Message-ID: <20140813174630.GM4525@ando> On Wed, Aug 13, 2014 at 09:59:52AM -0700, Chris Kaynor wrote: > The basic form I would use for an updated sum, which would often, but not > always, perform better would be something like (preferably, written in C, > not pure Python): > > def sum(items, start=0): > value = copy.copy(start) # Make sure that start is not mutated. > for item in items: > value += item > return value That's a semantic change from the existing sum: start does not have to be copyable. Currently I can write this: py> class A: ... def __copy__(self): ... raise TypeError("I am unique!") ... py> a = A() py> class B: ... def __radd__(self, other): ... return 23 ... py> b = B() py> sum([b, 1000, 2000], a) 3023 By changing the semantics of sum() to require start to be copyable, your revision has just broken my application. > To avoid needing to access copy.copy, something like the following would > also work, but with a bit more local complexity: > > def sum(items, start=0): > count = len(items) That's another semantic change: sum() currently accepts iterators (although not *infinite* iterators). This revision has now broken tens of thousands of applications. -- Steven From antoine at python.org Wed Aug 13 20:03:17 2014 From: antoine at python.org (Antoine Pitrou) Date: Wed, 13 Aug 2014 14:03:17 -0400 Subject: [Python-ideas] sum(...) limitation In-Reply-To: <20140813163729.GI4525@ando> References: <53E7CBA4.40105@g.nevcal.com> <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp> <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> Message-ID: Le 13/08/2014 12:37, Steven D'Aprano a ?crit : > > sum() is *not* the One Obvious Way to add arbitrary objects in every > domain. Well, it is. It's a builtin and it's called "sum", which makes it pretty clear it adds objects together. That could hardly be any more obvious, actually: all competitors are *less* obvious. Indeed it cannot *practically* be the most efficient or accurate to sum arbitrary objects, unless we manage to devise a clever protocol (__sum__? how would that work?) that can delegate to arbitrary third-party code. But people are bound to think, legitimately, that built-in sum() is the logical answer when wanting to sum a sequence of objects. (of course, whether or not "sum" is a reasonable notion when applied to strings - rather than lists or floats - is a separate question) Regards Antoine. From ckaynor at zindagigames.com Wed Aug 13 20:05:52 2014 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Wed, 13 Aug 2014 11:05:52 -0700 Subject: [Python-ideas] sum(...) limitation In-Reply-To: <20140813174630.GM4525@ando> References: <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> <20140813174630.GM4525@ando> Message-ID: On Wed, Aug 13, 2014 at 10:46 AM, Steven D'Aprano wrote: > That's another semantic change: sum() currently accepts iterators > (although not *infinite* iterators). This revision has now broken tens > of thousands of applications. > okay, how about: def sum(items, start=0): first = True for item in items: if first: start = start + item else: start += item return start I believe that has the same effect as my other two, but only adds the requirement that "value += item" behaves the same as "value = value + item". Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Aug 13 20:29:29 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 13 Aug 2014 14:29:29 -0400 Subject: [Python-ideas] sum(...) limitation In-Reply-To: References: <53E7CBA4.40105@g.nevcal.com> <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp> <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> Message-ID: On Wed, Aug 13, 2014 at 2:03 PM, Antoine Pitrou wrote: > Le 13/08/2014 12:37, Steven D'Aprano a ?crit : > > >> sum() is *not* the One Obvious Way to add arbitrary objects in every >> domain. >> > > Well, it is. It's a builtin and it's called "sum", which makes it pretty > clear it adds objects together. That could hardly be any more obvious, > actually: all competitors are *less* obvious. > When it comes to "adding" sequences or containers, "concat" would be a strong competitor. After all, this is exactly what + does on sequences under the hood. [1] While there are technical reasons for reusing the same infix operator for adding and concatenation, we are not so restricted when it comes to builtins. [1] https://docs.python.org/2/c-api/sequence.html#PySequence_Concat -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed Aug 13 21:04:25 2014 From: mertz at gnosis.cx (David Mertz) Date: Wed, 13 Aug 2014 12:04:25 -0700 Subject: [Python-ideas] sum(...) limitation In-Reply-To: References: <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> <20140813174630.GM4525@ando> Message-ID: This isn't the first time around in this discussion of (abusing) 'sum()'. Actually, I discussed the last round in my keynote at PyCon UK in 2013. In particular, I actually discussed there already how Chris' latest version of 'sum()' breaks *existing* and *widespread* code. My slides are at: http://gnosis.cx/pycon-uk-2013/Keynote-Ideas.pdf Look at slides 24-26. The TL;DR version: numpy arrays give different meanings to __add__() and __iadd__(). Chris' version breaks every program ever written that uses numpy. On Wed, Aug 13, 2014 at 11:05 AM, Chris Kaynor wrote: > > On Wed, Aug 13, 2014 at 10:46 AM, Steven D'Aprano > wrote: > >> That's another semantic change: sum() currently accepts iterators >> (although not *infinite* iterators). This revision has now broken tens >> of thousands of applications. >> > > okay, how about: > > def sum(items, start=0): > first = True > for item in items: > if first: > start = start + item > else: > start += item > return start > > I believe that has the same effect as my other two, but only adds the > requirement that "value += item" behaves the same as "value = value + item". > > Chris > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Aug 13 21:12:50 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 13 Aug 2014 12:12:50 -0700 Subject: [Python-ideas] sum(...) limitation In-Reply-To: References: <53E7CBA4.40105@g.nevcal.com> <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp> <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> Message-ID: <7FDFE116-9CDB-4EA5-842E-4356BF8DF217@yahoo.com> On Aug 13, 2014, at 11:03, Antoine Pitrou wrote: > Well, it is. It's a builtin and it's called "sum", which makes it pretty clear it adds objects together. That could hardly be any more obvious, actually: all competitors are *less* obvious. Have you ever heard of anyone wanting to "sum" a bunch of pieces of text? Or a bunch of lists? Actually, now that I think about it, although I was being somewhat flippant, I've heard both of those things plenty of times, and never once has it meant concatenation. It always means literal or figurative element-wise summing, or some other kind of aggregation. Which makes it even _less_ obvious as a one true way to concatenate a bunch of things. Also, how often do you really want to concatenate a bunch of lists into a bigger list, as opposed to just chaining them together into a new iterable? It's not _never_, but I think it's less common. So if itertools.chain.from_iterable doesn't deserve to be a builtin (or even a top-level module function), is concat, a modified sum, or some other way to spell the same idea really necessary? From marky1991 at gmail.com Wed Aug 13 21:28:02 2014 From: marky1991 at gmail.com (Mark Young) Date: Wed, 13 Aug 2014 15:28:02 -0400 Subject: [Python-ideas] sum(...) limitation In-Reply-To: <7FDFE116-9CDB-4EA5-842E-4356BF8DF217@yahoo.com> References: <53E7CBA4.40105@g.nevcal.com> <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp> <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> <7FDFE116-9CDB-4EA5-842E-4356BF8DF217@yahoo.com> Message-ID: Have you ever heard someone wanting to "add" a bunch of pieces of text? Outside of programming, of course not. In our domain, adding strings makes perfect sense (it's concatenation), thus the idea of summing strings makes perfect sense, given that summation is defined as repeated addition. The same argument applies for lists. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Aug 13 21:44:21 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 12:44:21 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations Message-ID: [There is no TL;DR other than the subject line. Please read the whole thing before replying. I do have an appendix with some motivations for adding type annotations at the end.] Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can learn from Haskell (and other languages); yesterday he gave the same talk at Dropbox. The talk is online ( https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad strokes comes down to three suggestions: (a) Python should adopt mypy's syntax for function annotations (b) Python's use of mutabe containers by default is wrong (c) Python should adopt some kind of Abstract Data Types Proposals (b) and (c) don't feel particularly actionable (if you disagree please start a new thread, I'd be happy to discuss these further if there's interest) but proposal (a) feels right to me. So what is mypy? It is a static type checker for Python written by Jukka for his Ph.D. thesis. The basic idea is that you add type annotations to your program using some custom syntax, and when running your program using the mypy interpreter, type errors will be found during compilation (i.e., before the program starts running). The clever thing here is that the custom syntax is actually valid Python 3, using (mostly) function annotations: your annotated program will still run with the regular Python 3 interpreter. In the latter case there will be no type checking, and no runtime overhead, except to evaluate the function annotations (which are evaluated at function definition time but don't have any effect when the function is called). In fact, it is probably more useful to think of mypy as a heavy-duty linter than as a compiler or interpreter; leave the type checking to mypy, and the execution to Python. It is easy to integrate mypy into a continuous integration setup, for example. To read up on mypy's annotation syntax, please see the mypy-lang.org website. Here's just one complete example, to give a flavor: from typing import List, Dict def word_count(input: List[str]) -> Dict[str, int]: result = {} #type: Dict[str, int] for line in input: for word in line.split(): result[word] = result.get(word, 0) + 1 return result Note that the #type: comment is part of the mypy syntax; mypy uses comments to declare types in situations where no syntax is available -- although this particular line could also be written as follows: result = Dict[str, int]() Either way the entire function is syntactically valid Python 3, and a suitable implementation of typing.py (containing class definitions for List and Dict, for example) can be written to make the program run correctly. One is provided as part of the mypy project. I should add that many of mypy's syntactic choices aren't actually new. The basis of many of its ideas go back at least a decade: I blogged about this topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the two followup posts linked from the top there). I'll emphasize once more that mypy's type checking happens in a separate pass: no type checking happens at run time (other than what the interpreter already does, like raising TypeError on expressions like 1+"1"). There's a lot to this proposal, but I think it's possible to get a PEP written, accepted and implemented in time for Python 3.5, if people are supportive. I'll go briefly over some of the action items. *(1) A change of direction for function annotations* PEP 3107 , which introduced function annotations, is intentional non-committal about how function annotations should be used. It lists a number of use cases, including but not limited to type checking. It also mentions some rejected proposals that would have standardized either a syntax for indicating types and/or a way for multiple frameworks to attach different annotations to the same function. AFAIK in practice there is little use of function annotations in mainstream code, and I propose a conscious change of course here by stating that annotations should be used to indicate types and to propose a standard notation for them. (We may have to have some backwards compatibility provision to avoid breaking code that currently uses annotations for some other purpose. Fortunately the only issue, at least initially, will be that when running mypy to type check such code it will produce complaints about the annotations; it will not affect how such code is executed by the Python interpreter. Nevertheless, it would be good to deprecate such alternative uses of annotations.) *(2) A specification for what to add to Python 3.5* There needs to be at least a rough consensus on the syntax for annotations, and the syntax must cover a large enough set of use cases to be useful. Mypy is still under development, and some of its features are still evolving (e.g. unions were only added a few weeks ago). It would be possible to argue endlessly about details of the notation, e.g. whether to use 'list' or 'List', what either of those means (is a duck-typed list-like type acceptable?) or how to declare and use type variables, and what to do with functions that have no annotations at all (mypy currently skips those completely). I am proposing that we adopt whatever mypy uses here, keeping discussion of the details (mostly) out of the PEP. The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter. The actual type checker will not be integrated with the Python interpreter, and it will not be checked into the CPython repository. The only thing that needs to be added to the stdlib is a copy of mypy's typing.py module. This module defines several dozen new classes (and a few decorators and other helpers) that can be used in expressing argument types. If you want to type-check your code you have to download and install mypy and run it separately. The curious thing here is that while standardizing a syntax for type annotations, we technically still won't be adopting standard rules for type checking. This is intentional. First of all, fully specifying all the type checking rules would make for a really long and boring PEP (a much better specification would probably be the mypy source code). Second, I think it's fine if the type checking algorithm evolves over time, or if variations emerge. The worst that can happen is that you consider your code correct but mypy disagrees; your code will still run. That said, I don't want to *completely* leave out any specification. I want the contents of the typing.py module to be specified in the PEP, so that it can be used with confidence. But whether mypy will complain about your particular form of duck typing doesn't have to be specified by the PEP. Perhaps as mypy evolves it will take options to tell it how to handle certain edge cases. Forks of mypy (or entirely different implementations of type checking based on the same annotation syntax) are also a possibility. Maybe in the distant future a version of Python will take a different stance, once we have more experience with how this works out in practice, but for Python 3.5 I want to restrict the scope of the upheaval. *Appendix -- Why Add Type Annotations?* The argument between proponents of static typing and dynamic typing has been going on for many decades. Neither side is all wrong or all right. Python has traditionally fallen in the camp of extremely dynamic typing, and this has worked well for most users, but there are definitely some areas where adding type annotations would help. - Editors (IDEs) can benefit from type annotations; they can call out obvious mistakes (like misspelled method names or inapplicable operations) and suggest possible method names. Anyone who has used IntelliJ or Xcode will recognize how powerful these features are, and type annotations will make such features more useful when editing Python source code. - Linters are an important tool for teams developing software. A linter doesn't replace a unittest, but can find certain types of errors better or quicker. The kind of type checking offered by mypy works much like a linter, and has similar benefits; but it can find problems that are beyond the capabilities of most linters. - Type annotations are useful for the human reader as well! Take the above word_count() example. How long would it have taken you to figure out the types of the argument and return value without annotations? Currently most people put the types in their docstrings; developing a standard notation for type annotations will reduce the amount of documentation that needs to be written, and running the type checker might find bugs in the documentation, too. Once a standard type annotation syntax is introduced, it should be simple to add support for this notation to documentation generators like Sphinx. - Refactoring. Bob's talk has a convincing example of how type annotations help in (manually) refactoring code. I also expect that certain automatic refactorings will benefit from type annotations -- imagine a tool like 2to3 (but used for some other transformation) augmented by type annotations, so it will know whether e.g. x.keys() is referring to the keys of a dictionary or not. - Optimizers. I believe this is actually the least important application, certainly initially. Optimizers like PyPy or Pyston wouldn't be able to fully trust the type annotations, and they are better off using their current strategy of optimizing code based on the types actually observed at run time. But it's certainly feasible to imagine a future optimizer also taking type annotations into account. -- --Guido "I need a new hobby" van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Aug 13 21:59:54 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 13 Aug 2014 12:59:54 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <53EBC3BA.10808@stoneleaf.us> On 08/13/2014 12:44 PM, Guido van Rossum wrote: > > [There is no TL;DR other than the subject line. Please read the whole thing before replying. I do have an appendix with > some motivations for adding type annotations at the end.] +0 on the proposal as a whole. It is not something I'm likely to use, but I'm not opposed to it, so long as it stays optional. > Nevertheless, it would be good to deprecate such alternative uses of annotations. -1 on deprecating alternative uses of annotations. -- ~Ethan~ From raymond.hettinger at gmail.com Wed Aug 13 22:18:18 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 13 Aug 2014 13:18:18 -0700 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: References: Message-ID: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> On Aug 13, 2014, at 12:44 PM, python-ideas-request at python.org wrote: > The goal is to make it possible to add > type checking annotations to 3rd party modules (and even to the stdlib) > while allowing unaltered execution of the program by the (unmodified) > Python 3.5 interpreter. Is the goal to "make it possible" or would it quickly become required (i.e. any time you write normal, readable Python, it would break someone's optimizer, refactorer, linter, etc.?) Are these annotations going to be pushed into every nook and cranny in the standard library and wend their way into the C code? Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Aug 13 22:19:01 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 13:19:01 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EBC3BA.10808@stoneleaf.us> References: <53EBC3BA.10808@stoneleaf.us> Message-ID: On Wed, Aug 13, 2014 at 12:59 PM, Ethan Furman wrote: > On 08/13/2014 12:44 PM, Guido van Rossum wrote: > >> >> [There is no TL;DR other than the subject line. Please read the whole >> thing before replying. I do have an appendix with >> some motivations for adding type annotations at the end.] >> > > +0 on the proposal as a whole. It is not something I'm likely to use, but > I'm not opposed to it, so long as it stays optional. > > > > Nevertheless, it would be good to deprecate such alternative uses of >> annotations. >> > > -1 on deprecating alternative uses of annotations. > Do you have a favorite alternative annotation use that you actually use (or are likely to)? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Wed Aug 13 22:19:48 2014 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 13 Aug 2014 15:19:48 -0500 Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...) limitation) In-Reply-To: References: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp> <53E93147.4050504@stoneleaf.us> <53E93736.8030105@stoneleaf.us> <53E983F9.1090802@stoneleaf.us> <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp> <53E9BF39.6060601@biologie.uni-freiburg.de> <87ha1hhum6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 08/13/2014 12:50 AM, Nick Coghlan wrote: > On 13 August 2014 14:38, Stephen J. Turnbull wrote: >> >Wolfgang Maier writes: >> > >> > > Exactly. So my point was that when you don't subclass str, but instead >> > > use a wrapper around it, you can give it a as str-like interface as you >> > > want so the thing looks and feels like a string to users, it will still >> > > not work as part of an iterable passed to .join >> > >> >You mean this behavior? >> > >> >wideload:~ 12:42$ python3.2 >>>>> >>>> >> >... >>>>> >>>>class N: >> >... def __init__(self, s=''): >> >... self.s = s >> >... def __str__(self): >> >... return self.s >> >... >>>>> >>>>" ".join(['a', N('b')]) >> >Traceback (most recent call last): >> > File "", line 1, in >> >TypeError: sequence item 1: expected str instance, N found >>>>> >>>>' '.join(str(x) for x in ['a', N('b')]) >> >'a b' >>>>> >>>> >> > >> >Given the fact that every object is str-able, I don't think we want to >> >give "str(x) for x in" semantics to str.join. So I think the answer >> >is "if you want Nasty to automatically acquire all the behaviors of >> >str, make it a subclass of str". > Note that this is a general problem - it is quite common to use > explicit type checks against str rather than relying on ducktyping. In > theory, a suitable ABC could be defined (using collections.UserString > as a starting point), but nobody has ever found it a pressing enough > problem to take the time to do so - it's generally easier to just > inherit from str. Is there a way to select a method more specifically on it's mixin? thing.method # any like named method thing.method|mixin # Only if it's from mixin Where method is spelled the same on differnt types, but it's actual operation may be different. Obviously that spelling won't work, but the idea is to allow a more fine grained method selection. thing.__add__|number(other) thing.__add__|sequence(other) thing.__add__|container(other) thing.__add__|str(other) So if I wanted to join strings but not containers or numbers, I could use the __add__|str method. And conversely if I wanted to iterate nested containers without iterating strings too, I could use __iter__|container method. Now that I think about it a bit more, it probably would be spelled.. mixin.method(thing, other) But maybe for the same reason we don't normally call a class method directly applies? class.method(thing, other) I think in most cases, the difference might be getting an attribute error early, vs a type error a bit later. But it seems to me there may be other differences/gotcha's in the case of calling a mixin or class method directly. Currently I'd add a type check before calling the method, but I'd like the finer grained method resolution over the type check. Cheers, Ron From alex.gaynor at gmail.com Wed Aug 13 22:29:38 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 13 Aug 2014 20:29:38 +0000 (UTC) Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: Message-ID: I'm strongly opposed this, for a few reasons. First, I think that standardizing on a syntax, without a semantics is incredibly confusing, and I can't imagine how having *multiple* competing implementations would be a boon for anyone. This proposal seems to be built around the idea that we should have a syntax, and then people can write third party tools, but Python itself won't really do anything with them. Fundamentally, this seems like a very confusing approach. How we write a type, and what we do with that information are fundamentally connected. Can I cast a ``List[str]`` to a ``List[object]`` in any way? If yes, what happens when I go to put an ``int`` in it? There's no runtime checking, so the type system is unsound, on the other hand, disallowing this prevents many types of successes. Both solutions have merit, but the idea of some implementations of the type checker having covariance and some contravariance is fairly disturbing. Another concern I have is that analysis based on these types is making some pretty strong assumptions about static-ness of Python programs that aren't valid. While existing checkers like ``flake8`` also do this, their assumptions are basically constrained to the symbol table, while this is far deeper. For example, can I annotate somethign as ``six.text_type``? What about ``django.db.models.sql.Query`` (keep in mind that this class is redefined based on what database you're using (not actually true, but it used to be))? Python's type system isn't very good. It lacks many features of more powerful systems such as algebraic data types, interfaces, and parametric polymorphism. Despite this, it works pretty well because of Python's dynamic typing. I strongly believe that attempting to enforce the existing type system would be a real shame. Alex PS: You're right. None of this would provide *any* value for PyPy. From christian at python.org Wed Aug 13 22:29:48 2014 From: christian at python.org (Christian Heimes) Date: Wed, 13 Aug 2014 22:29:48 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <53EBCABC.7010801@python.org> On 13.08.2014 21:44, Guido van Rossum wrote: > Yesterday afternoon I had an inspiring conversation with Bob Ippolito > (man of many trades, author of simplejson) and Jukka Lehtosalo (author > of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about > what Python can learn from Haskell (and other languages); yesterday he > gave the same talk at Dropbox. The talk is online > (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad > strokes comes down to three suggestions: > > (a) Python should adopt mypy's syntax for function annotations > (b) Python's use of mutabe containers by default is wrong > (c) Python should adopt some kind of Abstract Data Types I was at Bob's talk during EP14 and really liked the idea. A couple of colleagues and other attendees also said it's a good and useful proposal. I also like your proposal to standardize the type annotations first without a full integration of mypy. In general I'm +1 but I like to discuss two aspects: 1) I'm not keen with the naming of mypy's typing classes. The visual distinction between e.g. dict() and Dict() is too small and IMHO confusing for newcomers. How about an additional 'T' prefix to make clear that the objects are referring to typing objects? from typing import TList, TDict def word_count(input: TList[str]) -> TDict[str, int]: ... 2) PEP 3107 only specifies arguments and return values but not exceptions that can be raised by a function. Java has the "throws" syntax to list possible exceptions: public void readFile() throws IOException {} May I suggest that we also standardize a way to annotate the exceptions that can be raised by a function? It's a very useful piece of information and commonly requested information on the Python user mailing list. It doesn't have to be a new syntax element, a decorator in the typing module would suffice, too. For example: from typing import TList, TDict, raises @raises(RuntimeError, (ValueError, "is raised when input is empty")) def word_count(input: TList[str]) -> TDict[str, int]: ... Regards, Christian From ethan at stoneleaf.us Wed Aug 13 22:50:06 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 13 Aug 2014 13:50:06 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53EBC3BA.10808@stoneleaf.us> Message-ID: <53EBCF7E.2050909@stoneleaf.us> On 08/13/2014 01:19 PM, Guido van Rossum wrote: > On Wed, Aug 13, 2014 at 12:59 PM, Ethan Furman wrote: >> >> -1 on deprecating alternative uses of annotations. > > Do you have a favorite alternative annotation use that you actually use (or are likely to)? My script argument parser [1] uses annotations to figure out how to parse the cli parameters and cast them to appropriate values (copied the idea from one of Michele Simionato's projects... plac [2], I believe). I could store the info in some other structure besides 'annotations', but it's there and it fits the bill conceptually. Amusingly, it's a form of type info, but instead of saying what it has to already be, says what it will become. -- ~Ethan~ [1] https://pypi.python.org/pypi/scription (due for an overhaul now I've used it for awhile ;) [2] https://pypi.python.org/pypi/plac/0.9.1 From donald at stufft.io Wed Aug 13 22:53:35 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 13 Aug 2014 16:53:35 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: > On Aug 13, 2014, at 4:29 PM, Alex Gaynor wrote: > > I'm strongly opposed this, for a few reasons. > > First, I think that standardizing on a syntax, without a semantics is > incredibly confusing, and I can't imagine how having *multiple* competing > implementations would be a boon for anyone. > > This proposal seems to be built around the idea that we should have a syntax, > and then people can write third party tools, but Python itself won't really do > anything with them. > > Fundamentally, this seems like a very confusing approach. How we write a type, > and what we do with that information are fundamentally connected. Can I cast a > ``List[str]`` to a ``List[object]`` in any way? If yes, what happens when I go > to put an ``int`` in it? There's no runtime checking, so the type system is > unsound, on the other hand, disallowing this prevents many types of successes. > > Both solutions have merit, but the idea of some implementations of the type > checker having covariance and some contravariance is fairly disturbing. > > Another concern I have is that analysis based on these types is making some > pretty strong assumptions about static-ness of Python programs that aren't > valid. While existing checkers like ``flake8`` also do this, their assumptions > are basically constrained to the symbol table, while this is far deeper. For > example, can I annotate somethign as ``six.text_type``? What about > ``django.db.models.sql.Query`` (keep in mind that this class is redefined based > on what database you're using (not actually true, but it used to be))? > > Python's type system isn't very good. It lacks many features of more powerful > systems such as algebraic data types, interfaces, and parametric polymorphism. > Despite this, it works pretty well because of Python's dynamic typing. I > strongly believe that attempting to enforce the existing type system would be a > real shame. > > Alex > > PS: You're right. None of this would provide *any* value for PyPy. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ I agree with Alex that I think leaving the actual semantics of what these things mean up to a third party, which can possibly be swapped out by individual end users, is terribly confusing. I don?t think I agree though that this is a bad idea in general, I think that we should just add it for real and skip the indirection. IOW I'm not sure I see the benefit of defining the syntax but not the semantics when it seems this is already completely possible given the fact that mypy exists. The only real benefits I can see from doing it are that the stdlib can use it, and the ``import typing`` aspect. I don't believe that the stdlib benefits are great enough to get the possible confusion of multiple different implementations and I think that the typing import could easily be provided as a project on PyPI that people can depend on if they want to use this in their code. So my vote would be to add mypy semantics to the language itself. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrey.vlasovskikh at gmail.com Wed Aug 13 23:08:33 2014 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Thu, 14 Aug 2014 01:08:33 +0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53EBC3BA.10808@stoneleaf.us> Message-ID: 2014-08-14, 0:19, Guido van Rossum wrote: > Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can learn from Haskell (and other languages); yesterday he gave the same talk at Dropbox. The talk is online (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad strokes comes down to three suggestions: > > (a) Python should adopt mypy's syntax for function annotations +1. I'm a developer of the code analysis engine of PyCharm. I have discussed this idea with Jukka Lehtosalo and recently with Dave Halter, the author of Jedi code completion library. Standardized type annotations would be very useful for code analysis tools and IDEs such as PyCharm, Jedi and pylint. Type annotations would be especially great for third-party libraries. The idea is that most Python programmers don't have to write annotations in order to benefit from them. Annotated libraries are often enough for good code analysis. We (PyCharm) and Jukka have made some initial steps in this direction, including thoughts on semantics of annotations (https://github.com/pytypes/pytypes). Feedback is welcome. Here are slides from my talk about optional typing in Python, that show how Mypy types can be used in both static and dynamic type checking (http://blog.pirx.ru/media/files/2013/python-optional-typing/), Mypy-related part starts from slide 14. We are interested in getting type annotations standardized and we would like to help developing and testing type annotations proposals. -- Andrey Vlasovskikh Web: http://pirx.ru/ From guido at python.org Wed Aug 13 23:09:55 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 14:09:55 -0700 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> Message-ID: On Wed, Aug 13, 2014 at 1:18 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > > On Aug 13, 2014, at 12:44 PM, python-ideas-request at python.org wrote: > > The goal is to make it possible to add > type checking annotations to 3rd party modules (and even to the stdlib) > while allowing unaltered execution of the program by the (unmodified) > Python 3.5 interpreter. > > > Is the goal to "make it possible" or would it quickly become required > (i.e. any time you write normal, readable Python, it would break > someone's optimizer, refactorer, linter, etc.?) > Whoa, whoa. That's not at all the idea. Currently *nobody* uses type annotations because there's no standard notation. My goal is to enable their use by proposing a standard, nothing more. > Are these annotations going to be pushed into every nook and cranny > in the standard library and wend their way into the C code? > There's no need for that. Mypy lets you create stub modules that shadow stdlib (and other) modules during typechecking, and it comes with a modest set of standard stubs for some of the most popular stdlib modules. In most cases it's actually much more effective to create a stub than to add the annotations to the stdlib source code -- the stdlib code itself often is difficult to type check due to various optimizations or backwards compatibility concerns, but writing stubs is relatively straightforward, and the stubs will give useful guidance to users of the stdlib who care to run mypy over their own code. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Wed Aug 13 23:12:26 2014 From: antoine at python.org (Antoine Pitrou) Date: Wed, 13 Aug 2014 17:12:26 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: Hello, First, as a disclaimer, I am currently working on Numba for Continuum Analytics. Numba has its own type inference system which it applies to functions decorated with the @jit decorator. Due to Numba's objectives, the type inference system is heavily geared towards numerical computing, but it is conceptually (and a bit concretely) able to represent more generic information, such as "an enumerate() over an iterator of a complex128 numpy array". There are two sides to type inference: 1) first the (optional) annotations (I'm saying "optional" because in the most basic usage, a JIT compiler is normally able to defer compilation until the first function or method call, and to deduce input types from that) 2) second the inference engine properly, which walks the code (in whatever form the tool's developer has chosen: bytecode, AST, IR) and deduces types for any intermediate values Now only #1 is implied by this PEP proposal, but it also sounds like we should take into account the desired properties of #2 (for example, being able to express "an iterator of three-tuples" can be important for a JIT compiler - or not, perhaps, depending on the JIT compiler :-)). What #2 wants to do will differ depending on the use case: e.g. a code checker may need less type granularity than a JIT compiler. Therefore, regardless of mypy's typesystem's completeness and granularity, one requirement is for it to be easily extensible. By extensible I mean not only being able to define new type descriptions, but being able to do so for existing third-party libraries you don't want to modify. I'm saying that because I'm looking at http://mypy-lang.org/tutorial.html#genericclasses , and it's not clear from this example whether the typing code has to be interwoven with the collection's implementation, or can be written as a separate code module entirely (*). Ideally both should probably be possible (in the same vein as being able to subclass an ABC, or register an existing class with it). This also includes being to type-declare functions and types from C extension modules. In Numba, this would be typically required to write typing descriptions for Numpy arrays and functions; but also to derive descriptions for fixed-width integers, single-precision floats, etc. (this also means some form of subclassing for type descriptions themselves). (*) (actually, I'm a bit worried when I see that "List[int]()" instantiates an actual list; calling a type description class should give you a parametered type description, not an object; the [] notation is in general not powerful enough if you want several type parameters, possibly keyword-only) At some point, it will be even better if the typing system is powerful enough to remember properties of the *values* (for example not only "a string", but "a one-character string, or even "one of the 'Y', 'M', 'D' strings"). Think about type-checking / type-infering calls to the struct module. I may come back with more comments once I've read the mypy docs and/or code in detail. Regards Antoine. From guido at python.org Wed Aug 13 23:46:08 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 14:46:08 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 1:29 PM, Alex Gaynor wrote: > I'm strongly opposed this, for a few reasons. > > First, I think that standardizing on a syntax, without a semantics is > incredibly confusing, and I can't imagine how having *multiple* competing > implementations would be a boon for anyone. > That part was probably overly vague in my original message. I actually do want to standardize on semantics, but I think the semantics will prove controversial (they already have :-) and I think it's better to standardize the syntax and *some* semantics first rather than having to wait another decade for the debate over the semantics to settle. I mostly want to leave the door open for mypy to become smarter. But it might make sense to have a "weaker" interpretation in some cases too (e.g. an IDE might use a weaker type system in order to avoid overwhelming users with warnings). > This proposal seems to be built around the idea that we should have a > syntax, > and then people can write third party tools, but Python itself won't > really do > anything with them. > Right. > Fundamentally, this seems like a very confusing approach. How we write a > type, > and what we do with that information are fundamentally connected. Can I > cast a > ``List[str]`` to a ``List[object]`` in any way? If yes, what happens when > I go > to put an ``int`` in it? There's no runtime checking, so the type system is > unsound, on the other hand, disallowing this prevents many types of > successes. > Mypy has a cast() operator that you can use to shut it up when you (think you) know the conversion is safe. > Both solutions have merit, but the idea of some implementations of the type > checker having covariance and some contravariance is fairly disturbing. > Yeah, that wouldn't be good. ;-) > Another concern I have is that analysis based on these types is making some > pretty strong assumptions about static-ness of Python programs that aren't > valid. While existing checkers like ``flake8`` also do this, their > assumptions > are basically constrained to the symbol table, while this is far deeper. > For > example, can I annotate something as ``six.text_type``? What about > ``django.db.models.sql.Query`` (keep in mind that this class is redefined > based > on what database you're using (not actually true, but it used to be))? > Time will have to tell. Stubs can help. I encourage you to try annotating a medium-sized module. It's likely that you'll find a few things: maybe a bug in mypy, maybe a missing mypy feature, maybe a bug in your code, maybe a shady coding practice in your code or a poorly documented function (I know I found several of each during my own experiments so far). > Python's type system isn't very good. It lacks many features of more > powerful > systems such as algebraic data types, interfaces, and parametric > polymorphism. > Despite this, it works pretty well because of Python's dynamic typing. I > strongly believe that attempting to enforce the existing type system would > be a > real shame. > Mypy shines in those areas of Python programs that are mostly statically typed. There are many such areas in most large systems. There are usually also some areas where mypy's type system is inadequate. It's easy to shut it up for those cases (in fact, mypy is silent unless you use at least one annotation for a function). But that's the case with most type systems. Even Haskell sometimes calls out to C. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 00:00:20 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 15:00:20 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EBCF7E.2050909@stoneleaf.us> References: <53EBC3BA.10808@stoneleaf.us> <53EBCF7E.2050909@stoneleaf.us> Message-ID: On Wed, Aug 13, 2014 at 1:50 PM, Ethan Furman wrote: > On 08/13/2014 01:19 PM, Guido van Rossum wrote: > > On Wed, Aug 13, 2014 at 12:59 PM, Ethan Furman wrote: >> >>> >>> -1 on deprecating alternative uses of annotations. >>> >> >> Do you have a favorite alternative annotation use that you actually use >> (or are likely to)? >> > > My script argument parser [1] uses annotations to figure out how to parse > the cli parameters and cast them to appropriate values (copied the idea > from one of Michele Simionato's projects... plac [2], I believe). > > I could store the info in some other structure besides 'annotations', but > it's there and it fits the bill conceptually. Amusingly, it's a form of > type info, but instead of saying what it has to already be, says what it > will become. > I couldn't find any docs for scription (the tarball contains just the source code, not even an example), although I did find some for plac. I expect using type annotations to the source of scription.py might actually make it easier to grok what it does. :-) But really, I'm sure that in Python 3.5, scription and mypy can coexist. If the mypy idea takes off you might eventually be convinced to use a different convention. But you'd get plenty of warning. > [1] https://pypi.python.org/pypi/scription (due for an overhaul now I've > used it for awhile ;) > [2] https://pypi.python.org/pypi/plac/0.9.1 > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 00:05:53 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 15:05:53 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 1:53 PM, Donald Stufft wrote: > I agree with Alex that I think leaving the actual semantics of what these > things > mean up to a third party, which can possibly be swapped out by individual > end > users, is terribly confusing. I don?t think I agree though that this is a > bad > idea in general, I think that we should just add it for real and skip the > indirection. > Yeah, I probably overstated the option of alternative interpretations. I just don't want to have to write a PEP that specifies every little detail of mypy's type checking algorithm, and I don't think anyone would want to have to read such a PEP either. But maybe we can compromise on something that sketches broad strokes and leaves the details up to the team that maintains mypy (after all that tactic has worked pretty well for Python itself :-). > IOW I'm not sure I see the benefit of defining the syntax but not the > semantics > when it seems this is already completely possible given the fact that mypy > exists. > > The only real benefits I can see from doing it are that the stdlib can use > it, > and the ``import typing`` aspect. I don't believe that the stdlib benefits > are > great enough to get the possible confusion of multiple different > implementations > and I think that the typing import could easily be provided as a project > on PyPI > that people can depend on if they want to use this in their code. > > So my vote would be to add mypy semantics to the language itself. > What exactly would that mean? I don't think the Python interpreter should reject programs that fail the type check -- in fact, separating the type check from run time is the most crucial point of my proposal. I'm fine to have a discussion on things like covariance vs. contravariance, or what form of duck typing are acceptable, etc. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 00:07:19 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 15:07:19 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53EBC3BA.10808@stoneleaf.us> Message-ID: Wow. Awesome. I will make time to study what you have already done! On Wed, Aug 13, 2014 at 2:08 PM, Andrey Vlasovskikh < andrey.vlasovskikh at gmail.com> wrote: > 2014-08-14, 0:19, Guido van Rossum wrote: > > > Yesterday afternoon I had an inspiring conversation with Bob Ippolito > (man of many trades, author of simplejson) and Jukka Lehtosalo (author of > mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about what > Python can learn from Haskell (and other languages); yesterday he gave the > same talk at Dropbox. The talk is online ( > https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad > strokes comes down to three suggestions: > > > > (a) Python should adopt mypy's syntax for function annotations > > > +1. I'm a developer of the code analysis engine of PyCharm. I have > discussed this idea with Jukka Lehtosalo and recently with Dave Halter, the > author of Jedi code completion library. Standardized type annotations would > be very useful for code analysis tools and IDEs such as PyCharm, Jedi and > pylint. Type annotations would be especially great for third-party > libraries. The idea is that most Python programmers don't have to write > annotations in order to benefit from them. Annotated libraries are often > enough for good code analysis. > > We (PyCharm) and Jukka have made some initial steps in this direction, > including thoughts on semantics of annotations ( > https://github.com/pytypes/pytypes). Feedback is welcome. > > Here are slides from my talk about optional typing in Python, that show > how Mypy types can be used in both static and dynamic type checking ( > http://blog.pirx.ru/media/files/2013/python-optional-typing/), > Mypy-related part starts from slide 14. > > We are interested in getting type annotations standardized and we would > like to help developing and testing type annotations proposals. > > -- > Andrey Vlasovskikh > Web: http://pirx.ru/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Thu Aug 14 00:21:40 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Wed, 13 Aug 2014 17:51:40 -0430 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 3:14 PM, Guido van Rossum wrote: > I am proposing that we adopt whatever mypy uses here, keeping discussion > of the details (mostly) out of the PEP. The goal is to make it possible to > add type checking annotations to 3rd party modules (and even to the stdlib) > while allowing unaltered execution of the program by the (unmodified) > Python 3.5 interpreter. I'll comment later on the core subject. For now, I think this deserves some thought: Function annotations are not available in Python 2.7, so promoting widespread use of annotations in 3.5 would be promoting code that is compatible only with 3.x, when the current situation is that much effort is being spent on writing code that works on both 2.7 and 3.4 (most libraries?). Independently of its core merits, this proposal should fail unless annotations are added to Python 2.8. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Aug 14 00:28:15 2014 From: toddrjen at gmail.com (Todd) Date: Thu, 14 Aug 2014 00:28:15 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Aug 13, 2014 9:45 PM, "Guido van Rossum" wrote: > (1) A change of direction for function annotations > > PEP 3107, which introduced function annotations, is intentional non-committal about how function annotations should be used. It lists a number of use cases, including but not limited to type checking. It also mentions some rejected proposals that would have standardized either a syntax for indicating types and/or a way for multiple frameworks to attach different annotations to the same function. AFAIK in practice there is little use of function annotations in mainstream code, and I propose a conscious change of course here by stating that annotations should be used to indicate types and to propose a standard notation for them. > > (We may have to have some backwards compatibility provision to avoid breaking code that currently uses annotations for some other purpose. Fortunately the only issue, at least initially, will be that when running mypy to type check such code it will produce complaints about the annotations; it will not affect how such code is executed by the Python interpreter. Nevertheless, it would be good to deprecate such alternative uses of annotations.) I watched the original talk and read your proposal. I think type annotations could very very useful in certain contexts. However, I still don't get this bit. Why would allowing type annotations automatically imply that no other annotations would be possible? Couldn't we formalize what would be considered a type annotation while still allowing annotations that don't fit this criteria to be used for other things? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ceronman at gmail.com Thu Aug 14 00:27:54 2014 From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=) Date: Thu, 14 Aug 2014 00:27:54 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 9:44 PM, Guido van Rossum wrote: > [There is no TL;DR other than the subject line. Please read the whole > thing before replying. I do have an appendix with some motivations for > adding type annotations at the end.] > This is a very interesting idea. I played a bit with function annotations ( https://github.com/ceronman/typeannotations) and I gave a talk about them at EuroPython 2013. Certainly static type analysis is probably the best use case. The curious thing here is that while standardizing a syntax for type > annotations, we technically still won't be adopting standard rules for type > checking. This is intentional. First of all, fully specifying all the type > checking rules would make for a really long and boring PEP (a much better > specification would probably be the mypy source code). Second, I think it's > fine if the type checking algorithm evolves over time, or if variations > emerge. The worst that can happen is that you consider your code correct > but mypy disagrees; your code will still run. > > That said, I don't want to *completely* leave out any specification. I > want the contents of the typing.py module to be specified in the PEP, so > that it can be used with confidence. But whether mypy will complain about > your particular form of duck typing doesn't have to be specified by the > PEP. Perhaps as mypy evolves it will take options to tell it how to handle > certain edge cases. Forks of mypy (or entirely different implementations of > type checking based on the same annotation syntax) are also a possibility. > Maybe in the distant future a version of Python will take a different > stance, once we have more experience with how this works out in practice, > but for Python 3.5 I want to restrict the scope of the upheaval. > The type checking algorithm might evolve over the time, but by including typing.py in the stdlib, the syntax for annotations would be almost frozen and that will be a limitation. In other projects such as TypeScript ( http://www.typescriptlang.org/), that the syntax usually evolves alongside the algorithms. Is the syntax specifyed in typing.py mature enough to put it in the stdlib and expect users to start annotating their projects without worrying too much about future changes? Is there enough feedback from users using mypy in their projects? I think that rushing typing.py into 3.5 is not a good idea. However, It'd be nice to add some notes in PEP8, encourage it's use as an external library, let some projects and tools (e.g. PyCharm) use it. It's not that bad if mypy lives 100% outside the Python distribution for a while. Just like TypeScript to JavaScript. After getting some user base, part of it (typing.py) could be moved to the stdlib. Manuel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu Aug 14 00:34:07 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 13 Aug 2014 17:34:07 -0500 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EBCABC.7010801@python.org> References: <53EBCABC.7010801@python.org> Message-ID: On Wed, Aug 13, 2014 at 3:29 PM, Christian Heimes wrote: > On 13.08.2014 21:44, Guido van Rossum wrote: > > Yesterday afternoon I had an inspiring conversation with Bob Ippolito > > (man of many trades, author of simplejson) and Jukka Lehtosalo (author > > of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about > > what Python can learn from Haskell (and other languages); yesterday he > > gave the same talk at Dropbox. The talk is online > > (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad > > strokes comes down to three suggestions: > > > > (a) Python should adopt mypy's syntax for function annotations > > (b) Python's use of mutabe containers by default is wrong > > (c) Python should adopt some kind of Abstract Data Types > > I was at Bob's talk during EP14 and really liked the idea. A couple of > colleagues and other attendees also said it's a good and useful > proposal. I also like your proposal to standardize the type annotations > first without a full integration of mypy. > > In general I'm +1 but I like to discuss two aspects: > > 1) I'm not keen with the naming of mypy's typing classes. The visual > distinction between e.g. dict() and Dict() is too small and IMHO > confusing for newcomers. How about an additional 'T' prefix to make > clear that the objects are referring to typing objects? > > from typing import TList, TDict > > def word_count(input: TList[str]) -> TDict[str, int]: > ... > Eeewwwww. That's way too Pascal-ish. > 2) PEP 3107 only specifies arguments and return values but not > exceptions that can be raised by a function. Java has the "throws" > syntax to list possible exceptions: > > public void readFile() throws IOException {} > > May I suggest that we also standardize a way to annotate the exceptions > that can be raised by a function? It's a very useful piece of > information and commonly requested information on the Python user > mailing list. It doesn't have to be a new syntax element, a decorator in > the typing module would suffice, too. For example: > > from typing import TList, TDict, raises > > @raises(RuntimeError, (ValueError, "is raised when input is empty")) > def word_count(input: TList[str]) -> TDict[str, int]: > ... > That was a disaster in C++. It's confusing, especially since Python uses exceptions more than most other languages do. > > Regards, > Christian > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Aug 14 00:44:18 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 13 Aug 2014 18:44:18 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: > On Aug 13, 2014, at 6:05 PM, Guido van Rossum wrote: > > On Wed, Aug 13, 2014 at 1:53 PM, Donald Stufft > wrote: > I agree with Alex that I think leaving the actual semantics of what these things > mean up to a third party, which can possibly be swapped out by individual end > users, is terribly confusing. I don?t think I agree though that this is a bad > idea in general, I think that we should just add it for real and skip the > indirection. > > Yeah, I probably overstated the option of alternative interpretations. I just don't want to have to write a PEP that specifies every little detail of mypy's type checking algorithm, and I don't think anyone would want to have to read such a PEP either. But maybe we can compromise on something that sketches broad strokes and leaves the details up to the team that maintains mypy (after all that tactic has worked pretty well for Python itself :-). > > IOW I'm not sure I see the benefit of defining the syntax but not the semantics > when it seems this is already completely possible given the fact that mypy > exists. > > The only real benefits I can see from doing it are that the stdlib can use it, > and the ``import typing`` aspect. I don't believe that the stdlib benefits are > great enough to get the possible confusion of multiple different implementations > and I think that the typing import could easily be provided as a project on PyPI > that people can depend on if they want to use this in their code. > > So my vote would be to add mypy semantics to the language itself. > > What exactly would that mean? I don't think the Python interpreter should reject programs that fail the type check -- in fact, separating the type check from run time is the most crucial point of my proposal. I don?t know exactly :) Some ideas: 1) Raise a warning when the type check fails, but allow it happen. This would have the benefit of possibly catching bugs, but it's still opt in in the sense that you have to write the annotations for anything to happen. This would also enable people to turn on enforced type checking by raising the warning level to an exception. Even if this was off by default it would make it easy to enable it during test runs and also enable easier/better quickcheck like functionality. 2) Simply add a flag to the interpreter that turns on type checking. 3) Add a stdlib module that would run the program under type checking, like ``python -m typing myprog`` instead of ``python -m myprog``. Really I think a lot of the benefit is likely to come in the form of linting and during test runs. However if I have to run a seperate Python interpreter to actually do the run then I risk getting bad results through varying things like interpreter differences, language level differences, etc. Although I wouldn't complain if it meant that Python had actual type checking at the run time if a function had type annotations :) > > I'm fine to have a discussion on things like covariance vs. contravariance, or what form of duck typing are acceptable, etc. I?m not particularly knowledgable about the actual workings of a type system and covariance vs contravariance and the like. My main concern there is having a single reality. The meaning of something shouldn't change because I used a different interpreter/linter/whatever. Beyond that I don't know enough to have an opinion on the actual semantics. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 01:11:46 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 16:11:46 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 3:21 PM, Juancarlo A?ez wrote: > > > On Wed, Aug 13, 2014 at 3:14 PM, Guido van Rossum > wrote: > >> I am proposing that we adopt whatever mypy uses here, keeping discussion >> of the details (mostly) out of the PEP. The goal is to make it possible to >> add type checking annotations to 3rd party modules (and even to the stdlib) >> while allowing unaltered execution of the program by the (unmodified) >> Python 3.5 interpreter. > > > I'll comment later on the core subject. > > For now, I think this deserves some thought: > > Function annotations are not available in Python 2.7, so promoting > widespread use of annotations in 3.5 would be promoting code that is > compatible only with 3.x, when the current situation is that much effort is > being spent on writing code that works on both 2.7 and 3.4 (most > libraries?). > > Independently of its core merits, this proposal should fail unless > annotations are added to Python 2.8. > Actually, mypy already has a solution. There's a codec ( https://github.com/JukkaL/mypy/tree/master/mypy/codec) that you can use which transforms Python-2-with-annotations into vanilla Python 2. It's not an ideal solution, but it can work in cases where you absolutely have to have state of the art Python 3.5 type checking *and* backwards compatibility with Python 2. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 01:24:43 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 16:24:43 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 3:26 PM, Manuel Cer?n wrote: > The type checking algorithm might evolve over the time, but by including > typing.py in the stdlib, the syntax for annotations would be almost frozen > and that will be a limitation. In other projects such as TypeScript ( > http://www.typescriptlang.org/), that the syntax usually evolves > alongside the algorithms. > What kind of evolution did TypeScript experience? > Is the syntax specifyed in typing.py mature enough to put it in the stdlib > and expect users to start annotating their projects without worrying too > much about future changes? > This is a good question. I do think it is good enough as a starting point for future evolution. Perhaps the biggest question is how fast will the annotation syntax need to evolve? If it needs to evolve significantly faster than Python 3 feature releases come out (every 18 months, approximately) then it may be better to hold off and aim for inclusion in the 3.6 standard library. That would allow more time to reach agreement (though I'm not sure that's a good thing :-), and in the mean time typing.py could be distributed as a 3rd party module on PyPI. > Is there enough feedback from users using mypy in their projects? > > I think that rushing typing.py into 3.5 is not a good idea. However, It'd > be nice to add some notes in PEP8, encourage it's use as an external > library, let some projects and tools (e.g. PyCharm) use it. It's not that > bad if mypy lives 100% outside the Python distribution for a while. Just > like TypeScript to JavaScript. > Well, JavaScript's evolution is tied up forever in a standards body, so TypeScript realistically had no choice in the matter. But are there actually people writing TypeScript? I haven't heard from them yet (people at Dropbox seem to rather like CoffeeScript). Anyway, the situation isn't quite the same -- you wouldn't make any friends in the Python world if you wrote your code in an incompatible dialect that could only be executed after a translation step, but in the JavaScript world that's how all alternative languages work (and they even manage to interoperate). > After getting some user base, part of it (typing.py) could be moved to the > stdlib. > I'm still hopeful that we can get a sufficient user base and agreement on mypy's features for inclusion in 3.5 (extrapolating the 3.4 release schedule by 18 months, 3.5 alpha 1 would go out around February 2015; the feature freeze cut-off date, beta 1, would around May thereafter). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Thu Aug 14 01:27:20 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 14 Aug 2014 09:27:20 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: <53EBCABC.7010801@python.org> Message-ID: <85vbpw6kdj.fsf@benfinney.id.au> Christian Heimes writes: > 1) I'm not keen with the naming of mypy's typing classes. The visual > distinction between e.g. dict() and Dict() is too small and IMHO > confusing for newcomers. How about an additional 'T' prefix to make > clear that the objects are referring to typing objects? To this reader, ?dict? and ?list? *are* ?typing objects? ? they are objects that are types. Seeing code that referred to something else as ?typing objects? would be an infitation to confusion, IMO. You could argue ?that's because you don't know the special meaning of ?typing object? being discussed here?. To which my response would be, for a proposal to add something else as meaningful Python syntax, the jargon is poorly chosen and needlessly confusing with established terms in Python. If there's going to be a distinction between the types (?dict?, ?list?, etc.) and something else, I'd prefer it to be based on a clearer terminology distinction. -- \ ?Simplicity and elegance are unpopular because they require | `\ hard work and discipline to achieve and education to be | _o__) appreciated.? ?Edsger W. Dijkstra | Ben Finney From guido at python.org Thu Aug 14 01:30:10 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 16:30:10 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 3:28 PM, Todd wrote: > However, I still don't get this bit. Why would allowing type annotations > automatically imply that no other annotations would be possible? Couldn't > we formalize what would be considered a type annotation while still > allowing annotations that don't fit this criteria to be used for other > things? > We certainly *could* do that. However, I haven't seen sufficient other uses of annotations. If there is only one use for annotations (going forward), annotations would be unambiguous. If we allow different types of annotations, there would have to be a way to tell whether a particular annotation is intended as a type annotation or not. Currently mypy ignores all modules that don't import typing.py (using any form of import statement), and we could continue this convention. But it would mean that something like this would still require the typing import in order to be checked by mypy: import typing def gcd(int a, int b) -> int: The (necessary) import would be flagged as unused by every linter in the world... :-( -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 01:43:56 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 16:43:56 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 3:44 PM, Donald Stufft wrote: > On Aug 13, 2014, at 6:05 PM, Guido van Rossum wrote: > > On Wed, Aug 13, 2014 at 1:53 PM, Donald Stufft wrote: >> >> >> So my vote would be to add mypy semantics to the language itself. >> > > What exactly would that mean? I don't think the Python interpreter should > reject programs that fail the type check -- in fact, separating the type > check from run time is the most crucial point of my proposal. > > > I don?t know exactly :) > > Some ideas: > > 1) Raise a warning when the type check fails, but allow it happen. This > would > have the benefit of possibly catching bugs, but it's still opt in in the > sense that you have to write the annotations for anything to happen. > This > would also enable people to turn on enforced type checking by raising > the > warning level to an exception. > I don't think that's going to happen. It would require the entire mypy implementation to be checked into the stdlib. It would also require all sorts of hacks in that implementation to deal with dynamic (or just delayed) imports. Mypy currently doesn't handle any of that -- it must be able to find all imported modules before it starts executing even one line of code. > Even if this was off by default it would make it easy to enable it > during > test runs and also enable easier/better quickcheck like functionality. > It would *have* to be off by default -- it's way too slow to be on by default (note that some people are already fretting out today about a 25 msec process start-up time). > 2) Simply add a flag to the interpreter that turns on type checking. > > 3) Add a stdlib module that would run the program under type checking, like > ``python -m typing myprog`` instead of ``python -m myprog``. > > Really I think a lot of the benefit is likely to come in the form of > linting > and during test runs. However if I have to run a separate Python > interpreter > to actually do the run then I risk getting bad results through varying > things > like interpreter differences, language level differences, etc. > Yeah, but I just don't think it's realistic to do anything about that for 3.5 (or 3.6 for that matter). In a decade... Who knows! :-) > Although I wouldn't complain if it meant that Python had actual type > checking > at the run time if a function had type annotations :) > It's probably possibly to write a decorator that translates annotations into assertions that are invoked when a function is called. But in most cases it would be way too slow to turn on everywhere. > I'm fine to have a discussion on things like covariance vs. > contravariance, or what form of duck typing are acceptable, etc. > > I?m not particularly knowledgable about the actual workings of a type > system and > covariance vs contravariance and the like. My main concern there is having > a > single reality. The meaning of something shouldn't change because I used a > different interpreter/linter/whatever. Beyond that I don't know enough to > have > an opinion on the actual semantics. > Yeah, I regret writing it so vaguely already. Having Alex Gaynor open with "I'm strongly opposed [to] this" is a great joy killer. :-) I just really don't want to have to redundantly write up a specification for all the details of mypy's type checking rules in PEP-worthy English. But I'm fine with discussing whether List[str] is a subclass or a superclass of List[object] and how to tell the difference. Still, different linters exist and I don't hear people complain about that. I would also be okay if PyCharm's interpretation of the finer points of the type checking syntax was subtly different from mypy's. In fact I would be surprised if they weren't sometimes in disagreement. Heck, PyPy doesn't give *every* Python program the same meaning as CPython, and that's a feature. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Aug 14 01:58:59 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 13 Aug 2014 19:58:59 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <378F9047-5A37-49C8-A229-55904F8B2240@stufft.io> > On Aug 13, 2014, at 7:43 PM, Guido van Rossum wrote: > > On Wed, Aug 13, 2014 at 3:44 PM, Donald Stufft > wrote: >> On Aug 13, 2014, at 6:05 PM, Guido van Rossum > wrote: >> >> On Wed, Aug 13, 2014 at 1:53 PM, Donald Stufft > wrote: >> >> So my vote would be to add mypy semantics to the language itself. >> >> What exactly would that mean? I don't think the Python interpreter should reject programs that fail the type check -- in fact, separating the type check from run time is the most crucial point of my proposal. > > I don?t know exactly :) > > Some ideas: > > 1) Raise a warning when the type check fails, but allow it happen. This would > have the benefit of possibly catching bugs, but it's still opt in in the > sense that you have to write the annotations for anything to happen. This > would also enable people to turn on enforced type checking by raising the > warning level to an exception. > > I don't think that's going to happen. It would require the entire mypy implementation to be checked into the stdlib. It would also require all sorts of hacks in that implementation to deal with dynamic (or just delayed) imports. Mypy currently doesn't handle any of that -- it must be able to find all imported modules before it starts executing even one line of code. > > Even if this was off by default it would make it easy to enable it during > test runs and also enable easier/better quickcheck like functionality. > > It would *have* to be off by default -- it's way too slow to be on by default (note that some people are already fretting out today about a 25 msec process start-up time). > > 2) Simply add a flag to the interpreter that turns on type checking. > > 3) Add a stdlib module that would run the program under type checking, like > ``python -m typing myprog`` instead of ``python -m myprog``. > > Really I think a lot of the benefit is likely to come in the form of linting > and during test runs. However if I have to run a separate Python interpreter > to actually do the run then I risk getting bad results through varying things > like interpreter differences, language level differences, etc. > > Yeah, but I just don't think it's realistic to do anything about that for 3.5 (or 3.6 for that matter). In a decade... Who knows! :-) > > Although I wouldn't complain if it meant that Python had actual type checking > at the run time if a function had type annotations :) > > It's probably possibly to write a decorator that translates annotations into assertions that are invoked when a function is called. But in most cases it would be way too slow to turn on everywhere. >> I'm fine to have a discussion on things like covariance vs. contravariance, or what form of duck typing are acceptable, etc. > > I?m not particularly knowledgable about the actual workings of a type system and > covariance vs contravariance and the like. My main concern there is having a > single reality. The meaning of something shouldn't change because I used a > different interpreter/linter/whatever. Beyond that I don't know enough to have > an opinion on the actual semantics. > > Yeah, I regret writing it so vaguely already. Having Alex Gaynor open with "I'm strongly opposed [to] this" is a great joy killer. :-) > > I just really don't want to have to redundantly write up a specification for all the details of mypy's type checking rules in PEP-worthy English. But I'm fine with discussing whether List[str] is a subclass or a superclass of List[object] and how to tell the difference. Understood! And really the most important thing I'm worried about isn?t that there is some sort of code in the stdlib or in the interpreter just that there is an authoritative source of what stuff means. > > Still, different linters exist and I don't hear people complain about that. I would also be okay if PyCharm's interpretation of the finer points of the type checking syntax was subtly different from mypy's. In fact I would be surprised if they weren't sometimes in disagreement. Heck, PyPy doesn't give *every* Python program the same meaning as CPython, and that's a feature. :-) > Depends on what is meant by "meaning" I suppose. Generally in those linters or PyPy itself if there is a different *meaningful* result (for instance if print was defaulting to sys.stderr) then CPython (incl docs) acts as the authoritative source of what ``print()`` means (in this case writing to sys.stdout). I'm also generally OK with deferring possible code/interpreter changes to add actual type checking until a later point in time. If there's a defined semantics to what those annotations mean than third parties can experiment and do things with it and those different things can be looked at adding/incorporating into Python proper in 3.6 (or 3.7, or whatever). Honestly I think that probably the things I was worried about is sufficiently allayed given that it appears I was reading more into the vaguness and the optionally different interpretations than what was meant and I don't want to keep harping on it :) As long as there's some single source of what List[str] or what have you means than I'm pretty OK with it all. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Aug 14 02:32:04 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 14 Aug 2014 10:32:04 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Thu, Aug 14, 2014 at 5:44 AM, Guido van Rossum wrote: > from typing import List, Dict > > def word_count(input: List[str]) -> Dict[str, int]: > result = {} #type: Dict[str, int] > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result I strongly support the concept of standardized typing information. There'll be endless bikeshedding on names, though - personally, I don't like the idea of "from typing import ..." as there's already a "types" module and I think it'd be confusing. (Also, "mypy" sounds like someone's toy reimplementation of Python, which it does seem to be :) but that's not really well named for "type checker using stdlib annotations".) But I think the idea is excellent, and it deserves stdlib support. The cast notation sounds to me like it's what Pike calls a "soft cast" - it doesn't actually *change* anything (contrast a C or C++ type cast, where (float)42 is 42.0), it just says to the copmiler/type checker "this thing is actually now this type". If the naming is clear on this point, it leaves open the possibility of actual recursive casting - where casting a List[str] to List[int] is equivalent to [int(x) for x in lst]. Whether or not that's a feature worth adding can be decided in the distant future :) +1 on the broad proposal. +0.5 on defining the notation while leaving the actual type checking to an external program. ChrisA From haoyi.sg at gmail.com Thu Aug 14 02:53:26 2014 From: haoyi.sg at gmail.com (Haoyi Li) Date: Wed, 13 Aug 2014 17:53:26 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: > Both solutions have merit, but the idea of some implementations of the type checker having covariance and some contravariance is fairly disturbing. Why can't we have both? That's the only way to properly type things, since immutable-get-style APIs are always going to be convariant, set-only style APIs (e.g. a function that takes 1 arg and returns None) are going to be contravariant and mutable get-set APIs (like most python collections) should really be invariant. On Wed, Aug 13, 2014 at 5:32 PM, Chris Angelico wrote: > On Thu, Aug 14, 2014 at 5:44 AM, Guido van Rossum > wrote: > > from typing import List, Dict > > > > def word_count(input: List[str]) -> Dict[str, int]: > > result = {} #type: Dict[str, int] > > for line in input: > > for word in line.split(): > > result[word] = result.get(word, 0) + 1 > > return result > > I strongly support the concept of standardized typing information. > There'll be endless bikeshedding on names, though - personally, I > don't like the idea of "from typing import ..." as there's already a > "types" module and I think it'd be confusing. (Also, "mypy" sounds > like someone's toy reimplementation of Python, which it does seem to > be :) but that's not really well named for "type checker using stdlib > annotations".) But I think the idea is excellent, and it deserves > stdlib support. > > The cast notation sounds to me like it's what Pike calls a "soft cast" > - it doesn't actually *change* anything (contrast a C or C++ type > cast, where (float)42 is 42.0), it just says to the copmiler/type > checker "this thing is actually now this type". If the naming is clear > on this point, it leaves open the possibility of actual recursive > casting - where casting a List[str] to List[int] is equivalent to > [int(x) for x in lst]. Whether or not that's a feature worth adding > can be decided in the distant future :) > > +1 on the broad proposal. +0.5 on defining the notation while leaving > the actual type checking to an external program. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Thu Aug 14 02:59:28 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 13 Aug 2014 17:59:28 -0700 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> Message-ID: <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> On Aug 13, 2014, at 2:09 PM, Guido van Rossum wrote: >> The goal is to make it possible to add >> type checking annotations to 3rd party modules (and even to the stdlib) >> while allowing unaltered execution of the program by the (unmodified) >> Python 3.5 interpreter. > > Is the goal to "make it possible" or would it quickly become required > (i.e. any time you write normal, readable Python, it would break > someone's optimizer, refactorer, linter, etc.?) > > Whoa, whoa. That's not at all the idea. Currently *nobody* uses type annotations because there's no standard notation. My goal is to enable their use by proposing a standard, nothing more. > ... > Mypy lets you create stub modules that shadow stdlib (and other) modules during typechecking, and it comes with a modest set of standard stubs for some of the most popular stdlib modules. In most cases it's actually much more effective to create a stub than to add the annotations to the stdlib source code -- the stdlib code itself often is difficult to type check due to various optimizations or backwards compatibility concerns, but writing stubs is relatively straightforward, and the stubs will give useful guidance to users of the stdlib who care to run mypy over their own code. +1 from me :-) In the PEP, please make it clear that you don't want any overly enthusiastic coredevs injecting the annotations into the standard library (have no doubt, some one would do it) and demanding changes to existing APIs where annotations didn't fit neatly (like Larry's proposal to change the long-standing APIs to use his nullable ints by adding None to optional int signatures). Raymond P.S. I would really like for the annotations to grow some way to communicate exceptions as well as return types (i.e. that list.index can raise a ValueError and list.pop can raise an IndexError). This would be only for the exceptions directly added by a function or method, not ones raised by the data (which is something the function can't control). -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Thu Aug 14 03:00:53 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 13 Aug 2014 18:00:53 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <75CA8559-0A4C-4BA0-94AA-A69C0CD56AE7@langa.pl> It?s great to see this finally happening! I did some research on existing optional-typing approaches [1]. What I learned in the process was that linting is the most important use case for optional typing; runtime checks is too little, too late. That being said, having optional runtime checks available *is* also important. Used in staging environments and during unit testing, this case is able to cover cases obscured by meta-programming. Implementations like ?obiwan? and ?pytypedecl? show that providing a runtime type checker is absolutely feasible. The function annotation syntax currently supported in Python 3.4 is not well-suited for typing. This is because users expect to be able to operate on the types they know. This is currently not feasible because: 1. forward references are impossible 2. generics are impossible without custom syntax (which is the reason Mypy?s Dict exists) 3. optional types are clumsy to express (Optional[int] is very verbose for a use case this common) 4. union types are clumsy to express All those problems are elegantly solved by Google?s pytypedecl via moving type information to a separate file. Because for our use case that would not be an acceptable approach, my intuition would be to: 1. Provide support for generics (understood as an answer to the question: ?what does this collection contain??) in Abstract Base Classes. That would be a PEP in itself. 2. Change the function annotation syntax so that it?s not executed at import time but rather treated as strings. This solves forward references and enables us to? 3. Extend the function annotation syntax with first-class generics support (most languages like "list?) 4. Extend the function annotation syntax with first-class union type support. pytypedecl simply uses ?int or None?, which I find very elegant. 5. Speaking of None, possibly further extend the function annotation syntax with first-class optionality support. In the Facebook codebase in Hack we have tens of thousands of optional ints (nevermind other optional types!), this is a case that?s going to be used all the time. Hack uses ?int, that?s the most succinct style you can get. Yes, it?s special but None is a special type, too. All in all, I believe Mypy has the highest chance of becoming our typing linter, which is great! I just hope we can improve on the syntax, which is currently lacking. Also, reusing our existing ABCs where applicable would be nice. With Mypy?s typing module I feel like we?re going to get a new, orthogonal set of ABCs, which will confuse users to no end. Finally, the runtime type checker would make the ecosystem complete. This is just the beginning of the open issues I was juggling with and the reason my own try at the PEP was coming up slower than I?d like. [1] You can find a summary of examples I looked at here: http://lukasz.langa.pl/typehinting/ -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev On Aug 13, 2014, at 12:44 PM, Guido van Rossum wrote: > [There is no TL;DR other than the subject line. Please read the whole thing before replying. I do have an appendix with some motivations for adding type annotations at the end.] > > Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can learn from Haskell (and other languages); yesterday he gave the same talk at Dropbox. The talk is online (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad strokes comes down to three suggestions: > > (a) Python should adopt mypy's syntax for function annotations > (b) Python's use of mutabe containers by default is wrong > (c) Python should adopt some kind of Abstract Data Types > > Proposals (b) and (c) don't feel particularly actionable (if you disagree please start a new thread, I'd be happy to discuss these further if there's interest) but proposal (a) feels right to me. > > So what is mypy? It is a static type checker for Python written by Jukka for his Ph.D. thesis. The basic idea is that you add type annotations to your program using some custom syntax, and when running your program using the mypy interpreter, type errors will be found during compilation (i.e., before the program starts running). > > The clever thing here is that the custom syntax is actually valid Python 3, using (mostly) function annotations: your annotated program will still run with the regular Python 3 interpreter. In the latter case there will be no type checking, and no runtime overhead, except to evaluate the function annotations (which are evaluated at function definition time but don't have any effect when the function is called). > > In fact, it is probably more useful to think of mypy as a heavy-duty linter than as a compiler or interpreter; leave the type checking to mypy, and the execution to Python. It is easy to integrate mypy into a continuous integration setup, for example. > > To read up on mypy's annotation syntax, please see the mypy-lang.org website. Here's just one complete example, to give a flavor: > > from typing import List, Dict > > def word_count(input: List[str]) -> Dict[str, int]: > result = {} #type: Dict[str, int] > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result > > Note that the #type: comment is part of the mypy syntax; mypy uses comments to declare types in situations where no syntax is available -- although this particular line could also be written as follows: > > result = Dict[str, int]() > > Either way the entire function is syntactically valid Python 3, and a suitable implementation of typing.py (containing class definitions for List and Dict, for example) can be written to make the program run correctly. One is provided as part of the mypy project. > > I should add that many of mypy's syntactic choices aren't actually new. The basis of many of its ideas go back at least a decade: I blogged about this topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the two followup posts linked from the top there). > > I'll emphasize once more that mypy's type checking happens in a separate pass: no type checking happens at run time (other than what the interpreter already does, like raising TypeError on expressions like 1+"1"). > > There's a lot to this proposal, but I think it's possible to get a PEP written, accepted and implemented in time for Python 3.5, if people are supportive. I'll go briefly over some of the action items. > > (1) A change of direction for function annotations > > PEP 3107, which introduced function annotations, is intentional non-committal about how function annotations should be used. It lists a number of use cases, including but not limited to type checking. It also mentions some rejected proposals that would have standardized either a syntax for indicating types and/or a way for multiple frameworks to attach different annotations to the same function. AFAIK in practice there is little use of function annotations in mainstream code, and I propose a conscious change of course here by stating that annotations should be used to indicate types and to propose a standard notation for them. > > (We may have to have some backwards compatibility provision to avoid breaking code that currently uses annotations for some other purpose. Fortunately the only issue, at least initially, will be that when running mypy to type check such code it will produce complaints about the annotations; it will not affect how such code is executed by the Python interpreter. Nevertheless, it would be good to deprecate such alternative uses of annotations.) > > (2) A specification for what to add to Python 3.5 > > There needs to be at least a rough consensus on the syntax for annotations, and the syntax must cover a large enough set of use cases to be useful. Mypy is still under development, and some of its features are still evolving (e.g. unions were only added a few weeks ago). It would be possible to argue endlessly about details of the notation, e.g. whether to use 'list' or 'List', what either of those means (is a duck-typed list-like type acceptable?) or how to declare and use type variables, and what to do with functions that have no annotations at all (mypy currently skips those completely). > > I am proposing that we adopt whatever mypy uses here, keeping discussion of the details (mostly) out of the PEP. The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter. The actual type checker will not be integrated with the Python interpreter, and it will not be checked into the CPython repository. The only thing that needs to be added to the stdlib is a copy of mypy's typing.py module. This module defines several dozen new classes (and a few decorators and other helpers) that can be used in expressing argument types. If you want to type-check your code you have to download and install mypy and run it separately. > > The curious thing here is that while standardizing a syntax for type annotations, we technically still won't be adopting standard rules for type checking. This is intentional. First of all, fully specifying all the type checking rules would make for a really long and boring PEP (a much better specification would probably be the mypy source code). Second, I think it's fine if the type checking algorithm evolves over time, or if variations emerge. The worst that can happen is that you consider your code correct but mypy disagrees; your code will still run. > > That said, I don't want to completely leave out any specification. I want the contents of the typing.py module to be specified in the PEP, so that it can be used with confidence. But whether mypy will complain about your particular form of duck typing doesn't have to be specified by the PEP. Perhaps as mypy evolves it will take options to tell it how to handle certain edge cases. Forks of mypy (or entirely different implementations of type checking based on the same annotation syntax) are also a possibility. Maybe in the distant future a version of Python will take a different stance, once we have more experience with how this works out in practice, but for Python 3.5 I want to restrict the scope of the upheaval. > > Appendix -- Why Add Type Annotations? > > The argument between proponents of static typing and dynamic typing has been going on for many decades. Neither side is all wrong or all right. Python has traditionally fallen in the camp of extremely dynamic typing, and this has worked well for most users, but there are definitely some areas where adding type annotations would help. > > - Editors (IDEs) can benefit from type annotations; they can call out obvious mistakes (like misspelled method names or inapplicable operations) and suggest possible method names. Anyone who has used IntelliJ or Xcode will recognize how powerful these features are, and type annotations will make such features more useful when editing Python source code. > > - Linters are an important tool for teams developing software. A linter doesn't replace a unittest, but can find certain types of errors better or quicker. The kind of type checking offered by mypy works much like a linter, and has similar benefits; but it can find problems that are beyond the capabilities of most linters. > > - Type annotations are useful for the human reader as well! Take the above word_count() example. How long would it have taken you to figure out the types of the argument and return value without annotations? Currently most people put the types in their docstrings; developing a standard notation for type annotations will reduce the amount of documentation that needs to be written, and running the type checker might find bugs in the documentation, too. Once a standard type annotation syntax is introduced, it should be simple to add support for this notation to documentation generators like Sphinx. > > - Refactoring. Bob's talk has a convincing example of how type annotations help in (manually) refactoring code. I also expect that certain automatic refactorings will benefit from type annotations -- imagine a tool like 2to3 (but used for some other transformation) augmented by type annotations, so it will know whether e.g. x.keys() is referring to the keys of a dictionary or not. > > - Optimizers. I believe this is actually the least important application, certainly initially. Optimizers like PyPy or Pyston wouldn't be able to fully trust the type annotations, and they are better off using their current strategy of optimizing code based on the types actually observed at run time. But it's certainly feasible to imagine a future optimizer also taking type annotations into account. > > -- > --Guido "I need a new hobby" van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Thu Aug 14 03:09:23 2014 From: greg at krypto.org (Gregory P. Smith) Date: Wed, 13 Aug 2014 18:09:23 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 12:44 PM, Guido van Rossum wrote: > [There is no TL;DR other than the subject line. Please read the whole > thing before replying. I do have an appendix with some motivations for > adding type annotations at the end.] > > Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man > of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: > http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python > can learn from Haskell (and other languages); yesterday he gave the same > talk at Dropbox. The talk is online ( > https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad > strokes comes down to three suggestions: > > (a) Python should adopt mypy's syntax for function annotations > (b) Python's use of mutabe containers by default is wrong > (c) Python should adopt some kind of Abstract Data Types > > Proposals (b) and (c) don't feel particularly actionable (if you disagree > please start a new thread, I'd be happy to discuss these further if there's > interest) but proposal (a) feels right to me. > > So what is mypy? It is a static type checker for Python written by Jukka > for his Ph.D. thesis. The basic idea is that you add type annotations to > your program using some custom syntax, and when running your program using > the mypy interpreter, type errors will be found during compilation (i.e., > before the program starts running). > > The clever thing here is that the custom syntax is actually valid Python > 3, using (mostly) function annotations: your annotated program will still > run with the regular Python 3 interpreter. In the latter case there will be > no type checking, and no runtime overhead, except to evaluate the function > annotations (which are evaluated at function definition time but don't have > any effect when the function is called). > > In fact, it is probably more useful to think of mypy as a heavy-duty > linter than as a compiler or interpreter; leave the type checking to mypy, > and the execution to Python. It is easy to integrate mypy into a continuous > integration setup, for example. > > To read up on mypy's annotation syntax, please see the mypy-lang.org > website. Here's just one complete example, to give a flavor: > > from typing import List, Dict > > def word_count(input: List[str]) -> Dict[str, int]: > result = {} #type: Dict[str, int] > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result > > Note that the #type: comment is part of the mypy syntax; mypy uses > comments to declare types in situations where no syntax is available -- > although this particular line could also be written as follows: > > result = Dict[str, int]() > > Either way the entire function is syntactically valid Python 3, and a > suitable implementation of typing.py (containing class definitions for List > and Dict, for example) can be written to make the program run correctly. > One is provided as part of the mypy project. > > I should add that many of mypy's syntactic choices aren't actually new. > The basis of many of its ideas go back at least a decade: I blogged about > this topic in 2004 ( > http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the > two followup posts linked from the top there). > > I'll emphasize once more that mypy's type checking happens in a separate > pass: no type checking happens at run time (other than what the interpreter > already does, like raising TypeError on expressions like 1+"1"). > > There's a lot to this proposal, but I think it's possible to get a PEP > written, accepted and implemented in time for Python 3.5, if people are > supportive. I'll go briefly over some of the action items. > > *(1) A change of direction for function annotations* > > PEP 3107 , which introduced > function annotations, is intentional non-committal about how function > annotations should be used. It lists a number of use cases, including but > not limited to type checking. It also mentions some rejected proposals that > would have standardized either a syntax for indicating types and/or a way > for multiple frameworks to attach different annotations to the same > function. AFAIK in practice there is little use of function annotations in > mainstream code, and I propose a conscious change of course here by stating > that annotations should be used to indicate types and to propose a standard > notation for them. > > (We may have to have some backwards compatibility provision to avoid > breaking code that currently uses annotations for some other purpose. > Fortunately the only issue, at least initially, will be that when running > mypy to type check such code it will produce complaints about the > annotations; it will not affect how such code is executed by the Python > interpreter. Nevertheless, it would be good to deprecate such alternative > uses of annotations.) > > *(2) A specification for what to add to Python 3.5* > > There needs to be at least a rough consensus on the syntax for > annotations, and the syntax must cover a large enough set of use cases to > be useful. Mypy is still under development, and some of its features are > still evolving (e.g. unions were only added a few weeks ago). It would be > possible to argue endlessly about details of the notation, e.g. whether to > use 'list' or 'List', what either of those means (is a duck-typed list-like > type acceptable?) or how to declare and use type variables, and what to do > with functions that have no annotations at all (mypy currently skips those > completely). > > I am proposing that we adopt whatever mypy uses here, keeping discussion > of the details (mostly) out of the PEP. The goal is to make it possible to > add type checking annotations to 3rd party modules (and even to the stdlib) > while allowing unaltered execution of the program by the (unmodified) > Python 3.5 interpreter. The actual type checker will not be integrated with > the Python interpreter, and it will not be checked into the CPython > repository. The only thing that needs to be added to the stdlib is a copy > of mypy's typing.py module. This module defines several dozen new classes > (and a few decorators and other helpers) that can be used in expressing > argument types. If you want to type-check your code you have to download > and install mypy and run it separately. > > The curious thing here is that while standardizing a syntax for type > annotations, we technically still won't be adopting standard rules for type > checking. This is intentional. First of all, fully specifying all the type > checking rules would make for a really long and boring PEP (a much better > specification would probably be the mypy source code). Second, I think it's > fine if the type checking algorithm evolves over time, or if variations > emerge. The worst that can happen is that you consider your code correct > but mypy disagrees; your code will still run. > > That said, I don't want to *completely* leave out any specification. I > want the contents of the typing.py module to be specified in the PEP, so > that it can be used with confidence. But whether mypy will complain about > your particular form of duck typing doesn't have to be specified by the > PEP. Perhaps as mypy evolves it will take options to tell it how to handle > certain edge cases. Forks of mypy (or entirely different implementations of > type checking based on the same annotation syntax) are also a possibility. > Maybe in the distant future a version of Python will take a different > stance, once we have more experience with how this works out in practice, > but for Python 3.5 I want to restrict the scope of the upheaval. > > > *Appendix -- Why Add Type Annotations?* > The argument between proponents of static typing and dynamic typing has > been going on for many decades. Neither side is all wrong or all right. > Python has traditionally fallen in the camp of extremely dynamic typing, > and this has worked well for most users, but there are definitely some > areas where adding type annotations would help. > > - Editors (IDEs) can benefit from type annotations; they can call out > obvious mistakes (like misspelled method names or inapplicable operations) > and suggest possible method names. Anyone who has used IntelliJ or Xcode > will recognize how powerful these features are, and type annotations will > make such features more useful when editing Python source code. > > - Linters are an important tool for teams developing software. A linter > doesn't replace a unittest, but can find certain types of errors better or > quicker. The kind of type checking offered by mypy works much like a > linter, and has similar benefits; but it can find problems that are beyond > the capabilities of most linters. > > - Type annotations are useful for the human reader as well! Take the above > word_count() example. How long would it have taken you to figure out the > types of the argument and return value without annotations? Currently most > people put the types in their docstrings; developing a standard notation > for type annotations will reduce the amount of documentation that needs to > be written, and running the type checker might find bugs in the > documentation, too. Once a standard type annotation syntax is introduced, > it should be simple to add support for this notation to documentation > generators like Sphinx. > > - Refactoring. Bob's talk has a convincing example of how type annotations > help in (manually) refactoring code. I also expect that certain automatic > refactorings will benefit from type annotations -- imagine a tool like 2to3 > (but used for some other transformation) augmented by type annotations, so > it will know whether e.g. x.keys() is referring to the keys of a dictionary > or not. > > - Optimizers. I believe this is actually the least important application, > certainly initially. Optimizers like PyPy or Pyston > wouldn't be able to fully trust the > type annotations, and they are better off using their current strategy of > optimizing code based on the types actually observed at run time. But it's > certainly feasible to imagine a future optimizer also taking type > annotations into account. > > -- > --Guido "I need a new hobby" van Rossum (python.org/~guido) > > First, I am really happy that you are interested in this and that your point (2) of what you want to see done is very limited and acknowledges that it isn't going to specify everything! Because that isn't possible. :) Unfortunately I feel that adding syntax like this to the language itself is not useful without enforcement because it that leads to code being written with unintentionally incorrect annotations that winds up deployed in libraries that later become a problem as soon as an actual analysis tool attempts to run over something that uses that unknowingly incorrectly specified code in a place where it cannot be easily updated (like the standard library). At the summit in Montreal earlier this year ?ukasz Langa (cc'd) volunteered to lead writing the PEP on Python type hinting based on the many existing implementations of such things (including mypy, cython, numba and pytypedecl ). I believe he has an initial draft he intends to send out soon. I'll let him speak to that. Looks like ?ukasz already responded, I'll stop writing now and go read that. :) Personal opinion from experience trying: You can't express the depth of types for an interface within the Python language syntax itself (assuming hacks such as specially formatted comments, strings or docstrings do not count). Forward references to things that haven't even been defined yet are common. You often want an ability to specify a duck type interface rather than a specific type. I think he has those points covered better than I do. -gps PS If anyone want to see a run time type checker make code run at half speed, look at the one pytypedecl offers. I'm sure it could be sped up, but run-time checkers in an interpreter are always likely to be slow. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Aug 14 03:28:16 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 14 Aug 2014 13:28:16 +1200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <53EC10B0.9020105@canterbury.ac.nz> On 08/14/2014 12:32 PM, Chris Angelico wrote: > I don't like the idea of "from typing import ..." as there's already a > "types" module and I think it'd be confusing. Maybe from __statictyping__ import ... More explicit, and being a dunder name suggests that it's something special that linters should ignore if they don't understand it. -- Greg From abarnert at yahoo.com Thu Aug 14 03:26:56 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 13 Aug 2014 18:26:56 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com> On Wednesday, August 13, 2014 1:30 PM, Alex Gaynor wrote: >I'm strongly opposed this, for a few reasons. [...] >Python's type system isn't very good. It lacks many features of more powerful >systems such as algebraic data types, interfaces, and parametric polymorphism. >Despite this, it works pretty well because of Python's dynamic typing. I >strongly believe that attempting to enforce the existing type system would be a >real shame. This is my main concern, but I'd phrase it very differently. First, Python's type system _is_ powerful, but only because it's dynamic. Duck typing simulates parametric polymorphism perfectly, disjunction types as long as they don't include themselves recursively,?algebraic data types in some but not all cases, etc. Simple (Java-style) generics, of the kind that Guido seems to be proposing, are not nearly as flexible. That's the problem. On the other hand, even though these types only cover a small portion of the space of Python's implicit type system, a lot of useful functions fall within that small portion. As long as you can just leave the rest of the program untyped, and there are no boundary problems, there's no real risk. On the third hand, what worries me is this: > Mypy has a cast() operator that you can use to shut it up when you (think you) know the conversion is safe. Why do we need casts? You shouldn't be trying to enforce static typing in a part of the program whose static type isn't sound. Languages like Java and C++ have no choice; Python does, so why not take advantage of it? The standard JSON example seems appropriate here. What's the return type of?json.loads? In Haskell, you write a pretty trivial JSONThing ADT, and you return a JSONThing that's an Object (which means its value maps String to JSONThing). In Python today, you return a dict, and use it exactly the same as in Haskell, except that you can't verify its soundness at compile time. In Java or C++, it's? what??The sound option is a?special JSONThing that has separate getObjectMemberString and getArrayMemberString and getObjectMemberInt, which is incredibly painful to use. A plain old Dict[String, Object] looks simple, but it means you have to downcast all over the place to do anything, making it completely unsound, and still unpleasant.?The official Java json.org library gives you a hybrid between the two that manages to be neither sound nor user-friendly. And of course there are libraries for many poor static languages (especially C++) that try to fake duck typing as far as possible for their JSON objects, which is of course nowhere near as far as Python gets for free. From greg.ewing at canterbury.ac.nz Thu Aug 14 03:33:16 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 14 Aug 2014 13:33:16 +1200 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> Message-ID: <53EC11DC.7070007@canterbury.ac.nz> On 08/14/2014 12:59 PM, Raymond Hettinger wrote: > > I would really like for the annotations to grow some way > to communicate exceptions as well as return types I'd hate for exception typing to be compulsory, though, since that turned out to be extremely annoying in Java. It would be even worse in Python, since there's no branch of the exception hierarchy corresponding to Java's RuntimeError for things that can be raised from anywhere. -- Greg From abarnert at yahoo.com Thu Aug 14 03:39:15 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 13 Aug 2014 18:39:15 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum wrote: >? def word_count(input: List[str]) -> Dict[str, int]: >????? result = {}? #type: Dict[str, int] >????? for line in input: >????????? for word in line.split(): >????????????? result[word] = result.get(word, 0) + 1 >????? return result I just realized why this bothers me. This function really, really ought to be taking an Iterable[String] (except that we don't have a String ABC). If you hadn't statically typed it, it would work just fine with, say, a text file?or, for that matter, a binary file. By restricting it to List[str], you've made it a lot less usable, for no visible benefit. And, while this is less serious, I don't think it should be guaranteeing that the result is a Dict rather than just some kind of Mapping. If you want to change the implementation tomorrow to return some kind of proxy or a tree-based sorted mapping, you can't do so without breaking all the code that uses your function. And if even Guido, in the motivating example for this feature, is needlessly restricting the usability and future flexibility of a function, I suspect it may be a much bigger problem in practice. This example also shows exactly what's wrong with simple generics: if this function takes an Iterable[String], it doesn't just return a Mapping[String, int], it returns a Mapping of _the same String type_. If your annotations can't express that, any value that passes through this function loses type information.? And not being able to tell whether the keys in word_count(f) are str or bytes *even if you know that f was a text file* seems like a pretty major loss. From guido at python.org Thu Aug 14 03:42:38 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 18:42:38 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 5:53 PM, Haoyi Li wrote: > > Both solutions have merit, but the idea of some implementations of the > type checker having covariance and some contravariance is fairly > disturbing. > > Why can't we have both? That's the only way to properly type things, since > immutable-get-style APIs are always going to be convariant, set-only style > APIs (e.g. a function that takes 1 arg and returns None) are going to be > contravariant and mutable get-set APIs (like most python collections) > should really be invariant. > That makes sense. Can you put something in the mypy tracker about this? (Or send a pull request. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Thu Aug 14 03:43:54 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Wed, 13 Aug 2014 21:13:54 -0430 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 6:41 PM, Guido van Rossum wrote: > Actually, mypy already has a solution. There's a codec ( > https://github.com/JukkaL/mypy/tree/master/mypy/codec) that you can use > which transforms Python-2-with-annotations into vanilla Python 2. It's not > an ideal solution, but it can work in cases where you absolutely have to > have state of the art Python 3.5 type checking *and* backwards > compatibility with Python 2. > It can't be a solution because it's a hack... Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Aug 14 03:44:13 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 14 Aug 2014 13:44:13 +1200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com> References: <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <53EC146D.3090402@canterbury.ac.nz> On 08/14/2014 01:26 PM, Andrew Barnert wrote: > In Java or C++, it's? what? The sound option is a special JSONThing that > has separate getObjectMemberString and getArrayMemberString and > getObjectMemberInt, which is incredibly painful to use. That's mainly because Java doesn't let you define your own types that use convenient syntax such as [] for indexing. Python doesn't have that problem, so a decent static type system for Python should let you define a JSONThing class that's fully type-safe while having a standard mapping interface. -- Greg From lukasz at langa.pl Thu Aug 14 03:51:37 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 13 Aug 2014 18:51:37 -0700 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: <53EC11DC.7070007@canterbury.ac.nz> References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> <53EC11DC.7070007@canterbury.ac.nz> Message-ID: On Aug 13, 2014, at 6:33 PM, Greg Ewing wrote: > On 08/14/2014 12:59 PM, Raymond Hettinger wrote: >> >> I would really like for the annotations to grow some way >> to communicate exceptions as well as return types > > I'd hate for exception typing to be compulsory, though, > since that turned out to be extremely annoying in Java. > > It would be even worse in Python, since there's no > branch of the exception hierarchy corresponding to > Java's RuntimeError for things that can be raised > from anywhere. As I understand it, that would serve mostly documentational purpose. The only analysis we could do on that would be to check if an exception is ever caught by the caller somewhere up the stack. This could already be done by finding ?raise? statements and going from there. But even with well annotated types that would be often hard to infer, limiting the usefulness of that analysis. Moreover, it wouldn?t surface problems when the code is using any type of framework that swallows exceptions to process them for the user. So that leaves documentation. Now this information already lands in docstrings, which lacks the ?one-obvious-way-to-do-it? feel. However, thanks to the hackiness of that approach, users don?t expect the provided list of exceptions to be comprehensive. That would change with a first-class syntax for them. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Thu Aug 14 03:56:54 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 13 Aug 2014 18:56:54 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl> On Aug 13, 2014, at 6:39 PM, Andrew Barnert wrote: > On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum wrote: > >> def word_count(input: List[str]) -> Dict[str, int]: >> result = {} #type: Dict[str, int] >> for line in input: >> for word in line.split(): >> result[word] = result.get(word, 0) + 1 >> return result > > I just realized why this bothers me. > > This function really, really ought to be taking an Iterable[String] You do realize String also happens to be an Iterable[String], right? One of my big dreams about Python is that one day we'll drop support for strings being iterable. Nothing of value would be lost and that would enable us to use isinstance(x, Iterable) and more importantly isinstance(x, Sequence). Funny that this surfaces now, too. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Aug 14 04:27:25 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 13 Aug 2014 22:27:25 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: Guido, as requesting, I read your whole post before replying. Please to the same. This response is both critical and supportive. On 8/13/2014 3:44 PM, Guido van Rossum wrote: > Yesterday afternoon I had an inspiring conversation with Bob Ippolito > (man of many trades, author of simplejson) and Jukka Lehtosalo (author > of mypy: http://mypy-lang.org/). My main concern with static typing is that it tends to be anti-duck-typing, while I consider duck-typing to be a major *feature* of Python. The example in the page above is "def fib(n: int):". Fib should get an count (non-negative integer) value, but it need not be an int, and 'half' the ints do not qualify. Reading the tutorial, I could not tell if it supports numbers.Number (which should approximate the domain from above.) Now consider an extended version (after Lucas). def fib(n, a, b): i = 0 while i <= n: print(i,a) i += 1 a, b = b, a+b The only requirement of a, b is that they be addable. Any numbers should be allowed, as in fib(10, 1, 1+1j), but so should fib(5, '0', '1'). Addable would be approximated from below by Union(Number, str). > Bob gave a talk at EuroPython about > what Python can learn from Haskell (and other languages); yesterday he > gave the same talk at Dropbox. The talk is online > (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad > strokes comes down to three suggestions: > > (a) Python should adopt mypy's syntax for function annotations -+ Syntax with no meaning is a bit strange. On the other hand, syntax not bound to semantics, or at least not bound to just one meaning is quite pythonic. '+' has two standard meanings, plus custom meanings embodied in .__add__ methods. + The current semantics of annotations is that they are added to functions objects as .__annotations__ (for whatever use) *and* used as part of inspect.signature and included in help(ob) responses. In other words, annotations are already used in the stdlib. >>> def f(i:int) -> float: pass >>> from inspect import signature as sig >>> str(sig(f)) '(i:int) -> float' >>> help(f) Help on function f in module __main__: f(i:int) -> float Idle calltips include them also. A appropriately flexible standardized notation would enhance this usage and many others. +-+ I see the point of "The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter." On the other hand, "pip install mypytyping" is not a huge burden. On the third hand, in the stdlib allows use in the stdlib. > (b) Python's use of mutabe [mutable] containers by default is wrong The premise of this is partly wrong and partly obsolete. As far as I can remember, Python *syntax* only use tuples, not lists: "except (ex1, ex2):", "s % (val1, val2)", etc. The use of lists as the common format for data interchange between functions has largely been replaced by iterators. This fact makes Python code much more generic, and anti-generic static typing more wrong. In remaining cases, 'wrong' is as much a philosophical opinion as a fact. > (c) Python should adopt some kind of Abstract Data Types I would have to look at the talk to know what Jukka means. > Proposals (b) and (c) don't feel particularly actionable (if you > disagree please start a new thread, I'd be happy to discuss these > further if there's interest) but proposal (a) feels right to me. > So what is mypy? It is a static type checker for Python written by > Jukka for his Ph.D. thesis. The basic idea is that you add type > annotations to your program using some custom syntax, and when running > your program using the mypy interpreter, type errors will be found > during compilation (i.e., before the program starts running). > > The clever thing here is that the custom syntax is actually valid Python > 3, using (mostly) function annotations: your annotated program will > still run with the regular Python 3 interpreter. In the latter case > there will be no type checking, and no runtime overhead, except to > evaluate the function annotations (which are evaluated at function > definition time but don't have any effect when the function is called). > > In fact, it is probably more useful to think of mypy as a heavy-duty > linter than as a compiler or interpreter; leave the type checking to > mypy, and the execution to Python. It is easy to integrate mypy into a > continuous integration setup, for example. > > To read up on mypy's annotation syntax, please see the mypy-lang.org > website. I did not see a 'reference' page, but the tutorial comes pretty close. http://mypy-lang.org/tutorial.html Beyond that, typings.py would be definitive, https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py > Here's just one complete example, to give a flavor: > from typing import List, Dict > > def word_count(input: List[str]) -> Dict[str, int]: The input annotation should be Iterable[str], which mypy does have. > result = {} #type: Dict[str, int] > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result The information that input is an Iterable[str] can be used either within the definition of word_count or at places where word_count is called. A type aware checker, either in the editor or compiler, could check that the only uses of 'input' within the function is as input to functions declared to accept an Iterable or in for statements. Checking that the input to word_count is specifically Iterable[str] as opposed to any other Iterable may not be possible. But I think what can be done, including enhancing help information, might be worth it. For instance, the parameter to s.join is named 'iterable'. Something more specific, either 'iterable_of_strings' or 'strings: Iterable[str]' would be more helpful. Indeed, there have been people posting on python list who thought that 'iterable' means iterable and that .join would call str() on each object. I think there are other cases where a parameter is given a bland under-informative type name instead of a context-specific semantic name just because there was no type annotation available. There are places where the opposite problem occurs, too specific instead of too general, where iterable parameters are still called 'list'. > Note that the #type: comment is part of the mypy syntax; mypy uses > comments to declare types in situations where no syntax is available -- > although this particular line could also be written as follows: > > result = Dict[str, int]() > > Either way the entire function is syntactically valid Python 3, and a > suitable implementation of typing.py (containing class definitions for > List and Dict, for example) can be written to make the program run > correctly. One is provided as part of the mypy project. > > I should add that many of mypy's syntactic choices aren't actually new. > The basis of many of its ideas go back at least a decade: I blogged > about this topic in 2004 > (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the > two followup posts linked from the top there). > > I'll emphasize once more that mypy's type checking happens in a separate > pass: no type checking happens at run time (other than what the > interpreter already does, like raising TypeError on expressions like 1+"1"). > > There's a lot to this proposal, but I think it's possible to get a PEP > written, accepted and implemented in time for Python 3.5, if people are > supportive. I'll go briefly over some of the action items. > > *(1) A change of direction for function annotations* > > PEP 3107 , which introduced > function annotations, is intentional non-committal about how function > annotations should be used. It lists a number of use cases, including > but not limited to type checking. It also mentions some rejected > proposals that would have standardized either a syntax for indicating > types and/or a way for multiple frameworks to attach different > annotations to the same function. AFAIK in practice there is little use > of function annotations in mainstream code, and I propose a conscious > change of course here by stating that annotations should be used to > indicate types and to propose a standard notation for them. There are many uses for type information and I think Python should remain neutral among them. > (We may have to have some backwards compatibility provision to avoid > breaking code that currently uses annotations for some other purpose. > Fortunately the only issue, at least initially, will be that when > running mypy to type check such code it will produce complaints about > the annotations; it will not affect how such code is executed by the > Python interpreter. Nevertheless, it would be good to deprecate such > alternative uses of annotations.) I can imagine that people who have used annotations might feel a bit betrayed by deprecation of a new-in-py3 feature. But I do not think it necessary to do so. Tools that work with mypy annotations, including mypy itself, should only assume mypy typing if typing is imported. No 'import typing', no 'Warning: annotation does not follow typing rules." If 'typing' were a package with a 'mypy' module, the door would be left open to other 'blessed' typing modules. > *(2) A specification for what to add to Python 3.5* > > There needs to be at least a rough consensus on the syntax for > annotations, and the syntax must cover a large enough set of use cases > to be useful. Mypy is still under development, and some of its features > are still evolving (e.g. unions were only added a few weeks ago). It > would be possible to argue endlessly about details of the notation, e.g. > whether to use 'list' or 'List', what either of those means (is a > duck-typed list-like type acceptable?) or how to declare and use type > variables, and what to do with functions that have no annotations at all > (mypy currently skips those completely). > > I am proposing that we adopt whatever mypy uses here, keeping discussion > of the details (mostly) out of the PEP. The goal is to make it possible > to add type checking annotations to 3rd party modules (and even to the > stdlib) while allowing unaltered execution of the program by the > (unmodified) Python 3.5 interpreter. The actual type checker will not be > integrated with the Python interpreter, and it will not be checked into > the CPython repository. The only thing that needs to be added to the > stdlib is a copy of mypy's typing.py module. This module defines several > dozen new classes (and a few decorators and other helpers) that can be > used in expressing argument types. If you want to type-check your code > you have to download and install mypy and run it separately. > > The curious thing here is that while standardizing a syntax for type > annotations, we technically still won't be adopting standard rules for > type checking. Fine with me, as that is not the only use. And even for type checking, there is the choice between accept unless clearly wrong, versus reject unless clearly right. > This is intentional. First of all, fully specifying all > the type checking rules would make for a really long and boring PEP (a > much better specification would probably be the mypy source code). > Second, I think it's fine if the type checking algorithm evolves over > time, or if variations emerge. As in the choice between accept unless clearly wrong, versus reject unless clearly right. > The worst that can happen is that you > consider your code correct but mypy disagrees; your code will still run. > > That said, I don't want to /completely/ leave out any specification. I > want the contents of the typing.py module to be specified in the PEP, so > that it can be used with confidence. But whether mypy will complain > about your particular form of duck typing doesn't have to be specified > by the PEP. Perhaps as mypy evolves it will take options to tell it how > to handle certain edge cases. Forks of mypy (or entirely different > implementations of type checking based on the same annotation syntax) > are also a possibility. Maybe in the distant future a version of Python > will take a different stance, once we have more experience with how this > works out in practice, but for Python 3.5 I want to restrict the scope > of the upheaval. As usual, we should review the code before acceptance. It is not clear to me how much of the tutorial is implemented, as it says "Some of these features might never see the light of day. " ??? > *Appendix -- Why Add Type Annotations? > * > The argument between proponents of static typing and dynamic typing has > been going on for many decades. Neither side is all wrong or all right. > Python has traditionally fallen in the camp of extremely dynamic typing, > and this has worked well for most users, but there are definitely some > areas where adding type annotations would help. The answer to why on the mypy page is 'easier to find bugs', 'easier maintenance'. I find this under-convincing as sufficient justification in itself. I don't think there are many bugs on the tracker due to calling functions with the wrong type of object. Logic errors, ignored corner cases, and system idiosyncrasies are much more of a problem. Your broader list is more convincing. > - Editors (IDEs) can benefit from type annotations; they can call out > obvious mistakes (like misspelled method names or inapplicable > operations) and suggest possible method names. Anyone who has used > IntelliJ or Xcode will recognize how powerful these features are, and > type annotations will make such features more useful when editing Python > source code. > > - Linters are an important tool for teams developing software. A linter > doesn't replace a unittest, but can find certain types of errors better > or quicker. The kind of type checking offered by mypy works much like a > linter, and has similar benefits; but it can find problems that are > beyond the capabilities of most linters. Currently, Python linters do not have standard type annotations to work with. I suspect that programs other than mypy would use them if available. > - Type annotations are useful for the human reader as well! Take the > above word_count() example. How long would it have taken you to figure > out the types of the argument and return value without annotations? Under a minute, including the fact the the annotation was overly restrictive. But then I already know that only a mutation method can require a list. > Currently most people put the types in their docstrings; developing a > standard notation for type annotations will reduce the amount of > documentation that needs to be written, and running the type checker > might find bugs in the documentation, too. Once a standard type > annotation syntax is introduced, it should be simple to add support for > this notation to documentation generators like Sphinx. > > - Refactoring. Bob's talk has a convincing example of how type > annotations help in (manually) refactoring code. I also expect that > certain automatic refactorings will benefit from type annotations -- > imagine a tool like 2to3 (but used for some other transformation) > augmented by type annotations, so it will know whether e.g. x.keys() is > referring to the keys of a dictionary or not. > > - Optimizers. I believe this is actually the least important > application, certainly initially. Optimizers like PyPy or Pyston > wouldn't be able to fully trust the > type annotations, and they are better off using their current strategy > of optimizing code based on the types actually observed at run time. But > it's certainly feasible to imagine a future optimizer also taking type > annotations into account. -- Terry Jan Reedy From abarnert at yahoo.com Thu Aug 14 04:58:21 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 13 Aug 2014 19:58:21 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EC146D.3090402@canterbury.ac.nz> References: <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com> <53EC146D.3090402@canterbury.ac.nz> Message-ID: <3C196232-D06D-4EFC-81BB-58FCBBF86DC2@yahoo.com> On Aug 13, 2014, at 18:44, Greg Ewing wrote: > On 08/14/2014 01:26 PM, Andrew Barnert wrote: > >> In Java or C++, it's? what? The sound option is a special JSONThing that >> has separate getObjectMemberString and getArrayMemberString and >> getObjectMemberInt, which is incredibly painful to use. > > That's mainly because Java doesn't let you define your own > types that use convenient syntax such as [] for indexing. No it's not, or other languages like C++ (which has operator methods and overloading) wouldn't have the exact same problem, but they do. Look at JsonCpp, for example: https://github.com/open-source-parsers/jsoncpp/ > Python doesn't have that problem, so a decent static type > system for Python should let you define a JSONThing class > that's fully type-safe while having a standard mapping > interface. How? If you go with a single JSONThing type that represents an object, array, number, bool, string, or null, then it can't have a standard mapping interface, because it also needs to have a standard sequence interface, and they conflict. Likewise for number vs. string. The only fully type-safe interface it can have is as_string, as_number, etc. methods (which of course can only check at runtime, so it's no better than using isinstance from Python, and you're forced to do it for every single access.) What if you go the other way and have separate JSONObject, JSONArray, etc. types? Then all of those problems go away; you can define an unambiguous __getitem__. But what is its return value? The only possibility is a union of all the various types mentioned above, and such a union type has no interface at all. It's only useful if people subvert the type safety by casting. (I guess you could argue that returning a union type makes your JSON library type safe, it's only every program that ever uses it for anything that's unsafe. But where does that get you?) The only usable type safe interface is separate get_string, get_number, etc. methods in place of __getitem__. Or you can merge the two together and have a single JSONThing that has both as methods and, for convenience, combined as_object+get, or even as_object+get+as_str. Also, look at the mutation interfaces for these libraries. They're only marginally tolerable because all variable have obligatory types that you can overload on, which wouldn't be the case in Python. The alternative is, of course, to come up with a way to avoid type safety. In Swift, you parse a JSON object into a Cocoa NSDictionary, which is a dynamically-typed heterogeneous collection just like a Python dict. There are C++ libraries with a bunch of types that effectively act like Python dict, list, float, str, etc. and try to magically cast when they come into contact with native types. That's the best solution anyone has to dealing with even a dead-simple algebraic data type like JSON in a static language whose type system isn't powerful enough: to try to fake being a duck typed language. -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlopires at gmail.com Thu Aug 14 05:06:56 2014 From: carlopires at gmail.com (Carlo Pires) Date: Thu, 14 Aug 2014 00:06:56 -0300 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: I'm very happy to see this happening. "Optional" type checking for python would be a great addition to the language. For large codebases, use of type checking can really help. I also like the idea of using annotations instead of decorators (or something like that). I'm already using this for python3 [1] in a non intrusive way, by using assert to disable it on production. [1] https://pypi.python.org/pypi/optypecheck -- Carlo Pires -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Aug 14 05:08:12 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 13 Aug 2014 20:08:12 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl> References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl> Message-ID: On Aug 13, 2014, at 18:56, ?ukasz Langa wrote: > On Aug 13, 2014, at 6:39 PM, Andrew Barnert wrote: > >> On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum wrote: >> >>> def word_count(input: List[str]) -> Dict[str, int]: >>> result = {} #type: Dict[str, int] >>> for line in input: >>> for word in line.split(): >>> result[word] = result.get(word, 0) + 1 >>> return result >> >> I just realized why this bothers me. >> >> This function really, really ought to be taking an Iterable[String] > > You do realize String also happens to be an Iterable[String], right? Of course, but that's not a new problem, so I didn't want to bring it up. The fact that the static type checker couldn't reject word_count(f.read()) is annoying, but that's not the fault of the static type checking proposal. > One of my big dreams about Python is that one day we'll drop support for strings being iterable. Nothing of value would be lost and that would enable us to use isinstance(x, Iterable) and more importantly isinstance(x, Sequence). Funny that this surfaces now, too. IIRC, str doesn't implement Container, and therefore doesn't implement Sequence, because its __contains__ method is substring match instead of containment. So if you really want to treat sequences of strings separately from strings, you can. If only that really _were_ more important than Iterable, but I think the opposite is true. But anyway, this is probably off topic, so I'll stop here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Thu Aug 14 05:16:04 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 13 Aug 2014 20:16:04 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl> Message-ID: <2016271B-67D2-447D-8E51-369806CBA13A@langa.pl> On Aug 13, 2014, at 8:08 PM, Andrew Barnert wrote: > On Aug 13, 2014, at 18:56, ?ukasz Langa wrote: > >> One of my big dreams about Python is that one day we'll drop support for strings being iterable. Nothing of value would be lost and that would enable us to use isinstance(x, Iterable) and more importantly isinstance(x, Sequence). Funny that this surfaces now, too. > > IIRC, str doesn't implement Container, and therefore doesn't implement Sequence, because its __contains__ method is substring match instead of containment. So if you really want to treat sequences of strings separately from strings, you can. str and bytes objects respond True to both isinstance(x, Container) and isinstance(x, Sequence). But you?re right, off topic. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Aug 14 05:28:19 2014 From: mertz at gnosis.cx (David Mertz) Date: Wed, 13 Aug 2014 20:28:19 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <2016271B-67D2-447D-8E51-369806CBA13A@langa.pl> References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl> <2016271B-67D2-447D-8E51-369806CBA13A@langa.pl> Message-ID: A long while back I posted a recipe for using annotations for type checking. I'm certainly not the first person to do this, and what I did was deliberately simple: https://code.activestate.com/recipes/578528-type-checking-using-python-3x-annotations/?in=user-4173018 The approach I used was to use per-function decorators to say that a given function should be type checked. The type system I enforce in that recipe is much less than what mypy allows, but I can't see a real reason that it couldn't be extended to cover exactly the same range of type specifiers. The advantage I perceive in this approach is that it is purely optional, per module and per function. As well, it doesn't actually require making ANY change to Python 3.5 to implement it. Or as a minimal change, an extra decorator could simply be available in functools or elsewhere in the standard library, which implemented the full semantics of mypy. Now admittedly, this would be type checking, but not *static* type checking. There may not be an easy way to make a pre-runtime "lint" tool do the checking there. On the other hand, as a number of posters have noted, there's also no way to enforce, e.g. 'Iterable[String]' either statically. I'm not the BDFL of course, but I do not really get what advantage there is to the pre-runtime check that can catch a fairly small subset of type constraints rather than check at runtime everything that is available then (as the decorator approach could get you). On Wed, Aug 13, 2014 at 8:16 PM, ?ukasz Langa wrote: > On Aug 13, 2014, at 8:08 PM, Andrew Barnert wrote: > > On Aug 13, 2014, at 18:56, ?ukasz Langa wrote: > > One of my big dreams about Python is that one day we'll drop support for > strings being iterable. Nothing of value would be lost and that would > enable us to use isinstance(x, Iterable) and more importantly isinstance(x, > Sequence). Funny that this surfaces now, too. > > > IIRC, str doesn't implement Container, and therefore doesn't implement > Sequence, because its __contains__ method is substring match instead of > containment. So if you really want to treat sequences of strings separately > from strings, you can. > > > str and bytes objects respond True to both isinstance(x, Container) and > isinstance(x, Sequence). > > But you?re right, off topic. > > -- > Best regards, > ?ukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 05:41:30 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 20:41:30 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: On Wed, Aug 13, 2014 at 6:39 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum > wrote: > > > def word_count(input: List[str]) -> Dict[str, int]: > > result = {} #type: Dict[str, int] > > for line in input: > > for word in line.split(): > > result[word] = result.get(word, 0) + 1 > > return result > > I just realized why this bothers me. > > This function really, really ought to be taking an Iterable[String] > (except that we don't have a String ABC). If you hadn't statically typed > it, it would work just fine with, say, a text file?or, for that matter, a > binary file. By restricting it to List[str], you've made it a lot less > usable, for no visible benefit. > Heh. :-) I had wanted to write an additional paragraph explaining that it's easy to change this to use typing.Iterable instead of typing.List, but I forgot to add that. > And, while this is less serious, I don't think it should be guaranteeing > that the result is a Dict rather than just some kind of Mapping. If you > want to change the implementation tomorrow to return some kind of proxy or > a tree-based sorted mapping, you can't do so without breaking all the code > that uses your function. > Yeah, there's a typing.Mapping for that. > And if even Guido, in the motivating example for this feature, is > needlessly restricting the usability and future flexibility of a function, > I suspect it may be a much bigger problem in practice. > Well, so it was actually semi-intentional. :-) > This example also shows exactly what's wrong with simple generics: if this > function takes an Iterable[String], it doesn't just return a > Mapping[String, int], it returns a Mapping of _the same String type_. If > your annotations can't express that, any value that passes through this > function loses type information. > In most cases it really doesn't matter though -- some types are better left concrete, especially strings and numbers. If you read the mypy docs you'll find that there are generic types, so that it's possible to define a function as taking an Iterable[T] and returning a Mapping[T, int]. What's not currently possible is expressing additional constraints on T such as that it must be a String. When I last talked to Jukka he explained that he was going to add something for that too (@Jukka: structured types?). > And not being able to tell whether the keys in word_count(f) are str or > bytes *even if you know that f was a text file* seems like a pretty major > loss. > On this point one of us must be confused. Let's assume it's me. :-) Mypy has a few different IO types that can express the difference between text and binary files. I think there's some work that needs to be done (and of course the built-in open() function has a terribly ambiguous return type :-( ), but it should be possible to say that a text file is an Interable[str] and a binary file is an Iterable[bytes]. So together with the structured (?) types it should be possible to specify the signature of word_count() just as you want it. However, in most cases it's overkill, and you wouldn't want to do that for most code. Also, it probably wouldn't work for more realistic examples -- as soon as you replace the split() method call with something that takes punctuation into account, you're probably going to write it in a way that works only for text strings anyway, and very few people will want or need to write the polymorphic version. (But if they do, mypy has a handy @overload decorator that they can use. :-) Anyway, I agree it would be good to make sure that some of these more advanced things can actually be spelled before we freeze our commitment to a specific syntax, but let's not assume that just because you can't spell every possible generic use case it's no good. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlehtosalo at gmail.com Thu Aug 14 06:06:41 2014 From: jlehtosalo at gmail.com (Jukka Lehtosalo) Date: Wed, 13 Aug 2014 21:06:41 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: On Wed, Aug 13, 2014 at 6:39 PM, Andrew Barnert wrote: > On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum > wrote: > > > > def word_count(input: List[str]) -> Dict[str, int]: > > result = {} #type: Dict[str, int] > > for line in input: > > for word in line.split(): > > result[word] = result.get(word, 0) + 1 > > return result > > > I just realized why this bothers me. > > This function really, really ought to be taking an Iterable[String] > (except that we don't have a String ABC). If you hadn't statically typed > it, it would work just fine with, say, a text file?or, for that matter, a > binary file. By restricting it to List[str], you've made it a lot less > usable, for no visible benefit. > > And, while this is less serious, I don't think it should be guaranteeing > that the result is a Dict rather than just some kind of Mapping. If you > want to change the implementation tomorrow to return some kind of proxy or > a tree-based sorted mapping, you can't do so without breaking all the code > that uses your function. > I see this is a matter of programming style. In a library module, I'd usually use about as general types as feasible (without making them overly complex). However, if we have just a simple utility function that's only used within a single program, declaring everything using abstract types buys you little, IMHO, but may make things much more complicated. You can always refactor the code to use more general types if the need arises. Using simple, concrete types seems to decrease the cognitive load, but that's just my experience. Also, programmers don't always read documentation/annotations and can abuse the knowledge of the concrete return type of any function (they can figure this out easily by using repr()/type()). In general, as long as dynamically typed programs may call your function, changing the concrete return type of a library function risks breaking code that makes too many assumptions. Thus I'd rather use concrete types for function return types -- but of course everybody is free to not follow this convention. > And if even Guido, in the motivating example for this feature, is > needlessly restricting the usability and future flexibility of a function, > I suspect it may be a much bigger problem in practice. > > > This example also shows exactly what's wrong with simple generics: if this > function takes an Iterable[String], it doesn't just return a > Mapping[String, int], it returns a Mapping of _the same String type_. If > your annotations can't express that, any value that passes through this > function loses type information. > If I define a subclass X of str, split() still returns a List[str] rather than List[X], unless I override something, so this wouldn't work with the above example: >>> class X(str): pass ... >>> type(X('x y').split()[0]) > And not being able to tell whether the keys in word_count(f) are str or > bytes *even if you know that f was a text file* seems like a pretty major > loss. > Mypy considers bytes incompatible with str, and vice versa. The annotation Iterable[str] says that Iterable[bytes] (such as a binary file) would not be a valid argument. Text files and binary files have different types, though the return type of open(...) is not inferred correctly right now. It would be easy to fix this for the most common cases, though. You could use AnyStr to make the example work with bytes as well: def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]: result = {} #type: Dict[AnyStr, int] for line in input: for word in line.split(): result[word] = result.get(word, 0) + 1 return result Again, if this is just a simple utility function that you use once or twice, I see no reason to spend a lot of effort in coming up with the most general signature. Types are an abstraction and they can't express everything precisely -- there will always be a lot of cases where you can't express the most general type. However, I think that relatively simple types work well enough most of the time, and give the most bang for the buck. Jukka -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Aug 14 06:21:22 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 14 Aug 2014 00:21:22 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53EBC3BA.10808@stoneleaf.us> Message-ID: On 8/13/2014 5:08 PM, Andrey Vlasovskikh wrote: > Here are slides from my talk about optional typing in Python, that > show how Mypy types can be used in both static and dynamic type > checking > (http://blog.pirx.ru/media/files/2013/python-optional-typing/), I tried this on Windows 7in both Firefox and Internet Explorer and I cannot find any way to advance other than changing the page number on the url bar. -- Terry Jan Reedy From jlehtosalo at gmail.com Thu Aug 14 06:28:48 2014 From: jlehtosalo at gmail.com (Jukka Lehtosalo) Date: Wed, 13 Aug 2014 21:28:48 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: On Wed, Aug 13, 2014 at 8:41 PM, Guido van Rossum wrote: > On Wed, Aug 13, 2014 at 6:39 PM, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > >> This example also shows exactly what's wrong with simple generics: if >> this function takes an Iterable[String], it doesn't just return a >> Mapping[String, int], it returns a Mapping of _the same String type_. If >> your annotations can't express that, any value that passes through this >> function loses type information. >> > > In most cases it really doesn't matter though -- some types are better > left concrete, especially strings and numbers. If you read the mypy docs > you'll find that there are generic types, so that it's possible to define a > function as taking an Iterable[T] and returning a Mapping[T, int]. What's > not currently possible is expressing additional constraints on T such as > that it must be a String. When I last talked to Jukka he explained that he > was going to add something for that too (@Jukka: structured types?). > I wrote another message where I touched this. Mypy is likely to support something like this in the future, but I doubt it's usually worth the complexity. If a type signature is very general, at some point it describes the implementation in sufficient detail that you can't modify the code without changing the type. For example, we could plausibly allow anything that just supports split(), but if we change the implementation to use something other than split(), the signature would have to change. If we use more specific types (such as str), we leave us the freedom to modify the implementation within the bounds of the str interface. Standard library functions often only accept concrete str objects, so the moment you start using an abstract string type you lose access to much of the stdlib. > >> And not being able to tell whether the keys in word_count(f) are str or >> bytes *even if you know that f was a text file* seems like a pretty major >> loss. >> > > On this point one of us must be confused. Let's assume it's me. :-) Mypy > has a few different IO types that can express the difference between text > and binary files. I think there's some work that needs to be done (and of > course the built-in open() function has a terribly ambiguous return type > :-( ), but it should be possible to say that a text file is an > Interable[str] and a binary file is an Iterable[bytes]. So together with > the structured (?) types it should be possible to specify the signature of > word_count() just as you want it. However, in most cases it's overkill, and > you wouldn't want to do that for most code. > See my other message where I show that you can do this right now, except for the problem with open(). > > Also, it probably wouldn't work for more realistic examples -- as soon as > you replace the split() method call with something that takes punctuation > into account, you're probably going to write it in a way that works only > for text strings anyway, and very few people will want or need to write the > polymorphic version. (But if they do, mypy has a handy @overload decorator > that they can use. :-) > > Anyway, I agree it would be good to make sure that some of these more > advanced things can actually be spelled before we freeze our commitment to > a specific syntax, but let's not assume that just because you can't spell > every possible generic use case it's no good. > It's always easy to come up with interesting corner cases where a type system would break down, but luckily, these are often almost non-existent in the wild :-) I've learned that examples should be motivated by patterns in existing, 'real' code, as otherwise you'll waste your time on things that happen maybe once a million lines (or maybe only in code that *you* write). Jukka > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 06:55:11 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 21:55:11 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <75CA8559-0A4C-4BA0-94AA-A69C0CD56AE7@langa.pl> References: <75CA8559-0A4C-4BA0-94AA-A69C0CD56AE7@langa.pl> Message-ID: On Wed, Aug 13, 2014 at 6:00 PM, ?ukasz Langa wrote: > It?s great to see this finally happening! > Yes. :-) > I did some research on existing optional-typing approaches [1]. What I > learned in the process was that linting is the most important use case for > optional typing; runtime checks is too little, too late. > > That being said, having optional runtime checks available *is* also > important. Used in staging environments and during unit testing, this case > is able to cover cases obscured by meta-programming. Implementations like > ?obiwan? and ?pytypedecl? show that providing a runtime type checker is > absolutely feasible. > Yes. And the proposal here might well enable such applications (by providing a standard way to spell complex types). But I think it's going to be less important than good support for linting, so that's what I want to focus on first. > The function annotation syntax currently supported in Python 3.4 is not > well-suited for typing. This is because users expect to be able to operate > on the types they know. This is currently not feasible because: > 1. forward references are impossible > (Mypy's hack for this is that a string literal can be used as a forward reference.) 2. generics are impossible without custom syntax (which is the reason > Mypy?s Dict exists) > 3. optional types are clumsy to express (Optional[int] is very verbose for > a use case this common) > So define an alias 'oint'. :-) > 4. union types are clumsy to express > Aliasing can help. > All those problems are elegantly solved by Google?s pytypedecl via moving > type information to a separate file. > Mypy supports this too using stub files, but I think it is actually a strength that it doesn't require new syntax (although if the idea becomes popular we could certainly add syntax to support those things where mypy currently requires magic comments). Honestly I'm not sure what to do about mypy vs. pytypedecl. Should they compete, collaborate, converge? Do we need a bake-off or a joint hackathon? Food for thought. > Because for our use case that would not be an acceptable approach, my > intuition would be to: > > 1. Provide support for generics (understood as an answer to the question: > ?what does this collection contain??) in Abstract Base Classes. That would > be a PEP in itself. > 2. Change the function annotation syntax so that it?s not executed at > import time but rather treated as strings. This solves forward references > and enables us to? > 3. Extend the function annotation syntax with first-class generics support > (most languages like "list?) > 4. Extend the function annotation syntax with first-class union type > support. pytypedecl simply uses ?int or None?, which I find very elegant. > 5. Speaking of None, possibly further extend the function annotation syntax > with first-class optionality support. In the Facebook codebase in Hack we > have tens of thousands of optional ints (nevermind other optional types!), > this is a case that?s going to be used all the time. Hack uses ?int, that?s > the most succinct style you can get. Yes, it?s special but None is a > special type, too. > Hm. I think that selling such (IMO) substantial changes to Python's syntax is going to be much harder than just the idea of a standard typing syntax implemented as a new stdlib module. While mypy's syntax is perhaps not as concise or elegant as would be possible if we were to design the syntax from the ground up, it's actually pretty darn readable, and it is compatible with Python 3.2. It has decent ways to spell generics, forward references, unions and optional types already. And while I want to eventually phase out other uses of function annotations, your change #2 would break all existing packages that use them for other purposes (like Ethan Furman's scription). > All in all, I believe Mypy has the highest chance of becoming our typing > linter, which is great! I just hope we can improve on the syntax, which is > currently lacking. Also, reusing our existing ABCs where applicable would > be nice. With Mypy?s typing module I feel like we?re going to get a new, > orthogonal set of ABCs, which will confuse users to no end. Finally, the > runtime type checker would make the ecosystem complete. > We can discuss these things separately. Language evolution is an exercise in compromise. We may be able to reuse the existing ABCs, and mypy could still support Python 3.2 (or, with the codeck hack, 2.7) by having the typing module export aliases to those ABCs. I won't stop you from implementing a runtime type checker, but I think it should be a separate project. > This is just the beginning of the open issues I was juggling with and the > reason my own try at the PEP was coming up slower than I?d like. > Hopefully I've motivated you to speed up! > [1] You can find a summary of examples I looked at here: > http://lukasz.langa.pl/typehinting/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 07:11:17 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 22:11:17 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 6:09 PM, Gregory P. Smith wrote: > First, I am really happy that you are interested in this and that your > point (2) of what you want to see done is very limited and acknowledges > that it isn't going to specify everything! Because that isn't possible. :) > What a shame. :-) > Unfortunately I feel that adding syntax like this to the language itself > is not useful without enforcement because it that leads to code being > written with unintentionally incorrect annotations that winds up deployed > in libraries that later become a problem as soon as an actual analysis tool > attempts to run over something that uses that unknowingly incorrectly > specified code in a place where it cannot be easily updated (like the > standard library). > We could refrain from using type annotations in the stdlib (similar to how we refrain from using Unicode identifiers). Mypy's stubs mechanism makes it possible to ship the type declarations for stdlib modules with mypy instead of baking them into the stdlib. > At the summit in Montreal earlier this year ?ukasz Langa (cc'd) > volunteered to lead writing the PEP on Python type hinting based on the > many existing implementations of such things (including mypy, cython, numba > and pytypedecl ). I believe he has > an initial draft he intends to send out soon. I'll let him speak to that. > Mypy has a lot more than an initial draft. Don't be mistaken by its status as "one person's Ph.D. project" -- Jukka has been thinking about this topic for a decade, and mypy works remarkably well already. It also has some very active contributors already. > Looks like ?ukasz already responded, I'll stop writing now and go read > that. :) > > Personal opinion from experience trying: You can't express the depth of > types for an interface within the Python language syntax itself (assuming > hacks such as specially formatted comments, strings or docstrings do not > count). Forward references to things that haven't even been defined yet are > common. You often want an ability to specify a duck type interface rather > than a specific type. I think he has those points covered better than I do. > I think mypy has solutions for the syntactic issues, and the rest can be addressed by introducing a few more magic helper functions. It's remarkably readable. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 07:24:12 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 22:24:12 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo wrote: > > > You could use AnyStr to make the example work with bytes as well: > > def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]: > result = {} #type: Dict[AnyStr, int] > > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result > > Again, if this is just a simple utility function that you use once or > twice, I see no reason to spend a lot of effort in coming up with the most > general signature. Types are an abstraction and they can't express > everything precisely -- there will always be a lot of cases where you can't > express the most general type. However, I think that relatively simple > types work well enough most of the time, and give the most bang for the > buck. > I heartily agree. But just for the type theorists amongst us, if I really wanted to write the most general type, how would I express that the AnyStr in the return type matches the one in the argument? (I think pytypedecl would use something like T <= AnyStr.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From haoyi.sg at gmail.com Thu Aug 14 07:39:14 2014 From: haoyi.sg at gmail.com (Haoyi Li) Date: Wed, 13 Aug 2014 22:39:14 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: > I heartily agree. But just for the type theorists amongst us, if I really wanted to write the most general type, how would I express that the AnyStr in the return type matches the one in the argument? (I think pytypedecl would use something like T <= AnyStr.) To borrow Scala syntax, it would look something like def word_count[AnyStr <: String](input: Iterable[AnyStr]): Dict[AnyStr, Int] Where *word_count* is a generic function on the type *AnyStr*, which is not just *any* type but a type bound by the restriction it is a subclass of *String*. Thus you can force that the *AnyStr* going in and the *AnyStr* going out are the same one. I'm not sure if mypy allows for type-bounds, but that's the way you achieve what you want if it does, or will, in future =P On Wed, Aug 13, 2014 at 10:24 PM, Guido van Rossum wrote: > On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo > wrote: >> >> >> You could use AnyStr to make the example work with bytes as well: >> >> def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]: >> result = {} #type: Dict[AnyStr, int] >> >> for line in input: >> for word in line.split(): >> result[word] = result.get(word, 0) + 1 >> return result >> >> Again, if this is just a simple utility function that you use once or >> twice, I see no reason to spend a lot of effort in coming up with the most >> general signature. Types are an abstraction and they can't express >> everything precisely -- there will always be a lot of cases where you can't >> express the most general type. However, I think that relatively simple >> types work well enough most of the time, and give the most bang for the >> buck. >> > > I heartily agree. But just for the type theorists amongst us, if I really > wanted to write the most general type, how would I express that the AnyStr > in the return type matches the one in the argument? (I think pytypedecl > would use something like T <= AnyStr.) > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Aug 14 07:44:13 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Aug 2014 22:44:13 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: I'd be more interested in Jukka's specific proposal. (Note that in mypy, AnyStr is the type that you call String here -- it's either bytes or str; and it seems that you're introducing AnyStr here as a type variable -- mypy conventionally uses T for this, but you have to define it first.) On Wed, Aug 13, 2014 at 10:39 PM, Haoyi Li wrote: > > I heartily agree. But just for the type theorists amongst us, if I > really wanted to write the most general type, how would I express that the > AnyStr in the return type matches the one in the argument? (I think > pytypedecl would use something like T <= AnyStr.) > > To borrow Scala syntax, it would look something like > > > def word_count[AnyStr <: String](input: Iterable[AnyStr]): Dict[AnyStr, Int] > > Where *word_count* is a generic function on the type *AnyStr*, which is > not just *any* type but a type bound by the restriction it is a subclass of > *String*. Thus you can force that the *AnyStr* going in and the *AnyStr* > going out are the same one. > > I'm not sure if mypy allows for type-bounds, but that's the way you > achieve what you want if it does, or will, in future =P > > > On Wed, Aug 13, 2014 at 10:24 PM, Guido van Rossum > wrote: > >> On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo >> wrote: >>> >>> >>> You could use AnyStr to make the example work with bytes as well: >>> >>> def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]: >>> result = {} #type: Dict[AnyStr, int] >>> >>> for line in input: >>> for word in line.split(): >>> result[word] = result.get(word, 0) + 1 >>> return result >>> >>> Again, if this is just a simple utility function that you use once or >>> twice, I see no reason to spend a lot of effort in coming up with the most >>> general signature. Types are an abstraction and they can't express >>> everything precisely -- there will always be a lot of cases where you can't >>> express the most general type. However, I think that relatively simple >>> types work well enough most of the time, and give the most bang for the >>> buck. >>> >> >> I heartily agree. But just for the type theorists amongst us, if I really >> wanted to write the most general type, how would I express that the AnyStr >> in the return type matches the one in the argument? (I think pytypedecl >> would use something like T <= AnyStr.) >> >> -- >> --Guido van Rossum (python.org/~guido) >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Aug 14 08:48:07 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 14 Aug 2014 18:48:07 +1200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <3C196232-D06D-4EFC-81BB-58FCBBF86DC2@yahoo.com> References: <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com> <53EC146D.3090402@canterbury.ac.nz> <3C196232-D06D-4EFC-81BB-58FCBBF86DC2@yahoo.com> Message-ID: <53EC5BA7.10904@canterbury.ac.nz> Andrew Barnert wrote: > If you go with a single JSONThing type that represents an object, array, > number, bool, string, or null, then it can't have a standard mapping > interface, because it also needs to have a standard sequence interface, I didn't mean that. The most Pythonic way would probably be to have something like JSONThing = Union(JSONDict, JSONList, Int, Float, Str, Bool) JSONDict = Mapping[str, JSONThing] JSONList = Sequence[JSONThing] If you're concerned about type safety, at some point you need to introspect on what you've got to figure out what to do with it. This is inevitable, since a JSON object is inherently a dynamically-typed thing. This is true even in Haskell, you just don't notice it so much because you do it with pattern matching. > The only > possibility is a union of all the various types mentioned above, and > such a union type has no interface at all. It's only useful if people > subvert the type safety by casting. I don't know what mypy does with union types, but if I were designing a type system like this I would say that, e.g. if i is know to be of type Int and d of type JSONDict, then i = d['foo'] should be allowed, on the grounds that the return value *could* be an Int, with a run-time type check to make sure that it actually is. An implicit cast, in other words. As long as mypy is just a linter it's obviously not in a position to insert the runtime check, but it could be argued that it should allow the assignment anyway, since Python will end up raising a TypeError at some point if it's wrong. I've forgotten what the original point of all this was. If the point was that there's no benefit in trying to make JSON type-safe in Python, and you should just leave it all dynamically typed, maybe you're right. -- Greg From abarnert at yahoo.com Thu Aug 14 08:45:02 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 13 Aug 2014 23:45:02 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: <6CB4D512-E476-4E3B-AB2B-501C5E03FD28@yahoo.com> On Aug 13, 2014, at 21:06, Jukka Lehtosalo wrote: > You could use AnyStr to make the example work with bytes as well: > > def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]: > result = {} #type: Dict[AnyStr, int] > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result Defining a function as taking an Iterable[AnyStr] and returning a Dict[AnyStr, int], without any way to declare that the two AnyStr are the same type, is exactly what I meant by losing critical information. I pass in something that I know is a text file, and I get out something that are either strings or bytes and I don't know which; what am I going to do with that? If you can't propagate types, static typing doesn't help. My point is that the depth you seem to be reaching for is not actually a sweet spot for power vs. simplicity. You could make it simpler by just dropping generics in favor of a handful of special-purpose modifiers (optional, iterable-of), or you could make it more useful by having real parametric types, or at least the equivalent of C++ template templates or Swift arbitrary constraints. What makes Java-style simple generics with subclass constraints the right level of complexity? From abarnert at yahoo.com Thu Aug 14 09:37:14 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 14 Aug 2014 00:37:14 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EC5BA7.10904@canterbury.ac.nz> References: <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com> <53EC146D.3090402@canterbury.ac.nz> <3C196232-D06D-4EFC-81BB-58FCBBF86DC2@yahoo.com> <53EC5BA7.10904@canterbury.ac.nz> Message-ID: <13C23919-EF10-4A7F-870F-8561B70C4E99@yahoo.com> On Aug 13, 2014, at 23:48, Greg Ewing wrote: > I've forgotten what the original point of all this was. > If the point was that there's no benefit in trying to > make JSON type-safe in Python, and you should just leave > it all dynamically typed, maybe you're right. Well, that was my starting point, which I assumed people would take for granted, so I could use it to make my real point, which is that "just downcast it", which came up a few times in the first couple hours of this discussion, is a bad answer in general. If you start from the assumption that everything can and should be statically typed, but don't design a type system powerful enough to do that, you end up putting downcasts from void* or Object or whatever all over the place, and it's impossible to tell what parts of the program are actually safe, which defeats the entire purpose of static typing. If you start from the assumption that there will be some self-contained regions of your code that can be typed by your type system, and other parts just have to be dynamically typed and that's OK, you don't subvert the type system all over the place, and you know which parts of the code are or aren't known to be safe. Anyway, I think this is basically what you just said--you started explaining how we could effectively add C++ style implicit casts to hide the ugliness and unsafety of dealing with JSON, but then said that maybe it would be better to just leave the values dynamically typed. As long as the type checker can (not necessarily in the initial version, but at least reasonably plausibly) make it easy to tell which areas of your code have been "infected" by dynamic types, I think defaulting to "too hard, leave it dynamic" is the simplest, and probably right, thing to do. (Unless we actually want to design a sufficiently powerful type system, which I don't think we do.) There's a great paper on how this works in practice in an ML variant, but unfortunately googling any subset of the only keywords I can remember (ML type duck infect dynamic) just brings up papers about avian virology (even if I leave our "duck"). I think there's also a (much more recent and less technical) blog post by one of the Swift guys about a similar idea--making it easy to stay dynamic means you end up with cleanly walled off regions of safe and unsafe code. (Or you would, if Swift had a halfway-decent stdlib instead of forcing you to bridge to ObjC for almost anything, but that's not important here.) From ncoghlan at gmail.com Thu Aug 14 10:13:37 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Aug 2014 18:13:37 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On 14 August 2014 15:11, Guido van Rossum wrote: > On Wed, Aug 13, 2014 at 6:09 PM, Gregory P. Smith wrote: >> >> At the summit in Montreal earlier this year ?ukasz Langa (cc'd) >> volunteered to lead writing the PEP on Python type hinting based on the many >> existing implementations of such things (including mypy, cython, numba and >> pytypedecl). I believe he has an initial draft he intends to send out soon. >> I'll let him speak to that. > > > Mypy has a lot more than an initial draft. Don't be mistaken by its status > as "one person's Ph.D. project" -- Jukka has been thinking about this topic > for a decade, and mypy works remarkably well already. It also has some very > active contributors already. FWIW, I was strongly encouraging folks at SciPy that were interested in static typing to look at mypy as an example of a potentially acceptable syntax for standardised optional static typing. Aside from being +1 on the general idea of picking *something* as "good enough" and iterating from there, I don't have a strong personal opinion, though. (And yes, my main interest is in improving the ability to do effective linting for larger projects. "pylint -E" is a lot better than nothing, but it has its limits in the absence of additional static hints) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Aug 14 10:16:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Aug 2014 18:16:01 +1000 Subject: [Python-ideas] float comparison in doctest In-Reply-To: References: Message-ID: On 14 August 2014 01:30, Erik Bray wrote: > > That said, if anyone has any ideas for allowing tweaking the > tolerances for a doctest flag that would be great. If this otherwise > seems like a good idea in general to include in the doctest module I > will offer a patch. I think it sounds like a reasonable idea, even with the "can't be tuned" limitation. It does become a possible use case *for* a doctest flag tuning system, but that doesn't need to be a requirement. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Aug 14 10:37:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Aug 2014 18:37:55 +1000 Subject: [Python-ideas] sum(...) limitation In-Reply-To: <20140813163729.GI4525@ando> References: <53E7CBA4.40105@g.nevcal.com> <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp> <-2448384566377912251@unknownmsgid> <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp> <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp> <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando> Message-ID: On 14 August 2014 02:37, Steven D'Aprano wrote: > A bit of history, as I remember it: sum() exists because for half of > Python's lifetime, people were regularly defining this: > > def sum(numbers): > return reduce(lambda a, b: a+b, numbers) > > so they could easy add up a bunch of numbers: > > num_pages = sum([ch.pages() for ch in self.chapters]) > > sort of thing. Since this was a common need, it was decided to add it to > the built-ins. But sum() was never intended to replace str.join or > list.extend, let alone even more exotic cases. > > Built-in sum is aimed at sequences of numbers, not strings, lists, > tuples, or Widgets for that matter. Perhaps giving it a start parameter > was a mistake, but it is there and backwards compatibility says it isn't > going to be removed. But that doesn't mean that the use of sum() on > arbitrary types ought to be *encouraged*, even if it is *allowed*. Interesting point of history: Adding the sum() builtin is when Alex Martelli was given commit privileges (see http://bugs.python.org/issue724936 - found by putting "builtin sum" into the search field on bugs.python.org) Reviewing that issues, in one of the draft patches, all sequences were banned, but that was changed in order to allow sum() to handle sequences that define __add__ as an element-wise operation (since those won't exhibit the quadratic behaviour). The date of the issue also made it possible to find the thread Alex mentions in the list archives: https://mail.python.org/pipermail/python-dev/2003-April/034767.html And there we find that the original patch *did* automatically delegate to str.join, and after an intial review that expressed some reservations (https://mail.python.org/pipermail/python-dev/2003-April/034825.html), Guido specifically requested changing it to disallow strings: https://mail.python.org/pipermail/python-dev/2003-April/034853.html And Alex agreed: https://mail.python.org/pipermail/python-dev/2003-April/034855.html The idea of a join() builtin was also discussed and rejected in that thread. Regards, Nick. P.S. Note that this kind of thread (which I contributed to myself) is why I'm inclined to insist on a PEP for *any* new builtin these days, even non-controversial ones: so the rationale for the associated design decision is less likely to be rehashed at length more than a decade later. Remembering "there was a PEP for that feature, and it listed this as a rejected design alternative" is relatively easy. Remembering "that idea came up in that thread back in 2003" is much, much, harder :P -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Aug 14 10:44:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Aug 2014 18:44:53 +1000 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> Message-ID: On 14 August 2014 07:09, Guido van Rossum wrote: > On Wed, Aug 13, 2014 at 1:18 PM, Raymond Hettinger > wrote: >> >> >> On Aug 13, 2014, at 12:44 PM, python-ideas-request at python.org wrote: >> >> The goal is to make it possible to add >> type checking annotations to 3rd party modules (and even to the stdlib) >> while allowing unaltered execution of the program by the (unmodified) >> Python 3.5 interpreter. >> >> >> Is the goal to "make it possible" or would it quickly become required >> (i.e. any time you write normal, readable Python, it would break >> someone's optimizer, refactorer, linter, etc.?) > > > Whoa, whoa. That's not at all the idea. Currently *nobody* uses type > annotations because there's no standard notation. My goal is to enable their > use by proposing a standard, nothing more. Given the general lack of familiarity with ABCs, I don't have any great fear of optional static typing becoming standard practice either. Just like ABCs, it's a useful way to formalise some checks when scaling up a code base to larger development teams. Most code bases aren't going to be that large, so it often isn't going to be worth the hassle. On the other hand, it's going to make it easier to detect certain kinds of structural errors more easily than even extensive unit testing, as well as helping out IDEs to provide better prompts to users, so I'm personally a fan of the idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Thu Aug 14 11:08:43 2014 From: christian at python.org (Christian Heimes) Date: Thu, 14 Aug 2014 11:08:43 +0200 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> Message-ID: <53EC7C9B.1000806@python.org> On 14.08.2014 02:59, Raymond Hettinger wrote: > P.S. I would really like for the annotations to grow some way > to communicate exceptions as well as return types (i.e. that > list.index can raise a ValueError and list.pop can raise an IndexError). > This would be only for the exceptions directly added by a function or > method, > not ones raised by the data (which is something the function can't control). +1 I made the same suggestion a couple of hours earlier. A standardized and introspectable way to add exception annotation is a common request. It's especially useful for C code because it's very hard to impossible to deduce exceptions from C code. For now there is no need for a new syntax. IMHO a decorator and standard location for exception annotations are sufficient. Perhaps somebody is able to come up with a syntax later. Such a decorator should also include the reason for an exception, too. For example: class list: @raises(IndexError, "when list is empty") def pop(self, index=None): pass @raises(ValueError) def index(self, value): pass Christian From ceronman at gmail.com Thu Aug 14 12:28:26 2014 From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=) Date: Thu, 14 Aug 2014 12:28:26 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Thu, Aug 14, 2014 at 1:24 AM, Guido van Rossum wrote: > On Wed, Aug 13, 2014 at 3:26 PM, Manuel Cer?n wrote: > >> The type checking algorithm might evolve over the time, but by including >> typing.py in the stdlib, the syntax for annotations would be almost frozen >> and that will be a limitation. In other projects such as TypeScript ( >> http://www.typescriptlang.org/), that the syntax usually evolves >> alongside the algorithms. >> > > What kind of evolution did TypeScript experience? > There is a nice summary of breaking changes here: https://typescript.codeplex.com/wikipage?title=Known%20breaking%20changes%20between%200.8%20and%200.9 Most of these are in the way that the type checking engine works, but there are some syntax changes. While the basic types have remained stable, some special things have not. For example: generics, optional and default arguments, interfaces. I think it'd be valuable to learn from TypeScript as much as possible, it's the only project that I know that It's trying to bring static type analysis to a widely used dynamically typed language. One interesting feature of TypeScript is that it allows you to annotate existing code without modifying it, by using external definition files. In the JavaScript world, many people have contributed TypeScript annotation files for popular JS libraries (http://definitelytyped.org/). I think this is possible in Python as well doing something like this: @annotate('math.ciel') def ciel(x: float) -> int: pass I think this should be the way to go for annotating the stdlib. It has the advantage that if the type syntax changes, it's possible to provide new type annotations without changing the libraries at all, and even supporting older versions. In this way the code and type annotations can evolve separately. Does mypy support something like this? Manuel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Thu Aug 14 12:27:20 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 14 Aug 2014 05:57:20 -0430 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: <53EC7C9B.1000806@python.org> References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> <53EC7C9B.1000806@python.org> Message-ID: On Thu, Aug 14, 2014 at 4:38 AM, Christian Heimes wrote: > I made the same suggestion a couple of hours earlier. A standardized and > introspectable way to add exception annotation is a common request. It's > especially useful for C code because it's very hard to impossible to > deduce exceptions from C code. > > For now there is no need for a new syntax. IMHO a decorator and standard > location for exception annotations are sufficient. Perhaps somebody is > able to come up with a syntax later. > The problem with declaring raisable exceptions is that it invites static verification of callers doing "the right thing" with them, as in Java, and that introduces noise in the software development process that is almost impossible to clean up. @raises(ValueError) def index(self, value): pass def found(self, value): return self.index(value) >= 0 Is found() correct? If not, I assure you the most common (and probably incorrect) solution will be: def found(self, value): try: return self.index(value) >= 0 except ValueError: return False Or worse: def found(self, value): try: return self.index(value) >= 0 except: return False The right thing to do with an exception is to let it through, unless you know exactly what you have to do about it. In this interview, Anders Hejlsberg talks about why checked exceptions were left out of the design of C#: http://www.artima.com/intv/handcuffs.html Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From willvarfar at gmail.com Thu Aug 14 13:16:11 2014 From: willvarfar at gmail.com (willvarfar at gmail.com) Date: Thu, 14 Aug 2014 13:16:11 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations Message-ID: I fully support formalizing Python 3's annotations for type checking. I wrote - and use daily - my own type checker called obiwan https://pypi.python.org/pypi/obiwan Its a runtime type checker, and if enabled will check and enforce types on every call. I support the wider adoption and standardization of static type checkers, but runtime checkers are still wanted for very dynamic code and for checking external data e.g. I use obiwan for validating JSON. One small detail is that I feel the obiwan annotations are more pythonic than the mypy examples given. E.g. instead of: from typing import List, Dict def word_count(input: List[str]) -> Dict[str, int]: ... It would be: def word_count(input: [str]) -> {str, int}: ... Obiwan does not check types within functions; I was unwilling to try and overload comments! You can invoke obiwan to check things explicitly, but these are more as assertions. Anyway, when I look at the mypy in-function annotations (where comments are overloaded) I am cautious. It would be far nicer if we had annotations as part of the language instead, e.g. instead of: result = {} #type: Dict[str, int] It would be: result = {} -> {str, int} where we use the -> arrow again. I can see pros and cons for any implementation (as we'd want the annotation to be both to declare a type and to check a type, and want the annotation to be attached to the variable forever etc) so this would need a full PEP treatment and possibly be configurable as asserts are. But proper annotation support in the language rather than overloading comments would definitely be my preference. /Will /Will From andrey.vlasovskikh at gmail.com Thu Aug 14 13:36:26 2014 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Thu, 14 Aug 2014 15:36:26 +0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Thu, Aug 14, 2014 at 2:28 PM, Manuel Cer?n wrote: > One interesting feature of TypeScript is that it allows you to annotate > existing code without modifying it, by using external definition files. In > the JavaScript world, many people have contributed TypeScript annotation > files for popular JS libraries (http://definitelytyped.org/). > > I think this is possible in Python as well doing something like this: > > @annotate('math.ciel') > def ciel(x: float) -> int: > pass > > I think this should be the way to go for annotating the stdlib. It has the > advantage that if the type syntax changes, it's possible to provide new type > annotations without changing the libraries at all, and even supporting older > versions. In this way the code and type annotations can evolve separately. > > Does mypy support something like this? We use something quite similar to TypeScript's repository of annotations in PyCharm. Here is our external annotations proposal and stubs for some stdlib modules (https://github.com/JetBrains/python-skeletons) that uses a custom type syntax in docstrings due to lack of a better standard, see README. We state in our proposal that we would like the standard to emerge. The idea being discussed here about using Mypy's type system is not new to us. As I've mentioned in the original thread, we have discussed it with Jukka Lehtosalo, the author of Mypy. Some initial ideas are listed here (https://github.com/pytypes/pytypes). -- Andrey Vlasovskikh Web: http://pirx.ru/ From christian at python.org Thu Aug 14 13:38:45 2014 From: christian at python.org (Christian Heimes) Date: Thu, 14 Aug 2014 13:38:45 +0200 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> <53EC7C9B.1000806@python.org> Message-ID: <53EC9FC5.2010701@python.org> On 14.08.2014 12:27, Juancarlo A?ez wrote: > The problem with declaring raisable exceptions is that it invites > static verification of callers doing "the right thing" with them, > as in Java, and that introduces noise in the software development > process that is almost impossible to clean up. [snip] > The right thing to do with an exception is to let it through, > unless you know exactly what you have to do about it. > > In this interview, Anders Hejlsberg talks about why checked > exceptions were left out of the design of C#: Thanks for your input, Juancarlo. I don't think that anybody wants to add checked exceptions to Python at this point. It's one of my pain points with Java, too. As far as I can judge Raymond we both want to archive the same goal: improve introspection and self-documentation of code with a strong focus on bindings (e.g. C code). As I tried to explain earlier neither users nor tools are able to analyze code like builtin types and functions. Exception annotations could help IDEs to figure out what kind of exception could be raised. Does it sound better to you? The general idea is remotely related to mypy and should be addressed by a different PEP that accompanies the mypy PEP. Regards, Christian From kaiser.yann at gmail.com Thu Aug 14 15:59:33 2014 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Thu, 14 Aug 2014 15:59:33 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: I'd like to offer my perspective(and objections) on this: I author and maintain a reflective argument parser for python, clize[1], possibly in the same vein as Ethan's. While it originally did not support any Python 3 features, it has come to accept keyword-only parameters and annotations for alternate parameter names and conversion functions(which may look like type annotations to a reader looking for them). The upcoming version will completely deprecate the old "pass a bunch of unsightly arguments to a decorators[2]" API, and use annotations exclusively[3], using backports[4][5] to make them and keyword-only parameters available to Python 2 users. Clize has a handful of users, with 100 stars on GitHub, and ~75 downloads per day according to PyPI[1]. Needless to say, the idea of deprecating uses of parameter and return annotations that aren't this particular way of typechecking is upsetting to me. On to the most formal of formal complaints: PEP 3107[6] rejected standardizing typing-related annotations. What has changed since then, or what has otherwise invalidated that conclusion, that it should be backtracked on and changed? GvR says he knows of little practical use of function annotations in mainstream code. Here's the (unfortunate) reason: Mainstream code is either incompatible with Python 3 or maintaining compatibility between both Python 2 and Python 3. Standard library code which had no such concern for the most part stuck with what stood in PEP 3107, which was not to teach annotations as typing info. Short of resorting to tools like sigtools.modifiers.annotate[4], *mainstream code can't use function annotations*. You will argue that people are already coming up with type-checking libraries en masse, and that little alternative uses have come up. That's because type-checking is an idea that people have seen and had in mind well before now, PEP 3107, or Python itself. It is even one of the two examples in the PEP, the other being fairly irrelevant given docstrings, IMO. Mainstream code can't use annotations. New brains are mostly taught to use python 2[7] or keep compatibility with it, so no new ideas there(save for the endless stream of "I've used Python for a day and it needs static typing" coming from statically-typed languages.) IMO there simply hasn't been enough time for new uses to show up. There's the argument parsers me and a couple others have mentioned, and although the idea can be ported to other interfaces(web forms, GUI forms?), this is only one use yet, but I think it is a good one, one that is also fairly unique to Python. Do we want to close this window of opportunity by trying to imitate the 80's? Even if you don't deprecate other uses, putting this in the standard library sends a message to the mob: "Annotations are meant to express types and other uses should conform." It's already slightly against-current to propose reflective tools because "reflection is weird and unreliable," I'd hate to see where this would take us. Worse than shutting down a potential area of innovation, promoting type-checking in such a way would alienate what many experienced Python programmers view as a core tenet of Python: duck-typing. Ironically, GvR's example illustrates it perfectly: from typing import List, Dict def word_count(input: List[str]) -> Dict[str, int]: result = {} #type: Dict[str, int] for line in input: for word in line.split(): result[word] = result.get(word, 0) + 1 return result First, there's the obvious mistake others have noted: this thing does not need a list, it needs an iterable, and that iterable merely needs to yield objects that have a split method, which yields... hashable objects. That's it. This function could work just as well if line.split yielded integers. It's what's awesome about things like collections.Counter. You can give it whatever iterable you like, whether it yields characters, strings, integers, the types of objects tracked by the garbage collector, you name it. This is awesome. Declaring and casting between type templates is confusing, verbose, scary and not awesome in general. It just makes me think of C++ (sorry). Let's say that this function needs to operate on strings, because at some point in the future it will replace line.split() with something that accounts for punctuation and whatnot, and ignore that this would really mean it actually became a different function. (Do you expect str.split to begin handling punctuation overnight?) Now the only mistake is that it pretends it is specific to lists. This mistake is easily realized, so what happens when you write your fancy library that fully declares its typing, declaring generic interfaces where appropriate, and one of your dependencies finally makes the switch over and gets it wrong? Now your typechecker says your functions are making incorrect calls to that API because you passed the result of a generator function to something that expects a list. How often do you think this will happen? How many people eager to type-check everything will not know how duck-typing matters in their library or program and specify overly restrictive interfaces for their functions? How many will assume checking declared types is a substitute for unit testing? This is speculative, but being a regular participant of #python on FreeNode, I already dread this. Finally, if I abstract this proposal, it boils down to: The proposal wants to take what has been so far a fully free-for-all attribute of functions, and make it free-for-all-but-only-for-this-purpose. That's a bit... weird? Maybe the mypy project could split its declaration and checking components, sort of like I split clize(CLI argument parsing) and sigtools(high-level signature operations and reflection improvements)? Isn't the way those libraries declare types-to-be-checked the main way they will distinguish each other for developers? Is there a problem preventing IDEs from implementing a runner for mypy, much like eg. the way you can use pyflakes as your compiler in vim? Sorry for the long rant. [1] Clize on PyPI: https://pypi.python.org/pypi/clize [2] Old clize API: https://github.com/epsy/clize/blob/e84637a631574e793719114ad7e40d0b36df1a78/README.rst#using-clize-in-your-programs [3] New clize API involving annotations: http://clize.readthedocs.org/en/latest/basics.html#converting-arguments [4] Inspect.signature-compatible backport of annotations: http://sigtools.readthedocs.org/en/latest/#sigtools.modifiers.annotate [5] Backport of inspect.signature: https://pypi.python.org/pypi/funcsigs/0.4 [6] Rejected proposals from Function Annotations PEP: http://legacy.python.org/dev/peps/pep-3107/#rejected-proposals [7] Warnings for Beginners - Learn Python the Hard Way: http://learnpythonthehardway.org/book/ex0.html#warnings-for-beginners -Yann Kaiser On 14 August 2014 13:36, Andrey Vlasovskikh wrote: > On Thu, Aug 14, 2014 at 2:28 PM, Manuel Cer?n wrote: > > > One interesting feature of TypeScript is that it allows you to annotate > > existing code without modifying it, by using external definition files. > In > > the JavaScript world, many people have contributed TypeScript annotation > > files for popular JS libraries (http://definitelytyped.org/). > > > > I think this is possible in Python as well doing something like this: > > > > @annotate('math.ciel') > > def ciel(x: float) -> int: > > pass > > > > I think this should be the way to go for annotating the stdlib. It has > the > > advantage that if the type syntax changes, it's possible to provide new > type > > annotations without changing the libraries at all, and even supporting > older > > versions. In this way the code and type annotations can evolve > separately. > > > > Does mypy support something like this? > > We use something quite similar to TypeScript's repository of > annotations in PyCharm. Here is our external annotations proposal and > stubs for some stdlib modules > (https://github.com/JetBrains/python-skeletons) that uses a custom > type syntax in docstrings due to lack of a better standard, see > README. We state in our proposal that we would like the standard to > emerge. The idea being discussed here about using Mypy's type system > is not new to us. As I've mentioned in the original thread, we have > discussed it with Jukka Lehtosalo, the author of Mypy. Some initial > ideas are listed here (https://github.com/pytypes/pytypes). > > -- > Andrey Vlasovskikh > Web: http://pirx.ru/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Thu Aug 14 16:10:51 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 14 Aug 2014 09:40:51 -0430 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: <53EC9FC5.2010701@python.org> References: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com> <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> <53EC7C9B.1000806@python.org> <53EC9FC5.2010701@python.org> Message-ID: On Thu, Aug 14, 2014 at 7:08 AM, Christian Heimes wrote: > As I tried to explain earlier neither users nor tools are able to > analyze code like builtin types and functions. Exception annotations > could help IDEs to figure out what kind of exception could be raised. > Does it sound better to you? > It does, but it remains that annotations about exceptions will probably be inconsistent or incomplete: 1. If the function is changed, the annotation will remain valid even if the mentioned exceptions are not raised (this is the typical doc-comment problem). 2. The annotation cannot be validated unless the raisables of called functions (objects, actually) are analysed. Dealing with #2 means going the Java way and promoting incorrect exception handlers throughout. Not dealing with #2 means that the annotations are for documentation purposes only, so the same information can go in the doc-comment. This situation is unlike that for the proposed type signatures, because the intent for those is that they are strictly verified (with mypy or some other). In short, an exceptions signature would amount to: "This function may or may not raise the listed exceptions, and it may also raise other exceptions not in the list.". Yet, #2 can be dealt with by not being strict, and issuing warnings when things seem broken, taking note that the Python callables that can raise exceptions are not only functions and methods, but anything for which callable(o)==True. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Thu Aug 14 16:20:44 2014 From: antoine at python.org (Antoine Pitrou) Date: Thu, 14 Aug 2014 10:20:44 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: Hi, So a couple more comments: - there seems to be no detailed description of mypy's typing.py. There are some docstrings in the module but they don't really explain the intended semantics of using those types (and, most importantly, combining and extending them). Or is the tutorial the description? - mypy is a young module; is it actually proven on real-world codebases? if a PEP gets written, typing.py should be proposed on the "provisional" basis as described in PEP 411 - for the two reasons above I would expect a PEP to spell out and somehow motivate the typing semantics (by "typing" I do not mean the actual algorithm of a type checker, but the semantics of combining types and the mechanisms for extension), rather than defer to third-party docs and source code - many discussion topics seems to revolve around the syntax for declarations; besides syntactic sugar for declarations, though, there should probably be a hook to describe typing in a more programmatic way, especially when parametric types ("generic"?) are involved. I can't think of an example, so I realize this comment may not be very helpful (!), but I'm sure we'll hit the limits of what declarative syntax can give us, especially once people try it out on non-trivial code bases. Do such hooks already exist? - the typing module apparently has its own brand of generic functions (with multiple dispatch?). Are they actually necessary for typing declarations and type inference? If so, it would be nice if this could be unified with functools.singledispatch, or at least put in the functools namespace, and sufficiently similar API-wise that it doesn't stand out. - it would be nice if the PEP could discuss at least a little bit the performance expectations of using the type system, and if there are design decisions that can have a (positive or negative) performance impact on the algorithms using it. I realize this is not top-priority for the linting use case, but it's still important (especially for linting large code bases, which is one of the proclaimed use cases). (this reminds me how annoyed I am at the slowness of Sphinx on non-small reST docs) Regards Antoine. From rymg19 at gmail.com Thu Aug 14 16:37:56 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 14 Aug 2014 09:37:56 -0500 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 9:27 PM, Terry Reedy wrote: > Guido, as requesting, I read your whole post before replying. Please to > the same. This response is both critical and supportive. > > > On 8/13/2014 3:44 PM, Guido van Rossum wrote: > > Yesterday afternoon I had an inspiring conversation with Bob Ippolito >> (man of many trades, author of simplejson) and Jukka Lehtosalo (author >> of mypy: http://mypy-lang.org/). >> > > My main concern with static typing is that it tends to be > anti-duck-typing, while I consider duck-typing to be a major *feature* of > Python. The example in the page above is "def fib(n: int):". Fib should > get an count (non-negative integer) value, but it need not be an int, and > 'half' the ints do not qualify. Reading the tutorial, I could not tell if > it supports numbers.Number (which should approximate the domain from above.) > Very true; this is one of my fears. There are plenty of people who adore static typing and will make everything in all their libraries static with static this and static that. Maybe there could be a way to disable type checks? > > Now consider an extended version (after Lucas). > > def fib(n, a, b): > i = 0 > while i <= n: > print(i,a) > i += 1 > a, b = b, a+b > > The only requirement of a, b is that they be addable. Any numbers should > be allowed, as in fib(10, 1, 1+1j), but so should fib(5, '0', '1'). Addable > would be approximated from below by Union(Number, str). > > Unless MyPy added some sort of type classes... > > Bob gave a talk at EuroPython about >> what Python can learn from Haskell (and other languages); yesterday he >> gave the same talk at Dropbox. The talk is online >> (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad >> strokes comes down to three suggestions: >> >> (a) Python should adopt mypy's syntax for function annotations >> > > -+ Syntax with no meaning is a bit strange. On the other hand, syntax not > bound to semantics, or at least not bound to just one meaning is quite > pythonic. '+' has two standard meanings, plus custom meanings embodied in > .__add__ methods. > > + The current semantics of annotations is that they are added to functions > objects as .__annotations__ (for whatever use) *and* used as part of > inspect.signature and included in help(ob) responses. In other words, > annotations are already used in the stdlib. > > >>> def f(i:int) -> float: pass > > >>> from inspect import signature as sig > >>> str(sig(f)) > '(i:int) -> float' > >>> help(f) > Help on function f in module __main__: > > f(i:int) -> float > > Idle calltips include them also. A appropriately flexible standardized > notation would enhance this usage and many others. > > +-+ I see the point of "The goal is to make it possible to add type > checking annotations to 3rd party modules (and even to the stdlib) while > allowing unaltered execution of the program by the (unmodified) Python 3.5 > interpreter." On the other hand, "pip install mypytyping" is not a huge > burden. On the third hand, in the stdlib allows use in the stdlib. > > (b) Python's use of mutabe [mutable] containers by default is wrong >> > > The premise of this is partly wrong and partly obsolete. As far as I can > remember, Python *syntax* only use tuples, not lists: "except (ex1, ex2):", > "s % (val1, val2)", etc. The use of lists as the common format for data > interchange between functions has largely been replaced by iterators. This > fact makes Python code much more generic, and anti-generic static typing > more wrong. > > In remaining cases, 'wrong' is as much a philosophical opinion as a fact. > > > (c) Python should adopt some kind of Abstract Data Types >> > > I would have to look at the talk to know what Jukka means. > > > Proposals (b) and (c) don't feel particularly actionable (if you >> disagree please start a new thread, I'd be happy to discuss these >> further if there's interest) but proposal (a) feels right to me. >> > > So what is mypy? It is a static type checker for Python written by >> Jukka for his Ph.D. thesis. The basic idea is that you add type >> annotations to your program using some custom syntax, and when running >> your program using the mypy interpreter, type errors will be found >> during compilation (i.e., before the program starts running). >> >> The clever thing here is that the custom syntax is actually valid Python >> 3, using (mostly) function annotations: your annotated program will >> still run with the regular Python 3 interpreter. In the latter case >> there will be no type checking, and no runtime overhead, except to >> evaluate the function annotations (which are evaluated at function >> definition time but don't have any effect when the function is called). >> >> In fact, it is probably more useful to think of mypy as a heavy-duty >> linter than as a compiler or interpreter; leave the type checking to >> mypy, and the execution to Python. It is easy to integrate mypy into a >> continuous integration setup, for example. >> >> To read up on mypy's annotation syntax, please see the mypy-lang.org >> website. >> > > I did not see a 'reference' page, but the tutorial comes pretty close. > http://mypy-lang.org/tutorial.html > Beyond that, typings.py would be definitive, > https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py > > > > Here's just one complete example, to give a flavor: > > from typing import List, Dict >> >> def word_count(input: List[str]) -> Dict[str, int]: >> > > The input annotation should be Iterable[str], which mypy does have. > > > result = {} #type: Dict[str, int] >> for line in input: >> for word in line.split(): >> result[word] = result.get(word, 0) + 1 >> return result >> > > The information that input is an Iterable[str] can be used either within > the definition of word_count or at places where word_count is called. A > type aware checker, either in the editor or compiler, could check that the > only uses of 'input' within the function is as input to functions declared > to accept an Iterable or in for statements. > > Checking that the input to word_count is specifically Iterable[str] as > opposed to any other Iterable may not be possible. But I think what can be > done, including enhancing help information, might be worth it. > > For instance, the parameter to s.join is named 'iterable'. Something more > specific, either 'iterable_of_strings' or 'strings: Iterable[str]' would be > more helpful. Indeed, there have been people posting on python list who > thought that 'iterable' means iterable and that .join would call str() on > each object. I think there are other cases where a parameter is given a > bland under-informative type name instead of a context-specific semantic > name just because there was no type annotation available. There are places > where the opposite problem occurs, too specific instead of too general, > where iterable parameters are still called 'list'. > > Note that the #type: comment is part of the mypy syntax; mypy uses >> comments to declare types in situations where no syntax is available -- >> although this particular line could also be written as follows: >> >> result = Dict[str, int]() >> >> Either way the entire function is syntactically valid Python 3, and a >> suitable implementation of typing.py (containing class definitions for >> List and Dict, for example) can be written to make the program run >> correctly. One is provided as part of the mypy project. >> >> I should add that many of mypy's syntactic choices aren't actually new. >> The basis of many of its ideas go back at least a decade: I blogged >> about this topic in 2004 >> (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the >> two followup posts linked from the top there). >> >> I'll emphasize once more that mypy's type checking happens in a separate >> pass: no type checking happens at run time (other than what the >> interpreter already does, like raising TypeError on expressions like >> 1+"1"). >> >> There's a lot to this proposal, but I think it's possible to get a PEP >> written, accepted and implemented in time for Python 3.5, if people are >> supportive. I'll go briefly over some of the action items. >> >> *(1) A change of direction for function annotations* >> >> PEP 3107 , which introduced >> >> function annotations, is intentional non-committal about how function >> annotations should be used. It lists a number of use cases, including >> but not limited to type checking. It also mentions some rejected >> proposals that would have standardized either a syntax for indicating >> types and/or a way for multiple frameworks to attach different >> annotations to the same function. AFAIK in practice there is little use >> of function annotations in mainstream code, and I propose a conscious >> change of course here by stating that annotations should be used to >> indicate types and to propose a standard notation for them. >> > > There are many uses for type information and I think Python should remain > neutral among them. > > > (We may have to have some backwards compatibility provision to avoid >> breaking code that currently uses annotations for some other purpose. >> Fortunately the only issue, at least initially, will be that when >> running mypy to type check such code it will produce complaints about >> the annotations; it will not affect how such code is executed by the >> Python interpreter. Nevertheless, it would be good to deprecate such >> alternative uses of annotations.) >> > > I can imagine that people who have used annotations might feel a bit > betrayed by deprecation of a new-in-py3 feature. But I do not think it > necessary to do so. Tools that work with mypy annotations, including mypy > itself, should only assume mypy typing if typing is imported. No 'import > typing', no 'Warning: annotation does not follow typing rules." If > 'typing' were a package with a 'mypy' module, the door would be left open > to other 'blessed' typing modules. > > *(2) A specification for what to add to Python 3.5* >> >> >> There needs to be at least a rough consensus on the syntax for >> annotations, and the syntax must cover a large enough set of use cases >> to be useful. Mypy is still under development, and some of its features >> are still evolving (e.g. unions were only added a few weeks ago). It >> would be possible to argue endlessly about details of the notation, e.g. >> whether to use 'list' or 'List', what either of those means (is a >> duck-typed list-like type acceptable?) or how to declare and use type >> variables, and what to do with functions that have no annotations at all >> (mypy currently skips those completely). >> >> I am proposing that we adopt whatever mypy uses here, keeping discussion >> of the details (mostly) out of the PEP. The goal is to make it possible >> to add type checking annotations to 3rd party modules (and even to the >> stdlib) while allowing unaltered execution of the program by the >> (unmodified) Python 3.5 interpreter. The actual type checker will not be >> integrated with the Python interpreter, and it will not be checked into >> the CPython repository. The only thing that needs to be added to the >> stdlib is a copy of mypy's typing.py module. This module defines several >> dozen new classes (and a few decorators and other helpers) that can be >> used in expressing argument types. If you want to type-check your code >> you have to download and install mypy and run it separately. >> >> The curious thing here is that while standardizing a syntax for type >> annotations, we technically still won't be adopting standard rules for >> type checking. >> > > Fine with me, as that is not the only use. And even for type checking, > there is the choice between accept unless clearly wrong, versus reject > unless clearly right. > > > > This is intentional. First of all, fully specifying all > >> the type checking rules would make for a really long and boring PEP (a >> much better specification would probably be the mypy source code). >> Second, I think it's fine if the type checking algorithm evolves over >> time, or if variations emerge. >> > > As in the choice between accept unless clearly wrong, versus reject unless > clearly right. > > > > The worst that can happen is that you > >> consider your code correct but mypy disagrees; your code will still run. >> >> That said, I don't want to /completely/ leave out any specification. I >> >> want the contents of the typing.py module to be specified in the PEP, so >> that it can be used with confidence. But whether mypy will complain >> about your particular form of duck typing doesn't have to be specified >> by the PEP. Perhaps as mypy evolves it will take options to tell it how >> to handle certain edge cases. Forks of mypy (or entirely different >> implementations of type checking based on the same annotation syntax) >> are also a possibility. Maybe in the distant future a version of Python >> will take a different stance, once we have more experience with how this >> works out in practice, but for Python 3.5 I want to restrict the scope >> of the upheaval. >> > > As usual, we should review the code before acceptance. It is not clear to > me how much of the tutorial is implemented, as it says "Some of these > features might never see the light of day. " ??? > > *Appendix -- Why Add Type Annotations? >> >> * >> The argument between proponents of static typing and dynamic typing has >> been going on for many decades. Neither side is all wrong or all right. >> Python has traditionally fallen in the camp of extremely dynamic typing, >> and this has worked well for most users, but there are definitely some >> areas where adding type annotations would help. >> > > The answer to why on the mypy page is 'easier to find bugs', 'easier > maintenance'. I find this under-convincing as sufficient justification in > itself. I don't think there are many bugs on the tracker due to calling > functions with the wrong type of object. Logic errors, ignored corner > cases, and system idiosyncrasies are much more of a problem. > > Your broader list is more convincing. > > > - Editors (IDEs) can benefit from type annotations; they can call out >> obvious mistakes (like misspelled method names or inapplicable >> operations) and suggest possible method names. Anyone who has used >> IntelliJ or Xcode will recognize how powerful these features are, and >> type annotations will make such features more useful when editing Python >> source code. >> >> - Linters are an important tool for teams developing software. A linter >> doesn't replace a unittest, but can find certain types of errors better >> or quicker. The kind of type checking offered by mypy works much like a >> linter, and has similar benefits; but it can find problems that are >> beyond the capabilities of most linters. >> > > Currently, Python linters do not have standard type annotations to work > with. I suspect that programs other than mypy would use them if available. > > > - Type annotations are useful for the human reader as well! Take the >> above word_count() example. How long would it have taken you to figure >> out the types of the argument and return value without annotations? >> > > Under a minute, including the fact the the annotation was overly > restrictive. But then I already know that only a mutation method can > require a list. > > Currently most people put the types in their docstrings; developing a >> standard notation for type annotations will reduce the amount of >> documentation that needs to be written, and running the type checker >> might find bugs in the documentation, too. Once a standard type >> annotation syntax is introduced, it should be simple to add support for >> this notation to documentation generators like Sphinx. >> >> - Refactoring. Bob's talk has a convincing example of how type >> annotations help in (manually) refactoring code. I also expect that >> certain automatic refactorings will benefit from type annotations -- >> imagine a tool like 2to3 (but used for some other transformation) >> augmented by type annotations, so it will know whether e.g. x.keys() is >> referring to the keys of a dictionary or not. >> >> - Optimizers. I believe this is actually the least important >> application, certainly initially. Optimizers like PyPy or Pyston >> wouldn't be able to fully trust the >> >> type annotations, and they are better off using their current strategy >> of optimizing code based on the types actually observed at run time. But >> it's certainly feasible to imagine a future optimizer also taking type >> annotations into account. >> > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Thu Aug 14 16:46:52 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Thu, 14 Aug 2014 15:46:52 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On 14/08/2014 14:59, Yann Kaiser wrote: > New brains are mostly taught to use python 2[7] or keep compatibility with it, > [7] Warnings for Beginners - Learn Python the Hard Way: > http://learnpythonthehardway.org/book/ex0.html#warnings-for-beginners > > -Yann Kaiser This is the second time in a few days that I've seen a reference to that and IMHO it's simply plain wrong and should be ignored. Just my ?0.02p worth. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From encukou at gmail.com Thu Aug 14 16:52:33 2014 From: encukou at gmail.com (Petr Viktorin) Date: Thu, 14 Aug 2014 16:52:33 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: It seems to me that rather than adding a module which is only used by one project so far to the standard library is a bit premature. I support optional typing, but why push this to stdlib now? Wouldn't it be better to wait until most IDEs/linters all agree on this syntax, until freezing it in stdlib? So far typing seems to be a part of mypy, shouldn't it spend some time on PyPI first? I'm also sure about there not being other uses of annotations -- clize aside, there are not many widely used Python3-only 3rd party libraries, so it's no surprise that nothing big is built around Python 3 features. Maybe the way from PEP 3107's "here's a feature, use it for whatever you like" to "annotations are for typing declarations, using mypy/typing syntax" should include a step of "if you use annotations for typing, use mypy/typing syntax for it". (And perhaps it should end there.) From apalala at gmail.com Thu Aug 14 16:59:54 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 14 Aug 2014 10:29:54 -0430 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 3:14 PM, Guido van Rossum wrote: > I am proposing that we adopt whatever mypy uses here, keeping discussion > of the details (mostly) out of the PEP. The goal is to make it possible to > add type checking annotations to 3rd party modules (and even to the stdlib) > while allowing unaltered execution of the program by the (unmodified) > Python 3.5 interpreter. To the bottom of things... About the second time I wrote about Python ("Why not Python", 2007) I dismissed it as serious software development environment because the lack of static type checking hindered the creation of proper software development environments. http://blog.neogeny.org/why-not-python.html So, Why do I now have doubts about adding support for static type checking? I've been programming in almost-only Python for several years now, and this discussion had me think hard about "Why?". The answer is simple: I never was as productive as I've been since I've centered on Python. But, again Why? Despite what my '07 article says, the IDE I use is pythonized-VIM and the command line. Where does the productivity come from? 1. Readability with the right amount of succinctness. Python programs are very small, but understandable. 2. The breadth and design consistency of the standard library. Some 70%? of what I need is there, and the design consistency makes it easy (intiutive) to use. 3. PyPi covers another 28%. 4. The Zen of Python (import this) permeates all of the above, including most third-party packages. The ecosystem is consistent too. It's a culture. What do I fear? I think it is that Python be transformed into a programming language different from the one that now makes me so productive. I studied Ruby, and I don't like it. I've been studying Go, and I don't like it. One must like the concepts and the power, sure, but the syntax required for some day-to-day stuff stinks like trouble; simple stuff is so complicated to express and so easy to get wrong... I hate "List[str]" and "Dict[str:int]". Where did those come from? Shouldn't they (as others have proposed) be "[str]" and "{str:int}"? What about tuples? Why not write a similar, but different programming language that targets the Cython runtime and includes all the desired features? All said, this is my proposal. The PSF could support (even fund) MyPy and similar projects, promoting their maturity and their convergence. The changes in 3.5 would be limited but enough to enable those efforts, and those of the several IDE tool-smiths (changes in annotations, and maybe in ABCs). Basically, treat MyPy as PyPy or NumPy (which got '::'). It's in Python's history to enable third-party developments and then adopt what's mature or become the de-facto standard. Then, on a separate line of work, it would be good to think about how to enable different programming languages to target the CPython environment (because of #2, #3, and #4 above), maybe by improving AST creation and AST-to-bytecode? There could be other languages targeting the CPython runtime, which is the relationship that Scala, Jython, IronPython, and others have to their own runtimes. -1 for standardizing static type checking in 3.5 Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Aug 14 17:21:40 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 14 Aug 2014 08:21:40 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Aug 14, 2014, at 7:37, Ryan Gonzalez wrote: >> On 8/13/2014 3:44 PM, Guido van Rossum wrote: >> Now consider an extended version (after Lucas). >> >> def fib(n, a, b): >> i = 0 >> while i <= n: >> print(i,a) >> i += 1 >> a, b = b, a+b >> >> The only requirement of a, b is that they be addable. Any numbers should be allowed, as in fib(10, 1, 1+1j), but so should fib(5, '0', '1'). Addable would be approximated from below by Union(Number, str). > > Unless MyPy added some sort of type classes... By "type classes", do you mean this in the Haskell sense, or do you mean classes used just for typing--whether more granular ABCs (like an Addable which both Number and AnyStr and probably inherit) or typing.py type specifiers (like an Addable defined as Union(Number, AnyStr)? It's also worth noting that the idea that this function should take a Number or a str seems way off. It's questionable whether it should accept str, but if it does, shouldn't it also accept bytes, bytearray, and other string-like types? What about sequences? And meanwhile, whether or not it accepts str, it should probably accept np.ndarray and other types of element-wise adding types. If you create an Addable type, it has to define, globally, which of those counts as addable, but different functions will have different definitions that make sense. In fact, look at the other discussion going on. People want to ensure that sum only works on numbers or number-like types (and does that include NumPy arrays or not?), while others want to change it to work on all sequences, or only mutable sequences with += plus str because it effectively has magical += handling under the covers, etc. If we can't even decide what Addable means for one specific function that everyone has experience with... On the other hand, if sum could have been annotated to tell us the author's intention (or, rather, the consensus of the dev list), then all of these recurring arguments about summing str would go away. Until someone defined a number-like class (maybe even one that meets the ABC, but by calling register on it) and sum won't work for him. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Aug 14 18:01:08 2014 From: brett at python.org (Brett Cannon) Date: Thu, 14 Aug 2014 16:01:08 +0000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: Message-ID: On Thu Aug 14 2014 at 10:53:37 AM Petr Viktorin wrote: > It seems to me that rather than adding a module which is only used by > one project so far to the standard library is a bit premature. > > I support optional typing, but why push this to stdlib now? Wouldn't > it be better to wait until most IDEs/linters all agree on this syntax, > until freezing it in stdlib? So far typing seems to be a part of mypy, > shouldn't it spend some time on PyPI first? > Because as you have noticed in this thread there are already a ton of competing solutions and no consensus has been reached. Sometimes Guido and/or python-dev have to step in and simply say "there is obvious need and the community is not reaching consensus, so we will make the decision ourselves". > > I'm also sure about there not being other uses of annotations -- clize > aside, there are not many widely used Python3-only 3rd party > libraries, so it's no surprise that nothing big is built around Python > 3 features. > > Maybe the way from PEP 3107's "here's a feature, use it for whatever > you like" to "annotations are for typing declarations, using > mypy/typing syntax" should include a step of "if you use annotations > for typing, use mypy/typing syntax for it". (And perhaps it should end > there.) > That's a possibility. Another thing to support this approach is that if something like List[str] is used over `[str]` then the returned object can subclass some common superclass which can be typechecked for to know that the annotation is from typing.py and not clize/scription and continue to function. That way you can avoid any decorators adding some attribute on functions to know that types have been specified while allowing function annotations to be used for anything. Otherwise a @typing.ignore decorator could also exist for alternative formats to use (since typing has been the most common use case and decorating your single main() function with @typing.ignore is not exactly heavy-handed). -------------- next part -------------- An HTML attachment was scrubbed... URL: From varma.sunjay at gmail.com Thu Aug 14 18:01:37 2014 From: varma.sunjay at gmail.com (Sunjay Varma) Date: Thu, 14 Aug 2014 12:01:37 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations Message-ID: I am strongly opposed to this entire proposal. As Juancarlo points out, Python programs are small, but very understandable. I think this syntax detracts from that. I'll suggest an alternative further down in my reply. One benefit of Python that makes it so attractive for new programmers and even old programmers alike is that you can usually pick out any piece of Python code and begin to understand it immediately. Even if you come from a different programming language, Python is written in english explicitly using words like "and" and "or". Those constructs, as opposed to "&&" or "||" make the language less scary for new developers and in general easier to read as well. It's also easier to type regular english words (no need to use the shift key). Using the annotation syntax this heavily will detract very much from the readability of Python and from the overall usability as well. Programs are read more times than they are written. Several years ago, before I had any programming experience in any language at all, I needed to edit some Python code to make something I was doing work. Without any experience at all, I was able to look through the (small) program I was editing and figure out exactly what I needed to adjust. Without Python being such a clean, almost English language, that would have been impossible. Though the annotation syntax is already present in Python 3, I would argue that using this for type annotations will get very messy very quickly. If I'm understanding the syntax correctly, writing any function using a large library with many nested subpackages could result in code like this: import twisted.protocols.mice.mouseman def process_mouseman(inputMouseMan: twisted.protocols.mice.mouseman.MouseMan) -> twisted.protocols.mice.mouseman.MouseMan: pass That function definition is 122 characters long. Far more than what PEP8 recommends. Though this example was crafted to illustrate my point (I don't think most people would really write code like this), it is easy to see that this kind of code is possible and may sometimes be written by some less experienced programmers. It demonstrates how messy things can get even with just one parameter. It is also easy to see that it is very difficult to parse out what is going on in that function. Adding type annotations inline makes it very difficult to quickly get an idea of what arguments a function takes and in what order. It detracts from the overall readability of a program and can also lead to very poorly formatted programs that break the guidelines in PEP8. Though I have only demonstrated this for function declarations, the example could also be extended to inline statement comments as well. Things get too messy too quickly. My Alternative Proposal: As an alternative, I would like to propose a syntax that Pycharm already supports: http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html Since this type information isn't going to be used at runtime in the regular Python interpreter anyway, why not have it in the function docstring instead? This provides both readability and type checking. Standardizing that syntax or at least adding it as an optional way to check your program would in my opinion be a much better addition to the language. This approach needs no new syntax, keeps readability and allows the programmer to add additional documentation without going over the 80 character limit. Additionally, this approach can be used by documentation generators as well and removes any duplication from the function declaration and the docstring. Here's a taste of what that looks like: class SimpleEquation(object): def demo(self, a, b, c): """ This function returns the product of a, b and c @type self: SimpleEquation :param a: int - The first number :param b: int :param c: int - The third number should not be zero and should also only be -1 if you enjoy carrots (this comment spans 2 lines) :return: int """ return a * b * c Overall, I think overloading function declarations and inline comments is a bad idea. It promotes writing code with poor readability and in general adds a lot of extra bits to the language that (from the sounds of your proposal) aren't even going to be used by the main interpreter. On the original proposal: These changes really do seem to be overestimating the staticness of Python programs as well. What about functions that don't care about the type? What about functions that only want you to pass in an object that implements __iter__? Python should not become a language where developers are required to add hundreds of odd cast() calls every time they choose to pass a different, but still compatible type, to a function. This syntax makes too many assumptions about what developers know about their code. What if I develop a similar, but different class that is compatible with an existing function? If that function doesn't specify that my class can be used, my perfectly valid code will be rejected. -1 to adding mypy annotations to Python 3. Sunjay On Thu, Aug 14, 2014 at 10:59 AM, Juancarlo A?ez wrote: > > On Wed, Aug 13, 2014 at 3:14 PM, Guido van Rossum > wrote: > >> I am proposing that we adopt whatever mypy uses here, keeping discussion >> of the details (mostly) out of the PEP. The goal is to make it possible to >> add type checking annotations to 3rd party modules (and even to the stdlib) >> while allowing unaltered execution of the program by the (unmodified) >> Python 3.5 interpreter. > > > To the bottom of things... > > About the second time I wrote about Python ("Why not Python", 2007) I > dismissed it as serious software development environment because the lack > of static type checking hindered the creation of proper software > development environments. > > http://blog.neogeny.org/why-not-python.html > > So, Why do I now have doubts about adding support for static type checking? > > I've been programming in almost-only Python for several years now, and > this discussion had me think hard about "Why?". > > The answer is simple: I never was as productive as I've been since I've > centered on Python. > > But, again Why? > > Despite what my '07 article says, the IDE I use is pythonized-VIM and the > command line. Where does the productivity come from? > > 1. Readability with the right amount of succinctness. Python programs > are very small, but understandable. > 2. The breadth and design consistency of the standard library. Some > 70%? of what I need is there, and the design consistency makes it easy > (intiutive) to use. > 3. PyPi covers another 28%. > 4. The Zen of Python (import this) permeates all of the above, > including most third-party packages. The ecosystem is consistent too. It's > a culture. > > What do I fear? I think it is that Python be transformed into a > programming language different from the one that now makes me so productive. > > I studied Ruby, and I don't like it. I've been studying Go, and I don't > like it. One must like the concepts and the power, sure, but the syntax > required for some day-to-day stuff stinks like trouble; simple stuff is so > complicated to express and so easy to get wrong... > > I hate "List[str]" and "Dict[str:int]". Where did those come from? > Shouldn't they (as others have proposed) be "[str]" and "{str:int}"? What > about tuples? > Why not write a similar, but different programming language that targets > the Cython runtime and includes all the desired features? > > All said, this is my proposal. > > The PSF could support (even fund) MyPy and similar projects, promoting > their maturity and their convergence. The changes in 3.5 would be limited > but enough to enable those efforts, and those of the several IDE > tool-smiths (changes in annotations, and maybe in ABCs). Basically, treat > MyPy as PyPy or NumPy (which got '::'). It's in Python's history to enable > third-party developments and then adopt what's mature or become the > de-facto standard. > Then, on a separate line of work, it would be good to think about how to > enable different programming languages to target the CPython environment > (because of #2, #3, and #4 above), maybe by improving AST creation and > AST-to-bytecode? There could be other languages targeting the CPython > runtime, which is the relationship that Scala, Jython, IronPython, and > others have to their own runtimes. > > -1 for standardizing static type checking in 3.5 > > Cheers, > > -- > Juancarlo *A?ez* > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Sunjay Varma Python Programmer & Web Developer www.sunjay.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From ceronman at gmail.com Thu Aug 14 18:34:43 2014 From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=) Date: Thu, 14 Aug 2014 18:34:43 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Thu, Aug 14, 2014 at 4:27 AM, Terry Reedy wrote: > > My main concern with static typing is that it tends to be > anti-duck-typing, while I consider duck-typing to be a major *feature* of > Python. The example in the page above is "def fib(n: int):". Fib should > get an count (non-negative integer) value, but it need not be an int, and > 'half' the ints do not qualify. Reading the tutorial, I could not tell if > it supports numbers.Number (which should approximate the domain from above.) This is a good point. But I think that static typing and duck typing are not mutually exclusive. TypeScript does this very nicely by defining structural interfaces (http://www.typescriptlang.org/Handbook#interfaces). With them is possible to define a given behaviour and any object capable of providing that behaviour is accepted, without having to be part of any specific type hierarchy or having to explicitly register as implementation of certain specification. That's basically what duck typing means. For example: interface Named { name: string; say(): string; } function doSomething(x: Named) { console.log(x.name); } doSomething({name: "hello", say: function() { return this.name }}); // OK doSomething({something: "hello"}); // ERROR I think something like this is a must have for mypy. In Python, I've been playing with something similar (https://github.com/ceronman/typeannotations) but for runtime checking only: >>> class Person(Interface): ... name = str ... age = int ... def say_hello(name: str) -> str: ... pass Any object defining those the name, age and say_hello() members is a valid implementation of that interface. For example: >>> class Developer: ... def __init__(self, name, age): ... self.name = name ... self.age = age ... def say_hello(self, name: str) -> str: ... return 'hello ' + name ... >>> isinstance(Developer('bill', 20), Person) True Are there any plans for adding something like this to mypy? Manuel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Thu Aug 14 19:17:48 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 14 Aug 2014 12:47:48 -0430 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Thu, Aug 14, 2014 at 12:04 PM, Manuel Cer?n wrote: > For example: > > interface Named { > name: string; > say(): string; > } > > function doSomething(x: Named) { > console.log(x.name); > } > > doSomething({name: "hello", say: function() { return this.name }}); // OK > doSomething({something: "hello"}); // ERROR > That is so Java....! -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Aug 14 19:29:01 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 14 Aug 2014 10:29:01 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <53ECF1DD.8040808@stoneleaf.us> On 08/14/2014 09:01 AM, Sunjay Varma wrote: > > Additionally, this approach can be used by documentation generators as well and removes any duplication from the > function declaration and the docstring. > > Here's a taste of what that looks like: > class SimpleEquation(object): > def demo(self, a, b, c): > """ > This function returns the product of a, b and c > @type self: SimpleEquation > :param a: int - The first number > :param b: int > :param c: int - The third number should not be zero and should also > only be -1 if you enjoy carrots (this comment spans 2 lines) > :return: int > """ > return a * b * c +1 I like this much more. -- ~Ethan~ From steve at pearwood.info Thu Aug 14 19:31:03 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 03:31:03 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <20140814173103.GO4525@ando> As requested, I've read the whole post, and the whole thread, before responding :-) On Wed, Aug 13, 2014 at 12:44:21PM -0700, Guido van Rossum wrote: > (a) Python should adopt mypy's syntax for function annotations [...] I'm very excited to see functional annotations being treated seriously, I think the introduction of static typing, even optional, has the potential to radically change the nature of Python language and I'm not sure if that will be good or bad :-) but it is reassuring to hear that the intention is that it will be treated more like an optional linter than as a core part of the language. On the other hand, are you aware of Cobra, which explicitly was modelled on Python but with optional static typing? http://cobra-language.com/ [...] > *(1) A change of direction for function annotations* > [...] I propose a conscious change of course here by stating > that annotations should be used to indicate types and to propose a standard > notation for them. And in a later email, Guido also stated: > I want to eventually phase out other uses of function annotations That disappoints me and I hope you will reconsider. I've spent some time thinking about using annotations for purposes other than type checking, but because most of my code has to run on Python 2, there's nothing concrete. One example is that I started exploring ways to use annotations as documentation for the statistics module in 3.4, except that annotations are banned from the standard library. (Naturally I haven't spent a lot of time on something that I knew was going to be rejected.) I came up with ideas like this: def mean(data) -> '? = ?(x)/n': def pvariance(data) -> '?? = ?(x - ?)? ? n': which might have been a solution to this request: http://bugs.python.org/issue21046 had annotations been allowed in the stdlib. Regardless of whether this specific idea is a good one or not, I will be disappointed if annotations are limited to one and only one use. I don't mind if there is a standard, default, set of semantics so long as there is a way to opt-out and use something else: @use_spam_annotations def frobnicate(x: spam, y: eggs)->breakfast: ... for example. Whatever the mechanism, I think Python should not prohibit or deprecate other annotation semantics. > *(2) A specification for what to add to Python 3.5* > > There needs to be at least a rough consensus on the syntax for annotations, > and the syntax must cover a large enough set of use cases to be useful. > Mypy is still under development, and some of its features are still > evolving (e.g. unions were only added a few weeks ago). It would be > possible to argue endlessly about details of the notation, e.g. whether to > use 'list' or 'List', what either of those means (is a duck-typed list-like > type acceptable?) or how to declare and use type variables, and what to do > with functions that have no annotations at all (mypy currently skips those > completely). It doesn't sound to me like the mypy syntax is mature enough to bless, let alone to start using it in the standard library. > I am proposing that we adopt whatever mypy uses here, keeping discussion of > the details (mostly) out of the PEP. The goal is to make it possible to add > type checking annotations to 3rd party modules (and even to the stdlib) > while allowing unaltered execution of the program by the (unmodified) > Python 3.5 interpreter. The actual type checker will not be integrated with > the Python interpreter, and it will not be checked into the CPython > repository. The only thing that needs to be added to the stdlib is a copy > of mypy's typing.py module. What happens when the typing.py module in the standard library gets out of sync with the typing.py module in mypy? [...] > *Appendix -- Why Add Type Annotations?* > The argument between proponents of static typing and dynamic typing has > been going on for many decades. Neither side is all wrong or all right. > Python has traditionally fallen in the camp of extremely dynamic typing, > and this has worked well for most users, but there are definitely some > areas where adding type annotations would help. Some people have probably already seen this, but I have found this article to be very useful for understanding why static and dynamic type checking can be complementary rather than opposed: http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ -- Steven From ndbecker2 at gmail.com Thu Aug 14 19:33:47 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 14 Aug 2014 13:33:47 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: Message-ID: Does mypy support annotation of functions implemented in C code? If I extend cpython via C-API, does mypy provide a mechanism to annotate those functions? From steve at pearwood.info Thu Aug 14 19:35:19 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 03:35:19 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <20140814173519.GP4525@ando> On Thu, Aug 14, 2014 at 06:34:43PM +0200, Manuel Cer?n wrote: > On Thu, Aug 14, 2014 at 4:27 AM, Terry Reedy wrote: > > > > My main concern with static typing is that it tends to be > > anti-duck-typing, while I consider duck-typing to be a major *feature* of > > Python. The example in the page above is "def fib(n: int):". Fib should > > get an count (non-negative integer) value, but it need not be an int, and > > 'half' the ints do not qualify. Reading the tutorial, I could not tell if > > it supports numbers.Number (which should approximate the domain from above.) > > > This is a good point. But I think that static typing and duck typing are > not mutually exclusive. [...] > Are there any plans for adding something like this to mypy? The mypy FAQs claim to be focusing on nominative typing, but haven't ruled out structural typing in the future: http://www.mypy-lang.org/faq.html -- Steven From stefan_ml at behnel.de Thu Aug 14 19:37:55 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 14 Aug 2014 19:37:55 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: Guido van Rossum schrieb am 13.08.2014 um 21:44: > Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man > of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: > http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can > learn from Haskell (and other languages); yesterday he gave the same talk > at Dropbox. The talk is online ( > https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad > strokes comes down to three suggestions: > > (a) Python should adopt mypy's syntax for function annotations > [...] proposal (a) feels right to me. FWIW, Cython currently understands these forms of function argument annotations in Python code: x: dict x: {"type": dict} x: {"type": "dict"} x: {"ctype": "long double"} The "ctype" values that are usable here obviously only include those that can be converted from and to Python types, e.g. no arbitrary pointers. It'd be nice to have a way to declare container item types, but that's never really been a priority in the Cython project so far. Declaring protocols, OTOH, is pretty useless for a compiler, as it's obvious from the code which protocols are being used on a given value (iteration, item access, etc.). It's clear that restricting everything to one kind of annotation isn't enough, as there are use cases for a mix of different type systems, Python itself plus at least C/C++ in CPython and Cython, Java in Jython, C# in IronPython, plus others that people might want to interface with. C/C++ are generally interesting, for example, also for .NET or JVM users. Although I wasn't very impressed by Bob Ippolito's talk at EuroPython, I'm generally not opposed to type annotations to provide 1) additional contract information for libraries, 2) documentation, 3) better static analysis or 4) type hints for compilers. But these are actually quite different use cases that may each suggest a different strictness in the type declarations. For 3) and 4), function signatures aren't enough and should be accompanied by declarations for local variables. 4) should also support additional type systems for language integration. But 1) and 2) aren't completely overlapping either. 1) would need declarations that can be used for hard type checking, whereas 2) can be much more relaxed, generic and incomplete. Trying to get all of these under one umbrella might not be a good idea, but letting people add three different annotations for each function argument definitely isn't either. Stefan From steve at pearwood.info Thu Aug 14 19:45:12 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 03:45:12 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <20140814174512.GQ4525@ando> On Wed, Aug 13, 2014 at 05:51:40PM -0430, Juancarlo A?ez wrote: > Function annotations are not available in Python 2.7, so promoting > widespread use of annotations in 3.5 would be promoting code that is > compatible only with 3.x, Yes. You say that as if it were a bad thing. It is not. Python 3 is here to stay and we should be promoting Python 3 only code. There is absolutely no need to apologise for that fact. If people are happy with Python the way it is in 2.7, or 1.5 for that matter, that's great, they can stay on it for ever, but all new features are aimed at 3.x and not 2.x or 1.x. > when the current situation is that much effort is > being spent on writing code that works on both 2.7 and 3.4 (most > libraries?). There's no reason why all new code should be aimed at 2.x and 3.x. But even for code which is, the nice thing about this proposal is that it's optional, so you can run your type-check using mypy under Python 3.x and still get the benefit of it when running under 2.x. > Independently of its core merits, this proposal should fail unless > annotations are added to Python 2.8. There will be no Python 2.8, and no Python 2.9 either. New features go into 3.x. -- Steven From greg at krypto.org Thu Aug 14 19:46:43 2014 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 14 Aug 2014 10:46:43 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Thu, Aug 14, 2014 at 9:01 AM, Brett Cannon wrote: > > On Thu Aug 14 2014 at 10:53:37 AM Petr Viktorin wrote: > >> It seems to me that rather than adding a module which is only used by >> one project so far to the standard library is a bit premature. >> >> I support optional typing, but why push this to stdlib now? Wouldn't >> it be better to wait until most IDEs/linters all agree on this syntax, >> until freezing it in stdlib? So far typing seems to be a part of mypy, >> shouldn't it spend some time on PyPI first? >> > > Because as you have noticed in this thread there are already a ton of > competing solutions and no consensus has been reached. Sometimes Guido > and/or python-dev have to step in and simply say "there is obvious need and > the community is not reaching consensus, so we will make the decision > ourselves". > My overarching concern with the entire proposal is that adding this would just be yet more syntax added to the language with not much use that doesn't go far enough. We'd ultimately need pytd or something else regardless when it comes to full scale Python static analysis. But that *isn't necessarily* a bad thing. Specifying an actual basic annotation syntax that can do some subset of what you want to annotate in the language should in theory still be useful to a real code analysis tool. If it isn't, it will simply ignore it. If it is, it can use it and build on it even though it needs the ability to specify on the side. If you do add a module for this, at least consider hiding it behind a "from __future__ import some_module_full_of_annotation_related_things" instead of making it a new no-op top level module. -gps > > >> >> I'm also sure about there not being other uses of annotations -- clize >> aside, there are not many widely used Python3-only 3rd party >> libraries, so it's no surprise that nothing big is built around Python >> 3 features. >> >> Maybe the way from PEP 3107's "here's a feature, use it for whatever >> you like" to "annotations are for typing declarations, using >> mypy/typing syntax" should include a step of "if you use annotations >> for typing, use mypy/typing syntax for it". (And perhaps it should end >> there.) >> > > That's a possibility. Another thing to support this approach is that if > something like List[str] is used over `[str]` then the returned object can > subclass some common superclass which can be typechecked for to know that > the annotation is from typing.py and not clize/scription and continue to > function. That way you can avoid any decorators adding some attribute on > functions to know that types have been specified while allowing function > annotations to be used for anything. Otherwise a @typing.ignore decorator > could also exist for alternative formats to use (since typing has been the > most common use case and decorating your single main() function with > @typing.ignore is not exactly heavy-handed). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Aug 14 19:53:44 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 14 Aug 2014 19:53:44 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: Neal Becker schrieb am 14.08.2014 um 19:33: > Does mypy support annotation of functions implemented in C code? > > If I extend cpython via C-API, does mypy provide a mechanism to annotate those > functions? No, mypy isn't about C or even CPython. However, you can already do that, although not easily, I think. What you'd need is an __annotations__ dict on the function object and a bit of trickery to make CPython believe it's a function. Cython gives you that for free (by simply providing the normal Python semantics), but you can get the same thing with some additional work when writing your own C code. There's also the argument clinic, but IIRC it doesn't support signature annotations for some reason, guess it wasn't considered relevant (yet). Stefan From rymg19 at gmail.com Thu Aug 14 20:05:58 2014 From: rymg19 at gmail.com (Ryan) Date: Thu, 14 Aug 2014 13:05:58 -0500 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <87d5d44f-f48e-4df0-a4c7-2a8d7c6f4996@email.android.com> Andrew Barnert wrote: >On Aug 14, 2014, at 7:37, Ryan Gonzalez wrote: > >>> On 8/13/2014 3:44 PM, Guido van Rossum wrote: > >>> Now consider an extended version (after Lucas). >>> >>> def fib(n, a, b): >>> i = 0 >>> while i <= n: >>> print(i,a) >>> i += 1 >>> a, b = b, a+b >>> >>> The only requirement of a, b is that they be addable. Any numbers >should be allowed, as in fib(10, 1, 1+1j), but so should fib(5, '0', >'1'). Addable would be approximated from below by Union(Number, str). >> >> Unless MyPy added some sort of type classes... > >By "type classes", do you mean this in the Haskell sense, or do you >mean classes used just for typing--whether more granular ABCs (like an >Addable which both Number and AnyStr and probably inherit) or typing.py >type specifiers (like an Addable defined as Union(Number, AnyStr)? The Haskell way. Having ABCs and type classes can get confusing, but, when I can, I use type classes for the more unrelated concepts(such as Addable) and ABCs for the more parent-child concepts(such as Node in an AST or Generator in a set of generators). The fine line might actually make it a bad choice to add to Python/mypy, though. > >It's also worth noting that the idea that this function should take a >Number or a str seems way off. It's questionable whether it should >accept str, but if it does, shouldn't it also accept bytes, bytearray, >and other string-like types? What about sequences? And meanwhile, >whether or not it accepts str, it should probably accept np.ndarray and >other types of element-wise adding types. If you create an Addable >type, it has to define, globally, which of those counts as addable, but >different functions will have different definitions that make sense. > >In fact, look at the other discussion going on. People want to ensure >that sum only works on numbers or number-like types (and does that >include NumPy arrays or not?), while others want to change it to work >on all sequences, or only mutable sequences with += plus str because it >effectively has magical += handling under the covers, etc. If we can't >even decide what Addable means for one specific function that everyone >has experience with... > >On the other hand, if sum could have been annotated to tell us the >author's intention (or, rather, the consensus of the dev list), then >all of these recurring arguments about summing str would go away. Until >someone defined a number-like class (maybe even one that meets the ABC, >but by calling register on it) and sum won't work for him. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. From steve at pearwood.info Thu Aug 14 20:15:54 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 04:15:54 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <20140814181554.GR4525@ando> On Thu, Aug 14, 2014 at 12:01:37PM -0400, Sunjay Varma wrote: > Though the annotation syntax is already present in Python 3, I would argue > that using this for type annotations will get very messy very quickly. If > I'm understanding the syntax correctly, writing any function using a large > library with many nested subpackages could result in code like this: > > import twisted.protocols.mice.mouseman > > def > process_mouseman(inputMouseMan: twisted.protocols.mice.mouseman.MouseMan) > -> twisted.protocols.mice.mouseman.MouseMan: > pass I would write that like this: from twisted.protocols.mice.mouseman import MouseMan def process_mouseman(inputMouseMan: MouseMan) -> MouseMan: pass > That function definition is 122 characters long. Or 58. > It is also easy to see that it is very difficult to parse out what is going > on in that function. Only because I have no idea what MouseMan means :-) > As an alternative, I would like to propose a syntax that Pycharm already > supports: > http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html [...] > Here's a taste of what that looks like: > class SimpleEquation(object): > > def demo(self, a, b, c): > """ > This function returns the product of a, b and c > @type self: SimpleEquation > :param a: int - The first number > :param b: int > :param c: int - The third number should not be zero and should > also > only be -1 if you enjoy carrots (this comment spans 2 lines) > :return: int > """ > return a * b * c I really dislike that syntax. I dislike adding cruft like "@type" and ":param" into docstrings, which should be written for human readers, not linters. I dislike that you have documented that self is a SimpleEquation. (What else could it be?) I dislike that the syntax clashes with ReST syntax. I dislike that it isn't obvious to me why the first parameter uses @type while the second parameter uses :param. > Overall, I think overloading function declarations and inline comments is a > bad idea. It promotes writing code with poor readability I like the annotation syntax. I'm not completely convinced that the mypy syntax is mature enough to bless, but the basic idea of type annotations is pretty common in dozens of languages. I think you are in a tiny minority if you think that putting the type declaration right next to the parameter make it *less* clear that putting the type declaration in a completely different part of the code. # the type is together with the parameter def frobinate(x: Spam, y: Egg)->Breakfast: # the type declaration and parameter are distantly apart def frobinate(x, y): """Return the frobinated x and y. Some more text goes here. Perhaps lots of text. :param x: Spam :param y: Eggs :return: Breakfast """ > On the original proposal: > These changes really do seem to be overestimating the staticness of Python > programs as well. What about functions that don't care about the type? They can declare that they are object. Or not declare a type at all. > What about functions that only want you to pass in an object that implements > __iter__? I would expect this should work: from typing import Iter def func(it:Iter): ... > Python should not become a language where developers are required > to add hundreds of odd cast() calls every time they choose to pass a > different, but still compatible type, to a function. I'm not sure how you go from *optional* static typing to developers being *required* to cast values. As I see it, one HUGE advantage of this proposal is that people who want strict static typing currently might write code like this: def make_sandwich(filling): if not isinstance(filling, Ham): raise TypeError ... With the new proposal, they will probably write this: def make_sandwich(filling: Ham): ... and allow the static type check to occur at compile time. That means that if I want to pass a Spam instance instead of a Ham instance, all I need do is disable the compile-time type check, and make_sandwich will happily accept anything that has the same duck-type interface as Ham, like Spam. If I pass an int instead, I'll get the same run-time error that I would have got if make_sandwich did not include an explicit type check. So, I think this proposal might actually lead to *more* duck typing rather than less, since you can always turn off the type checking. -- Steven From steve at pearwood.info Thu Aug 14 20:30:14 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 04:30:14 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <20140814183014.GS4525@ando> On Thu, Aug 14, 2014 at 10:46:43AM -0700, Gregory P. Smith wrote: > My overarching concern with the entire proposal is that adding this would > just be yet more syntax added to the language with not much use that > doesn't go far enough. This isn't new syntax. Functional annotations have been in Python since Python 3.0. What's new here is blessing one specific meaning for that syntax as the One Offical use for annotations. I'd rather: - bless function annotations for static type checking as the default meaning, but allowing code to opt-out and use annotations for something else; - encourage the various type checkers and linters to come up with a standard syntax for type annotations. I'm not convinced that mypy syntax is yet mature enough to be that standard. But, perhaps if the typing module is given provisional status, maybe it could be a good start. > If you do add a module for this, at least consider hiding it behind a "from > __future__ import some_module_full_of_annotation_related_things" instead of > making it a new no-op top level module. I'm not sure what this comment means. Did you read Guido's first post in this thread? I thought he was clear that to get type checking, you would do this: from typing import Dict def function(d: Dict[str, int])-> int: ... I bet you can guess what that does, but in case you can't, it declares that argument d is a Dict with str keys and int values, and the return result is an int. I'm not sure where you get the idea of a no-op top level module from. "from __future__ import ..." is inappropriate too, since that is intended for changes to compiler semantics (e.g. new syntax). This is existing syntax, and the actual type checking itself will be delegated to a separate product, mypy, which as Guido stated will behave as a linter. That means that the type checks won't do anything unless you have mypy installed, and you can still run your code under any Python compiler you like. As I understand it, CPython itself requires no changes to make this work, just the addition of typing.py from the mypy project and a policy change to the use of annotations. -- Steven From cory at lukasa.co.uk Thu Aug 14 20:25:04 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 14 Aug 2014 19:25:04 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814181554.GR4525@ando> References: <20140814181554.GR4525@ando> Message-ID: On 14 August 2014 19:15, Steven D'Aprano wrote: > I really dislike that syntax. I dislike adding cruft like "@type" and > ":param" into docstrings, which should be written for human readers, not > linters. That ship has long-since sailed. Sphinx uses exactly this :param: and :return: syntax for its docstring parsing. It is by now a common convention (at least, I see it all over the place in open source code), and should not be considered a surprise. I appreciate that it doesn't lead to clean docstrings, but I've found it leads to docstrings that are genuinely written to be read (because they're part of your documentation). > So, I think this proposal might actually lead to *more* duck typing > rather than less, since you can always turn off the type checking. I found this conclusion impossible to understand: have I missed something Steven? To my eyes, the fact that when run by a user who knows nothing about the static type checker much duck typing will fail will clearly not lead to more duck typing. It will lead either to a) less duck typing because of all the bug reports (your code breaks whenever I try to run it!), or b) everyone turning the static type checker off. That objection assumes the static checker would be on by default. If it were off by default but available, both of these problems go away but we're back in the situation we're in right now. In that case, I don't see why we'd add this to CPython. From ethan at stoneleaf.us Thu Aug 14 20:33:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 14 Aug 2014 11:33:45 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814181554.GR4525@ando> References: <20140814181554.GR4525@ando> Message-ID: <53ED0109.6070207@stoneleaf.us> On 08/14/2014 11:15 AM, Steven D'Aprano wrote: > > I like the annotation syntax. I'm not completely convinced that the mypy > syntax is mature enough to bless, but the basic idea of type annotations > is pretty common in dozens of languages. I think you are in a tiny > minority if you think that putting the type declaration right next to > the parameter make it *less* clear that putting the type declaration in > a completely different part of the code. > > # the type is together with the parameter > def frobinate(x: Spam, y: Egg)->Breakfast: > > # the type declaration and parameter are distantly apart > def frobinate(x, y): > """Return the frobinated x and y. > > Some more text goes here. Perhaps lots of text. > > :param x: Spam > :param y: Eggs > :return: Breakfast > """ Sure, keeping that info in the annotations makes more sense, but I'd rather see it in the doc string instead of ruling out all other possible uses of annotations -- particularly for something that's supposed to be /optional/. -- ~Ethan~ From steve at pearwood.info Thu Aug 14 20:52:45 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 04:52:45 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814181554.GR4525@ando> Message-ID: <20140814185245.GT4525@ando> On Thu, Aug 14, 2014 at 07:25:04PM +0100, Cory Benfield wrote: > On 14 August 2014 19:15, Steven D'Aprano wrote: > > I really dislike that syntax. I dislike adding cruft like "@type" and > > ":param" into docstrings, which should be written for human readers, not > > linters. > > That ship has long-since sailed. Sphinx uses exactly this :param: and > :return: syntax for its docstring parsing. It is by now a common > convention (at least, I see it all over the place in open source > code), and should not be considered a surprise. I've seen it too, but not in docstrings written in vanilla ReST. That's a disappointment to hear that Sphinx uses it, because I think it is hideously ugly :-( > > So, I think this proposal might actually lead to *more* duck typing > > rather than less, since you can always turn off the type checking. > > I found this conclusion impossible to understand: have I missed > something Steven? To my eyes, the fact that when run by a user who > knows nothing about the static type checker much duck typing will fail > will clearly not lead to more duck typing. It will lead either to a) > less duck typing because of all the bug reports (your code breaks > whenever I try to run it!), or b) everyone turning the static type > checker off. Let me explain my reasoning. Back in the Old Days, before Python 2.2, there was no isinstance(). We were strongly discouraged from doing type checks, instead we were encouraged to rely on duck-typing and that functions would fail loudly if passed the wrong argument. With the introduction of isinstance, Python code has slowly, gradually, begun using more and more run-time explicit type checks with isinstance. Some people do this more than others. Let's consider Fred, who is a Java programmer at heart and so writes code like this: def foo(x): if not instance(x, float): raise TypeError("Why doesn't python check this for me?") return (x+1)/2 I want to pass a Decimal to foo(), but can't, because of the explicit type check. I am sad. But with this proposal, Fred may write his function like this: def foo(x:float)->float: return (x+1)/2 and rely on mypy to check the types at compile time. Fred is happy: he has static type checks, Python does it automatically for him (once he has set up his build system to call mypy), and he is now convinced that foo() is type-safe and an isinstance check at run-time would be a waste of cycles. I want to pass a Decimal to foo(). All I have to do is *not* install mypy, or disable it, and lo and behold, like magic, the type checking doesn't happen, and foo() operates by duck-typing just like in the glory days of Python 1.5. Both Fred and I are now happy, and with the explicit isinstance check removed, the only type checking that occurs when I run Fred's library are the run-time duck-typing checks. -- Steven From steve at pearwood.info Thu Aug 14 20:55:55 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 04:55:55 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <20140814185555.GU4525@ando> On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote: > One interesting feature of TypeScript is that it allows you to annotate > existing code without modifying it, by using external definition files. In > the JavaScript world, many people have contributed TypeScript annotation > files for popular JS libraries (http://definitelytyped.org/). > > I think this is possible in Python as well doing something like this: > > @annotate('math.ciel') > def ciel(x: float) -> int: > pass I'm afraid I don't understand what the annotate decorator is doing here. Can you explain please? -- Steven From steve at pearwood.info Thu Aug 14 21:02:38 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 05:02:38 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EBCABC.7010801@python.org> References: <53EBCABC.7010801@python.org> Message-ID: <20140814190237.GV4525@ando> On Wed, Aug 13, 2014 at 10:29:48PM +0200, Christian Heimes wrote: > 1) I'm not keen with the naming of mypy's typing classes. The visual > distinction between e.g. dict() and Dict() is too small and IMHO > confusing for newcomers. How about an additional 'T' prefix to make > clear that the objects are referring to typing objects? > > from typing import TList, TDict > > def word_count(input: TList[str]) -> TDict[str, int]: > ... Would it be possible, and desirable, to modify the built-in types so that we could re-use them in the type annotations? def word_count(input: list[str]) -> dict[str, int]: Since types are otherwise unlikely to be indexable like that, I think that might work. > 2) PEP 3107 only specifies arguments and return values but not > exceptions that can be raised by a function. Java has the "throws" > syntax to list possible exceptions: > > public void readFile() throws IOException {} I understand that this is called a "checked exception" in Java. I also understand that they are hated and derided as useless or even counter-productive: http://literatejava.com/exceptions/checked-exceptions-javas-biggest-mistake/ > May I suggest that we also standardize a way to annotate the exceptions > that can be raised by a function? It's a very useful piece of > information and commonly requested information on the Python user > mailing list. And as frequently explained on the python-list, it's almost impossible to answer. Or rather, the answer is usually no more specific than "raises Exception". There are very few guarantees you can reliably make about what exceptions *cannot* be raised by a function. To put it simply, given almost any operation in your function, say, x+1, there's no limit on what x.__add__ might raise. Even if you know x is a subclass of int, it could do anything in its __add__ method. Only if you know x is *exactly* a builtin int can you be confident that it won't raise (say) ImportError. Perhaps with a few years of experience, we might be able to extend this to exceptions without making the same mistakes as Java's checked exceptions, but I wouldn't rush into it. -- Steven From varma.sunjay at gmail.com Thu Aug 14 21:21:59 2014 From: varma.sunjay at gmail.com (Sunjay Varma) Date: Thu, 14 Aug 2014 15:21:59 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: Frankly, I'd just really like to get all of this noise out of the function declaration. Any reasonable, readable and consistent documentation format is fine with me. I chose the sphinx format because it is already well supported in pycharm and that was mentioned in the first few responses. I actually don't like the colon syntax very much (it's awkward and unnatural to type), so if anyone has a different suggestion I'd be very open to that. Mainly I want to ensure that Python doesn't sacrifice readability and line length (which is part of readability) just because annotations are already built in. I suggest we decide on a standard format that can be used in documentation strings and also used with type checking. Let's enhance our documentation with types, not obfuscate function declarations. Sunjay On Aug 14, 2014 3:14 PM, "Nathaniel Smith" wrote: > On 14 Aug 2014 17:02, "Sunjay Varma" wrote: > > Here's a taste of what that looks like: > > class SimpleEquation(object): > > > > def demo(self, a, b, c): > > """ > > This function returns the product of a, b and c > > @type self: SimpleEquation > > :param a: int - The first number > > :param b: int > > :param c: int - The third number should not be zero and > should also > > only be -1 if you enjoy carrots (this comment spans 2 > lines) > > :return: int > > """ > > return a * b * c > > There are at least three existing, popular, standardized syntaxes for > these kinds of docstring annotations in use: plain ReST, Google's docstring > standard, and numpy's docstring standard. All are supported by Sphinx out > of the box. (The latter two require enabling the "napolean" extension, but > this is literally a one line config file switch.) > > Would you suggest that python-dev should pick one of these and declare it > to be the official standard, or...? > > -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Aug 14 21:14:27 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 14 Aug 2014 20:14:27 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On 14 Aug 2014 17:02, "Sunjay Varma" wrote: > Here's a taste of what that looks like: > class SimpleEquation(object): > > def demo(self, a, b, c): > """ > This function returns the product of a, b and c > @type self: SimpleEquation > :param a: int - The first number > :param b: int > :param c: int - The third number should not be zero and should also > only be -1 if you enjoy carrots (this comment spans 2 lines) > :return: int > """ > return a * b * c There are at least three existing, popular, standardized syntaxes for these kinds of docstring annotations in use: plain ReST, Google's docstring standard, and numpy's docstring standard. All are supported by Sphinx out of the box. (The latter two require enabling the "napolean" extension, but this is literally a one line config file switch.) Would you suggest that python-dev should pick one of these and declare it to be the official standard, or...? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From brakhane at googlemail.com Thu Aug 14 21:28:11 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Thu, 14 Aug 2014 21:28:11 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53EBC3BA.10808@stoneleaf.us> Message-ID: <53ED0DCB.2000306@googlemail.com> Allow me to chime in. Am 13.08.2014 22:19, schrieb Guido van Rossum: > On Wed, Aug 13, 2014 at 12:59 PM, Ethan Furman > wrote: > > -1 on deprecating alternative uses of annotations. > > > Do you have a favorite alternative annotation use that you actually > use (or are likely to)? > I would be very sad to see annotations being limited to convey type information. But I think I have a solution that will make all happy (see end of mail) I've programmed Java web application for many years now (hoping to finally switch to Python), and "method parameter annotations" as they are called in Java would be one thing I'd really miss, as they can be very useful. Let me give you two examples: 1. Annotations can be used to communicate additional restrictions on values that must be checked on run time Let's assume a simple Web service that is called via HTTP to register a user, and the underlying framework decodes the request and finally calls a simple controller function, it could look like this (Java code, @ signifies an annotation) public Response register(@Range(18,100) int age, @ValidEmail String email) { ... } The framework would check the range of the age parameter and the validity of the email and if there are validation errors, refusing the request with a suitable error message without ever calling our function. Even if we assume that mypy's type system will be incredibly complex and allowing to specify all kinds of restrictions on a type, it won't help because those checks have to be done at run time, and are not optional. Of course those checks could be hard coded into the function, but using annotation also provides a simple and immediate documentation about the allowed values, and avoids boilerplate (I do not have to write "if (emailNotvalie(email) throw ValidationError("Field email is not a valid email")" in every method that uses an email) 2. They can give meta information to a framework An admittedly contrieved example, let's expand our register function: public Response register( int age, @Inject @Scope("session") UserService userService, @Authenticated User user) .... Here I can tell the dependency injection framework that I want an instance of the UserService injected, but one instance that has session scope instead of the normal "singleton" scope. I also ask the framework to inject me the currently authenticated user object (let's assume if I'd write "@Authenticated String user" I could get the login name as string etc.) The flexibility annotations give in Java makes programming much less frustrating. It also took quite some time before annotations were widly used (they were only available starting in Java 5) and people started finding more uses for them. I think this will also be true for Python, it will take time before people find useful ways for them. Redefining them now to be "not much more" than static type information feels wrong to me. My proposed solution: If an annotation is a tuple, mypy will take a look at each item and do it's usual thing. If it doesn't recognise an item, it is skipped. Every framework that uses annotations should also iterate over entries of a tuple to find the ones it is interested in. This also allows more than one annotation at a time, and is completely backwards compatible (as far as Python itself is concerned) for example, my first example could be written as def register(age: (int, range(18,100)), email: (str, ValidEmail)) also, it will allow me to add annotations to existing "typed" functions, def foo(bar: int) -> int could become def foo(bar: (int, MyAnnotation)) -> (int, AnotherOfMyAnnotations) I'm not sure what should happen if two conflicting types are given, like (int, str); I think it should be treated as a union type (either int or str). -- Dennis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: From skip at pobox.com Thu Aug 14 21:34:51 2014 From: skip at pobox.com (Skip Montanaro) Date: Thu, 14 Aug 2014 14:34:51 -0500 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53ED0DCB.2000306@googlemail.com> References: <53EBC3BA.10808@stoneleaf.us> <53ED0DCB.2000306@googlemail.com> Message-ID: On Thu, Aug 14, 2014 at 2:28 PM, Dennis Brakhane wrote: > public Response register(@Range(18,100) int age, @ValidEmail String > email) { ... } > > The framework would check the range of the age parameter and the > validity of the email and if there are validation errors, > refusing the request with a suitable error message without ever calling > our function. Couldn't you do that today in Python with a suitably sophisticated function decorator? The range/type checking would happen before the user's actual function is called. Skip From lukasz at langa.pl Thu Aug 14 21:35:38 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Thu, 14 Aug 2014 12:35:38 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814190237.GV4525@ando> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> Message-ID: On Aug 14, 2014, at 12:02 PM, Steven D'Aprano wrote: > Would it be possible, and desirable, to modify the built-in types so > that we could re-use them in the type annotations? > > def word_count(input: list[str]) -> dict[str, int]: > > > Since types are otherwise unlikely to be indexable like that, I think > that might work. -1 on that idea. Actually, -1 on List, Dict and friends as well. Square brackets are for lookup (indexing, key-based, or slicing). Saying here that you?re ?looking up? a subtype of list that holds strings is a far stretch. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Thu Aug 14 21:23:34 2014 From: phd at phdru.name (Oleg Broytman) Date: Thu, 14 Aug 2014 21:23:34 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814185245.GT4525@ando> References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> Message-ID: <20140814192334.GA27244@phdru.name> On Fri, Aug 15, 2014 at 04:52:45AM +1000, Steven D'Aprano wrote: > But with this proposal, Fred may write his function like this: > > def foo(x:float)->float: > return (x+1)/2 > > and rely on mypy to check the types at compile time. Fred is happy: he > has static type checks, Python does it automatically for him (once he > has set up his build system to call mypy), and he is now convinced that > foo() is type-safe and an isinstance check at run-time would be a waste > of cycles. > > I want to pass a Decimal to foo(). All I have to do is *not* install > mypy, or disable it, and lo and behold, like magic, the type checking > doesn't happen, and foo() operates by duck-typing just like in the glory > days of Python 1.5. Both Fred and I are now happy, and with the explicit > isinstance check removed, the only type checking that occurs when I run > Fred's library are the run-time duck-typing checks. Well, that's funny. Static type checking as a way to subvert type checking! (-: Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ethan at stoneleaf.us Thu Aug 14 21:43:22 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 14 Aug 2014 12:43:22 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814185555.GU4525@ando> References: <20140814185555.GU4525@ando> Message-ID: <53ED115A.1090306@stoneleaf.us> On 08/14/2014 11:55 AM, Steven D'Aprano wrote: > On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote: > >> One interesting feature of TypeScript is that it allows you to annotate >> existing code without modifying it, by using external definition files. In >> the JavaScript world, many people have contributed TypeScript annotation >> files for popular JS libraries (http://definitelytyped.org/). >> >> I think this is possible in Python as well doing something like this: >> >> @annotate('math.ciel') >> def ciel(x: float) -> int: >> pass > > I'm afraid I don't understand what the annotate decorator is doing here. > Can you explain please? My understanding is it's using the 'math.ciel' file in order to understand how it should treat the annotations of 'float' and 'int'. -- ~Ethan~ From brakhane at googlemail.com Thu Aug 14 21:43:39 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Thu, 14 Aug 2014 21:43:39 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53EBC3BA.10808@stoneleaf.us> <53ED0DCB.2000306@googlemail.com> Message-ID: <53ED116B.2010300@googlemail.com> Am 14.08.2014 21:34, schrieb Skip Montanaro: > Couldn't you do that today in Python with a suitably sophisticated > function decorator? The range/type checking would happen before the > user's actual function is called. I suppose so, but I'd have to repeat myself and it would look ugly, because I have to tell the decorator which parameter I'm talking about. Something like @do_range_check('age', 18, 100) @do_email_check('email') def register(age: int, email: str): looks not nearly as nice. Furthermore, my proposal allows multiple uses of annotations, without restricting them to a single use. If you only use mypy, you can keep using annotations as type declarations, when you use some other framework that uses annotations for a different thing, you can still use them, only once you want to use both *and* you have a method that needs both types of annotations are you forced to use the tuple notation. From varma.sunjay at gmail.com Thu Aug 14 21:40:26 2014 From: varma.sunjay at gmail.com (Sunjay Varma) Date: Thu, 14 Aug 2014 15:40:26 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53ED0DCB.2000306@googlemail.com> References: <53EBC3BA.10808@stoneleaf.us> <53ED0DCB.2000306@googlemail.com> Message-ID: See responses inline On Aug 14, 2014 3:28 PM, "Dennis Brakhane" wrote: > 1. Annotations can be used to communicate additional restrictions on > values that must be checked on run time > > Let's assume a simple Web service that is called via HTTP to register a > user, and the underlying framework decodes > the request and finally calls a simple controller function, it could > look like this > > (Java code, @ signifies an annotation) > > public Response register(@Range(18,100) int age, @ValidEmail String > email) { ... } > > The framework would check the range of the age parameter and the > validity of the email and if there are validation errors, > refusing the request with a suitable error message without ever calling > our function. > This is exactly the kind of syntax I want to avoid. Python should not attempt to just blindly become like Java or any other language. Though Dennis was just illustrating his point (this was not a suggestion of an alternate syntax), I feel like Python is moving further and further towards code like this. Python programmers should not be forcing as much as possible into each line. Explicit is better than implicit. > Even if we assume that mypy's type system will be incredibly complex and > allowing to specify all kinds of restrictions on a type, > it won't help because those checks have to be done at run time, and are > not optional. This is a great point. Regardless of the nature of the annotation, we can't let this become a way out for lazy programmers. Having annotations is no excuse for poor error checking. > 2. They can give meta information to a framework > > An admittedly contrieved example, let's expand our register function: > > public Response register( int age, @Inject @Scope("session") UserService > userService, @Authenticated User user) .... This is too much to put on one line in Python. We should be aiming to make code cleaner and promote proper error checking and documentation. > The flexibility annotations give in Java makes programming much less > frustrating. It also took quite some time before > annotations were widly used (they were only available starting in Java > 5) and people started finding more uses for them. > I think this will also be true for Python, it will take time before > people find useful ways for them. Redefining them now to > be "not much more" than static type information feels wrong to me. I agree that annotations can be useful, but I don't think they should be used for type checking at this scale. Sunjay _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Thu Aug 14 21:48:50 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 15 Aug 2014 05:48:50 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> Message-ID: <851tsi7syl.fsf@benfinney.id.au> Steven D'Aprano writes: > On Wed, Aug 13, 2014 at 10:29:48PM +0200, Christian Heimes wrote: > > > 1) I'm not keen with the naming of mypy's typing classes. The visual > > distinction between e.g. dict() and Dict() is too small and IMHO > > confusing for newcomers. [?] > > Would it be possible, and desirable, to modify the built-in types so > that we could re-use them in the type annotations? That would address my concern. With that change, when the programmer who reads the code encounters mention of a type, it means precisely the same type object as it appears to be and not some special beast. -- \ ?Science embraces facts and debates opinion; religion embraces | `\ opinion and debates the facts.? ?Tom Heehler, _The Well-Spoken | _o__) Thesaurus_ | Ben Finney From tjreedy at udel.edu Thu Aug 14 21:59:39 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 14 Aug 2014 15:59:39 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On 8/14/2014 11:21 AM, Andrew Barnert wrote: > On Aug 14, 2014, at 7:37, Ryan Gonzalez > > wrote: > >> On 8/13/2014 3:44 PM, Guido van Rossum wrote: > >> Now consider an extended version (after Lucas). >> >> def fib(n, a, b): >> i = 0 >> while i <= n: >> print(i,a) >> i += 1 >> a, b = b, a+b >> >> The only requirement of a, b is that they be addable. Any numbers >> should be allowed, as in fib(10, 1, 1+1j), but so should fib(5, >> '0', '1'). Addable would be approximated from below by >> Union(Number, str). >> >> >> Unless MyPy added some sort of type classes... > > By "type classes", do you mean this in the Haskell sense, or do you mean > classes used just for typing--whether more granular ABCs (like an > Addable which both Number and AnyStr and probably inherit) or typing.py > type specifiers (like an Addable defined as Union(Number, AnyStr)? What I like is the idea of protocol based types, as in Andrey Vlasovskikh's slide 26 http://blog.pirx.ru/media/files/2013/python-optional-typing/#26 class Addable(Protocol): @abstractmethod def __add__(self, other): pass Even this does not capture the actual requirement that a and b be addable together, in that order. Addable_together is inherently a pair concept. Syntactically, that could be handled by passing a pair. def fib(n:Comparable_to_int, pair:Addable_together) -> Type resulting from pair: However, the actual Python rules for Addable_together are rather complex. A static type checker for this would be extremely difficult to write. The best dynamic type checker is to try and let Python say no by raising. > It's also worth noting that the idea that this function should take a > Number or a str seems way off. As written, it *does* take any pair that can be repeatedly added together. > It's questionable whether it should accept str, I recently read Douglas Hofstadler's Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought. In the first chapter he discussed puzzles like: A series begins 0, 1, 01, 101, ..., what is the next item. He started with number sequences and moved on to digit string sequences like the above, where the digit strings are *not* interpreted as number. Generic functions and duck-typing encourages this sort of creative thinking. > but if it does, shouldn't it also accept bytes, > bytearray, and other string-like types? Of course. A descriptive type should not invalidate anything that is allowed. > What about sequences? And meanwhile, > whether or not it accepts str, it should probably accept np.ndarray and > other types of element-wise adding types. The function above already does accept such. > If you create an Addable type, > it has to define, globally, which of those counts as addable, but > different functions will have different definitions that make sense. A single arg having .__(r)add__ is trivial. The real difficultly is expressing *addable to each other* and relating the result type to the types of the members of the pair. -- Terry Jan Reedy From brakhane at googlemail.com Thu Aug 14 22:05:59 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Thu, 14 Aug 2014 22:05:59 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <53ED16A7.5080007@googlemail.com> Am 13.08.2014 23:46, schrieb Guido van Rossum: > > > Mypy has a cast() operator that you can use to shut it up when you > (think you) know the conversion is safe. > Does Mypy provide a way to "fix/monkeypatch" incorrect type declarations in function signatures? For example, by modifying __annotations__? My pet peeve of static languages is that programmers are often too fixated on their particular problem that they don't think about alternate uses for their code and make the type declarations uncessarily complex. For example, in Java nearly every method in Java that deals with character strings uses "String" as parameter type, while they should have used "CharSequence". Having to read an entire input stream and storing it in a String just to be able to use a method is not fun (String is final in Java) I'm worried that in Python we will have utility functions that declare they require a List[int], when in fact they actually only require a Sequence[int] or Sequence[Number]. While Mypy's cast is nice in that I won't have to wrap my Integer Tuple in list like object, having to cast it every time I use a particular broken utility method feels very ugly to me; and defining a wrapper function with the correct type information feels like unnecessary run time overhead for no gain. Don't get me wrong, I'm not entirely against some kind of type checking, but I fear that there must exist possible workarounds for badly written code. From varma.sunjay at gmail.com Thu Aug 14 22:16:20 2014 From: varma.sunjay at gmail.com (Sunjay Varma) Date: Thu, 14 Aug 2014 16:16:20 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53ED16A7.5080007@googlemail.com> References: <53ED16A7.5080007@googlemail.com> Message-ID: +1 This is definitely something to consider. One of the many benefits of Python is that you can use objects with equivalent interfaces in functions that may not have expected that type while they were being written. One thing to note: if we try to make this syntax too verbose, it may lose all of its purpose all together. For example: one (bad) solution to support what I described above is to define some sort of convoluted syntax for specifying the exact interface a function supports. At this point, any addition to the language would do nothing but hinder it. Anything that verbose loses is too complex to be valuable. We have to be careful with this. If we do accept it (or any of the many alternatives suggested so far), then we should choose a few use cases and focus on solving them as best as possible. On Aug 14, 2014 4:06 PM, "Dennis Brakhane" wrote: > > Am 13.08.2014 23:46, schrieb Guido van Rossum: > > > > > > Mypy has a cast() operator that you can use to shut it up when you > > (think you) know the conversion is safe. > > > Does Mypy provide a way to "fix/monkeypatch" incorrect type declarations > in function signatures? For example, by modifying __annotations__? > > My pet peeve of static languages is that programmers are often too > fixated on their particular problem that they don't think about alternate > uses for their code and make the type declarations uncessarily complex. > > For example, in Java nearly every method in Java that deals with > character strings uses "String" as parameter type, while they should > have used "CharSequence". Having to read an entire input stream and > storing it in a String just to be able to use a method is not fun > (String is final in Java) > > I'm worried that in Python we will have utility functions that declare > they require a List[int], when in fact they actually only require a > Sequence[int] or > Sequence[Number]. > > While Mypy's cast is nice in that I won't have to wrap my Integer Tuple > in list like object, having to cast it every time I use a particular > broken utility method > feels very ugly to me; and defining a wrapper function with the correct > type information feels like unnecessary run time overhead for no gain. > > Don't get me wrong, I'm not entirely against some kind of type checking, > but I fear that there must exist possible workarounds for badly written > code. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ceronman at gmail.com Thu Aug 14 22:29:31 2014 From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=) Date: Thu, 14 Aug 2014 22:29:31 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814185555.GU4525@ando> References: <20140814185555.GU4525@ando> Message-ID: On Thu, Aug 14, 2014 at 8:55 PM, Steven D'Aprano wrote: > On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote: > > > One interesting feature of TypeScript is that it allows you to annotate > > existing code without modifying it, by using external definition files. > In > > the JavaScript world, many people have contributed TypeScript annotation > > files for popular JS libraries (http://definitelytyped.org/). > > > > I think this is possible in Python as well doing something like this: > > > > @annotate('math.ciel') > > def ciel(x: float) -> int: > > pass > > I'm afraid I don't understand what the annotate decorator is doing here. > Can you explain please? > The idea is to add type annotations to modules without modifying them. For example, in this case, the stdlib math module is defined and implemented in C, but you still want to have annotations for it so that if you write math.ciel('foo'), the static type analyzer gives you a error or warning. By defining a new module, for example math_annotations.py with empty functions with annotated signatures, you can let the static analyzer know what are the annotations for another module. In this example, the annotate decorator is just a way of telling the static analyzer that these annotations apply to the math.ceil function, not math_annotations.ceil. This is what TypeScript does to annotate popular libraries written in plain JavaScript with zero type information. Manuel. > > > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Aug 14 22:33:00 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 14 Aug 2014 21:33:00 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814185245.GT4525@ando> References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> Message-ID: On 14 August 2014 19:52, Steven D'Aprano wrote: > I want to pass a Decimal to foo(). All I have to do is *not* install > mypy, or disable it, and lo and behold, like magic, the type checking > doesn't happen, and foo() operates by duck-typing just like in the glory > days of Python 1.5. Both Fred and I are now happy, and with the explicit > isinstance check removed, the only type checking that occurs when I run > Fred's library are the run-time duck-typing checks. Thanks for explaining Steven, that's a lot clearer. I understand where you're coming from now. I still don't agree, however. I suspect what's more likely to happen is that Fred writes his code, a user goes to run it with duck typing, and it breaks. Assuming the static checker is in CPython and on by default, there are a number of options here, most of which are bad: 1. The user doesn't know about the type checker and Googles the problem. They find there's a flag they can pass to make the problem go away, so they do. They have now learned a bad habit: to silence these errors, pass this flag. They can no longer gain any benefits from the type checker: it may as well have been not there (or off by default). 2. The user doesn't know about the type checker and blames Fred's library, opening a bug report. In extreme cases, for popular libraries, this will happen so often that Fred will either relent and remove the annotations or get increasingly frustrated and take it out on the users. (I'm speaking from experience in this regard.) 3. The user knows about the type checker and isn't using it. They turn it off. Fine, this is ok. 4. The user knows about the type checker but is using it for their own code in the same program. They're between a rock and a hard place: either they turn off the checker and lose the benefit in their own code, or they stop duck typing. This is actually the worst of these cases. Basically, my objection is to the following (admittedly extreme) case: a static type checker that is a) present in the core distribution, b) on by default, and c) with the only available scope being per-program. I think that such an implementation is a recipe for having everyone learn to turn the checker off, wasting all the effort associated with it. I am much happier if (b) goes away. Off by default is fine. Not in the core distribution at all is also fine (because it's effectively off-by-default). Allowing refined scopes is also a good idea, but doesn't solve the core problem: people will continue to just turn it off. I am not averse to having static checking be an option for Python and for annotations to be the mechanism by which such typing is done. I just think we should be really cautious about ever including it in CPython. From apalala at gmail.com Thu Aug 14 22:12:09 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 14 Aug 2014 15:42:09 -0430 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814174512.GQ4525@ando> References: <20140814174512.GQ4525@ando> Message-ID: On Thu, Aug 14, 2014 at 1:15 PM, Steven D'Aprano wrote: > Yes. You say that as if it were a bad thing. It is not. Python 3 is > here to stay and we should be promoting Python 3 only code. There is > absolutely no need to apologise for that fact. If people are happy with > Python the way it is in 2.7, or 1.5 for that matter, that's great, they > can stay on it for ever, but all new features are aimed at 3.x and not > 2.x or 1.x. > That's reasonable..., in theory. Reality is that most of the people most supportive of the migration towards Python 3 are currently writing code that is compatible with both 3.x and 2.[67]. You don't leave your people behind (not without a lifeboat). Since its decided there will not be a 2.8, the right thing to do is to delay decisions about static typing (or restrictions on annotations) till 4.0. It would be "a bad thing" to break or deprecate existing 3.x code with 3.5. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Aug 14 22:44:56 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 14 Aug 2014 13:44:56 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814185555.GU4525@ando> Message-ID: <53ED1FC8.1040102@stoneleaf.us> On 08/14/2014 01:29 PM, Manuel Cer?n wrote: > On Thu, Aug 14, 2014 at 8:55 PM, Steven D'Aprano wrote: >> On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote: >>> >>> >>> @annotate('math.ciel') >>> def ciel(x: float) -> int: >>> pass >> >> I'm afraid I don't understand what the annotate decorator is doing here. >> Can you explain please? > > The idea is to add type annotations to modules without modifying them. For example, in this case, the stdlib math module > is defined and implemented in C, but you still want to have annotations for it so that if you write math.ciel('foo'), > the static type analyzer gives you a error or warning. By defining a new module, for example math_annotations.py with > empty functions with annotated signatures, you can let the static analyzer know what are the annotations for another > module. In this example, the annotate decorator is just a way of telling the static analyzer that these annotations > apply to the math.ceil function, not math_annotations.ceil. To make sure I understand: The above snippet is located in a file named 'math_annotations.py', and the annotate decorator says "the following function annotation should be stored against the 'ceil' function in the 'math' module, not the 'ceil' function in this current module" ? -- ~Ethan~ From ethan at stoneleaf.us Thu Aug 14 22:51:53 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 14 Aug 2014 13:51:53 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> Message-ID: <53ED2169.6080800@stoneleaf.us> On 08/14/2014 01:33 PM, Cory Benfield wrote: > On 14 August 2014 19:52, Steven D'Aprano wrote: >> >> I want to pass a Decimal to foo(). All I have to do is *not* install >> mypy, or disable it, and lo and behold, like magic, the type checking >> doesn't happen, and foo() operates by duck-typing just like in the glory >> days of Python 1.5. Both Fred and I are now happy, and with the explicit >> isinstance check removed, the only type checking that occurs when I run >> Fred's library are the run-time duck-typing checks. > > Thanks for explaining Steven, that's a lot clearer. I understand where > you're coming from now. > > I still don't agree, however. I suspect what's more likely to happen > is that Fred writes his code, a user goes to run it with duck typing, > and it breaks. Assuming the static checker is in CPython and on by > default, there are a number of options here, most of which are bad: These are bad assumptions, since the PEP is about defining how annotations are to be used and specifically states there will be *no run-time checking*. To be of use at all you have to get a third-party program (mypy at this point) and use it. So the scenario you list simply isn't going to happen... at least, not like that. What could happen is newbie team member tries to check something in, but mypy and annotations are in the pre-check, no-one has told newbie team member about mypy or newbie forgot and is too embarrassed to go ask someone, so same basic problem arises. That, however, is mostly outside the concerns of developing Python. -- ~Ethan~ From apalala at gmail.com Thu Aug 14 23:03:13 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 14 Aug 2014 16:33:13 -0430 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814185245.GT4525@ando> References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> Message-ID: On Thu, Aug 14, 2014 at 2:22 PM, Steven D'Aprano wrote: > def foo(x:float)->float: > return (x+1)/2 > > and rely on mypy to check the types at compile time. Fred is happy: he > has static type checks, Python does it automatically for him (once he > has set up his build system to call mypy), and he is now convinced that > foo() is type-safe and an isinstance check at run-time would be a waste > of cycles. > The foo() kind of examples won't cut it. The standard library and other important Python libraries will take an argument of any of several "reasonable" but otherwise unrelated types... and do the right thing. Such is duck-typing. Trying to find the "common abstract type" to cast it in stone is a waste of time. For example, see https://docs.python.org/3.3/library/json.html#json.dump, or take a look at http://pyyaml.org/. In fact, Why aren't we instead discussing the much more interesting topic of *type inference*, as it's going on in projects like PyDev, PyCharm, Rope, and others? I would be all-in for an approach that helps make code-completion tools more precise and more available, and that would enable linters to tell me why a certain call will probably not work. It could be a milder approach, with "type hinting" for when the tools can't infer the type, instead of one of "type specification". def set_color(self, color): ... o.set_color("red") It could well be that 'color' should actually be one of the constants in a Color enum, and the call should fail statically... if it was Java. In the Python way, the method will probably figure out the caller's intentions, and do the right thing. Cheers, p.s. I just had an epiphany: This discussion is not about empowering Python programmers, but again about finding "magic" to allow bad (read "cheap") programmers to write good programs. I'm out! -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob at redivi.com Thu Aug 14 22:54:20 2014 From: bob at redivi.com (Bob Ippolito) Date: Thu, 14 Aug 2014 13:54:20 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> Message-ID: On Thu, Aug 14, 2014 at 1:33 PM, Cory Benfield wrote: > On 14 August 2014 19:52, Steven D'Aprano wrote: > > I want to pass a Decimal to foo(). All I have to do is *not* install > > mypy, or disable it, and lo and behold, like magic, the type checking > > doesn't happen, and foo() operates by duck-typing just like in the glory > > days of Python 1.5. Both Fred and I are now happy, and with the explicit > > isinstance check removed, the only type checking that occurs when I run > > Fred's library are the run-time duck-typing checks. > > Thanks for explaining Steven, that's a lot clearer. I understand where > you're coming from now. > > I still don't agree, however. I suspect what's more likely to happen > is that Fred writes his code, a user goes to run it with duck typing, > and it breaks. Assuming the static checker is in CPython and on by > default, there are a number of options here, most of which are bad: > These assumptions are incorrect. * Adding the checker to CPython is not part of this proposal. * Turning it on by default is not part of this proposal. * Having the type checker in any way associated with the runtime is not part of this proposal. What is part of this proposal? * An effort to standardize on a particular syntax for optional type annotations using Python 3 function annotations * The syntax will be handwavingly based on what's in mypy, and thus implies some semantics but not the implementation of the type checker/inference/etc. * A suggestion to use tools such as mypy to provide a static type checking pass in much the same way that people use linters today * A suggestion that IDEs, documentation tools, etc. take advantage of the information provided by the type annotations * Deprecation of using function annotations for any other purpose. They can't really be composed, and ideally type checkers can work primarily at the syntax level and not have to evaluate the module with a full Python interpreter in order to extract the annotations, so it's best to keep the syntax uniform. > I am not averse to having static checking be an option for Python and > for annotations to be the mechanism by which such typing is done. I > just think we should be really cautious about ever including it in > CPython. So, it sounds like you're not averse to what is has been proposed. :) -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From brakhane at googlemail.com Fri Aug 15 00:11:16 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 00:11:16 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53ED16A7.5080007@googlemail.com> Message-ID: <53ED3404.7080309@googlemail.com> Am 14.08.2014 22:16, schrieb Sunjay Varma: > One of the many benefits of Python is that you can use objects with > equivalent interfaces in functions that may not have expected that > type while they were being written. Exactly. As others have already noted, Guido's example is actually restricting code from calling it with a file object for no particular reason. Another thing is that it restricts the entries of the iterables to be str, while it actually only requires them to provide a split method. For example, and I'm aware that this is a contrieved case, let's assume word_count would actually be somehow extremely optimized code that I don't want to rewrite. I now want to count all prime factors in a list of integers. Without annotations I *could* do something like this ("Language for consenting adults"): class IntWrapper(int): def split(self): return primefactors(self) word_count(IntWrapper(i) for i in my_integer_list) I don't want to argue whether that's good code or not (it probably isn't), it would be possible to write such code and it will work correctly. For mypy to accept such code, the type declaration of word_count would have to be something like def word_count(input: Iterable[has("split() -> T")]) -> Dict[T, int] Which would require a very complex type system. If I understand MyPy correctly, the above would be possible, but it would have to look something like cast(Dict[int, int], word_count(cast(List[str], (IntWrapper(i) for i in my_integer_list))) One could argue that this ugly piece of code is the rightful punishment for abusing duck typing, but I'm not convinced you should be forced to make the code unreadable (and therefore refactor your code and reimplement word_count yourself). > We have to be careful with this. If we do accept it (or any of the > many alternatives suggested so far), then we should choose a few use > cases and focus on solving them as best as possible. Yes, my fear is that we will either end up with a broken type system like Java (which provides more problems than it solves) or end up with a hugely complex type system and type hierachy like Scala (see http://www.scala-lang.org/api/2.11.2/#scala.collection.MapLike for a quick example what you end up with) From ahammel87 at gmail.com Fri Aug 15 00:35:04 2014 From: ahammel87 at gmail.com (Alex Hammel) Date: Thu, 14 Aug 2014 15:35:04 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53ED3404.7080309@googlemail.com> References: <53ED16A7.5080007@googlemail.com> <53ED3404.7080309@googlemail.com> Message-ID: If we're looking for precedents in other languages, the Dialyzer project for Erlang is worth a look. It does completely optional static type checking without affecting the runtime at all. It was pretty widely adopted and it works well in my experience. I think it's a bit different from what's being proposed here (it does type inference as well, for instance), but it is a more or less successful example of adding optional static checks to a dynamically typed language. Based on my experiences with Dialyzer and the way I've used function annotations in Python to date, I'm really excited about this proposal. On Thu, Aug 14, 2014 at 3:11 PM, Dennis Brakhane wrote: > Am 14.08.2014 22:16, schrieb Sunjay Varma: > > One of the many benefits of Python is that you can use objects with > > equivalent interfaces in functions that may not have expected that > > type while they were being written. > Exactly. As others have already noted, Guido's example is actually > restricting code from calling it with a file object for no particular > reason. > > Another thing is that it restricts the entries of the iterables to be > str, while it actually only requires them to provide a split method. > > For example, and I'm aware that this is a contrieved case, let's assume > word_count would actually be somehow extremely optimized code that > I don't want to rewrite. > > I now want to count all prime factors in a list of integers. > > Without annotations I *could* do something like this ("Language for > consenting adults"): > > class IntWrapper(int): > def split(self): > return primefactors(self) > > word_count(IntWrapper(i) for i in my_integer_list) > > I don't want to argue whether that's good code or not (it probably > isn't), it would be possible to write such code and it will work correctly. > > For mypy to accept such code, the type declaration of word_count would > have to be something like > > def word_count(input: Iterable[has("split() -> T")]) -> Dict[T, int] > > Which would require a very complex type system. > > If I understand MyPy correctly, the above would be possible, but it > would have to look something like > > cast(Dict[int, int], word_count(cast(List[str], (IntWrapper(i) for i > in my_integer_list))) > > One could argue that this ugly piece of code is the rightful punishment > for abusing duck typing, but I'm not convinced > you should be forced to make the code unreadable (and therefore refactor > your code and reimplement word_count yourself). > > > > We have to be careful with this. If we do accept it (or any of the > > many alternatives suggested so far), then we should choose a few use > > cases and focus on solving them as best as possible. > Yes, my fear is that we will either end up with a broken type system > like Java (which provides more problems than it solves) or end up with a > hugely complex type system and type hierachy like Scala > (see http://www.scala-lang.org/api/2.11.2/#scala.collection.MapLike for > a quick example what you end up with) > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Aug 15 00:44:54 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 15 Aug 2014 08:44:54 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53ED16A7.5080007@googlemail.com> References: <53ED16A7.5080007@googlemail.com> Message-ID: On Fri, Aug 15, 2014 at 6:05 AM, Dennis Brakhane wrote: > Does Mypy provide a way to "fix/monkeypatch" incorrect type declarations > in function signatures? For example, by modifying __annotations__? AIUI there's a separation of execution and type checking here - you *either* run under mypy to check types, *or* run under Python to execute the code. If that's the case, it should be possible to create a magic module that you import which overwrites a few functions with stubs. # spam.py (which you can't change) def func(x: int) -> int: return x * 3 + 5 # type_fixes.py in the regular PYTHONPATH """This is a stub; the mypy equivalent fixes some type annotations used in third-party libraries.""" # type_fixes.py in the mypy PYTHONPATH import spam def func(x: Number) -> Number: pass spam.func = func # in main_module.py import type_fixes import spam x = func(42.5) Monkey-patching entire functions shouldn't be a problem if they don't need bodies. ChrisA From brakhane at googlemail.com Fri Aug 15 00:59:36 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 00:59:36 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53ED16A7.5080007@googlemail.com> <53ED3404.7080309@googlemail.com> Message-ID: <53ED3F58.1040004@googlemail.com> Am 15.08.2014 00:35, schrieb Alex Hammel: > I think it's a bit different from what's being proposed here (it does > type inference as well, for instance) FWIW, David Halter is currently working on extending Jedi to include a linter that does type inference. For example, the current development version can do the following $ cat foo.py def foo(x): return x + "foo" foo(1) $ python -m jedi linter foo.py foo.py:2:13: E11 TypeError: unsupported operand type(s) for +: and From greg.ewing at canterbury.ac.nz Fri Aug 15 00:59:33 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 15 Aug 2014 10:59:33 +1200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <53ED3F55.6030502@canterbury.ac.nz> willvarfar at gmail.com wrote: > But proper annotation support in the language rather than overloading > comments would definitely be my preference. I've been thinking the same thing. -- Greg From bob at redivi.com Thu Aug 14 22:58:45 2014 From: bob at redivi.com (Bob Ippolito) Date: Thu, 14 Aug 2014 13:58:45 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53ED1FC8.1040102@stoneleaf.us> References: <20140814185555.GU4525@ando> <53ED1FC8.1040102@stoneleaf.us> Message-ID: On Thu, Aug 14, 2014 at 1:44 PM, Ethan Furman wrote: > On 08/14/2014 01:29 PM, Manuel Cer?n wrote: > >> On Thu, Aug 14, 2014 at 8:55 PM, Steven D'Aprano wrote: >> >>> On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote: >>> >>>> >>>> >>>> @annotate('math.ciel') >>>> def ciel(x: float) -> int: >>>> pass >>>> >>> >>> I'm afraid I don't understand what the annotate decorator is doing here. >>> Can you explain please? >>> >> >> The idea is to add type annotations to modules without modifying them. >> For example, in this case, the stdlib math module >> is defined and implemented in C, but you still want to have annotations >> for it so that if you write math.ciel('foo'), >> the static type analyzer gives you a error or warning. By defining a new >> module, for example math_annotations.py with >> empty functions with annotated signatures, you can let the static >> analyzer know what are the annotations for another >> module. In this example, the annotate decorator is just a way of telling >> the static analyzer that these annotations >> apply to the math.ceil function, not math_annotations.ceil. >> > > To make sure I understand: The above snippet is located in a file named > 'math_annotations.py', and the annotate decorator says "the following > function annotation should be stored against the 'ceil' function in the > 'math' module, not the 'ceil' function in this current module" ? > mypy has existing infrastructure in the form of stub modules to support this use case: https://github.com/JukkaL/mypy/blob/master/stubs/3.2/math.py I'm not sure if stubs can easily be pulled in from outside of mypy, but that would be straightforward to support if it doesn't work already. -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseph.martinot-lagarde at m4x.org Fri Aug 15 01:43:33 2014 From: joseph.martinot-lagarde at m4x.org (Joseph Martinot-Lagarde) Date: Fri, 15 Aug 2014 01:43:33 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814174512.GQ4525@ando> Message-ID: <53ED49A5.1070404@m4x.org> Le 14/08/2014 22:12, Juancarlo A?ez a ?crit : > > On Thu, Aug 14, 2014 at 1:15 PM, Steven D'Aprano > > wrote: > > Yes. You say that as if it were a bad thing. It is not. Python 3 is > here to stay and we should be promoting Python 3 only code. There is > absolutely no need to apologise for that fact. If people are happy with > Python the way it is in 2.7, or 1.5 for that matter, that's great, they > can stay on it for ever, but all new features are aimed at 3.x and not > 2.x or 1.x. > > > That's reasonable..., in theory. > > Reality is that most of the people most supportive of the migration > towards Python 3 are currently writing code that is compatible with both > 3.x and 2.[67]. > > You don't leave your people behind (not without a lifeboat). > > Since its decided there will not be a 2.8, the right thing to do is to > delay decisions about static typing (or restrictions on annotations) > till 4.0. It would be "a bad thing" to break or deprecate existing 3.x > code with 3.5. It breaks nothing since it's optional. What you say is true for every new feature in python: using a feature present in on version of python prevents the code to be compatible with previous versions. I mostly write code compatible with 2.7 and 3.x, I won't use annotations the same way I won't use asyncio or yield from... > > Cheers, > -- > Juancarlo *A?ez* > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From guido at python.org Fri Aug 15 01:56:39 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 14 Aug 2014 16:56:39 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads Message-ID: I have read pretty much the entire thread up and down, and I don't think I can keep up with responding to every individual piece of feedback. (Also, a lot of responses cancel each other out. :-) I think there are three broad categories of questions to think about next. (A) Do we even need this? (B) What syntax to use? (C) Does/should it support ? Taking these in turn: (A) Do we even need a standard for optional static typing? Many people have shown either support for the idea, or pointed to some other system that addresses the same issue. On the other hand, several people have claimed that they don't need it, or that they worry it will make Python less useful for them. (However, many of the detractors seem to have their own alternative proposal. :-) In the end I don't think we can ever know for sure -- but my intuition tells me that as long as we keep it optional, there is a real demand. In any case, if we don't start building something we'll never know whether it'll be useful, so I am going to take a leap of faith and continue to promote this idea. I am going to make one additional assumption: the main use cases will be linting, IDEs, and doc generation. These all have one thing in common: it should be possible to run a program even though it fails to type check. Also, adding types to a program should not hinder its performance (nor will it help :-). (B) What syntax should a standard system for optional static typing use? There are many interesting questions here, but at the highest level there are a few choices that constrain the rest of the discussion, and I'd like to start with these. I see three or four "families" of approaches, and I think the first order is to pick a family. (1) The mypy family. (http://mypy-lang.org/) This is characterized by its use of PEP 3107 function annotations and the constraint that its syntax must be valid (current) Python syntax that can be evaluated without errors at function definition time. However, mypy also supports collecting annotations in separate "stub" files; this is how it handles annotations for the stdlib and C extensions. When mypy annotations occur inline (not in a stub file) they are used to type check the body of the annotated function as well as input for type checking its callers. (2) The pytypedecl family. (https://github.com/google/pytypedecl) This is a custom syntax that can only be used in separate stub files. Because it is not constrained by Python's current syntax, its syntax is slightly more elegant than mypy. (3) The PyCharm family. ( http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html) This is a custom syntax that lives entirely in docstrings. There is also a way to use stub files with this. (In fact, every viable approach has to support some form of stub files, if only to describe signatures for C extensions.) (I suppose we could add a 4th family that puts everything in comments, but I don't think anyone is seriously working on such a thing, and I don't see any benefits.) There's also a variant of (1) that ?ukasz Langa would like to see -- use the syntactic position of function annotations but using a custom syntax (e.g. one similar to the pytypedecl syntax) that isn't evaluated at function-definition time. This would have to use "from __future__ import " for backward compatibility. I'm skeptical about this though; it is only slightly more elegant than mypy, and it would open the floodgates of unconstrained language design. So how to choose? I've read passionate attacks and defenses of each approach. I've got a feeling that the three projects aren't all that different in maturity (all are well beyond the toy stage, none are quite ready for prime time). In terms of specific type system features (e.g. forward references, generic types, duck typing) I expect they are all acceptable, and all probably need some work (and there's no reason to assume that work can't be done). All support stubs so you can specify signatures for code you can't edit (whether C extension, stdlib or just opaque 3rd party code). To me there is no doubt that (1) is the most Pythonic approach. When we discussed PEP 3107 (function annotations) it was always my goal that these would eventually be used for type annotations. There was no consensus at the time on what the rules for type checking should be, but their syntactic position was never in doubt. So we decided to introduce "annotations" in Python 3 in the hope that 3rd party experiments would eventually produce something satisfactory. Mypy is one such experiment. One of the important lessons I draw from mypy is that type annotations are most useful to linters, and should (normally) not be used to enforce types at run time. They are also not useful for code generation. None of that was obvious when we were discussing PEP 3107! I don't buy the argument that PEP 3107 promises that annotations are completely free of inherent semantics. It promises compatibility, and I take that very seriously, but I think it is reasonable to eventually deprecate other uses of annotations -- there aren't enough significant other uses for them to warrant crippling type annotations forever. In the meantime, we won't be breaking existing use of annotations -- but they may confuse a type checker, whether a stand-alone linter like mypy or built into an IDE like PyCharm, and that may serve as an encouragement to look for a different solution. Most of the thornier issues brought up against mypy wouldn't go away if we adopted another approach: whether to use concrete or abstract types, the use of type variables, how to define type equivalence, the relationship between a list of ints and a list of objects, how to spell "something that implements the buffer interface", what to do about JSON, binary vs. text I/O and the signature of open(), how to check code that uses isinstance(), how to shut up the type checker when you know better... The list goes on. There will be methods whose type signature can't be spelled (yet). There will be code distributed with too narrowly defined types. Some programmers will uglify their code to please the type checker. There are questions about what to do for older versions of Python. I find mypy's story here actually pretty good -- the mypy codec may be a hack, but so is any other approach. Only the __future__ approach really loses out here, because you can't add a new __future__ import to an old version. So there you have it. I am picking the mypy family and I hope we can start focusing on specific improvements to mypy. I also hope that somebody will write converters from pytypedecl and PyCharm stubs into mypy stubs, so that we can reuse the work already put into stub definitions for those two systems. And of course I hope that PyCharm and pytypedecl will adopt mypy's syntax (initially in addition to their native syntax, eventually as their sole syntax). PS. I realize I didn't discuss question (C) much. That's intentional -- we can now start discussing specific mypy features in separate threads (or in this one :-). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Aug 15 02:14:39 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 10:14:39 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> Message-ID: <20140815001439.GX4525@ando> On Thu, Aug 14, 2014 at 01:54:20PM -0700, Bob Ippolito wrote: > What is part of this proposal? > > * An effort to standardize on a particular syntax for optional type > annotations using Python 3 function annotations +1 on that. > * The syntax will be handwavingly based on what's in mypy, and thus implies > some semantics but not the implementation of the type checker/inference/etc. I have some reservations as to whether the mypy syntax is mature enough to bake in as standard, but other than that reservation, I prefer the mypy approach over putting type declarations in docstrings. +1 > * A suggestion to use tools such as mypy to provide a static type checking > pass in much the same way that people use linters today "Tools such as mypy" is a very important point. If there is a standard syntax for type annotations, there's no reason why other linters couldn't support it as well. +1 > * A suggestion that IDEs, documentation tools, etc. take advantage of the > information provided by the type annotations +1 > * Deprecation of using function annotations for any other purpose. They > can't really be composed, and ideally type checkers can work primarily at > the syntax level and not have to evaluate the module with a full Python > interpreter in order to extract the annotations, so it's best to keep the > syntax uniform. -1 I don't think this one is justified. At the very least, I think the decision to deprecate or not should be deferred until at least 3.7. It's enough to say that: - you can use function annotations for type checking; or - you can use function annotations for something else; but not both at the same time. I don't think it is a big burden to have mypy and other linters support an "opt-out" decorator, say, so that projects can use annotations for something else without confusing the linter. As I understand it, the current behaviour of mypy is that you have to import typing in the module before it will type check the module, so that already gives you a way to skip type annotations on a per-module basis: just don't import typing. -- Steven From rosuav at gmail.com Fri Aug 15 02:29:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 15 Aug 2014 10:29:21 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140815001439.GX4525@ando> References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> <20140815001439.GX4525@ando> Message-ID: On Fri, Aug 15, 2014 at 10:14 AM, Steven D'Aprano wrote: > I don't think this one is justified. At the very least, I think the > decision to deprecate or not should be deferred until at least 3.7. It's > enough to say that: > > - you can use function annotations for type checking; or > > - you can use function annotations for something else; But who is "you"? Presumably the application author. What about imported modules - what will they use annotations for? Will conflicting uses break stuff? Or, conversely, will every use of annotations have to be meta-annotated with its purpose, to try to avoid breaking things? Simpler to just say "this feature is standardly used for this purpose", and let people break that convention at their own risk if they like. Deprecation of other options fits this. ChrisA From bob at redivi.com Fri Aug 15 02:46:51 2014 From: bob at redivi.com (Bob Ippolito) Date: Thu, 14 Aug 2014 17:46:51 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140815001439.GX4525@ando> References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> <20140815001439.GX4525@ando> Message-ID: On Thu, Aug 14, 2014 at 5:14 PM, Steven D'Aprano wrote: > On Thu, Aug 14, 2014 at 01:54:20PM -0700, Bob Ippolito wrote: > > * Deprecation of using function annotations for any other purpose. They > > can't really be composed, and ideally type checkers can work primarily at > > the syntax level and not have to evaluate the module with a full Python > > interpreter in order to extract the annotations, so it's best to keep the > > syntax uniform. > > -1 > > I don't think this one is justified. At the very least, I think the > decision to deprecate or not should be deferred until at least 3.7. It's > enough to say that: > > - you can use function annotations for type checking; or > > - you can use function annotations for something else; > > but not both at the same time. I don't think it is a big burden to have > mypy and other linters support an "opt-out" decorator, say, so that > projects can use annotations for something else without confusing the > linter. > Certainly using annotations for purposes other than type checking would be permissible, but discouraged. No enforcement of any kind is proposed, just language to go in the documentation. It would be non-trivial for the Python compiler or runtime to enforce it anyway! For that sort of annotation, a decorator probably isn't best, some module-level syntax would likely be more appropriate. > As I understand it, the current behaviour of mypy is that you have to > import typing in the module before it will type check the module, so > that already gives you a way to skip type annotations on a per-module > basis: just don't import typing. > The current implementation of mypy will typecheck any module that uses annotations or imports from typing. -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Aug 15 03:13:14 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 11:13:14 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814181554.GR4525@ando> <20140814185245.GT4525@ando> <20140815001439.GX4525@ando> Message-ID: <20140815011314.GY4525@ando> On Fri, Aug 15, 2014 at 10:29:21AM +1000, Chris Angelico wrote: > On Fri, Aug 15, 2014 at 10:14 AM, Steven D'Aprano wrote: > > I don't think this one is justified. At the very least, I think the > > decision to deprecate or not should be deferred until at least 3.7. It's > > enough to say that: > > > > - you can use function annotations for type checking; or > > > > - you can use function annotations for something else; > > But who is "you"? Presumably the application author. The author of the code containing the annotations. > What about imported modules - what will they use annotations for? Whatever they choose. If they import typing and don't explicitly disable typechecking, they will get the default meaning of annotations, which is typechecking. If they perform whatever step is required to tell the linter "don't check here", the linter will treat that module as if it had no annotations. (That doesn't necessarily mean ignoring the module, it only means ignore the annotations. A type checker with type inference might still be able to work with the module, so long as it needs no type hints.) > Will > conflicting uses break stuff? Or, conversely, will every use of > annotations have to be meta-annotated with its purpose, to try to > avoid breaking things? No. I think Guido has the right instinct to make annotations' default purpose be for type-checking, but it's easy enough to make it opt-out. Since this is Python-Ideas, here are some ideas: * If "typing" is not imported at all in the module, linters should ignore annotations inside that module. * Have a decorator that tells linters and other tools "these are not type annotations": from typing import skip @skip def function(x: Spam, y: Eggs)->Cheese: pass Compliant linters and IDEs will now skip function, treating it as if it had no annotations at all, leaving the interpretation of the annotations up to the author of the module. Non-compliant linters, of course, should be beaten with a large halibut, like any other buggy software :-) The last one will probably require a standardized convention for how compliant tools recognise whether or not annotations are for them. Perhaps something like: if '+state' in func.__annotations__: # skip type-checking. I picked '+state' because it is an invalid identifier and so cannot clash with any parameter name. The value of __annotations__['+state'] can remain unspecified, different tools could use it for whatever they like without Python's blessing, only the existence of that key is enough to mark the function as "don't treat these as type annotations". -- Steven From stephen at xemacs.org Fri Aug 15 04:15:10 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 15 Aug 2014 11:15:10 +0900 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814190237.GV4525@ando> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> Message-ID: <87wqaah51t.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > > 2) PEP 3107 only specifies arguments and return values but not > > exceptions that can be raised by a function. Java has the "throws" > > syntax to list possible exceptions: > > > > public void readFile() throws IOException {} > > I understand that this is called a "checked exception" in Java. I also > understand that they are hated and derided Sure, but that's because it's hard for a human to guess what might happen down in lower-level functions, let alone 3rd-party libraries and the runtime -- and your program fails if you guess wrong. Maybe the fact that type-checking is going to be optional mitigates that. I suspect it's not terrible useful, but another idea is to invert the sense, i.e. say what exceptions the function believes it handles, and therefore cannot be raised. > Perhaps with a few years of experience, we might be able to extend this > to exceptions without making the same mistakes as Java's checked > exceptions, but I wouldn't rush into it. +1 to that! From stephen at xemacs.org Fri Aug 15 04:39:55 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 15 Aug 2014 11:39:55 +0900 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814174512.GQ4525@ando> Message-ID: <87vbpuh3wk.fsf@uwakimon.sk.tsukuba.ac.jp> Juancarlo A?ez writes: > Reality is that most of the people most supportive of the migration > towards Python 3 are currently writing code that is compatible with > both 3.x and 2.[67]. And under Guido's proposal, they can continue to do so. They just have to live without the benefits of function annotations in that code, same as they already do. Or they can write a 3to2 tool to strip the annotations. It's just that the benefits to using Python 3 vs. 2/3-compatible syntax may take a quantum leap upward. I don't see a real problem here. All that Guido is proposing to do is to rationalize development effort on something people are already doing by standardizing a good-enough approach, on an opt-in basis. Sure, there are people who want something more coercive, but Python developers aren't going to stand for that, and Guido himself is the first line of defense. He's already said that's not going to happen, not now (and IIUC probably never). From apalala at gmail.com Fri Aug 15 05:01:26 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 14 Aug 2014 22:31:26 -0430 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <87vbpuh3wk.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20140814174512.GQ4525@ando> <87vbpuh3wk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Aug 14, 2014 at 10:09 PM, Stephen J. Turnbull wrote: > All that Guido is proposing to do is to rationalize development effort > on something people are already doing by standardizing a good-enough > approach, on an opt-in basis. Sure, there are people who want > something more coercive, but Python developers aren't going to stand > for that, and Guido himself is the first line of defense. He's > already said that's not going to happen, not now (and IIUC probably > never). > The change has been decreed, so it will be. It's all right, because (as Guido suggests in the decree) we can now take the discussion to how to make it good for all involved. The Python community will likely succeed at that. It is a tautology that only experience will reveal the truth, at least so in our endeavour. Signing off this thread, yours truly, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at ryanhiebert.com Fri Aug 15 05:33:24 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Thu, 14 Aug 2014 22:33:24 -0500 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814190237.GV4525@ando> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> Message-ID: <7CFFAB82-8F95-41FA-A356-F5D1CAE2DA1C@ryanhiebert.com> > On Aug 14, 2014, at 2:02 PM, Steven D'Aprano wrote: > > Would it be possible, and desirable, to modify the built-in types so > that we could re-use them in the type annotations? > > def word_count(input: list[str]) -> dict[str, int]: > > > Since types are otherwise unlikely to be indexable like that, I think > that might work. Huge +1 from me. I know that extending types to do more might not be the first thing someone would want to do, but I think describing types well is exactly the right thing to extend type instances to do. If that were to be considered plausible, it would also make sense to my mind to implement the union operator (|) on types, so that we could use: int | None Which I think would be quite beautiful. (or would it need to be NoneType? Perhaps we can give some special consideration to the None singleton) It?s only tangentially related, but my (likely broken) work on a TypeSet metaclass might also be informative. It?s goal was to bring set operations to types, which has some overlap with the above discussion. https://github.com/ryanhiebert/typeset From abarnert at yahoo.com Fri Aug 15 05:51:16 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 14 Aug 2014 20:51:16 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814190237.GV4525@ando> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> Message-ID: <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> On Thursday, August 14, 2014 12:03 PM, Steven D'Aprano wrote: > > On Wed, Aug 13, 2014 at 10:29:48PM +0200, Christian Heimes wrote: > >> 1) I'm not keen with the naming of mypy's typing classes. The > visual >> distinction between e.g. dict() and Dict() is too small and IMHO >> confusing for newcomers. How about an additional 'T' prefix to make >> clear that the objects are referring to typing objects? >> >> ? from typing import TList, TDict >> >> ? def word_count(input: TList[str]) -> TDict[str, int]: >> ? ? ? ... > > Would it be possible, and desirable, to modify the built-in types so > that we could re-use them in the type annotations? > > ? ? def word_count(input: list[str]) -> dict[str, int]: > > > Since types are otherwise unlikely to be indexable like that, I think > that might work. I strongly agree with the basic sentiment here, but if you take it a little bit farther, it makes things a lot simpler. Most of the code in typing.py is a duplication of the ABCs in the collections.abc module (and io and maybe others). I understand that MyPy couldn't monkeypatch that code into the stdlib, so it had to fork the contents, but if this is going into the stdlib, can't we just modify collections/abc.py instead? Why not have the type information on collections.abc.Sequence be introspectable at runtime? More importantly, why not have the interface defined by collections.abc.Sequence be _exactly_ the same as the one checked by the static type checker, instead of just very similar? This would also make it easier for some libraries to document their types. For example, dbm.* could inherit from or register with MutableMapping[bytes, bytes] instead of just MutableMapping, and then it wouldn't have to explain in the docstring that the keys and values have to be bytes. Once you move the ABC typing to the actual ABCs, the only problem left is the built-in (concrete) collections. But I still don't understand why people even _want_ those to be checked. Does anyone have a good example of a function that needs to restrict its arguments to list[str] instead of Sequence[str]? Guido gave us a _terrible_ example: a function whose most obvious use was on text files, but he added a static type check that prevents it being used that way. And I think that will be the case the vast majority of the time. When I see documentation in a PyPI library or a colleague's code that says a function takes a list, it's almost always a lie; the function in fact takes any iterable (or occasionally any sequence or mutable sequence). (In fact, every exception I can think of is written in C, so it couldn't be easily annotated anyway.) It seems like we're making things a lot harder for ourselves, just for a handful of types that people are almost always going to use wrong. (As a side note, generic ABCs might even be the answer to the AnyStr problem: str implements String[str], bytes implements String[bytes], bytearray implements MutableString[bytes], and classes like PyObjC's NSString can now document that they implement String[str] or String[bytes] as appropriate.) Anyway, these leaves typing.py as nothing but functions (Union, Optional, etc.) on types defined elsewhere, in the rest of the stdlib. From stefan_ml at behnel.de Fri Aug 15 06:12:18 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 15 Aug 2014 06:12:18 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: Guido van Rossum schrieb am 14.08.2014 um 07:24: > On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo wrote: >> You could use AnyStr to make the example work with bytes as well: >> >> def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]: >> result = {} #type: Dict[AnyStr, int] >> >> for line in input: >> for word in line.split(): >> result[word] = result.get(word, 0) + 1 >> return result >> >> Again, if this is just a simple utility function that you use once or >> twice, I see no reason to spend a lot of effort in coming up with the most >> general signature. Types are an abstraction and they can't express >> everything precisely -- there will always be a lot of cases where you can't >> express the most general type. However, I think that relatively simple >> types work well enough most of the time, and give the most bang for the >> buck. > > I heartily agree. But just for the type theorists amongst us, if I really > wanted to write the most general type, how would I express that the AnyStr > in the return type matches the one in the argument? (I think pytypedecl > would use something like T <= AnyStr.) That's how Cython's "fused types" (generics) work, at least. They go by name: same name of the type, same type. Otherwise, use alias names, which make the types independent from each other. http://docs.cython.org/src/userguide/fusedtypes.html While it's a matter of definition what way to go here (same type or not), practice has shown that it's clearly the right decision to make identical types the default. Stefan From guido at python.org Fri Aug 15 06:15:15 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 14 Aug 2014 21:15:15 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: On Thu, Aug 14, 2014 at 8:51 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > > Most of the code in typing.py is a duplication of the ABCs in the > collections.abc module (and io and maybe others). I understand that MyPy > couldn't monkeypatch that code into the stdlib, so it had to fork the > contents, but if this is going into the stdlib, can't we just modify > collections/abc.py instead? Why not have the type information on > collections.abc.Sequence be introspectable at runtime? More importantly, > why not have the interface defined by collections.abc.Sequence be _exactly_ > the same as the one checked by the static type checker, instead of just > very similar? > > This would also make it easier for some libraries to document their types. > For example, dbm.* could inherit from or register with > MutableMapping[bytes, bytes] instead of just MutableMapping, and then it > wouldn't have to explain in the docstring that the keys and values have to > be bytes. > > Once you move the ABC typing to the actual ABCs, the only problem left is > the built-in (concrete) collections. But I still don't understand why > people even _want_ those to be checked. Does anyone have a good example of > a function that needs to restrict its arguments to list[str] instead of > Sequence[str]? Guido gave us a _terrible_ example: a function whose most > obvious use was on text files, but he added a static type check that > prevents it being used that way. And I think that will be the case the vast > majority of the time. When I see documentation in a PyPI library or a > colleague's code that says a function takes a list, it's almost always a > lie; the function in fact takes any iterable (or occasionally any sequence > or mutable sequence). (In fact, every exception I can think of is written > in C, so it couldn't be easily annotated anyway.) It seems like we're > making things a lot harder for ourselves, just for a handful of types that > people are almost > always going to use wrong. > > (As a side note, generic ABCs might even be the answer to the AnyStr > problem: str implements String[str], bytes implements String[bytes], > bytearray implements MutableString[bytes], and classes like PyObjC's > NSString can now document that they implement String[str] or String[bytes] > as appropriate.) > > Anyway, these leaves typing.py as nothing but functions (Union, Optional, > etc.) on types defined elsewhere, in the rest of the stdlib. > I think I already responded to a similar proposal. I think modifying the existing ABCs is fine for Python 3.5 and beyond, but I think I'd like to keep aliasing in the typing module to make it easier to use annotations in earlier Python versions (using a typing.py installed from PyPI). I feel similar about using a|b as a concise way to spell Union[a, n], and int|None would be quite decent as a way to spell optional int. But again, these could only be used in code meant exclusively for Python 3.5 and later. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Aug 15 06:18:50 2014 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 15 Aug 2014 04:18:50 +0000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: <53ED16A7.5080007@googlemail.com> Message-ID: On Thu Aug 14 2014 at 1:06:20 PM Dennis Brakhane wrote: > Am 13.08.2014 23:46, schrieb Guido van Rossum: > > > > > > Mypy has a cast() operator that you can use to shut it up when you > > (think you) know the conversion is safe. > > > Does Mypy provide a way to "fix/monkeypatch" incorrect type declarations > in function signatures? For example, by modifying __annotations__? > > My pet peeve of static languages is that programmers are often too > fixated on their particular problem that they don't think about alternate > uses for their code and make the type declarations uncessarily complex. > > For example, in Java nearly every method in Java that deals with > character strings uses "String" as parameter type, while they should > have used "CharSequence". Having to read an entire input stream and > storing it in a String just to be able to use a method is not fun > (String is final in Java) > > I'm worried that in Python we will have utility functions that declare > they require a List[int], when in fact they actually only require a > Sequence[int] or > Sequence[Number]. > > This always happens, even in docstrings where people state types today. For example, I often correct people during code reviews to say sequence of instead of list of in docstrings. Nothing can really prevent that beyond documentation about annotations having good examples and promoting the use of concept or interface annotations over specific type annotations. But what's being proposed here isn't a strict static type check, it's an annotation. Sure, if someone misannotates their code and runs it through a "compile time" checker (aka superlint) such as mypy or something even more advanced, it will highlight problems where there may in fact be none. But the problem being highlighted there is that the annotation is wrong. Any system doing that level of code analysis will need a way to deal with loading corrected annotations for code that for some reason cannot be changed directly in the annotated files themselves (standard library, third party library from pypi, etc). But this is really no different than loading annotations for extension modules and the stdlib builtins. Those are already likely to be separate files declaring interface annotations only (there is a pile of these that we've generated via code analysis for builtins and parts of the stdlib in the pytypedecl repo, expect more of that to come from anyone working on this kind of thing). While Mypy's cast is nice in that I won't have to wrap my Integer Tuple > in list like object, having to cast it every time I use a particular > broken utility method > feels very ugly to me; and defining a wrapper function with the correct > type information feels like unnecessary run time overhead for no gain. > > Don't get me wrong, I'm not entirely against some kind of type checking, > but I fear that there must exist possible workarounds for badly written > code. > always. but this is more a task for all checker implementations. not something the syntax for annotations itself can prevent. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Aug 15 06:34:31 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 14 Aug 2014 21:34:31 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: On Thu, Aug 14, 2014 at 9:12 PM, Stefan Behnel wrote: > Guido van Rossum schrieb am 14.08.2014 um 07:24: > > On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo wrote: > >> You could use AnyStr to make the example work with bytes as well: > >> > >> def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]: > >> result = {} #type: Dict[AnyStr, int] > >> > >> for line in input: > >> for word in line.split(): > >> result[word] = result.get(word, 0) + 1 > >> return result > >> > >> Again, if this is just a simple utility function that you use once or > >> twice, I see no reason to spend a lot of effort in coming up with the > most > >> general signature. Types are an abstraction and they can't express > >> everything precisely -- there will always be a lot of cases where you > can't > >> express the most general type. However, I think that relatively simple > >> types work well enough most of the time, and give the most bang for the > >> buck. > > > > I heartily agree. But just for the type theorists amongst us, if I really > > wanted to write the most general type, how would I express that the > AnyStr > > in the return type matches the one in the argument? (I think pytypedecl > > would use something like T <= AnyStr.) > > That's how Cython's "fused types" (generics) work, at least. They go by > name: same name of the type, same type. Otherwise, use alias names, which > make the types independent from each other. > > http://docs.cython.org/src/userguide/fusedtypes.html > > While it's a matter of definition what way to go here (same type or not), > practice has shown that it's clearly the right decision to make identical > types the default. > I don't understand those docs at all, but I do think I understand the rule "same name, same type" and I think I like it. Let me be clear -- in this example: def word_count(input: Iterable[AnyStr]) -> Mapping[AnyStr, int]: ... the implication would be that if the input is Iterable[bytes] the output is Mapping[bytes, int] while if the input is Iterable[str] the output is Mapping[str, int]. Have I got that right? I hope so, because I think it is a nice simplifying rule that covers a lot of cases in practice. (Note: AnyStr is a predefined type in mypy that means "str or bytes".) BTW there are a lot of messy things to consider around bytes, and IIUC mypy currently doesn't really cover them. Often when you write code that accepts a bytes instance, in practice it will accept anything that supports the buffer protocol (e.g. bytearray and memoryview). Except when you are going to use it as a dict key, then bytearray won't work. And if you say that you are returning bytes, you probably shouldn't be returning a memoryview or bytearray. I don't expect that any type system we can come up with will be quite precise enough to cover all the cases, so we probably shouldn't lose too much sleep over this. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Aug 15 06:40:37 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Aug 2014 14:40:37 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On 15 August 2014 09:56, Guido van Rossum wrote: > > I don't buy the argument that PEP 3107 promises that annotations are > completely free of inherent semantics. It's also worth noting the corresponding bullet point in PEP 3100 (under http://www.python.org/dev/peps/pep-3100/#core-language): * Add optional declarations for static typing [45] [10] [done] [10] Guido's blog ("Python Optional Typechecking Redux") http://www.artima.com/weblogs/viewpost.jsp?thread=89161 [45] PEP 3107 (Function Annotations) http://www.python.org/dev/peps/pep-3107 > It promises compatibility, and I take > that very seriously, but I think it is reasonable to eventually deprecate > other uses of annotations -- there aren't enough significant other uses for > them to warrant crippling type annotations forever. In the meantime, we > won't be breaking existing use of annotations -- but they may confuse a type > checker, whether a stand-alone linter like mypy or built into an IDE like > PyCharm, and that may serve as an encouragement to look for a different > solution. Linters/checkers may also want to provide a configurable way to say "the presence of decorator means the annotations on that function aren't type markers". That ties in with the recommendation we added to PEP 8 a while back: "It is recommended that third party experiments with annotations use an associated decorator to indicate how the annotation should be interpreted." > So there you have it. I am picking the mypy family and I hope we can start > focusing on specific improvements to mypy. I also hope that somebody will > write converters from pytypedecl and PyCharm stubs into mypy stubs, so that > we can reuse the work already put into stub definitions for those two > systems. And of course I hope that PyCharm and pytypedecl will adopt mypy's > syntax (initially in addition to their native syntax, eventually as their > sole syntax). Having Argument Clinic generate appropriate annotations automatically could also be interesting. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Fri Aug 15 06:41:02 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 14 Aug 2014 21:41:02 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53ED16A7.5080007@googlemail.com> Message-ID: On Thu, Aug 14, 2014 at 9:18 PM, Gregory P. Smith wrote: > (there is a pile of these that we've generated via code analysis for > builtins and parts of the stdlib in the pytypedecl repo, expect more of > that to come from anyone working on this kind of thing). > This is exciting news! It means that we should be able to convert pytypedecl stubs to mypy stubs and mypy will get a lot of stdlib stubs for free. I filed https://github.com/JukkaL/mypy/issues/382 for this. Hopefully you are releasing Python 3 stubs along these lines too? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Aug 15 06:46:47 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 14 Aug 2014 21:46:47 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Thu, Aug 14, 2014 at 9:40 PM, Nick Coghlan wrote: > On 15 August 2014 09:56, Guido van Rossum wrote: > > > > I don't buy the argument that PEP 3107 promises that annotations are > > completely free of inherent semantics. > > It's also worth noting the corresponding bullet point in PEP 3100 > (under http://www.python.org/dev/peps/pep-3100/#core-language): > > * Add optional declarations for static typing [45] [10] [done] > > [10] Guido's blog ("Python Optional Typechecking Redux") > http://www.artima.com/weblogs/viewpost.jsp?thread=89161 > [45] PEP 3107 (Function Annotations) > http://www.python.org/dev/peps/pep-3107 > Such youthful optimism. :-) > Having Argument Clinic generate appropriate annotations automatically > could also be interesting. > How much of the 3.5 stdlib is currently covered by Argument Clinic? I thought there's still a lot left to do. Might it be possible to convert some of pytypedecl's stubs into AC stubs? Alternatively, the AC info could be turned into mypy stubs. I'm just really hoping that between AC, pytypedecl, PyCharm and mypy we have specs for most builtins and extension modules in machine-readable form already, and we could use this combined information to bootstrap mypy's collection of stubs. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Aug 15 06:43:54 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 14 Aug 2014 21:43:54 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53ED16A7.5080007@googlemail.com> Message-ID: <1408077834.74431.YahooMailNeo@web181002.mail.ne1.yahoo.com> On Thursday, August 14, 2014 9:20 PM, Gregory P. Smith wrote: >On Thu Aug 14 2014 at 1:06:20 PM Dennis Brakhane wrote: > >>My pet peeve of static languages is that programmers are often too >>fixated on their particular problem that they don't think about alternate >>uses for their code and make the type declarations uncessarily complex. >> >>For example, in Java nearly every method in Java that deals with >>character strings uses "String" as parameter type, while they should >>have used "CharSequence". Having to read an entire input stream and >>storing it in a String just to be able to use a method is not fun >>(String is final in Java) >> >>I'm worried that in Python we will have utility functions that declare >>they require a List[int], when in fact they actually only require a >>Sequence[int] or >>Sequence[Number]. > >This always happens, even in docstrings where people state types today. For example, I often correct people during code reviews to say sequence of instead of list of in docstrings. Nothing can really prevent that We can't _prevent_ it, but we certainly can _drastically reduce_ it, just by not providing List[int]. Again, I think 90% or more of the uses of List[int] in type annotations will be incorrect. Why go out of our way to make things more complicated just so people can make more mistakes? If anyone really does need a list of ints, implementing List should be pretty simple (in fact, it might even just be `class List(MutableSequence, list): pass`). Why bend over backwards to make it easier for people to make mistakes? From stefan_ml at behnel.de Fri Aug 15 07:35:28 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 15 Aug 2014 07:35:28 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: Guido van Rossum schrieb am 15.08.2014 um 06:34: > On Thu, Aug 14, 2014 at 9:12 PM, Stefan Behnel wrote: >> Guido van Rossum schrieb am 14.08.2014 um 07:24: >>> On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo wrote: >>>> You could use AnyStr to make the example work with bytes as well: >>>> >>>> def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]: >>>> result = {} #type: Dict[AnyStr, int] >>>> >>>> for line in input: >>>> for word in line.split(): >>>> result[word] = result.get(word, 0) + 1 >>>> return result >>>> >>>> Again, if this is just a simple utility function that you use once or >>>> twice, I see no reason to spend a lot of effort in coming up with the >> most >>>> general signature. Types are an abstraction and they can't express >>>> everything precisely -- there will always be a lot of cases where you >> can't >>>> express the most general type. However, I think that relatively simple >>>> types work well enough most of the time, and give the most bang for the >>>> buck. >>> >>> I heartily agree. But just for the type theorists amongst us, if I really >>> wanted to write the most general type, how would I express that the >> AnyStr >>> in the return type matches the one in the argument? (I think pytypedecl >>> would use something like T <= AnyStr.) >> >> That's how Cython's "fused types" (generics) work, at least. They go by >> name: same name of the type, same type. Otherwise, use alias names, which >> make the types independent from each other. >> >> http://docs.cython.org/src/userguide/fusedtypes.html >> >> While it's a matter of definition what way to go here (same type or not), >> practice has shown that it's clearly the right decision to make identical >> types the default. > > I don't understand those docs at all I'm not surprised. ;) The main idea is that you declare (typedef) a "fused" type that means "any of the following list of types". Then you use it in a function signature and the compiler explodes it into multiple specialised implementations that get separately optimised for the specific type(s) they use. Compile time generic functions, essentially. You get the cross product of all different fused types that your function uses, but in practice, you almost always want only one specialisation for each type, regardless of how often you used it in the argument list. > but I do think I understand the rule > "same name, same type" and I think I like it. Let me be clear -- in this > example: > > def word_count(input: Iterable[AnyStr]) -> Mapping[AnyStr, int]: > ... > > the implication would be that if the input is Iterable[bytes] the output is > Mapping[bytes, int] while if the input is Iterable[str] the output is > Mapping[str, int]. Have I got that right? I hope so, because I think it is > a nice simplifying rule that covers a lot of cases in practice. Yes, absolutely. One caveat for Python: static analysis tools (including Cython) will usually have the AST available and thus see the type name used. Once the annotation is in the __annotations__ dict, however, it's lost and reduced to the base type object instance. So renaming types would have to be an explicit operation that the type object knows about. Otherwise, you'd loose semantics at runtime (not sure it matters much in practice, but it would when used for documentation purposes). Not very DRY, but as I said, the hugely more normal case is to want all types the same. > BTW there are a lot of messy things to consider around bytes, and IIUC mypy > currently doesn't really cover them. Often when you write code that accepts > a bytes instance, in practice it will accept anything that supports the > buffer protocol (e.g. bytearray and memoryview). Yes, totally. I've been teaching people that for years now, but it's so much easier for them to write "x: bytes" than to remember to be forgiving about input and think about what that means for their specific code. Not typing the input at all is actually the best solution in many cases, but getting that into the head of users who are just discovering the beauty of an optionally typed Python language is a true up-hill battle. Sometimes I even encourage them to use memory views instead of expecting "real" byte string input, even though that can make working with the data less "stringish". But it's what the users of their code will want. Cython essentially uses NumPy-ish syntax for (compile time) memory views, i.e. you'd write "int[:] x" to unpack a 1-dimensional buffer of item type C int, or "unsigned char[:] b" for a plain uchar buffer. Here's a bunch of examples: http://docs.cython.org/src/userguide/memoryviews.html This is a very well established syntax by now, with lots of code out there using it. Makes working with arbitrary buffer providers a charm. Note that Cython has its own C level memory view implementation, so this is way more efficient than Python's generic memoryview objects (but PEP 3118 based, so compatible). > Except when you are going > to use it as a dict key, then bytearray won't work. And if you say that you > are returning bytes, you probably shouldn't be returning a memoryview or > bytearray. Right. Hashability, strict output, all that. > I don't expect that any type system we can come up with will be > quite precise enough to cover all the cases, so we probably shouldn't lose > too much sleep over this. Well, Cython's type system has pretty much all you'd need. But it's linked to the compiler in the sense that some features exist because Cython can do them at compile time. Not everything can be done in pure Python at runtime or import time. Stefan From stephen at xemacs.org Fri Aug 15 07:46:10 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 15 Aug 2014 14:46:10 +0900 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp> It was written: > a static type check that prevents it being used that way. Can we please refrain from using the word "prevent" in this thread? I suggest instead a static type check that generates [scads of|tons of|way too many] [spurious|bogus|annoying] warnings when used that way. which is IMO a genuine concern but also probably has technological fixes like the one Andrew proposes (automatically promoting many container types to the ABC actually needed). In some sense I'm not really bothered by this (I'd be willing to omit all the bracketed phrases :-). If an upstream author writes def somefilter(l_of_s: List[str]) -> List[int]: pass I think it's reasonable to assume that Ms. U. S. Author didn't think about the implications of tuples or dictionary views or iterators or whatever. I personally think a warning here is a *good* thing: if Ms. Author didn't intend those usages, why should she have to check them? If somebody else does the checking and believes the function is properly prepared for those usages, they should prepare an RFE (*not* a bug report! -- Ms. Author will have to recheck that the code works as claimed in the more general context, and perhaps fix bugs to ensure that it does work to *her* standards for code *she* distributes). From nicholas.cole at gmail.com Fri Aug 15 07:46:15 2014 From: nicholas.cole at gmail.com (Nicholas Cole) Date: Fri, 15 Aug 2014 06:46:15 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53ECF1DD.8040808@stoneleaf.us> References: <53ECF1DD.8040808@stoneleaf.us> Message-ID: On Thu, Aug 14, 2014 at 6:29 PM, Ethan Furman wrote: > On 08/14/2014 09:01 AM, Sunjay Varma wrote: >> >> >> Additionally, this approach can be used by documentation generators as >> well and removes any duplication from the >> function declaration and the docstring. >> >> Here's a taste of what that looks like: >> class SimpleEquation(object): >> def demo(self, a, b, c): >> """ >> This function returns the product of a, b and c >> @type self: SimpleEquation >> :param a: int - The first number >> :param b: int >> :param c: int - The third number should not be zero and >> should also >> only be -1 if you enjoy carrots (this comment spans 2 >> lines) >> :return: int >> """ >> return a * b * c > > > > +1 I like this much more. +1 from me as well. I like this much, much more as well. It is simply far more readable and easier on the eyes, especially for functions with complicated definitions. This feels like something I would actually use, without it being a burden. I think it could also work very nicely for keyword arguments. N. From abarnert at yahoo.com Fri Aug 15 08:08:15 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 14 Aug 2014 23:08:15 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com> On Thursday, August 14, 2014 10:46 PM, Stephen J. Turnbull wrote: ? > I suggest instead > > ? ? a static type check that generates [scads of|tons of|way too many] > ? ? [spurious|bogus|annoying] warnings when used that way. > > which is IMO a genuine concern but also probably has technological > fixes like the one Andrew proposes (automatically promoting many > container types to the ABC actually needed). > > In some sense I'm not really bothered by this (I'd be willing to omit > all the bracketed phrases :-).? If an upstream author writes > > ? ? def somefilter(l_of_s: List[str]) -> List[int]: > ? ? ? ? pass > > I think it's reasonable to assume that Ms. U. S. Author didn't think > about the implications of tuples or dictionary views or iterators or > whatever.? I personally think a warning here is a *good* thing: if > Ms. Author didn't intend those usages, why should she have to check > them? You're forgetting that linting is not the only purpose of static type annotations in this proposal. If you write what should be perfectly valid code and it spits out scads of warnings when linted, yes, you can quickly figure out that it's a bug in the library and report it upstream. But if, say, your IDE doesn't suggest a function that it could have, you may never notice the problem, and just find the library to be less fun to use than you expected. From abarnert at yahoo.com Fri Aug 15 08:28:11 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 14 Aug 2014 23:28:11 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <1408084091.95412.YahooMailNeo@web181006.mail.ne1.yahoo.com> On Thursday, August 14, 2014 9:15 PM, Guido van Rossum wrote: >On Thu, Aug 14, 2014 at 8:51 PM, Andrew Barnert wrote: > >>Most of the code in typing.py is a duplication of the ABCs in the collections.abc module (and io and maybe others). I understand that MyPy couldn't monkeypatch that code into the stdlib, so it had to fork the contents, but if this is going into the stdlib, can't we just modify collections/abc.py instead? Why not have the type information on collections.abc.Sequence be introspectable at runtime? More importantly, why not have the interface defined by collections.abc.Sequence be _exactly_ the same as the one checked by the static type checker, instead of just very similar? [snip] >I think I already responded to a similar proposal. I think modifying the existing ABCs is fine for Python 3.5 and beyond, but I think I'd like to keep aliasing in the typing module to make it easier to use annotations in earlier Python versions (using a typing.py installed from PyPI). That sounds like a great idea. My worry wasn't the extra few KB of code sitting on my computer, it was the extra set of not-quite-the-same parallel concepts that are in typing.py today. If it instead has exactly the same classes as collections.abc, io, etc., everything's great. >I feel similar about using a|b as a concise way to spell Union[a, n], and int|None would be quite decent as a way to spell optional int. Swift makes you sound like a 12-year-old girl? who makes every phrase? sound like a question? by writing `int?` everywhere? Haskell maybe makes you sound wishy-washy and maybe a little uncertain by having you say `Maybe int`. ?Python lets you declare, `int, or None` like Patrick Henry. Sounds good. From lukasz at langa.pl Fri Aug 15 09:48:12 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 15 Aug 2014 00:48:12 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Aug 14, 2014, at 4:56 PM, Guido van Rossum wrote: > There's also a variant of (1) that ?ukasz Langa would like to see -- use the syntactic position of function annotations but using a custom syntax (e.g. one similar to the pytypedecl syntax) that isn't evaluated at function-definition time. This would have to use "from __future__ import " for backward compatibility. I'm skeptical about this though; it is only slightly more elegant than mypy, and it would open the floodgates of unconstrained language design. I see the decision has been made. For the curious, the design would be as close as possible to PEP 3107. The biggest wins would be first-class annotations for variables (supported in Mypy as comments) and support for forward-references without the need to fall back to strings. I?m also not a fan of the square brackets for generics (those brackets mean lookup!) but a BDFL once said that ?language evolution is the art of compromise? and one cannot disagree with that. > So there you have it. I am picking the mypy family and I hope we can start focusing on specific improvements to mypy. Alright! That sounds good. With the syntax mostly out of the way, the next issue to handle is the typing module. dict, MutableMapping, and now Dict? One step too far. We should be able to re-use ABCs for that, e.g. to add support for union types and generics. Lots of decisions ahead (covariance, casting, multiple dispatch, etc.) but we?ll get there. The typing module as aliases for the updated ABCs sounds like a fair compromise, although I?m worried about the details of that approach (for starters: we need to bundle the new ABCMeta to support union types but having both implementations at runtime feels very wrong). Moreover, with dynamic type registration and duck typing isinstance(), there will be challenges to cover. I?m happy to go and slay that dragon. As a side note, I?m happy you?re willing to agree on str | None. This reads really well and is concise enough to not require aliasing to be usable. While we?re at slaying dragons, I?ll also silently make str non-iterable so that we can use Sequence[str] meaningfully from now on? How about that? -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Aug 15 11:11:50 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 15 Aug 2014 18:11:50 +0900 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp> <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <87k36aglrd.fsf@uwakimon.sk.tsukuba.ac.jp> Andrew Barnert writes: > On Thursday, August 14, 2014 10:46 PM, Stephen J. Turnbull wrote: > > I think it's reasonable to assume that Ms. U. S. Author didn't think > > about the implications of tuples or dictionary views or iterators or > > whatever.? I personally think a warning here is a *good* thing: if > > Ms. Author didn't intend those usages, why should she have to check > > them? > > You're forgetting that linting is not the only purpose of static > type annotations in this proposal. Hardly. > But if, say, your IDE doesn't suggest a function that it could > have, you may never notice the problem, and just find the library > to be less fun to use than you expected. But who decides it *should* be perfectly valid code? *Somebody* has to check that, and if code is used naively in a case where it shouldn't be (cf the "why doesn't sum() handle iter_of_str" thread!), the user is screwed again. Have you never written code that was perfectly sound for your purpose, but could be used in other contexts according to ducktyping? Are you *sure* that it is valid in those other contexts? (Think sum vs. math.fsum -- accuracy matters.) Of course it's a matter of balance, and *maybe* it tips more in the direction of "default to the most general type where the code will run without crashing or raising". I'm just saying it's not obvious to me that it's such a bad thing that one-off piece of junk libraries may not be as re-abusable if you use a type checker as they are when you don't. From tjreedy at udel.edu Fri Aug 15 11:38:03 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 15 Aug 2014 05:38:03 -0400 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On 8/15/2014 12:40 AM, Nick Coghlan wrote: > On 15 August 2014 09:56, Guido van Rossum wrote: >> >> I don't buy the argument that PEP 3107 promises that annotations are >> completely free of inherent semantics. > > It's also worth noting the corresponding bullet point in PEP 3100 > (under http://www.python.org/dev/peps/pep-3100/#core-language): > > * Add optional declarations for static typing [45] [10] [done] ... > Linters/checkers may also want to provide a configurable way to say > "the presence of decorator means the annotations on that function > aren't type markers". That ties in with the recommendation we added to > PEP 8 a while back: "It is recommended that third party experiments > with annotations use an associated decorator to indicate how the > annotation should be interpreted." Depending on the checker, this suggests that non-type-check annotations need not be deprecated. If a decorator wraps a function with an unannotated wrapper, then the checker should see the result as unannotated, rather than looking for a wrapped attribute. Also, a decorator can remove non-type annotations and act on them, store them in a closure variable, or store them on the function in a different name. For example. >>> def doodad(f): f.doodad = f.__annotations__ f.__annotations__ = {} return f >>> @doodad def f(x:'arg doodad')->'return:doodad': pass >>> f.__annotations__ {} >>> f.doodad {'x': 'arg doodad', 'return': 'return:doodad'} Given these possibilities, all that is needs be said is "After a function is post-processed by decorators, any remaining annotations should be for type-checking or documentation." For checkers that do look at the source, or the AST before compiling, the rule could be to ignore string annotations. Decorators can always eval, or perhaps safe_eval, strings. -- Terry Jan Reedy From ncoghlan at gmail.com Fri Aug 15 11:48:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Aug 2014 19:48:10 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On 15 August 2014 19:38, Terry Reedy wrote: > On 8/15/2014 12:40 AM, Nick Coghlan wrote: >> >> On 15 August 2014 09:56, Guido van Rossum wrote: >>> >>> >>> I don't buy the argument that PEP 3107 promises that annotations are >>> completely free of inherent semantics. >> >> >> It's also worth noting the corresponding bullet point in PEP 3100 >> (under http://www.python.org/dev/peps/pep-3100/#core-language): >> >> * Add optional declarations for static typing [45] [10] [done] > > ... > >> Linters/checkers may also want to provide a configurable way to say >> "the presence of decorator means the annotations on that function >> aren't type markers". That ties in with the recommendation we added to >> PEP 8 a while back: "It is recommended that third party experiments >> with annotations use an associated decorator to indicate how the >> annotation should be interpreted." > > > Depending on the checker, this suggests that non-type-check annotations need > not be deprecated. If a decorator wraps a function with an unannotated > wrapper, then the checker should see the result as unannotated, rather than > looking for a wrapped attribute. Also, a decorator can remove non-type > annotations and act on them, store them in a closure variable, or store them > on the function in a different name. No, many (most?) linters and IDEs will run off the AST without actually executing the code, so they'll see the annotations, even if they get stripped by the decorator at runtime. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brakhane at googlemail.com Fri Aug 15 12:02:21 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 12:02:21 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <87k36aglrd.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp> <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com> <87k36aglrd.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53EDDAAD.8040907@gmail.com> Am 15.08.2014 11:11, schrieb Stephen J. Turnbull: > Have you never written code that was perfectly sound for your purpose, > but could be used in other contexts according to ducktyping? Are you > *sure* that it is valid in those other contexts? (Think sum vs. > math.fsum -- accuracy matters.) > Well, if someone uses my frobnicate method that I intended to be only usable by lists, but he uses it with generators, it's his responsibility to write a unit test for his code (which he should do anyway). Therefore he's responsible for guaranteeing that his code continues to work, and a linter should not tell him "you can't do that, the author hasn't thought of that". At least I should be able to silence the warning *once* and for all. Furthermore, I want my IDE to suggest me the frobnicate method even if I'm working with generators, so there should be a way for me to tell the linter and IDE "frobnicate takes an iterable, ignore the silly original annotation" Unit testing your own code is the only way you can "guarantee" that your code runs correctly. After all, even if you use my frobnicate method only with lists, you will still have to write a test, maybe I introduce a subtle bug in the new version of my library. From ndbecker2 at gmail.com Fri Aug 15 12:56:46 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 15 Aug 2014 06:56:46 -0400 Subject: [Python-ideas] generic code and dependent types Message-ID: I'm interested in the proposals for adding type annotation. Coming from some experience with generic code in c++, one of the difficult issues has been reasoning about dependent types in generic code. In general, we need to be able to declare a type via an arbitrary metafunction some_metafunction::type F (... It is also useful to be able to write something like: typedef typeof (int() * float()) my_type; I wonder if any of the proposals will be able to handle this sort of algebra on types? I think it's needed to truly support generic code. From bunslow at gmail.com Fri Aug 15 13:01:23 2014 From: bunslow at gmail.com (Bill Winslow) Date: Fri, 15 Aug 2014 06:01:23 -0500 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: As a lurker who barely reads this list at all, let me just add here that many of the typing-related questions that the BDFL presented as follow-up are, in my opinion, questions worth asking even outside of the annotation issue. (Mini wall of text incoming:) What I mean is, one of my grievances with Python is that the type hierarchy is poorly defined and difficult to use, in that the "types" presented in collections.abc in no way whatsoever interact with the builtins, and considering how many library and user types inherit from those ABCs (as is the purpose of those ABCs), it seems to me like a rather serious issue with using the type system. Consider the following: >>> from collections import abc as types >>> isinstance(dict, types.Mapping) False >>> isinstance(types.Mapping, dict) False >>> isinstance(list, types.MutableSequence) False >>> isinstance(types.MutableSequence, list) False >>> isinstance(list, types.Sized) False Furthermore: >>> class DictThing(types.MutableMapping): pass #Easier to subclass ... >>> isinstance(DictThing, dict) False >>> class DicterThing(dict): pass # Simpler ... >>> isinstance(DicterThing, types.MutableMapping) False And finally: >>> from collections import defaultdict >>> isinstance(defaultdict, dict) False >>> isinstance(defaultdict, types.MutableMapping) False My conclusion is that to make lint-sort type-checkers worth their salt, the Python type hierarchy needs to be fixed and properly integrated. There is currently no obvious way to check if an object either is a list or behaves like a list (the duck typing philosophy equates the two). (The current shortest way is "isinstance(myvar, (list, types.MutableSequence)", but that's not very obvious or Pythonic IMO.) Before now I haven't bothered to put my thoughts to words, but this is pretty decent motivation. --------------------------------------------------------------------------- If people agree with my conclusion, then the ideal solution would be to somehow merge the builtin types and the collections.abc types into one single hierarchy, one single isinstance check, but that's probably impossible. The next thing would be to make one hierarchy the appropriate subclasses of the other hierarchy, but either of those solutions has its own problems. There're probably better solutions out there. --Bill On Fri, Aug 15, 2014 at 4:48 AM, Nick Coghlan wrote: > On 15 August 2014 19:38, Terry Reedy wrote: > > On 8/15/2014 12:40 AM, Nick Coghlan wrote: > >> > >> On 15 August 2014 09:56, Guido van Rossum wrote: > >>> > >>> > >>> I don't buy the argument that PEP 3107 promises that annotations are > >>> completely free of inherent semantics. > >> > >> > >> It's also worth noting the corresponding bullet point in PEP 3100 > >> (under http://www.python.org/dev/peps/pep-3100/#core-language): > >> > >> * Add optional declarations for static typing [45] [10] [done] > > > > ... > > > >> Linters/checkers may also want to provide a configurable way to say > >> "the presence of decorator means the annotations on that function > >> aren't type markers". That ties in with the recommendation we added to > >> PEP 8 a while back: "It is recommended that third party experiments > >> with annotations use an associated decorator to indicate how the > >> annotation should be interpreted." > > > > > > Depending on the checker, this suggests that non-type-check annotations > need > > not be deprecated. If a decorator wraps a function with an unannotated > > wrapper, then the checker should see the result as unannotated, rather > than > > looking for a wrapped attribute. Also, a decorator can remove non-type > > annotations and act on them, store them in a closure variable, or store > them > > on the function in a different name. > > No, many (most?) linters and IDEs will run off the AST without > actually executing the code, so they'll see the annotations, even if > they get stripped by the decorator at runtime. > > Cheers, > Nick. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Aug 15 13:50:31 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Aug 2014 21:50:31 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On 15 August 2014 21:01, Bill Winslow wrote: > Consider the following: > > >>>> from collections import abc as types >>>> isinstance(dict, types.Mapping) > False >>>> isinstance(types.Mapping, dict) > False >>>> isinstance(list, types.MutableSequence) > False >>>> isinstance(types.MutableSequence, list) > False >>>> isinstance(list, types.Sized) > False You're doing instance checks on subclasses - that's never going to work. Once you account for the type/instance distinction, you can see everything is correctly registered: >>> from collections import abc as cabc >>> issubclass(dict, cabc.Mapping) True >>> issubclass(list, cabc.Sequence) True >>> issubclass(list, cabc.Sized) True >>> isinstance(dict(), cabc.Mapping) True >>> isinstance(list(), cabc.Sequence) True >>> isinstance(list(), cabc.Sized) True (in Python 2, a couple of builtins claim ABCs they don't actually implement fully, but that's addressed in newer versions of Python 3) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bunslow at gmail.com Fri Aug 15 13:53:31 2014 From: bunslow at gmail.com (Bill Winslow) Date: Fri, 15 Aug 2014 06:53:31 -0500 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: Oh dear lord, it's like a nightmare where you're late to class without clothing... except it actually happened... I knew something like this would happen when I decided to actually do something. On Fri, Aug 15, 2014 at 6:50 AM, Nick Coghlan wrote: > On 15 August 2014 21:01, Bill Winslow wrote: > > Consider the following: > > > > > >>>> from collections import abc as types > >>>> isinstance(dict, types.Mapping) > > False > >>>> isinstance(types.Mapping, dict) > > False > >>>> isinstance(list, types.MutableSequence) > > False > >>>> isinstance(types.MutableSequence, list) > > False > >>>> isinstance(list, types.Sized) > > False > > You're doing instance checks on subclasses - that's never going to > work. Once you account for the type/instance distinction, you can see > everything is correctly registered: > > >>> from collections import abc as cabc > >>> issubclass(dict, cabc.Mapping) > True > >>> issubclass(list, cabc.Sequence) > True > >>> issubclass(list, cabc.Sized) > True > >>> isinstance(dict(), cabc.Mapping) > True > >>> isinstance(list(), cabc.Sequence) > True > >>> isinstance(list(), cabc.Sized) > True > > (in Python 2, a couple of builtins claim ABCs they don't actually > implement fully, but that's addressed in newer versions of Python 3) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Aug 15 14:01:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Aug 2014 22:01:30 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On 15 August 2014 21:53, Bill Winslow wrote: > Oh dear lord, it's like a nightmare where you're late to class without > clothing... except it actually happened... I knew something like this would > happen when I decided to actually do something. If it helps any, I had to type it into the interactive interpreter and get very confused for a moment before I realised what had happened. And I *knew* they were integrated, because I'd helped fix some of the bugs with the ABC non-conformance :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Fri Aug 15 14:26:09 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Aug 2014 22:26:09 +1000 Subject: [Python-ideas] generic code and dependent types In-Reply-To: References: Message-ID: <20140815122608.GZ4525@ando> On Fri, Aug 15, 2014 at 06:56:46AM -0400, Neal Becker wrote: > I'm interested in the proposals for adding type annotation. Coming from some > experience with generic code in c++, one of the difficult issues has been > reasoning about dependent types in generic code. > > In general, we need to be able to declare a type via an arbitrary metafunction > > some_metafunction::type F (... > > It is also useful to be able to write something like: > > typedef typeof (int() * float()) my_type; > > I wonder if any of the proposals will be able to handle this sort of algebra on > types? I think it's needed to truly support generic code. Let's pretend that this is a Python mailing list, and that some readers aren't familiar with C++ generic syntax :-) What does typedef typeof (int() * float()) my_type; do? From a Python perspecive, that looks like type(0*0.0) which will return float. Presumably that's not what you want, or you would have just said "float". So what am I missing? (Apart from years of C++ experience.) -- Steven From tinchester at gmail.com Fri Aug 15 14:32:04 2014 From: tinchester at gmail.com (=?UTF-8?Q?Tin_Tvrtkovi=C4=87?=) Date: Fri, 15 Aug 2014 14:32:04 +0200 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31 In-Reply-To: References: Message-ID: Hi, I realize I'm very late to the party, but just in the interest of completeness I thought I'd mention Typed Clojure (http://typedclojure.org/), which is an effort to add optional typing to Clojure. It has even had a successful crowdfunding campaign: https://www.indiegogo.com/projects/typed-clojure. I haven't had an opportunity of using it yet so this is all I know. P.S. Very excited about this effort! > Date: Wed, 13 Aug 2014 12:44:21 -0700 > From: Guido van Rossum > To: Python-Ideas > Cc: Jukka Lehtosalo , Bob Ippolito > > Subject: [Python-ideas] Proposal: Use mypy syntax for function > annotations > Message-ID: > < > CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > [There is no TL;DR other than the subject line. Please read the whole thing > before replying. I do have an appendix with some motivations for adding > type annotations at the end.] > > Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man > of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: > http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python > can > learn from Haskell (and other languages); yesterday he gave the same talk > at Dropbox. The talk is online ( > https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad > strokes comes down to three suggestions: > > (a) Python should adopt mypy's syntax for function annotations > (b) Python's use of mutabe containers by default is wrong > (c) Python should adopt some kind of Abstract Data Types > > Proposals (b) and (c) don't feel particularly actionable (if you disagree > please start a new thread, I'd be happy to discuss these further if there's > interest) but proposal (a) feels right to me. > > So what is mypy? It is a static type checker for Python written by Jukka > for his Ph.D. thesis. The basic idea is that you add type annotations to > your program using some custom syntax, and when running your program using > the mypy interpreter, type errors will be found during compilation (i.e., > before the program starts running). > > The clever thing here is that the custom syntax is actually valid Python 3, > using (mostly) function annotations: your annotated program will still run > with the regular Python 3 interpreter. In the latter case there will be no > type checking, and no runtime overhead, except to evaluate the function > annotations (which are evaluated at function definition time but don't have > any effect when the function is called). > > In fact, it is probably more useful to think of mypy as a heavy-duty linter > than as a compiler or interpreter; leave the type checking to mypy, and the > execution to Python. It is easy to integrate mypy into a continuous > integration setup, for example. > > To read up on mypy's annotation syntax, please see the mypy-lang.org > website. Here's just one complete example, to give a flavor: > > from typing import List, Dict > > def word_count(input: List[str]) -> Dict[str, int]: > result = {} #type: Dict[str, int] > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result > > Note that the #type: comment is part of the mypy syntax; mypy uses comments > to declare types in situations where no syntax is available -- although > this particular line could also be written as follows: > > result = Dict[str, int]() > > Either way the entire function is syntactically valid Python 3, and a > suitable implementation of typing.py (containing class definitions for List > and Dict, for example) can be written to make the program run correctly. > One is provided as part of the mypy project. > > I should add that many of mypy's syntactic choices aren't actually new. The > basis of many of its ideas go back at least a decade: I blogged about this > topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- > see also the two followup posts linked from the top there). > > I'll emphasize once more that mypy's type checking happens in a separate > pass: no type checking happens at run time (other than what the interpreter > already does, like raising TypeError on expressions like 1+"1"). > > There's a lot to this proposal, but I think it's possible to get a PEP > written, accepted and implemented in time for Python 3.5, if people are > supportive. I'll go briefly over some of the action items. > > *(1) A change of direction for function annotations* > > PEP 3107 , which introduced > function annotations, is intentional non-committal about how function > annotations should be used. It lists a number of use cases, including but > not limited to type checking. It also mentions some rejected proposals that > would have standardized either a syntax for indicating types and/or a way > for multiple frameworks to attach different annotations to the same > function. AFAIK in practice there is little use of function annotations in > mainstream code, and I propose a conscious change of course here by stating > that annotations should be used to indicate types and to propose a standard > notation for them. > > (We may have to have some backwards compatibility provision to avoid > breaking code that currently uses annotations for some other purpose. > Fortunately the only issue, at least initially, will be that when running > mypy to type check such code it will produce complaints about the > annotations; it will not affect how such code is executed by the Python > interpreter. Nevertheless, it would be good to deprecate such alternative > uses of annotations.) > > *(2) A specification for what to add to Python 3.5* > > There needs to be at least a rough consensus on the syntax for annotations, > and the syntax must cover a large enough set of use cases to be useful. > Mypy is still under development, and some of its features are still > evolving (e.g. unions were only added a few weeks ago). It would be > possible to argue endlessly about details of the notation, e.g. whether to > use 'list' or 'List', what either of those means (is a duck-typed list-like > type acceptable?) or how to declare and use type variables, and what to do > with functions that have no annotations at all (mypy currently skips those > completely). > > I am proposing that we adopt whatever mypy uses here, keeping discussion of > the details (mostly) out of the PEP. The goal is to make it possible to add > type checking annotations to 3rd party modules (and even to the stdlib) > while allowing unaltered execution of the program by the (unmodified) > Python 3.5 interpreter. The actual type checker will not be integrated with > the Python interpreter, and it will not be checked into the CPython > repository. The only thing that needs to be added to the stdlib is a copy > of mypy's typing.py module. This module defines several dozen new classes > (and a few decorators and other helpers) that can be used in expressing > argument types. If you want to type-check your code you have to download > and install mypy and run it separately. > > The curious thing here is that while standardizing a syntax for type > annotations, we technically still won't be adopting standard rules for type > checking. This is intentional. First of all, fully specifying all the type > checking rules would make for a really long and boring PEP (a much better > specification would probably be the mypy source code). Second, I think it's > fine if the type checking algorithm evolves over time, or if variations > emerge. The worst that can happen is that you consider your code correct > but mypy disagrees; your code will still run. > > That said, I don't want to *completely* leave out any specification. I want > the contents of the typing.py module to be specified in the PEP, so that it > can be used with confidence. But whether mypy will complain about your > particular form of duck typing doesn't have to be specified by the PEP. > Perhaps as mypy evolves it will take options to tell it how to handle > certain edge cases. Forks of mypy (or entirely different implementations of > type checking based on the same annotation syntax) are also a possibility. > Maybe in the distant future a version of Python will take a different > stance, once we have more experience with how this works out in practice, > but for Python 3.5 I want to restrict the scope of the upheaval. > > > *Appendix -- Why Add Type Annotations?* > The argument between proponents of static typing and dynamic typing has > been going on for many decades. Neither side is all wrong or all right. > Python has traditionally fallen in the camp of extremely dynamic typing, > and this has worked well for most users, but there are definitely some > areas where adding type annotations would help. > > - Editors (IDEs) can benefit from type annotations; they can call out > obvious mistakes (like misspelled method names or inapplicable operations) > and suggest possible method names. Anyone who has used IntelliJ or Xcode > will recognize how powerful these features are, and type annotations will > make such features more useful when editing Python source code. > > - Linters are an important tool for teams developing software. A linter > doesn't replace a unittest, but can find certain types of errors better or > quicker. The kind of type checking offered by mypy works much like a > linter, and has similar benefits; but it can find problems that are beyond > the capabilities of most linters. > > - Type annotations are useful for the human reader as well! Take the above > word_count() example. How long would it have taken you to figure out the > types of the argument and return value without annotations? Currently most > people put the types in their docstrings; developing a standard notation > for type annotations will reduce the amount of documentation that needs to > be written, and running the type checker might find bugs in the > documentation, too. Once a standard type annotation syntax is introduced, > it should be simple to add support for this notation to documentation > generators like Sphinx. > > - Refactoring. Bob's talk has a convincing example of how type annotations > help in (manually) refactoring code. I also expect that certain automatic > refactorings will benefit from type annotations -- imagine a tool like 2to3 > (but used for some other transformation) augmented by type annotations, so > it will know whether e.g. x.keys() is referring to the keys of a dictionary > or not. > > - Optimizers. I believe this is actually the least important application, > certainly initially. Optimizers like PyPy or Pyston > wouldn't be able to fully trust the > type annotations, and they are better off using their current strategy of > optimizing code based on the types actually observed at run time. But it's > certainly feasible to imagine a future optimizer also taking type > annotations into account. > > -- > --Guido "I need a new hobby" van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brakhane at googlemail.com Fri Aug 15 14:37:45 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 14:37:45 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EDFF19.7060305@gmail.com> Am 15.08.2014 01:56, schrieb Guido van Rossum: > PS. I realize I didn't discuss question (C) much. That's intentional -- we can now start discussing specific mypy features in separate threads (or in this one :-). So is this the place to discuss "the thornier issues brought up against mypy"? Because I think it's important that we get an idea of what we want mypy to be able to do and what not. After all, mypy will probably end up as a kind of reference implementation for static type checkers. And I'm worried there might be real damage to Python as a language if they aren't thought through: > Many people have shown either support for the idea, or pointed to some other system that addresses the same issue. On the other hand, several people have claimed > that they don't need it, or that they worry it will make Python less useful for them. (However, many of the detractors seem to have their own alternative proposal. :-) > In the end I don't think we can ever know for sure -- but my intuition tells me that as long as we keep it optional, there is a real demand. It won't be optional for programmers who work in a corporate environment where mypy happens to be required. Those teams will probably also use third party libraries, and they might want them to be "type safe" as well and file RFEs for it; in a few years, we might end up with a situation where "serious" code is expected to provide static type info. Even if mypy is "just a linter", a linter is supposed to find bugs and promote best practices. And if the Python reference doc somehow named mypy as an example for a static type checker, it will be probably seen as enforcing Python best practices. I really think there's a good chance/risk that mypy will change how Python programs are written in the future. For example, people wouldn't probably call the word_count method with a file object and write a test case, instead, they will read the file into a list and call it instead. After all, the linter would complain otherwise. IMO, the question is how much "staticness" we want to encourage. If we want a really useful and flexible static type checking system, we would need a very complex type system. If we'd go that route, I fear that one of Pythons main features, its dynamic nature will be seen by new programmers as kind of "deprecated legacy", and turning Python into some poor-man's-Scala. If we take the current mypy approach, the type system would probably end up a lot like Java's, useful for simple cases, and useless/a PITA otherwise. And if we make the system minimal by design, there's a good chance that it will be completely useless for static type checking for all but the most trivial cases, helping no one. I'm not sure which one I prefer, although I'm leaning into the minimal to Java level direction; that way, the type system might be "just bad enough" for people to see when dynamic typing is an advantage and using that instead. One thing where I do have a clear opinion is that it should/must be possible to override (bad) type annotations. Providing a seperate file seems ok to me. But the open problem with this approach is 1) how to tell IDEs and linters which overrides are in effect when and where (there might be different override for different modules) 2)how we should handle different versions of libraries As an example for 2), let's say I'm the author of the frobnicate library, which depends on spammatron 0.3 from pip. Let's also say that - because it's a good idea - mypy will check override modules for consistency: An override cannot declare a completely different signature than the original. Spammatro's author declared "def foo(x: float) -> float", but actually, it should have been "Number" instead of float, as I'm using Decimal. So I define a override module to fix it. Now spammatron 0.3.1 is released, and foo has gained an additional optional parameter: "def foo(x: float, y: float = 0) -> float". So I have to update my override module, but then my library isn't compatbile with 0.3 anymore; or at least will give linter errors. Providing both overrides would be very difficult; the static analyser would have to know which versions of the library it is using. I suppose it could look at the __version__ attribute, but what if it's missing? From brakhane at googlemail.com Fri Aug 15 14:44:04 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 14:44:04 +0200 Subject: [Python-ideas] generic code and dependent types In-Reply-To: <20140815122608.GZ4525@ando> References: <20140815122608.GZ4525@ando> Message-ID: <53EE0094.7080303@gmail.com> Am 15.08.2014 14:26, schrieb Steven D'Aprano: > > Let's pretend that this is a Python mailing list, and that some readers > aren't familiar with C++ generic syntax :-) What does > > typedef typeof (int() * float()) my_type; > > do? I think its "whatever type operator*(int a, float b)" returns. Or more generally: "whatever type a suitable * operator returns after, where suitable means after int and float might or might not have been casted into another type so that a matching operator was found" So, for example, if there were no operator*(int a, float b), it would look if there's at operator*(int a, int b), and if not, operator*(float, float) (or vice versa). The "advantage" is that you do not need to change your code if the meaning of multiplying an int with a float changes. From brakhane at googlemail.com Fri Aug 15 15:06:31 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 15:06:31 +0200 Subject: [Python-ideas] generic code and dependent types In-Reply-To: References: Message-ID: <53EE05D7.6010202@gmail.com> Am 15.08.2014 12:56, schrieb Neal Becker: > I wonder if any of the proposals will be able to handle this sort of algebra on > types? I think it's needed to truly support generic code. > I hope it won't ;-) I agree that it is needed for truly generic code, but if Python gains a truly complete type system, then what's the point of having dynamic typing at all? At this point, you might as well program in Scala instead. What's the point in trying to turn Python into a statically typed language that can optionally be dynamically typed? There are enough good static languages out there. IMHO, the type information in Python should be considered part of the documentation, if the type system gets too complex and can only be parsed by the linter, then what's the point? Is the information: "this function takes an Iterable of T and a Map[U,Map[U,T]|T|int] and returns the type of the result of foo applied to parameters of type T and U" really useful for a programmer anymore? Does it help me if the linter says that "Map[int,float]" is not compatible with "T|Map[U,V subtype str]|Map[U subtype str, V]"? I think if my unit test fails with a "TypeError: float has no method strip" that's a much more useful information to me. So I think limiting the power of the type system will actually be a good thing. From ndbecker2 at gmail.com Fri Aug 15 15:10:51 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 15 Aug 2014 09:10:51 -0400 Subject: [Python-ideas] generic code and dependent types References: <20140815122608.GZ4525@ando> <53EE0094.7080303@gmail.com> Message-ID: Dennis Brakhane wrote: > Am 15.08.2014 14:26, schrieb Steven D'Aprano: >> >> Let's pretend that this is a Python mailing list, and that some readers >> aren't familiar with C++ generic syntax :-) What does >> >> typedef typeof (int() * float()) my_type; >> >> do? > > I think its "whatever type operator*(int a, float b)" returns. > > Or more generally: "whatever type a suitable * operator returns after, > where suitable means after int and float might or might not have been > casted into another type so that a matching operator was found" > > So, for example, if there were no operator*(int a, float b), it would > look if there's at operator*(int a, int b), and if not, operator*(float, > float) (or vice versa). > > The "advantage" is that you do not need to change your code if the > meaning of multiplying an int with a float changes. I really meant more like typeof (A() * B()) where A and B are 2 types, and A() is a call to the default constructor for A. For example ret_t multiply (A a, B b): return a * b what is ret_t? From apalala at gmail.com Fri Aug 15 15:12:43 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Fri, 15 Aug 2014 08:42:43 -0430 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 7:23 AM, Bill Winslow wrote: > Oh dear lord, it's like a nightmare where you're late to class without > clothing... except it actually happened... I knew something like this would > happen when I decided to actually do something. Don't worry. It's a too common happening, in one of the few places in which Python has no Zen. -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Aug 15 16:04:52 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 16 Aug 2014 00:04:52 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EDDAAD.8040907@gmail.com> References: <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando> <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com> <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp> <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com> <87k36aglrd.fsf@uwakimon.sk.tsukuba.ac.jp> <53EDDAAD.8040907@gmail.com> Message-ID: <20140815140452.GB4525@ando> On Fri, Aug 15, 2014 at 12:02:21PM +0200, Dennis Brakhane wrote: > Unit testing your own code is the only way you can "guarantee" that your > code runs correctly. That is actually backwards. Unit testing cannot guarantee the code is correct. If you have 1000 tests, all that you have proved is that the code works for those 1000 cases. It tells you nothing about all the uncounted billions of other cases you haven't written tests for. Given some set of tests which pass, you can be confident that your code gets at least those things correct (assuming the tests are themselves correct -- you write tests for your tests, don't you? *wink*). But that's all. As Dijkstra said, ?Program testing can be used to show the presence of bugs, but never to show their absence.? To put it another way, there could be a million bugs hiding in all the corners of your code that you don't test, and you can never test *everything*. Static typing is complementary to unit testing. Static typing can eliminate a lot of the unit tests that you otherwise would have to write. For example, you might have unit tests that try to demonstrate that given arguments of type float, the function will return a float; given Fraction arguments, the function returns a Fraction; and so on. A static type checker can *prove* that[1] instead of merely testing it with a few dozen examples and hoping the result generalises to all the billions of other floats and Fractions not tested. Static typing is actually a form of automated correctness testing. [1] Assuming you trust that the checker itself is bug free. -- Steven From antoine at python.org Fri Aug 15 16:30:24 2014 From: antoine at python.org (Antoine Pitrou) Date: Fri, 15 Aug 2014 10:30:24 -0400 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: Le 15/08/2014 00:40, Nick Coghlan a ?crit : > > Having Argument Clinic generate appropriate annotations automatically > could also be interesting. That would be great actually. That's one of the things AC should eventually be able to bring to the table (thank you Larry :-)). Being able to access the concrete implementation (without the boxing / unboxing wrapper) could also be quite useful for people who try to shave off some interpretation overhead :-) Regards Antoine. From steve at pearwood.info Fri Aug 15 16:38:22 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 16 Aug 2014 00:38:22 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EDFF19.7060305@gmail.com> References: <53EDFF19.7060305@gmail.com> Message-ID: <20140815143821.GC4525@ando> On Fri, Aug 15, 2014 at 02:37:45PM +0200, Dennis Brakhane wrote: > I really think there's a good chance/risk that mypy will change how > Python programs are written in the future. I certainly hope so. We're wasting our time if it doesn't. Why go through all the time and effort if nobody uses it? > For example, people wouldn't probably call the word_count method with > a file object and write a test case, instead, they will read the file > into a list and call it instead. After all, the linter would complain > otherwise. How do you know the linter will complain? Many of the arguments against this proposal are based on the assumption that, given a type checker for Python, developers will suddenly abandon all the proven advantages of duck-typing and dynamic typing and rush to turn Python into a third-rate Java. I don't think this is a realistic fear. The mere fact we are having this argument proves that many Python developers will fight tooth and nail to keep using duck-typing and dynamic typing. If they do static type checks, they aren't going to give up those advantages. They'll check for Iterable, not list. And those who don't? Do the same thing you would do *right now* when those authors write code like this: if type(argument) is not list: raise TypeError("list expected") Report it as a bug, or request a feature enhancement. Patch the library. Use a different library. Or, *just don't use the linter*. > IMO, the question is how much "staticness" we want to encourage. If we > want a really useful and flexible static type checking system, we > would need a very complex type system. If we'd go that route, I fear > that one of Pythons main features, its dynamic nature will be seen by > new programmers as kind of "deprecated legacy", and turning Python > into some poor-man's-Scala. Static typing and dynamic typing are *not* opposites. The names are unfortunate, because they imply an opposition that doesn't necessarily exist. Both static and dynamic typing are attempts to solve certain problems in programming, and it is possible to do both at the same time. [...] > One thing where I do have a clear opinion is that it should/must be > possible to override (bad) type annotations. Why? Do you consider it a "must" to override functions that raise TypeError at run time? py> len(None) Traceback (most recent call last): File "", line 1, in TypeError: object of type 'NoneType' has no len() So why do you consider it a "must" to be able to override functions that report type errors at compile time? But for what it's worth, as this proposal has repeatedly said, the linter is optional. If you don't believe the linter, *just run the code* in Python like you have always done before. -- Steven From apalala at gmail.com Fri Aug 15 16:39:01 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Fri, 15 Aug 2014 10:09:01 -0430 Subject: [Python-ideas] generic code and dependent types In-Reply-To: <53EE05D7.6010202@gmail.com> References: <53EE05D7.6010202@gmail.com> Message-ID: On Fri, Aug 15, 2014 at 8:36 AM, Dennis Brakhane wrote: > What's the point in trying to turn Python into a statically typed > language that can optionally be dynamically typed? There are enough good > static languages out there. > That's important. Even within the standard lib, Python is a well behaved systems citizen, so people are free to write parts of their software in any language/environment, and Python will happily communicate with whatever. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Aug 15 17:10:58 2014 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 15 Aug 2014 15:10:58 +0000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: Message-ID: On Thu Aug 14 2014 at 9:02:54 AM Sunjay Varma wrote: > I am strongly opposed to this entire proposal. As Juancarlo points out, > Python programs are small, but very understandable. I think this syntax > detracts from that. I'll suggest an alternative further down in my reply. > Small? I've got tens of millions of lines of Python code to wrangle that says otherwise. We're trying to create an analyzer and type inferencer so that we can actually make sense of it all to make it easier to both (a) maintain and (b) migrate to Python 3. :) > One benefit of Python that makes it so attractive for new programmers and > even old programmers alike is that you can usually pick out any piece of > Python code and begin to understand it immediately. Even if you come from a > different programming language, Python is written in > english explicitly using words like "and" and "or". Those constructs, as > opposed to "&&" or "||" make the language less scary for new developers and > in general easier to read as well. It's also easier to type regular english > words (no need to use the shift key). Using the annotation syntax this > heavily will detract very much from the readability of Python and from the > overall usability as well. Programs are read more times than they are > written. > > Several years ago, before I had any programming experience in any language > at all, I needed to edit some Python code to make something I was doing > work. Without any experience at all, I was able to look through the (small) > program I was editing and figure out exactly what I needed to adjust. > Without Python being such a clean, almost English language, that would have > been impossible. > > Though the annotation syntax is already present in Python 3, I would argue > that using this for type annotations will get very messy very quickly. If > I'm understanding the syntax correctly, writing any function using a large > library with many nested subpackages could result in code like this: > > import twisted.protocols.mice.mouseman > > def > process_mouseman(inputMouseMan: twisted.protocols.mice.mouseman.MouseMan) > -> twisted.protocols.mice.mouseman.MouseMan: > pass > > That function definition is 122 characters long. Far more than what PEP8 > recommends. Though this example was crafted to illustrate my point (I don't > think most people would really write code like this), it is easy to see > that this kind of code is possible and may sometimes be written by some > less experienced programmers. It demonstrates how messy things can get even > with just one parameter. > > It is also easy to see that it is very difficult to parse out what is > going on in that function. Adding type annotations inline makes it very > difficult to quickly get an idea of what arguments a function takes and in > what order. It detracts from the overall readability of a program and can > also lead to very poorly formatted programs that break the guidelines in > PEP8. Though I have only demonstrated this for function declarations, the > example could also be extended to inline statement comments as well. Things > get too messy too quickly. > > My Alternative Proposal: > As an alternative, I would like to propose a syntax that Pycharm already > supports: > http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html > > Since this type information isn't going to be used at runtime in the > regular Python interpreter anyway, why not have it in the function > docstring instead? This provides both readability and type checking. > Standardizing that syntax or at least adding it as an optional way to check > your program would in my opinion be a much better addition to the language. > This approach needs no new syntax, keeps readability and allows the > programmer to add additional documentation without going over the 80 > character limit. > Without commenting on the specific format of the docstring, there is an added benefit to using docstrings for parameter and return value type information: It encourages people to write documentation. (the exact format of types in a docstring could turn into its own bikeshed even grander than this thread already is) JavaScript successfully uses this approach for type annotations. (sure, its in comments, but that's because they don't _have_ docstrings). In a way, using docstrings is similar to what argument clinic does for extension modules. We could provide a standard way to do it and have the language runtime parse them and turn them into actual annotation objects when the __annotations__ attribute is first accessed on anything. If someone really wanted to they could have it hide the information from the docstring at the same time (I don't recommend that. Argument clinic is "special" and had a real a need for this). That laziness *mostly* avoids the forward referencing or forward declaration mess that you'd otherwise have. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Aug 15 17:17:33 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 08:17:33 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 12:48 AM, ?ukasz Langa wrote: > On Aug 14, 2014, at 4:56 PM, Guido van Rossum wrote: > > There's also a variant of (1) that ?ukasz Langa would like to see -- use > the syntactic position of function annotations but using a custom syntax > (e.g. one similar to the pytypedecl syntax) that isn't evaluated at > function-definition time. This would have to use "from __future__ import > " for backward compatibility. I'm skeptical about this though; > it is only slightly more elegant than mypy, and it would open the > floodgates of unconstrained language design. > > > I see the decision has been made. For the curious, the design would be as > close as possible to PEP 3107. The biggest wins would be first-class > annotations for variables (supported in Mypy as comments) and support for > forward-references without the need to fall back to strings. > You can probably come up with a notation for first-class variable annotations, e.g. x: Sequence[int] = [] The value might be optional. The question is though, would the type (Sequence[int]) be stored anyway? Also, in a class body, does it define a class var or an instance var (or doesn't it matter?). Does this need a 'var' keyword to be ambiguous? I propose to disallow declaring multiple variables in this style, since it's hard to decide whether the comma should bind tighter than the '=' sign (as in assignments) or less tight (as in function headings). > > I?m also not a fan of the square brackets for generics (those brackets > mean lookup!) but a BDFL once said that ?language evolution is the art of > compromise? and one cannot disagree with that. > > > So there you have it. I am picking the mypy family and I hope we can start > focusing on specific improvements to mypy. > > > Alright! That sounds good. With the syntax mostly out of the way, the next > issue to handle is the typing module. dict, MutableMapping, and now Dict? > One step too far. We should be able to re-use ABCs for that, e.g. to add > support for union types and generics. Lots of decisions ahead (covariance, > casting, multiple dispatch, etc.) but we?ll get there. > > The typing module as aliases for the updated ABCs sounds like a fair > compromise, although I?m worried about the details of that approach (for > starters: we need to bundle the new ABCMeta to support union types but > having both implementations at runtime feels very wrong). Moreover, with > dynamic type registration and duck typing isinstance(), there will be > challenges to cover. I?m happy to go and slay that dragon. > > As a side note, I?m happy you?re willing to agree on str | None. This > reads really well and is concise enough to not require aliasing to be > usable. > > While we?re at slaying dragons, I?ll also silently make str non-iterable > so that we can use Sequence[str] meaningfully from now on? How about that? > I hope you meant that as a joke. We missed our chance for that one with Python 3.0. We must live with it. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrocklin at gmail.com Fri Aug 15 17:33:41 2014 From: mrocklin at gmail.com (Matthew Rocklin) Date: Fri, 15 Aug 2014 08:33:41 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch Message-ID: Hi all, I curate the multipledispatch library on PyPI. I believe that it is a natural continuation of singledispatch outlined in PEP 443 and included in functools 3.4. For those unaware, dispatching correctly and unambiguously on multiple inputs is a somewhat more complex problem than on a single input. I believe the approach taken in multipledispatch is fairly robust. I wrote a blogpost about MD a while ago: http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch/ The docs pages live here: http://multiple-dispatch.readthedocs.org/en/latest/ And the github page lives here: https://github.com/mrocklin/multipledispatch Recommendations for improvement welcome. Thoughts on whether or not this is appropriate to include in the standard library also welcome. Best, -Matthew Rocklin -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Aug 15 17:41:47 2014 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 15 Aug 2014 15:41:47 +0000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: <20140814174512.GQ4525@ando> <87vbpuh3wk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu Aug 14 2014 at 8:02:14 PM Juancarlo A?ez wrote: > > On Thu, Aug 14, 2014 at 10:09 PM, Stephen J. Turnbull > wrote: > >> All that Guido is proposing to do is to rationalize development effort >> on something people are already doing by standardizing a good-enough >> approach, on an opt-in basis. Sure, there are people who want >> something more coercive, but Python developers aren't going to stand >> for that, and Guido himself is the first line of defense. He's >> already said that's not going to happen, not now (and IIUC probably >> never). >> > > The change has been decreed, so it will be. > Where? Please link to that in the mailing list archives if so. > > It's all right, because (as Guido suggests in the decree) we can now take > the discussion to how to make it good for all involved. The Python > community will likely succeed at that. > > It is a tautology that only experience will reveal the truth, at least so > in our endeavour. > > Signing off this thread, yours truly, > > -- > Juancarlo *A?ez* > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Fri Aug 15 17:42:05 2014 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 15 Aug 2014 17:42:05 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 9:48 AM, ?ukasz Langa wrote: ... > As a side note, I?m happy you?re willing to agree on str | None. This reads > really well and is concise enough to not require aliasing to be usable. The common use is not all that concise: def foo(bar: int | None=None): pass Or alternatively it could be: def foo(bar: int=None): pass if the default was automatically allowed. Also... Does None magically mean NoneType in type definitions? From guido at python.org Fri Aug 15 17:55:36 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 08:55:36 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Aug 15, 2014 8:43 AM, "Petr Viktorin" wrote: > > On Fri, Aug 15, 2014 at 9:48 AM, ?ukasz Langa wrote: > ... > > As a side note, I?m happy you?re willing to agree on str | None. This reads > > really well and is concise enough to not require aliasing to be usable. > > The common use is not all that concise: > def foo(bar: int | None=None): pass > > Or alternatively it could be: > def foo(bar: int=None): pass > if the default was automatically allowed. Good idea. > Also... Does None magically mean NoneType in type definitions? Yes. --Guido -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Aug 15 18:15:17 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 15 Aug 2014 09:15:17 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EE3215.6030907@stoneleaf.us> On 08/15/2014 04:53 AM, Bill Winslow wrote: > > I knew something like this would happen when I decided to actually do something. Take heart, we've all done it, and you've learned something important that I bet you don't soon forget. ;) -- ~Ethan~ From greg at krypto.org Fri Aug 15 18:24:09 2014 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 15 Aug 2014 16:24:09 +0000 Subject: [Python-ideas] Optional static typing -- the crossroads References: Message-ID: On Thu Aug 14 2014 at 9:47:45 PM Guido van Rossum wrote: > On Thu, Aug 14, 2014 at 9:40 PM, Nick Coghlan wrote: > >> On 15 August 2014 09:56, Guido van Rossum wrote: >> > >> > I don't buy the argument that PEP 3107 promises that annotations are >> > completely free of inherent semantics. >> >> It's also worth noting the corresponding bullet point in PEP 3100 >> (under http://www.python.org/dev/peps/pep-3100/#core-language): >> >> * Add optional declarations for static typing [45] [10] [done] >> >> [10] Guido's blog ("Python Optional Typechecking Redux") >> http://www.artima.com/weblogs/viewpost.jsp?thread=89161 >> [45] PEP 3107 (Function Annotations) >> http://www.python.org/dev/peps/pep-3107 >> > > Such youthful optimism. :-) > > >> Having Argument Clinic generate appropriate annotations automatically >> could also be interesting. >> > > How much of the 3.5 stdlib is currently covered by Argument Clinic? I > thought there's still a lot left to do. Might it be possible to convert > some of pytypedecl's stubs into AC stubs? Alternatively, the AC info could > be turned into mypy stubs. I'm just really hoping that between AC, > pytypedecl, PyCharm and mypy we have specs for most builtins and extension > modules in machine-readable form already, and we could use this combined > information to bootstrap mypy's collection of stubs. > > I believe we have partially intersecting subsets of builtins and stdlib coverage, union them all together and sanity check them and it's a good start but will likely still have giant holes to be filled in. We've been concentrating on 2.7 with the code analysis to generate pytypedecl pytd's for but have always assumed that argument clinic would be useful in providing annotation details for 3.4 onwards. Should it generate annotation files itself? possibly, but I'm not sure it is expressive enough to generate an ideal annotation. To start with I'd leave generating annotations Python builtins, extensions and internals itself out of CPython itself in 3.5. Such things can be pulled in with tools to generate them in later release once we're happy it is easy to maintain via the tools without a much human tweaking being required. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From brakhane at googlemail.com Fri Aug 15 18:25:33 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 18:25:33 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <20140815143821.GC4525@ando> References: <53EDFF19.7060305@gmail.com> <20140815143821.GC4525@ando> Message-ID: <53EE347D.2040801@googlemail.com> On 15.08.2014 16:38, Steven D'Aprano wrote: > How do you know the linter will complain? Because a file object is not a list of strings, which is what word_count incorrectly declares it needs. (it actually requires a iterable of things that can be split()ed and doesn't return a Dict[str, int] but a Dict[return_type_of_split(), int]) > So why do you consider it a "must" to be able to override functions > that report type errors at compile time? The difference in that latter case is that the code runs perfectly fine and correctly. It's just that the linter implies you've done something bad. Using isinstance checks is kinda frowned upon, using type annotations will be probably considered to be totally acceptable practise (if not, what's the point of this proposal). It's much harder to argue with the original author that it's a bug in the latter case. Furthermore, the author might not want to loose the type requirement, because he doesn't want to guarantee those semantics, for example. If I'm willing to take that risk (and have a test case in my code) why shouldn't I be allowed to silence those errors in a simple way that doesn't require casting at every method call? To me, one of the things that sets Python apart from other languages is the fact that code will and can be used in a way the original author might not have thought of. Giving the original author of methods the means to dictate what types are acceptable and not with no clean and simple way of overriding it just feels Java-esque to me. (Again, isinstance checks are considered a bad practise, but I doubt declaring too restrictive types will be) I do not want to be forced to litter my code with casts, making it ugly and feel bad about what seems to me a reasonable method call. > But for what it's worth, as this proposal has repeatedly said, the > linter is optional. If you don't believe the linter, *just run the > code* in Python like you have always done before. As already said, I might be forced to run it because of company policy. I might contribute to a library that uses mypy. I don't have a big problem with complying with strange code style requirements enforced by a linter ("in this project, all variables must begin with foobar and end with a number"), that's just naming. I do have a problem when a linter will de facto enforce rules like "all code must only call methods in the way the original author thought of", as this might lead to more ugly code because of workarounds/casts. Cheers, Dennis From greg at krypto.org Fri Aug 15 18:35:54 2014 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 15 Aug 2014 16:35:54 +0000 Subject: [Python-ideas] Optional static typing -- the crossroads References: <53EDFF19.7060305@gmail.com> Message-ID: On Fri Aug 15 2014 at 5:46:12 AM Dennis Brakhane wrote: > Am 15.08.2014 01:56, schrieb Guido van Rossum: > > > PS. I realize I didn't discuss question (C) much. That's intentional > -- we can now start discussing specific mypy features in separate > threads (or in this one :-). > > So is this the place to discuss "the thornier issues brought up against > mypy"? Because I think it's important that we get an idea of what we > want mypy to be able to do and what > not. After all, mypy will probably end up as a kind of reference > implementation for static type checkers. And I'm worried there might be > real damage to Python as a language > if they aren't thought through: > I'm not concerned about that myself. If this syntax doesn't work out, the the existing status quo prevails and the syntax is ignored by other tools that need more than it can provide. That still leads to the same feedback cycle we already have today such that the language syntax for type annotations can evolve and improve again in the future. [Caution: I'm commenting about mypy below after having spent less than 10 minutes looking at its website to pretend I know what it can and can't do already. Assume I'm wrong.] ie: there are things I don't know about mypy. Does it have the ability to specify that the return type of "def foo(A, B)" is the same type as whatever the caller passed in for parameter B? That is a pretty common thing in Python. Even if it doesn't have it today, I suspect it can be added in the future. There are other things mypy didn't appear to deal with at first glance either, specific sets of possible inputs -> outputs rather than always listing inputs as a union of all possible types for that parameter and outputs as a union of all possible types to be output. I may well be wrong about the above. But even if I'm not, I'm not worried. Deeper analysis and annotation tools will simply do what they are already doing: plowing on ahead with their own extended annotation format. I understand your concerns (in the rest of our message that i've elided) but I think a "try it and see" approach will actually work here. Libraries are already released where people have gone overboard with incorrect overly strict isinstance or issubclass checks. Those are _worse_ than something that merely lists overly strict types as you literally cannot use them without modifying the code or complying. Something any code analyzer implementation needs is an ability to be told "ignore this module, it's full of crap." as part of its analysis process. :) -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From brakhane at googlemail.com Fri Aug 15 18:43:02 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 18:43:02 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EE3896.2050804@googlemail.com> On 15.08.2014 17:42, Petr Viktorin wrote: > > The common use is not all that concise: > def foo(bar: int | None=None): pass > > Or alternatively it could be: > def foo(bar: int=None): pass > if the default was automatically allowed. (Assuming you mean "the type of the default") While I like the second form a bit more, it kinda goes against "explicit is better than implicit". Also, if I change the default value from None to 42, I've either changed the allowable types, or need to remember to turn "bar: int=None" into "bar: int|None = 42". Furthermore, what should happen in the following case: # no annotations here def foo(): ... def bar(evil: int = foo()): ... Should this be disallowed, as the type checker will not be able to know what type foo is? Should it just assume int? And in the latter case, what should happen if foo now gains a "-> float" annotation? From encukou at gmail.com Fri Aug 15 18:48:54 2014 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 15 Aug 2014 18:48:54 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum wrote: ... >> Also... Does None magically mean NoneType in type definitions? > > Yes. This would mean either that `(None | None) is None`, or that (x | None) is not always "optional x". And if type objects grow any other common functionality, None will have to support that as well. From guido at python.org Fri Aug 15 18:56:35 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 09:56:35 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EE347D.2040801@googlemail.com> References: <53EDFF19.7060305@gmail.com> <20140815143821.GC4525@ando> <53EE347D.2040801@googlemail.com> Message-ID: On Fri, Aug 15, 2014 at 9:25 AM, Dennis Brakhane wrote: > On 15.08.2014 16:38, Steven D'Aprano wrote: > > How do you know the linter will complain? > Because a file object is not a list of strings, which is what word_count > incorrectly declares it needs. (it actually requires a iterable of > things that can be split()ed and doesn't return a Dict[str, int] but a > Dict[return_type_of_split(), int]) > > > So why do you consider it a "must" to be able to override functions > > that report type errors at compile time? > > The difference in that latter case is that the code runs perfectly fine > and correctly. It's just that the linter implies you've done something bad. > > Using isinstance checks is kinda frowned upon, using type annotations > will be probably considered to be totally acceptable practise (if not, > what's the point of this proposal). It's much harder to argue with the > original author that it's a bug in the latter case. Furthermore, the > author might not want to loose the type requirement, because he doesn't > want to guarantee those semantics, for example. If I'm willing to take > that risk (and have a test case in my code) why shouldn't I be allowed > to silence those errors in a simple way that doesn't require casting at > every method call? > > To me, one of the things that sets Python apart from other languages is > the fact that code will and can be used in a way the original author > might not have thought of. > > Giving the original author of methods the means to dictate what types > are acceptable and not with no clean and simple way of overriding it > just feels Java-esque to me. (Again, isinstance checks are considered a > bad practise, but I doubt declaring too restrictive types will be) > > I do not want to be forced to litter my code with casts, making it ugly > and feel bad about what seems to me a reasonable method call. > > > > But for what it's worth, as this proposal has repeatedly said, the > > linter is optional. If you don't believe the linter, *just run the > > code* in Python like you have always done before. > As already said, I might be forced to run it because of company policy. > I might contribute to a library that uses mypy. > > I don't have a big problem with complying with strange code style > requirements enforced by a linter ("in this project, all variables must > begin with foobar and end with a number"), that's just naming. > > I do have a problem when a linter will de facto enforce rules like "all > code must only call methods in the way the original author thought of", > as this might lead to more ugly code because of workarounds/casts. > This attitude will not help you when interviewing at such a company. Did it occur to you that there might actually be a good reason for the lint rule, and that you, as a new hire, might not yet be aware of that reason? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Aug 15 19:00:13 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 10:00:13 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin wrote: > On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum > wrote: > ... > >> Also... Does None magically mean NoneType in type definitions? > > > > Yes. > > This would mean either that `(None | None) is None`, or that (x | > None) is not always "optional x". > And if type objects grow any other common functionality, None will > have to support that as well. > Perhaps None itself should not implement any of this, and the __ror__ method on ABCs should implement it. That way, None|Mapping and Mapping|None would both work, yet None|None would still be the TypeError it is today. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Fri Aug 15 19:11:38 2014 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 15 Aug 2014 19:11:38 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EE3896.2050804@googlemail.com> References: <53EE3896.2050804@googlemail.com> Message-ID: On Fri, Aug 15, 2014 at 6:43 PM, Dennis Brakhane wrote: > On 15.08.2014 17:42, Petr Viktorin wrote: >> >> The common use is not all that concise: >> def foo(bar: int | None=None): pass >> >> Or alternatively it could be: >> def foo(bar: int=None): pass >> if the default was automatically allowed. > (Assuming you mean "the type of the default") I really meant *only* the default. This really only works for None, but that's a good thing, since something like: def foo(bar:int=''): pass looks very suspicious. I'd be fine with the linter complaining about foo('hello'). Of course you can always do: def foo(bar: (int | str)=''): pass > While I like the second form a bit more, it kinda goes against "explicit > is better than implicit". > > Also, if I change the default value from None to 42, I've either changed > the allowable types, > or need to remember to turn "bar: int=None" into "bar: int|None = 42". > > Furthermore, what should happen in the following case: > > > # no annotations here > def foo(): ... > > def bar(evil: int = foo()): ... > > > Should this be disallowed, as the type checker will not be able to know > what type foo is? Should it just assume int? > And in the latter case, what should happen if foo now gains a "-> float" > annotation? If your linter can't figure it out, just specify the default's type explicitly. Always a good thing to do when something's not immediately obvious. From brakhane at googlemail.com Fri Aug 15 19:12:04 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 15 Aug 2014 19:12:04 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EDFF19.7060305@gmail.com> Message-ID: <53EE3F64.5090600@googlemail.com> On 15.08.2014 18:35, Gregory P. Smith wrote: > > Libraries are already released where people have gone overboard with > incorrect overly strict isinstance or issubclass checks. Those are > _worse_ than something that merely lists overly strict types as you > literally cannot use them without modifying the code or complying. > I agree. > That still leads to the same feedback cycle we already have today such that the language syntax for > type annotations can evolve and improve again in the future. > > Does it have the ability to specify that the return type of "def > foo(A, B)" is the same type as whatever the caller passed in for > parameter B? That is a pretty common thing in Python. Even if it > doesn't have it today, I suspect it can be added in the future. That's a good example of what I'm worried about: to be really useful, the type declarations have to be really flexible. My (maybe irrational) fear is that we end up with a turing complete type system. If a programmers needs to spend 10 minutes thinking about how exactly he has to declare his method parameters or "class interfaces" (does my container type behave covariantly or contravariantly?), and all that just so his code passes a linter, something went wrong ;) We shouldn't try to make the signatures carry all information needed for type checkers, if some things can only be found out by code analysis, that's fine by me. For example, Jedi can handle the following silly example: def foo(): return [os, sys] x = foo() Jedi knows that x[1] is sys and will only propose members of sys, and only os members for x[0]. And the linter of the current development version will actually barf on x[1].walk(): /tmp/a.py:12:5: E1 AttributeError: > has no attribute walk. Trying to get this kind information encoded into some kind of type signature would be insane, IMO. From encukou at gmail.com Fri Aug 15 19:19:38 2014 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 15 Aug 2014 19:19:38 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum wrote: > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin wrote: >> >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum >> wrote: >> ... >> >> Also... Does None magically mean NoneType in type definitions? >> > >> > Yes. >> >> This would mean either that `(None | None) is None`, or that (x | >> None) is not always "optional x". >> And if type objects grow any other common functionality, None will >> have to support that as well. > > > Perhaps None itself should not implement any of this, and the __ror__ method > on ABCs should implement it. That way, None|Mapping and Mapping|None would > both work, yet None|None would still be the TypeError it is today. ... and that (x|None) does not always mean "optional x". Is this case special enough? From guido at python.org Fri Aug 15 19:28:20 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 10:28:20 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: Have you written anything significant that you're distributing with it? Has anyone else? Are there good uses of singledispatch in the wild even? On Fri, Aug 15, 2014 at 8:33 AM, Matthew Rocklin wrote: > Hi all, > > I curate the multipledispatch library on PyPI. I believe that it is a > natural continuation of singledispatch outlined in PEP 443 and included in > functools 3.4. > > For those unaware, dispatching correctly and unambiguously on multiple > inputs is a somewhat more complex problem than on a single input. I > believe the approach taken in multipledispatch is fairly robust. > > I wrote a blogpost about MD a while ago: > http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch/ > > The docs pages live here: > http://multiple-dispatch.readthedocs.org/en/latest/ > > And the github page lives here: > https://github.com/mrocklin/multipledispatch > > Recommendations for improvement welcome. Thoughts on whether or not this > is appropriate to include in the standard library also welcome. > > Best, > -Matthew Rocklin > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Aug 15 19:34:12 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 15 Aug 2014 10:34:12 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: <53EE4494.8050609@stoneleaf.us> On 08/15/2014 10:28 AM, Guido van Rossum wrote: > > Are there good uses of singledispatch in the wild even? Well, I don't know if it's a "good use", but I have used singledispatch to override 'float' in certain modules so that I can say, for example, float(DateTime.Time) or float("time string"). It's really quite handy. -- ~Ethan~ From guido at python.org Fri Aug 15 19:36:50 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 10:36:50 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 10:19 AM, Petr Viktorin wrote: > On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum > wrote: > > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin > wrote: > >> > >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum > >> wrote: > >> ... > >> >> Also... Does None magically mean NoneType in type definitions? > >> > > >> > Yes. > >> > >> This would mean either that `(None | None) is None`, or that (x | > >> None) is not always "optional x". > >> And if type objects grow any other common functionality, None will > >> have to support that as well. > > > > > > Perhaps None itself should not implement any of this, and the __ror__ > method > > on ABCs should implement it. That way, None|Mapping and Mapping|None > would > > both work, yet None|None would still be the TypeError it is today. > > ... and that (x|None) does not always mean "optional x". > Is this case special enough? > I'm not following. The proposal seems to be to add __or__ and __ror__ methods to type itself requiring the other argument to be also a type, or the special case None (which is a value, not a type). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Aug 15 19:47:22 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 15 Aug 2014 13:47:22 -0400 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum wrote: > Are there good uses of singledispatch in the wild even? In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution. Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch. -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Fri Aug 15 19:51:12 2014 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 15 Aug 2014 19:51:12 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 7:36 PM, Guido van Rossum wrote: > On Fri, Aug 15, 2014 at 10:19 AM, Petr Viktorin wrote: >> >> On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum >> wrote: >> > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin >> > wrote: >> >> >> >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum >> >> wrote: >> >> ... >> >> >> Also... Does None magically mean NoneType in type definitions? >> >> > >> >> > Yes. >> >> >> >> This would mean either that `(None | None) is None`, or that (x | >> >> None) is not always "optional x". >> >> And if type objects grow any other common functionality, None will >> >> have to support that as well. >> > >> > >> > Perhaps None itself should not implement any of this, and the __ror__ >> > method >> > on ABCs should implement it. That way, None|Mapping and Mapping|None >> > would >> > both work, yet None|None would still be the TypeError it is today. >> >> ... and that (x|None) does not always mean "optional x". >> Is this case special enough? > > > I'm not following. The proposal seems to be to add __or__ and __ror__ > methods to type itself requiring the other argument to be also a type, or > the special case None (which is a value, not a type). My concern is that if someone does programmatic type declaration manipulation/generation, there's now a special case to keep in mind. Instead of def optional(t): return t | None it's now: def optional(t): if t is None: return t else: return t | None because unlike other type declarations, None doesn't have __or__, or any other operation that types will gain in the future as this proposal matures. But maybe this is will never be a valid use case? From guido at python.org Fri Aug 15 19:55:06 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 10:55:06 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: Doesn't seem a big deal. The only place where you'd see an implementation of optional() would be in typing.py, and optional(None) is redundant anyway. On Fri, Aug 15, 2014 at 10:51 AM, Petr Viktorin wrote: > On Fri, Aug 15, 2014 at 7:36 PM, Guido van Rossum > wrote: > > On Fri, Aug 15, 2014 at 10:19 AM, Petr Viktorin > wrote: > >> > >> On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum > >> wrote: > >> > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin > >> > wrote: > >> >> > >> >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum > >> >> wrote: > >> >> ... > >> >> >> Also... Does None magically mean NoneType in type definitions? > >> >> > > >> >> > Yes. > >> >> > >> >> This would mean either that `(None | None) is None`, or that (x | > >> >> None) is not always "optional x". > >> >> And if type objects grow any other common functionality, None will > >> >> have to support that as well. > >> > > >> > > >> > Perhaps None itself should not implement any of this, and the __ror__ > >> > method > >> > on ABCs should implement it. That way, None|Mapping and Mapping|None > >> > would > >> > both work, yet None|None would still be the TypeError it is today. > >> > >> ... and that (x|None) does not always mean "optional x". > >> Is this case special enough? > > > > > > I'm not following. The proposal seems to be to add __or__ and __ror__ > > methods to type itself requiring the other argument to be also a type, or > > the special case None (which is a value, not a type). > > My concern is that if someone does programmatic type declaration > manipulation/generation, there's now a special case to keep in mind. > Instead of > > def optional(t): > return t | None > > it's now: > > def optional(t): > if t is None: > return t > else: > return t | None > > because unlike other type declarations, None doesn't have __or__, or > any other operation that types will gain in the future as this > proposal matures. > > But maybe this is will never be a valid use case? > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrocklin at gmail.com Fri Aug 15 19:55:57 2014 From: mrocklin at gmail.com (Matthew Rocklin) Date: Fri, 15 Aug 2014 10:55:57 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: I've definitely seen homegrown solutions like Alexander talks about in a variety of projects. It has popped up a few times in SymPy. My implementation was influenced somewhat by Julia's solution which seems fairly sober. Multiple dispatch has demonstrated value in that community. I used the multipledispatch library in a few of my projects. I can provide examples of where it's been helpful if desired. On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum > wrote: > >> Are there good uses of singledispatch in the wild even? > > > In one of my projects, I have a todo item to convert a homegrown > {type:func} dictionary based dispatch to singledispatch. It has been open > for 5 months already. I wouldn't be surprised if others had something > similar: a good-enough (and probably buggy) solution that is not bad enough > to justify adding extra dependency in 2.x or replacing with a 3.x only > solution. > > Multiple dispatch is much harder to get right or even "good enough," so I > don't think singledispatch popularity or lack thereof is a good predictor > for multipledispatch. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Aug 15 20:01:07 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 11:01:07 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: Please do write about non-toy examples! On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin wrote: > I've definitely seen homegrown solutions like Alexander talks about in a > variety of projects. It has popped up a few times in SymPy. > > My implementation was influenced somewhat by Julia's solution which seems > fairly sober. Multiple dispatch has demonstrated value in that community. > > I used the multipledispatch library in a few of my projects. I can > provide examples of where it's been helpful if desired. > > > On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> >> On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum >> wrote: >> >>> Are there good uses of singledispatch in the wild even? >> >> >> In one of my projects, I have a todo item to convert a homegrown >> {type:func} dictionary based dispatch to singledispatch. It has been open >> for 5 months already. I wouldn't be surprised if others had something >> similar: a good-enough (and probably buggy) solution that is not bad enough >> to justify adding extra dependency in 2.x or replacing with a 3.x only >> solution. >> >> Multiple dispatch is much harder to get right or even "good enough," so I >> don't think singledispatch popularity or lack thereof is a good predictor >> for multipledispatch. >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Fri Aug 15 20:25:35 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 15 Aug 2014 11:25:35 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: It has over 25k downloads from PyPI per month. It?s used in OpenStack?s Nova. It?s used at Facebook as part of a deadline based, resource aware, dependency based, fault tolerant scheduler. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev On Aug 15, 2014, at 11:01 AM, Guido van Rossum wrote: > Please do write about non-toy examples! > > > On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin wrote: > I've definitely seen homegrown solutions like Alexander talks about in a variety of projects. It has popped up a few times in SymPy. > > My implementation was influenced somewhat by Julia's solution which seems fairly sober. Multiple dispatch has demonstrated value in that community. > > I used the multipledispatch library in a few of my projects. I can provide examples of where it's been helpful if desired. > > > On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky wrote: > > On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum wrote: > Are there good uses of singledispatch in the wild even? > > In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution. > > Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch. > > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrocklin at gmail.com Fri Aug 15 20:33:06 2014 From: mrocklin at gmail.com (Matthew Rocklin) Date: Fri, 15 Aug 2014 11:33:06 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: I assume that Lukasz is referrring to singledispatch? multipledispatch doesn't yet have that level of adoption as far as I'm aware. On Fri, Aug 15, 2014 at 11:25 AM, ?ukasz Langa wrote: > It has over 25k downloads from PyPI per month. It?s used in OpenStack?s > Nova. It?s used at Facebook as part of a deadline based, resource aware, > dependency based, fault tolerant scheduler. > > -- > Best regards, > ?ukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > > > > On Aug 15, 2014, at 11:01 AM, Guido van Rossum wrote: > > Please do write about non-toy examples! > > > On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin > wrote: > >> I've definitely seen homegrown solutions like Alexander talks about in a >> variety of projects. It has popped up a few times in SymPy. >> >> My implementation was influenced somewhat by Julia's solution which seems >> fairly sober. Multiple dispatch has demonstrated value in that community. >> >> I used the multipledispatch library in a few of my projects. I can >> provide examples of where it's been helpful if desired. >> >> >> On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky < >> alexander.belopolsky at gmail.com> wrote: >> >>> >>> On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum >>> wrote: >>> >>>> Are there good uses of singledispatch in the wild even? >>> >>> >>> In one of my projects, I have a todo item to convert a homegrown >>> {type:func} dictionary based dispatch to singledispatch. It has been open >>> for 5 months already. I wouldn't be surprised if others had something >>> similar: a good-enough (and probably buggy) solution that is not bad enough >>> to justify adding extra dependency in 2.x or replacing with a 3.x only >>> solution. >>> >>> Multiple dispatch is much harder to get right or even "good enough," so >>> I don't think singledispatch popularity or lack thereof is a good predictor >>> for multipledispatch. >>> >> >> > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Fri Aug 15 20:38:02 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 15 Aug 2014 11:38:02 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: <967197E8-94BD-45C3-BCD8-BFD0690EFF12@langa.pl> Ah, that?s right, I meant singledispatch. Sorry for the confusion! :) -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev On Aug 15, 2014, at 11:33 AM, Matthew Rocklin wrote: > I assume that Lukasz is referrring to singledispatch? multipledispatch doesn't yet have that level of adoption as far as I'm aware. > > > On Fri, Aug 15, 2014 at 11:25 AM, ?ukasz Langa wrote: > It has over 25k downloads from PyPI per month. It?s used in OpenStack?s Nova. It?s used at Facebook as part of a deadline based, resource aware, dependency based, fault tolerant scheduler. > > -- > Best regards, > ?ukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > > > > On Aug 15, 2014, at 11:01 AM, Guido van Rossum wrote: > >> Please do write about non-toy examples! >> >> >> On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin wrote: >> I've definitely seen homegrown solutions like Alexander talks about in a variety of projects. It has popped up a few times in SymPy. >> >> My implementation was influenced somewhat by Julia's solution which seems fairly sober. Multiple dispatch has demonstrated value in that community. >> >> I used the multipledispatch library in a few of my projects. I can provide examples of where it's been helpful if desired. >> >> >> On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky wrote: >> >> On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum wrote: >> Are there good uses of singledispatch in the wild even? >> >> In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution. >> >> Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch. >> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varma.sunjay at gmail.com Fri Aug 15 20:43:23 2014 From: varma.sunjay at gmail.com (Sunjay Varma) Date: Fri, 15 Aug 2014 14:43:23 -0400 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: Hi all, Has the syntax for specifying type been fully decided on already? Using brackets may confuse new Python programmers. Since specifying type in Python is fairly new anyway, what do you all think of introducing angle brackets into Python instead? Other languages use angle brackets to specify types. It provides a good separation between type specification and list indexing. I'm also worried that using square brackets will cause confusion as that notation is generally associated with array declarations in other languages. Even in Python, MyClass[int] may be confused with getting a key called int from some MyClass. dict seems to tell me more explicitly that I'm dealing with a declaration of an expected type. dict[str, int] looks very much like I'm getting an item (str, int) from some class. The angle bracket (or any other suggestions you have in mind) provides a more concrete separation between when we are performing item indexing and when we're specifying a type to validate. Sunjay On Aug 15, 2014 1:55 PM, "Guido van Rossum" wrote: > Doesn't seem a big deal. The only place where you'd see an implementation > of optional() would be in typing.py, and optional(None) is redundant anyway. > > > On Fri, Aug 15, 2014 at 10:51 AM, Petr Viktorin wrote: > >> On Fri, Aug 15, 2014 at 7:36 PM, Guido van Rossum >> wrote: >> > On Fri, Aug 15, 2014 at 10:19 AM, Petr Viktorin >> wrote: >> >> >> >> On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum >> >> wrote: >> >> > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin >> >> > wrote: >> >> >> >> >> >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum > > >> >> >> wrote: >> >> >> ... >> >> >> >> Also... Does None magically mean NoneType in type definitions? >> >> >> > >> >> >> > Yes. >> >> >> >> >> >> This would mean either that `(None | None) is None`, or that (x | >> >> >> None) is not always "optional x". >> >> >> And if type objects grow any other common functionality, None will >> >> >> have to support that as well. >> >> > >> >> > >> >> > Perhaps None itself should not implement any of this, and the __ror__ >> >> > method >> >> > on ABCs should implement it. That way, None|Mapping and Mapping|None >> >> > would >> >> > both work, yet None|None would still be the TypeError it is today. >> >> >> >> ... and that (x|None) does not always mean "optional x". >> >> Is this case special enough? >> > >> > >> > I'm not following. The proposal seems to be to add __or__ and __ror__ >> > methods to type itself requiring the other argument to be also a type, >> or >> > the special case None (which is a value, not a type). >> >> My concern is that if someone does programmatic type declaration >> manipulation/generation, there's now a special case to keep in mind. >> Instead of >> >> def optional(t): >> return t | None >> >> it's now: >> >> def optional(t): >> if t is None: >> return t >> else: >> return t | None >> >> because unlike other type declarations, None doesn't have __or__, or >> any other operation that types will gain in the future as this >> proposal matures. >> >> But maybe this is will never be a valid use case? >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Fri Aug 15 20:55:44 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 15 Aug 2014 11:55:44 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <2971B072-78B1-4E51-9B0F-50BFFDD66535@langa.pl> On Aug 15, 2014, at 11:43 AM, Sunjay Varma wrote: > Hi all, > Has the syntax for specifying type been fully decided on already? > > Using brackets may confuse new Python programmers. (?) Other languages use angle brackets to specify types. > I also agree that angle brackets would be nicer. Guido decided against it for pragmatic reasons: 1. angle brackets would create Python source code incompatible with any version lower than 3.5 2. angle brackets would complicate the lexer (normally you expect < and > to be spaced, in this case it wouldn?t) 3. angle brackets would require a new mechanism in Python to store this kind of expression within the type; this is still true for generics expressed with square brackets but at least you can use the existing nuts and bolts of Python classes All in all, this is more trouble than it?s worth. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Fri Aug 15 20:56:22 2014 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 15 Aug 2014 20:56:22 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 8:43 PM, Sunjay Varma wrote: > Hi all, > Has the syntax for specifying type been fully decided on already? > > Using brackets may confuse new Python programmers. Since specifying type in > Python is fairly new anyway, what do you all think of introducing angle > brackets into Python instead? Other languages use angle brackets to specify > types. It provides a good separation between type specification and list > indexing. > > I'm also worried that using square brackets will cause confusion as that > notation is generally associated with array declarations in other languages. > Even in Python, MyClass[int] may be confused with getting a key called int > from some MyClass. > > dict seems to tell me more explicitly that I'm dealing with a > declaration of an expected type. dict[str, int] looks very much like I'm > getting an item (str, int) from some class. > > The angle bracket (or any other suggestions you have in mind) provides a > more concrete separation between when we are performing item indexing and > when we're specifying a type to validate. Square brackets have the advantage of being valid Python now, so typed code would be backwards compatible. If the syntax was to change, what about a new operator? def sum(seq: Iterable of Number, start: Number): def print_grades(p: Mapping of (Student, Grade)): Just an idea. From kaiser.yann at gmail.com Fri Aug 15 20:57:04 2014 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Fri, 15 Aug 2014 20:57:04 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EE3896.2050804@googlemail.com> Message-ID: So, if you need to change how many things you specify typing information for depending on which parser you will use, how does putting those typing information stub objects into the standard library advance anyone? On 15 August 2014 19:11, Petr Viktorin wrote: > On Fri, Aug 15, 2014 at 6:43 PM, Dennis Brakhane > wrote: > > On 15.08.2014 17:42, Petr Viktorin wrote: > >> > >> The common use is not all that concise: > >> def foo(bar: int | None=None): pass > >> > >> Or alternatively it could be: > >> def foo(bar: int=None): pass > >> if the default was automatically allowed. > > (Assuming you mean "the type of the default") > > I really meant *only* the default. This really only works for None, > but that's a good thing, since something like: > def foo(bar:int=''): pass > looks very suspicious. I'd be fine with the linter complaining about > foo('hello'). > > Of course you can always do: > def foo(bar: (int | str)=''): pass > > > While I like the second form a bit more, it kinda goes against "explicit > > is better than implicit". > > > > Also, if I change the default value from None to 42, I've either changed > > the allowable types, > > or need to remember to turn "bar: int=None" into "bar: int|None = 42". > > > > Furthermore, what should happen in the following case: > > > > > > # no annotations here > > def foo(): ... > > > > def bar(evil: int = foo()): ... > > > > > > Should this be disallowed, as the type checker will not be able to know > > what type foo is? Should it just assume int? > > And in the latter case, what should happen if foo now gains a "-> float" > > annotation? > > If your linter can't figure it out, just specify the default's type > explicitly. Always a good thing to do when something's not immediately > obvious. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at ryanhiebert.com Fri Aug 15 20:56:17 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Fri, 15 Aug 2014 13:56:17 -0500 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> > On Aug 15, 2014, at 1:43 PM, Sunjay Varma wrote: > > Using brackets may confuse new Python programmers. Since specifying type in Python is fairly new anyway, what do you all think of introducing angle brackets into Python instead? Other languages use angle brackets to specify types. It provides a good separation between type specification and list indexing. Angle brackets already have meaning in Python, as comparison operators. The current surrounding operators ([], (), {}) require a matched pair in all cases. Breaking that rule would be confusing, though I know there are languages that do that. > > I'm also worried that using square brackets will cause confusion as that notation is generally associated with array declarations in other languages. Even in Python, MyClass[int] may be confused with getting a key called int from some MyClass. > > dict seems to tell me more explicitly that I'm dealing with a declaration of an expected type. dict[str, int] looks very much like I'm getting an item (str, int) from some class. > Getting an item from a class has no meaning for any classes that I?ve ever used, and I haven?t come up with any hypothetical one that would want to do that. I think that the parallel between item access and item declaration is a great argument in favor of using the dict[str, int] (or perhaps dict[str: int]) syntax as a type declaration. Ryan From ethan at stoneleaf.us Fri Aug 15 21:06:55 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 15 Aug 2014 12:06:55 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> References: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> Message-ID: <53EE5A4F.5060501@stoneleaf.us> On 08/15/2014 11:56 AM, Ryan Hiebert wrote: > > Getting an item from a class has no meaning for any classes that I?ve ever used, and I haven?t come up with any hypothetical one that would want to do that. --> class Foo(Enum): ... spam = 'meat flavored' ... eggs = 'chicken by-product' ... --> Foo --> Foo['spam'] -- ~Ethan~ From varma.sunjay at gmail.com Fri Aug 15 21:11:49 2014 From: varma.sunjay at gmail.com (Sunjay Varma) Date: Fri, 15 Aug 2014 15:11:49 -0400 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <2971B072-78B1-4E51-9B0F-50BFFDD66535@langa.pl> References: <2971B072-78B1-4E51-9B0F-50BFFDD66535@langa.pl> Message-ID: On Aug 15, 2014 2:55 PM, "?ukasz Langa" wrote: > > On Aug 15, 2014, at 11:43 AM, Sunjay Varma wrote: >> >> Hi all, >> Has the syntax for specifying type been fully decided on already? >> >> Using brackets may confuse new Python programmers. (?) Other languages use angle brackets to specify types. > > > I also agree that angle brackets would be nicer. Guido decided against it for pragmatic reasons: > 1. angle brackets would create Python source code incompatible with any version lower than 3.5 I'm all for compatibility, but Python 3 already breaks compatibility with Python 2. Why not add this feature in Python 4 (or whatever the next breaking release is) and do it "right" the first time. I don't think it makes sense to start muddling up the different semantic meanings of Python's operations just because we think it will break in an older version. This is such a big and important change. It deserves its own syntax (and if necessary a new version number as well). > 2. angle brackets would complicate the lexer (normally you expect < and > to be spaced, in this case it wouldn?t) > 3. angle brackets would require a new mechanism in Python to store this kind of expression within the type; this is still true for generics expressed with square brackets but at least you can use the existing nuts and bolts of Python classes Angle brackets were just a suggestion as they are used frequently by other languages. Even braces would be more appropriate as they're already built into the lexer and dict{int, str} clearly means something different than dict[int, str]. > > All in all, this is more trouble than it?s worth. I can understand that it's easier to use what's already there, but I don't agree with doing something just because it's easier. Especially when the side effects are not at all appealing. Sunjay -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Fri Aug 15 21:13:39 2014 From: skip at pobox.com (Skip Montanaro) Date: Fri, 15 Aug 2014 14:13:39 -0500 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 1:43 PM, Sunjay Varma wrote: > Using brackets may confuse new Python programmers. Since specifying type > in Python is fairly new anyway, what do you all think of introducing angle > brackets into Python instead? Is this a facility which new programmers are likely to encounter right off the bat or is it going to mostly be buried from casual view? The use of paired angle brackets has been suggested over the years for other purposes in Python. I no longer recall the arguments against them, but ISTR issues with grammar complexity and syntax highlighting in editors. Still, since C++ somehow managed to use them that way, perhaps all the various tools which might be exposed to them have been fixed by now. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Fri Aug 15 21:14:33 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 15 Aug 2014 12:14:33 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EE5A4F.5060501@stoneleaf.us> References: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> <53EE5A4F.5060501@stoneleaf.us> Message-ID: On Aug 15, 2014, at 12:06 PM, Ethan Furman wrote: > On 08/15/2014 11:56 AM, Ryan Hiebert wrote: >> >> Getting an item from a class has no meaning for any classes that I?ve ever used, and I haven?t come up with any hypothetical one that would want to do that. > > --> class Foo(Enum): > ... spam = 'meat flavored' > ... eggs = 'chicken by-product' > ... > --> Foo > > > --> Foo['spam'] > I also thought of enums. Looks fairly innocent to me, though. Do you see any cases where the two would conflict? -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Fri Aug 15 21:24:10 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 15 Aug 2014 12:24:10 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Aug 15, 2014, at 8:17 AM, Guido van Rossum wrote: > You can probably come up with a notation for first-class variable annotations, e.g. > > x: Sequence[int] = [] Yes, that syntax is out of scope for now, though, right? If I understand your reasoning behind choosing Mypy?s function annotation syntax, we don?t want to create programs that require Python 3.5+ just to be parsed. If we were to introduce first-class variable typing, yes, the syntax you propose is what I also had in mind. > The value might be optional. The question is though, would the type (Sequence[int]) be stored anyway? Also, in a class body, does it define a class var or an instance var (or doesn't it matter?). I wouldn?t change the current behaviour: class C: cls_member: str = ?on the class? def __init__(self): self.obj_member: str = ?on the instance' self.cls_member = 2 # that?s the real question: type error or an instance member? For that last case, even though it?s currently valid Python, my intuition tells me for Mypy to treat it as an error. > Does this need a 'var' keyword to be ambiguous? I fail to see any additional value provided by such keyword. What would stop people from doing var i = 1 I don?t think we want to end up with that. > I propose to disallow declaring multiple variables in this style, since it's hard to decide whether the comma should bind tighter than the '=' sign (as in assignments) or less tight (as in function headings). Right. I wonder if we even need this. For lines that use multiple assignment just for brevity, they can switch to multiple lines for typing. For common cases like: host, port = origin.rsplit(?:?, 1) successful, errors = query_the_world(hostnames) I think the types can be easily inferred (assuming rsplit and query_the_world are annotated). > While we?re at slaying dragons, I?ll also silently make str non-iterable so that we can use Sequence[str] meaningfully from now on? How about that? > > I hope you meant that as a joke. We missed our chance for that one with Python 3.0. We must live with it. Yes, that was obviously just a joke. By the way, is the PEP number 4000 free? Asking for a friend. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From varma.sunjay at gmail.com Fri Aug 15 21:25:53 2014 From: varma.sunjay at gmail.com (Sunjay Varma) Date: Fri, 15 Aug 2014 15:25:53 -0400 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> <53EE5A4F.5060501@stoneleaf.us> Message-ID: Another such example, since names are just names: dict = {"a": 2} print(dict["a"]) # 2 Overwriting a built in type name is bad, but entirely possible. dict["a"] here is also confusing with dict[str]. This kind of use could also potentially throw off a type linter too. These probably aren't the best examples out there, but I can definitely see this operator's meaning becoming very confused as more people start to apply it in different ways. We should not be just using something because it's there. Especially if it causes other problems. list[str] may be valid syntax in old Python 3 versions, but it's still not going to be correct if used in those versions. You're going to get some breakage no matter what. This feature is very new to Python as a whole, why not give it a syntax that provides a proper separation from what already was? Sunjay On Aug 15, 2014 3:14 PM, "?ukasz Langa" wrote: > On Aug 15, 2014, at 12:06 PM, Ethan Furman wrote: > > On 08/15/2014 11:56 AM, Ryan Hiebert wrote: > > > Getting an item from a class has no meaning for any classes that I?ve ever > used, and I haven?t come up with any hypothetical one that would want to do > that. > > > --> class Foo(Enum): > ... spam = 'meat flavored' > ... eggs = 'chicken by-product' > ... > --> Foo > > > --> Foo['spam'] > > > > I also thought of enums. Looks fairly innocent to me, though. Do you see > any cases where the two would conflict? > > -- > Best regards, > ?ukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at ryanhiebert.com Fri Aug 15 21:24:09 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Fri, 15 Aug 2014 14:24:09 -0500 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EE5A4F.5060501@stoneleaf.us> References: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> <53EE5A4F.5060501@stoneleaf.us> Message-ID: > On Aug 15, 2014, at 2:06 PM, Ethan Furman wrote: > > On 08/15/2014 11:56 AM, Ryan Hiebert wrote: >> >> Getting an item from a class has no meaning for any classes that I?ve ever used, and I haven?t come up with any hypothetical one that would want to do that. > > --> class Foo(Enum): > ... spam = 'meat flavored' > ... eggs = 'chicken by-product' > ... > --> Foo > > > --> Foo['spam'] > Well thanks for that ;-) I don?t think it would conflict in this case, since I don?t think there?d be a reason to mix Enum item access with Type item access. Especially because the Enum itself might be used in the type signature (though not item access on the Enum, I?d think). I think it?s enough of an edge-case that I still like the parallel it provides, but I appreciate the reality check. From guido at python.org Fri Aug 15 21:31:26 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 12:31:26 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EE5A4F.5060501@stoneleaf.us> References: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> <53EE5A4F.5060501@stoneleaf.us> Message-ID: On Fri, Aug 15, 2014 at 12:06 PM, Ethan Furman wrote: > On 08/15/2014 11:56 AM, Ryan Hiebert wrote: > >> >> Getting an item from a class has no meaning for any classes that I?ve >> ever used, and I haven?t come up with any hypothetical one that would want >> to do that. >> > > --> class Foo(Enum): > ... spam = 'meat flavored' > ... eggs = 'chicken by-product' > ... > --> Foo > > > --> Foo['spam'] > > That's a little unfortunate, but I don't think it's harmful, as I don't expect a use case for parametrized enums. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Aug 15 21:33:07 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 12:33:07 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <2971B072-78B1-4E51-9B0F-50BFFDD66535@langa.pl> Message-ID: On Fri, Aug 15, 2014 at 12:11 PM, Sunjay Varma wrote: > > On Aug 15, 2014 2:55 PM, "?ukasz Langa" wrote: > > > > On Aug 15, 2014, at 11:43 AM, Sunjay Varma > wrote: > >> > >> Hi all, > >> Has the syntax for specifying type been fully decided on already? > >> > >> Using brackets may confuse new Python programmers. (?) Other languages > use angle brackets to specify types. > > > > > > I also agree that angle brackets would be nicer. Guido decided against > it for pragmatic reasons: > > 1. angle brackets would create Python source code incompatible with any > version lower than 3.5 > > I'm all for compatibility, but Python 3 already breaks compatibility with > Python 2. Why not add this feature in Python 4 (or whatever the next > breaking release is) and do it "right" the first time. I don't think it > makes sense to start muddling up the different semantic meanings of > Python's operations just because we think it will break in an older > version. > There won't *be* a "next breaking release". > This is such a big and important change. It deserves its own syntax (and > if necessary a new version number as well). > > > 2. angle brackets would complicate the lexer (normally you expect < and > > to be spaced, in this case it wouldn?t) > > 3. angle brackets would require a new mechanism in Python to store this > kind of expression within the type; this is still true for generics > expressed with square brackets but at least you can use the existing nuts > and bolts of Python classes > > Angle brackets were just a suggestion as they are used frequently by other > languages. Even braces would be more appropriate as they're already built > into the lexer and dict{int, str} clearly means something different than > dict[int, str]. > > > > > All in all, this is more trouble than it?s worth. > > I can understand that it's easier to use what's already there, but I don't > agree with doing something just because it's easier. Especially when the > side effects are not at all appealing. > I don't think you quite appreciate the art of language evolution. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From haoyi.sg at gmail.com Fri Aug 15 21:33:20 2014 From: haoyi.sg at gmail.com (Haoyi Li) Date: Fri, 15 Aug 2014 12:33:20 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> <53EE5A4F.5060501@stoneleaf.us> Message-ID: > This feature is very new to Python as a whole, why not give it a syntax that provides a proper separation from what already was? Mainly because angle brackets are one of C++'s worst mistakes . Well, worst among many other equally-worst things, but it's pretty bad. Say goodbye to your simple LL1 parser! On Fri, Aug 15, 2014 at 12:25 PM, Sunjay Varma wrote: > Another such example, since names are just names: > > dict = {"a": 2} > print(dict["a"]) # 2 > > Overwriting a built in type name is bad, but entirely possible. > > dict["a"] here is also confusing with dict[str]. This kind of use could > also potentially throw off a type linter too. > > These probably aren't the best examples out there, but I can definitely > see this operator's meaning becoming very confused as more people start to > apply it in different ways. > > We should not be just using something because it's there. Especially if it > causes other problems. > list[str] may be valid syntax in old Python 3 versions, but it's still not > going to be correct if used in those versions. You're going to get some > breakage no matter what. > > This feature is very new to Python as a whole, why not give it a syntax > that provides a proper separation from what already was? > > Sunjay > On Aug 15, 2014 3:14 PM, "?ukasz Langa" wrote: > >> On Aug 15, 2014, at 12:06 PM, Ethan Furman wrote: >> >> On 08/15/2014 11:56 AM, Ryan Hiebert wrote: >> >> >> Getting an item from a class has no meaning for any classes that I?ve >> ever used, and I haven?t come up with any hypothetical one that would want >> to do that. >> >> >> --> class Foo(Enum): >> ... spam = 'meat flavored' >> ... eggs = 'chicken by-product' >> ... >> --> Foo >> >> >> --> Foo['spam'] >> >> >> >> I also thought of enums. Looks fairly innocent to me, though. Do you see >> any cases where the two would conflict? >> >> -- >> Best regards, >> ?ukasz Langa >> >> WWW: http://lukasz.langa.pl/ >> Twitter: @llanga >> IRC: ambv on #python-dev >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Aug 15 21:46:27 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 15 Aug 2014 12:46:27 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Fri, Aug 15, 2014 at 12:24 PM, ?ukasz Langa wrote: > On Aug 15, 2014, at 8:17 AM, Guido van Rossum wrote: > > You can probably come up with a notation for first-class variable > annotations, e.g. > > x: Sequence[int] = [] > > Yes, that syntax is out of scope for now, though, right? If I understand > your reasoning behind choosing Mypy?s function annotation syntax, we don?t > want to create programs that require Python 3.5+ just to be parsed. > That's stronger than I meant it. The List proposal would completely prevent the new typing syntax from being backported. Having to refrain from adding types for variables (or being required to use the inferior "magic comment" syntax) is a much smaller burden. > If we were to introduce first-class variable typing, yes, the syntax you > propose is what I also had in mind. > It might be a separate PEP. > The value might be optional. The question is though, would the type > (Sequence[int]) be stored anyway? Also, in a class body, does it define a > class var or an instance var (or doesn't it matter?). > > > I wouldn?t change the current behaviour: > > class C: > cls_member: str = ?on the class? > > def __init__(self): > self.obj_member: str = ?on the instance' > self.cls_member = 2 # that?s the real question: type error or an > instance member? > > For that last case, even though it?s currently valid Python, my intuition > tells me for Mypy to treat it as an error. > I disagree -- it's a very common idiom to set (immutable) default values on the class for what is meant to be an instance variable. This is why I called it out as a question we need to answer. > Does this need a 'var' keyword to be ambiguous? > > I fail to see any additional value provided by such keyword. What would > stop people from doing > > var i = 1 > > I don?t think we want to end up with that. > In a different world it could be used to address the issue of typos going unnoticed, but I think it would be too big a departure from current PYthon practice. > I propose to disallow declaring multiple variables in this style, since > it's hard to decide whether the comma should bind tighter than the '=' sign > (as in assignments) or less tight (as in function headings). > > > Right. I wonder if we even need this. For lines that use multiple > assignment just for brevity, they can switch to multiple lines for typing. > For common cases like: > > host, port = origin.rsplit(?:?, 1) > successful, errors = query_the_world(hostnames) > > I think the types can be easily inferred (assuming rsplit and > query_the_world are annotated). > Sure. It's just that people would be expecting it to work based on generalizations from other forms -- it's just that if you generalize from argument lists you end up with something different than when you generalize from assignment. I think it's reasonable to disallow a, b: Tuple[int, float] = 42, 3.14 but to allow (a, b): Tuple[int, float] = (42, 3.14) > While we?re at slaying dragons, I?ll also silently make str non-iterable >> so that we can use Sequence[str] meaningfully from now on? How about that? >> > > I hope you meant that as a joke. We missed our chance for that one with > Python 3.0. We must live with it. > > > Yes, that was obviously just a joke. By the way, is the PEP number 4000 > free? Asking for a friend. > I totally missed the joke. :-( -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Aug 15 23:18:11 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 15 Aug 2014 17:18:11 -0400 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On 8/15/2014 5:48 AM, Nick Coghlan wrote: > On 15 August 2014 19:38, Terry Reedy wrote: >> On 8/15/2014 12:40 AM, Nick Coghlan wrote: >>> >>> On 15 August 2014 09:56, Guido van Rossum wrote: >>>> >>>> >>>> I don't buy the argument that PEP 3107 promises that annotations are >>>> completely free of inherent semantics. >>> >>> >>> It's also worth noting the corresponding bullet point in PEP 3100 >>> (under http://www.python.org/dev/peps/pep-3100/#core-language): >>> >>> * Add optional declarations for static typing [45] [10] [done] >> >> ... >> >>> Linters/checkers may also want to provide a configurable way to say >>> "the presence of decorator means the annotations on that function >>> aren't type markers". That ties in with the recommendation we added to >>> PEP 8 a while back: "It is recommended that third party experiments >>> with annotations use an associated decorator to indicate how the >>> annotation should be interpreted." I claim that the mere presence of a decorator in the *source* is not enough. The decorator for non-type markers should do something with .__annotations__ -- in particular, remove the non-type markers. The presence of a decorator in the source does not necessarily leave a trace on the returned function object for runtime detection. >> Depending on the checker, this suggests that non-type-check annotations need >> not be deprecated. If a decorator wraps a function with an unannotated >> wrapper, then the checker should see the result as unannotated, rather than >> looking for a wrapped attribute. Also, a decorator can remove non-type >> annotations and act on them, store them in a closure variable, or store them >> on the function in a different name. "Depending on the checker" alludes to the fact that there are two types of annotation consumers: those that read the source or parsed source (ast) and the annotations therein and those that look at the function .__annotations__ attribute after compilation. Inspect.signature is an example of the latter. (If there were none, there would be no purpose to .__annotations__!) Suppose an integer square root function were annotated with with type and non-type information. >>> def nsqrt(n:(int, 'random non-type info'))->(int, 'more non-type info'): pass To me, inspect.signature already assumes that annotations are about type, which means that this horse has already left barn. In 3.4.1: >>> from inspect import signature as sig >>> str(sig(nsqrt)) "(n:(, 'random non-type info')) -> (, 'more non-type info')" Typing 'nsqrt(' in Idle and pausing a fraction of a second brings up a calltip with the same string (without the outer quotes). To me, having random non-type info in the signature and calltip is noise and therefore wrong. So I agree that the non-standard annotation should be signaled by a decorator *and* suggest that the decorator should remove the non-standard annotation, which is easily done, so that the signature string for the above would be "(n: " I summarized the above, perhaps not the best I could have, with "Given these possibilities, all that is needs be said is "After a function is post-processed by decorators, any remaining annotations should be for type-checking or documentation." > No, many (most?) The future relative proportion of pre- and post-compile annotation consumers is not relevant to my argument. The stdlib already has a important post-compile consumer that is already somewhat broken by non-type info remaining in .__annotations__. > linters and IDEs will run off the AST without > actually executing the code, so they'll see the annotations, even if > they get stripped by the decorator at runtime. Being aware of this, I concluded the post with the following that already said this. "For checkers that do look at the source, or the AST before compiling," *and* I went on to suggest a solution. "the rule could be to ignore string annotations. Decorators can always eval, or perhaps safe_eval, strings." In other words, if type annotations were to be classes, as proposed, then non-type annotations should not be classes, so that pre-compile annotation consumers could easily ignore them. In particular, I suggested literal strings, which are easily recognized in source, as well as in asts. To put this all another way -- The new-in-3.0 annotation feature has two components: a python function source syntax, and a function compile behavior of adding a dict attribute as .__annotations__. If I understand correctly, Argument Clinic piggybacks on this by adding a mechanism to produce .__annotations__ from C source -- mainly for use by .signature, but also by another other .__annotations__ users. The two components -- source annotations and .__annotations__ dict -- each have their consumers. Currently, annotation values are untyped (or AnyTyped). Guido has proposed favoring type annotations. I support that and suggest that such favoritism is already needed for the annotation dict. So I would strengthen the PEP 8 recommendation. Guido has also suggested 'deprecating' non-type annotations. That would literally mean raising an error either when source is compiled or when def statements are executed. I think deprecation in this sense is both unwise, since non-type annotation were explicitly invited, and unnecessary for the purpose of favoring type annotations. The point of my previous post was to explore what restrictions *are* necessary. In summary, I suggest 1. use distinct syntax (this depends on what is adopted for type annotations); 2. decorate (as already suggested in PEP 8); 3. clean .__annotations__ (which should also be suggested in PEP 8). -- Terry Jan Reedy From barry at python.org Sat Aug 16 00:17:37 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 15 Aug 2014 18:17:37 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: Message-ID: <20140815181737.1e1279b4@anarchist.localdomain> On Aug 13, 2014, at 10:11 PM, Guido van Rossum wrote: >We could refrain from using type annotations in the stdlib (similar to how >we refrain from using Unicode identifiers). Mypy's stubs mechanism makes it >possible to ship the type declarations for stdlib modules with mypy instead >of baking them into the stdlib. That would address my visceral reaction to the proposal. I haven't had time to read the entire thread, or supporting material, but my main concern is about readability. I don't think it's clear whether the impact of adding type annotations is a pure win or loss. OT1H maybe it helps because you would have to read less of the body of the code to guess what the implicit types are for the arguments and return values of a function. I myself often have to "annotate" (i.e. add comments) around code such as: # Map UUIDs to user object. users_by_uid = {} Admittedly, it's often quite difficult when reading foreign code to understand what are the types of the various constructs involved. It can involve reading more deeply into the call structure, or as I usually find, writing a bit of exploratory code and then just stepping through with the debugger. So type annotations could be a win here. OTOH there *is* a cost that could negatively impact readability. It's more text visually assaulting you. :) It can reduce the speed at which you can scan code. I'd say most of the time, you can pretty much deduce what's happening fairly easily (with molasses-inducing outliers like above) so type annotations may make things more confusing, especially if you have to mentally map between annotation types and run time types that are similar but not quite the same. If even more complex types are used, then you have to go digging into other material (docs or code) to understand what the author meant by some of the annotations. Type annotations likely increase line length or vertical real-estate, both of which can negatively impact readability. And if annotations are buggy, then it imposes even more cognitive dissonance. In practice, who knows whether wholesale adoption of type annotation in the stdlib *code* would be a net win or loss? I don't think it's at all clear. The other concern I'd have is whether annotating new stdlib code would become mandatory, and whether there'd be a wholesale campaign to annotate existing code. Using stubs and not baking them into the stdlib alleviates those concerns too. My gut reaction is more negative than positive but I remain open minded until I've had time to read more. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From barry at python.org Sat Aug 16 00:33:00 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 15 Aug 2014 18:33:00 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: Message-ID: <20140815183300.04a3ad33@anarchist.localdomain> On Aug 14, 2014, at 12:01 PM, Sunjay Varma wrote: >Here's a taste of what that looks like: > class SimpleEquation(object): > > def demo(self, a, b, c): > """ > This function returns the product of a, b and c > @type self: SimpleEquation > :param a: int - The first number > :param b: int > :param c: int - The third number should not be zero and should >also > only be -1 if you enjoy carrots (this comment spans 2 lines) > :return: int > """ > return a * b * c I use this all the time. I think I originally adopted this from epydoc: http://epydoc.sourceforge.net/manual-fields.html (i.e. the reStructuredText flavor of epydoc fields.) E.g. def inject_message(mlist, msg, recipients=None, switchboard=None, **kws): """Inject a message into a queue. If the message does not have a Message-ID header, one is added. An X-Message-Id-Hash header is also always added. :param mlist: The mailing list this message is destined for. :type mlist: IMailingList :param msg: The Message object to inject. :type msg: a Message object :param recipients: Optional set of recipients to put into the message's metadata. :type recipients: sequence of strings :param switchboard: Optional name of switchboard to inject this message into. If not given, the 'in' switchboard is used. :type switchboard: string :param kws: Additional values for the message metadata. :type kws: dictionary """ With perhaps a little more formalism (or care on the part of the author ), e.g. a better spelling of "sequence of strings", this format seems much more readable to me. And being tucked away in a docstring, it can really be safely ignored. It also seems like a processor like mypy could use this information just as easily as type annotations. For whatever reason, this style seems much more comfortable to me than trying to encode everything in the function signature. it also has the added benefit of actually describing the purpose and detail of the arguments to a human reader rather than leaving it up to a mental mapping to translate annotated types to semantics or content detail. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From barry at python.org Sat Aug 16 00:40:44 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 15 Aug 2014 18:40:44 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: <20140814181554.GR4525@ando> Message-ID: <20140815184044.2aaeca8e@anarchist.localdomain> On Aug 15, 2014, at 04:15 AM, Steven D'Aprano wrote: >> Here's a taste of what that looks like: >> class SimpleEquation(object): >> >> def demo(self, a, b, c): >> """ >> This function returns the product of a, b and c >> @type self: SimpleEquation >> :param a: int - The first number >> :param b: int >> :param c: int - The third number should not be zero and should >> also >> only be -1 if you enjoy carrots (this comment spans 2 lines) >> :return: int >> """ >> return a * b * c > >I really dislike that syntax. I dislike adding cruft like "@type" and >":param" into docstrings, which should be written for human readers, not >linters. For me, the whole point of using syntax like this is for the human reader, especially because I don't have a linter that parses this... yet. Used judiciously, this syntax could benefit both the human reader *and* automated tools. >I dislike that you have documented that self is a SimpleEquation. (What else >could it be?) FWIW, I never document 'self'. >I dislike that the syntax clashes with ReST syntax. It needn't. http://epydoc.sourceforge.net/manual-fields.html >I dislike that it isn't obvious to me why the first parameter uses @type >while the second parameter uses :param. It needn't. >I like the annotation syntax. I'm not completely convinced that the mypy >syntax is mature enough to bless, but the basic idea of type annotations is >pretty common in dozens of languages. I think you are in a tiny minority if >you think that putting the type declaration right next to the parameter make >it *less* clear that putting the type declaration in a completely different >part of the code. Of course, it's not a completely different part of the code. docstrings naturally live right after the function signature (indeed, or it wouldn't get stuffed into __doc__), so it's always close to the source. That makes it quite easy for the third party human reader, but also for the author to keep up-to-date. Plus, you can *always* fit these reST-ish epydoc field descriptions in PEP 8 line lengths. It always bugs me to see multiline function signatures. Currently those are pretty rare (and IMHO are a code smell), but with type annotations, I suspect they'll be the norm. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From barry at python.org Sat Aug 16 00:49:09 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 15 Aug 2014 18:49:09 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us> Message-ID: <20140815184909.61c4ee53@anarchist.localdomain> On Aug 14, 2014, at 11:33 AM, Ethan Furman wrote: >> def frobinate(x, y): >> """Return the frobinated x and y. >> >> Some more text goes here. Perhaps lots of text. >> >> :param x: Spam >> :param y: Eggs >> :return: Breakfast >> """ > >Sure, keeping that info in the annotations makes more sense, but I'd rather >see it in the doc string instead of ruling out all other possible uses of >annotations -- particularly for something that's supposed to be /optional/. Docstring annotations almost by definition can contain more information useful to the human reader than type annotations can, especially if you carefully use the reST-ish epydoc convention of both :param: and :type:. The latter contains the type annotation (which an automated system could utilize) while the former contains the exposition (for the benefit of the human reader). It's the explanations that are missing from any type annotations. I suppose you could intersperse comments with your type annotations, resulting in a long multiline function signature. I doubt that would be a readability improvement. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From greg.ewing at canterbury.ac.nz Sat Aug 16 00:51:38 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 16 Aug 2014 10:51:38 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EE8EFA.7040005@canterbury.ac.nz> ?ukasz Langa wrote: > I?m also not a fan of the square brackets for generics (those brackets > mean lookup!) but a BDFL once said that ?language evolution is the art > of compromise? and one cannot disagree with that. I think there's sufficient precedent in other languages to justify using square brackets that way. I much prefer them over the angle brackets of C++ and Java, which are ugly to my eyes and make the code hard to read. They also make parsing tricky if you have "<<" and ">>" operators. -- Greg From kaiser.yann at gmail.com Sat Aug 16 00:44:27 2014 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Sat, 16 Aug 2014 00:44:27 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On 15 August 2014 23:18, Terry Reedy wrote: > > > I claim that the mere presence of a decorator in the *source* is not > enough. The decorator for non-type markers should do something with > .__annotations__ -- in particular, remove the non-type markers. The > presence of a decorator in the source does not necessarily leave a trace on > the returned function object for runtime detection. > > The way I understand it, mypy, which is what Guido's proposal sees its main potential user, operates at a stage similar to compilation in CPython. At which point anything after the colon in a parameter would be in its equivalent of __annotation__, regardless of what decorators add or remove. How would your other-annotation-removing decorator help mypy at all? By having it examine the decorator source and infer what kind of operation the decorator does? That doesn't sound very compile-stagey at all. > > To me, inspect.signature already assumes that annotations are about type, > which means that this horse has already left barn. In 3.4.1: > > >>> from inspect import signature as sig > >>> str(sig(nsqrt)) > "(n:(, 'random non-type info')) -> (, 'more > non-type info')" > > `inspect.signature` makes no such assumption, it only relays what it found on the function's __annotation__ attribute. I don't know where this nsqrt function comes from, but it is responsible for having set up those annotations, which seem to be mere documentation if the "random non-type info" is a string. > "the rule could be to ignore string annotations. Decorators can always > eval, or perhaps safe_eval, strings." > > In other words, if type annotations were to be classes, as proposed, then > non-type annotations should not be classes, so that pre-compile annotation > consumers could easily ignore them. In particular, I suggested literal > strings, which are easily recognized in source, as well as in asts. > > "Oh, everything that's not type-checking can be expressed in a string, maybe to be eval'ed, say, with sys,_getframe(-1) locals." No, and that's incredibly ugly and may not work on alternate implementations of Python. If a PEP is made to standardize typing attributes for annotations, perhaps in the form of a typing type hierarchy or a type decorator(ie. typing.Like(AClass)), or annotation namespacing of some sort(decorate the function with "hey! please typecheck me!"?), then couldn't tools that rely on those attributes pick out what's relevant to them on their own, thanks to the standardization? I find the whole idea of having so far equal uses of function annotations be bullied aside for the good of a concept yet foreign to Python very arrogant and unnecessary. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Aug 16 01:46:55 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 15 Aug 2014 16:46:55 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140815184909.61c4ee53@anarchist.localdomain> References: <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us> <20140815184909.61c4ee53@anarchist.localdomain> Message-ID: <53EE9BEF.8040001@stoneleaf.us> On 08/15/2014 03:49 PM, Barry Warsaw wrote: > On Aug 14, 2014, at 11:33 AM, Ethan Furman wrote: > >>> def frobinate(x, y): >>> """Return the frobinated x and y. >>> >>> Some more text goes here. Perhaps lots of text. >>> >>> :param x: Spam >>> :param y: Eggs >>> :return: Breakfast >>> """ >> >> Sure, keeping that info in the annotations makes more sense, but I'd rather >> see it in the doc string instead of ruling out all other possible uses of >> annotations -- particularly for something that's supposed to be /optional/. > > Docstring annotations almost by definition can contain more information useful > to the human reader than type annotations can, especially if you carefully use > the reST-ish epydoc convention of both :param: and :type:. The latter > contains the type annotation (which an automated system could utilize) while > the former contains the exposition (for the benefit of the human reader). > It's the explanations that are missing from any type annotations. > > I suppose you could intersperse comments with your type annotations, resulting > in a long multiline function signature. I doubt that would be a readability > improvement. Sounds like what we *really* need is a decorator that will parse the docstring and fill in the annotations automatically. :) -- ~Ethan~ From greg.ewing at canterbury.ac.nz Sat Aug 16 01:57:48 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 16 Aug 2014 11:57:48 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EE9E7C.6000004@canterbury.ac.nz> Petr Viktorin wrote: > Instead of > > def optional(t): > return t | None > > it's now: > > def optional(t): > if t is None: > return t > else: > return t | None I don't think so. Code that's manipulating types shouldn't be using None as a stand-in for NoneType in the first place. Think of it this way: None is *not* a type, just a special-case value of x in " | x". So optional(None) is an error, just as much as optional(42) would be. -- Greg From greg.ewing at canterbury.ac.nz Sat Aug 16 02:10:54 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 16 Aug 2014 12:10:54 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EEA18E.3050600@canterbury.ac.nz> Sunjay Varma wrote: > dict seems to tell me more explicitly that I'm dealing with a > declaration of an expected type. In addition to my earlier objections to angle brackets, there would be a big problem with parsing this notation in Python. In languages that use syntax like this, there is a clear division between type descriptions and expressions -- they belong to completely separate areas of the grammar. However, we need to be able to parse our type descriptions as expressions, because they *are* expressions, just like any other. Now consider: dict ^^^^^^^^^^^^^ Everything up to the final '>' is a valid Python expression, equivalent to ((dict < str), int) so the parser would have to be capable of completely changing its mind about how to interpret all of that when it saw the '>'. I'm fairly sure that Python's mostly-LL parser can't do that. -- Greg From greg.ewing at canterbury.ac.nz Sat Aug 16 02:14:36 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 16 Aug 2014 12:14:36 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EEA26C.3040907@canterbury.ac.nz> Sunjay Varma wrote: > dict[str, int] looks very much like I'm > getting an item (str, int) from some class. If you consider that 'dict' on its own represents a set of possible types, then it's not unreasonable that 'dict[str, int]' selects one of the types from that set. In other words, dict is a lazy collection of types. -- Greg From greg.ewing at canterbury.ac.nz Sat Aug 16 02:20:11 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 16 Aug 2014 12:20:11 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com> <53EE5A4F.5060501@stoneleaf.us> Message-ID: <53EEA3BB.5030304@canterbury.ac.nz> Ryan Hiebert wrote: > I don?t think it would conflict in this case, since I don?t think there?d be > a reason to mix Enum item access with Type item access. It would conflict if you somehow needed to define an Enum subclass with type parameters. I'm having trouble thinking of a use case for such a thing, though. -- Greg From greg.ewing at canterbury.ac.nz Sat Aug 16 02:26:09 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 16 Aug 2014 12:26:09 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EEA521.7020406@canterbury.ac.nz> ?ukasz Langa wrote: > class C: > cls_member: str = ?on the class? > > def __init__(self): > self.obj_member: str = ?on the instance' > self.cls_member = 2 # that?s the real question: type error or an > instance member? I think not treating it as an error would make it hard to reason about the type of x.cls_member for an instance x of C. Its type would depend on whether del x.cls_member had been performed on x. Code which relied on them being different types would be rather confusing to a human reader too, so it's probably fine to discourage that. -- Greg From alexander.belopolsky at gmail.com Sat Aug 16 04:07:45 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 15 Aug 2014 22:07:45 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EE9BEF.8040001@stoneleaf.us> References: <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us> <20140815184909.61c4ee53@anarchist.localdomain> <53EE9BEF.8040001@stoneleaf.us> Message-ID: On Fri, Aug 15, 2014 at 7:46 PM, Ethan Furman wrote: > Sounds like what we *really* need is a decorator that will parse the > docstring and fill in the annotations automatically. :) While it may be contrary to the TOOWTDI principle, but I think decorator- and annotations-based type specifications can co-exist. I have been using the following decorator for about a year def returns(type): """emulate -> attribute setting in python 2""" def decorator(func): func.__dict__.setdefault('__attributes__', {})['return'] = type return func return decorator and I often find code that uses for example @returns(int) more readable than the code that uses .. -> int: The advantage of argument annotations over any decorator-based solution is avoidance of repetition, but I often miss K&R-style declarations in C. Probably because int copy(from, to) char *from; char *to; {..} looks more "pythonic" than int copy(char *from, char *to) {} In python, what is more readable: def copy(from:Sequence, to:MutableSequence): .. or @arg_types(from=Sequence, to=MutableSequence) def copy(from, to): .. or def copy(from, to): .. set_arg_types(copy, from=Sequence, to=MutableSequence) ? I believe that having type specifications out of the way in the last variant more than outweighs the need to repeat the names of arguments. Even the repetition can be avoided if we do something like @arg_types(Sequence, MutableSequence) def copy(from, to): .. but this is still more intrusive than the after-body variant where function name repetition is unavoidable and argument names repetition improves readability. I can imagine cases where having type specifications follow each argument is helpful. For example def configure(d:Drawing, c:Configuration): d.background = c.get_parameter('background') d.height = c.get_parameter('height') .. It really depends on the problem at hand and on the coding style whether you may want type specification before or within the function declaration or out of the way in docstring, after the function body or in a separate "stubs" file altogether. If Python gets an official standard for specifying types in function attributes, it should not be hard to standardize docstring and decorator alternatives as well. As a bonus, these alternatives will be immediately available to 2.x users. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat Aug 16 05:07:29 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 16 Aug 2014 12:07:29 +0900 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140815181737.1e1279b4@anarchist.localdomain> References: <20140815181737.1e1279b4@anarchist.localdomain> Message-ID: <87egwhgmj2.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > OTOH there *is* a cost that could negatively impact readability. > It's more text visually assaulting you. :) Not if you have a syntax-highlighting editor that has a no-see-um face or text property. From wes.turner at gmail.com Sat Aug 16 05:16:30 2014 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 15 Aug 2014 22:16:30 -0500 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <53EBC3BA.10808@stoneleaf.us> <53ED0DCB.2000306@googlemail.com> Message-ID: > Couldn't you do that today in Python with a suitably sophisticated > function decorator? The range/type checking would happen before the > user's actual function is called. PyContracts does this; with (1) an @contract decorator, (2) Python 3 annotations, (3) with :type: and :rtype: docstrings ... http://www.reddit.com/r/Python/comments/2dh036/pythonideas_proposal_use_mypy_syntax_for_function/#cjq09bu : > +1 for static typing for certain problems. > [...] > Static type checking at compile time (linting) looks really neat. > [...] > Do we need a separate approach for actual type assertions > at runtime? Will that ever be in scope for mypy? From ncoghlan at gmail.com Sat Aug 16 06:07:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Aug 2014 14:07:12 +1000 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: <967197E8-94BD-45C3-BCD8-BFD0690EFF12@langa.pl> References: <967197E8-94BD-45C3-BCD8-BFD0690EFF12@langa.pl> Message-ID: On 16 August 2014 04:38, ?ukasz Langa wrote: > Ah, that?s right, I meant singledispatch. Sorry for the confusion! :) I think "singledispatch" is still potentially relevant to Matthew's idea of a library specifically devoted to dispatch and signature matching. My rationale is that "functools.singledispatch" actually has some pretty neat funcationality behind it. I'm talking things like functools._c3_merge, functools._compose_mro, functools._find_impl - all the machinery that is needed to take the complexity of ABC registration and turn it into something that can be checked quickly at runtime. Even the trick with using abc.get_cache_token() to invalidate the dispatch caches whenever the object graph changes via explicit ABC registration is worth illuminating further. Decoupling the underlying dispatch machinery from the specific functools use case also opens up additional possibilities for type based dispatch. The way Julia uses multiple dispatch for binary operators is interesting (http://julia.readthedocs.org/en/latest/manual/methods/), although not directly applicable to Python's binary operators, since we use the "return NotImplemented" dance to decide which implementation to use. That's where I can see a possible fit for something like multiple dispatch support in the standard library: making it easier to write binary operator overloads correctly. A *lot* of folks make the mistake of raising TypeError or NotImplementedError directly in their operator overload implementations, rather than returning the NotImplemented singleton that tells the interpreter to try the other type. Even some of the CPython builtins get that wrong, since the sq_concat and sq_repeat slots in C don't properly support the type coercion dance, so you *have* to raise the exception yourself if you're only implementing those without implementing nb_add and nb_mul (types defined in Python automatically populate both sets of C level slots if you define __add__ or __mul__). Dealing with the "NotImplemented dance" properly is also why functools.total_ordering got substantially slower in Python 3.4 - it isn't at risk of blowing up with RecursionError in some cases any more, but it paid a hefty price in speed to get there. Notation wise, I strongly encourage going with the format defined in PEP 443: a "default implementation" that defines the namespace all the other implementations will hook into, along with explicit registration of additional overloads. If there's no sensible default implementation, it can be written to raise an appropriate exception (a dispatch library could even help with raising an appropriately formatted type error). For example, here's how a PEP 443 inspired notation would handle the task of defining a stricter version of sequence repetition than the default "*" binary operator: @functools.multidispatch(2) # Multiple dispatch on the first 2 positional arguments def repeat(lhs, rhs): raise BinaryDispatchError('repeat', lhs, rhs) # See below for possible definition @example.register(numbers.Integral, collections.abc.Sequence): @example.register(collections.abc.Sequence, numbers.Integral): def repeat(lhs, rhs): return lhs * rhs # Assume this useful helper class is defined somewhere... class BinaryDispatchError(TypeError): def __init__(self, fname, lhs, rhs): self.fname = fname self.lhs_kind = lhs_kind = lhs.__class__.__name__ self.rhs_kind = rhs_kind = rhs.__class__.__name__ msg = "Unsupported operand type(s) for {!r}: {!r} and {!r}".format(fname, lhs_kind, rhs_kind) super().__init__(msg) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From antoine at python.org Sat Aug 16 06:24:19 2014 From: antoine at python.org (Antoine Pitrou) Date: Sat, 16 Aug 2014 00:24:19 -0400 Subject: [Python-ideas] generic code and dependent types In-Reply-To: References: Message-ID: Le 15/08/2014 06:56, Neal Becker a ?crit : > I'm interested in the proposals for adding type annotation. Coming from some > experience with generic code in c++, one of the difficult issues has been > reasoning about dependent types in generic code. > > In general, we need to be able to declare a type via an arbitrary metafunction > > some_metafunction::type F (... What is a metafunction? From steve at pearwood.info Sat Aug 16 07:04:33 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 16 Aug 2014 15:04:33 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140815184044.2aaeca8e@anarchist.localdomain> References: <20140814181554.GR4525@ando> <20140815184044.2aaeca8e@anarchist.localdomain> Message-ID: <20140816050431.GE4525@ando> Some people like the epydoc-style convention of putting type annotations in docstrings: [...] > >> def demo(self, a, b, c): > >> """ > >> This function returns the product of a, b and c > >> @type self: SimpleEquation > >> :param a: int - The first number > >> :param b: int > >> :param c: int - The third number should not be zero and should > >> also > >> only be -1 if you enjoy carrots (this comment spans 2 lines) > >> :return: int > >> """ One issue I haven't see raised is that annotations are available at runtime, whereas docstrings may not be. (The -OO switch removes docstrings.) A linter may be able to parse the docstrings at compile time before the docstrings are discarded, or it may not, but using docstrings means the information is not always available for introspection at runtime. I think that's a major disadvantage. Although I admit I don't always remember to test my code using -O and -OO, I do try very hard to do this and I have found bugs in my code from doing so. I think anything which makes testing -O and -OO modes harder is a bad thing. [Quoting Barry Warsaw] > docstrings > naturally live right after the function signature (indeed, or it wouldn't get > stuffed into __doc__), so it's always close to the source. That makes it > quite easy for the third party human reader, but also for the author to keep > up-to-date. *Close to the source* is not the same as *part of the source*. In the example above, the difference is as high as eight lines, compared to zero: # Function annotations def demo(self, a:int, b:int, c:int)->int: Using docstring annotations splits the information about parameters into two places. Those two places might be close, but there are still two sources of ultimate truth instead of one. You have the name of the parameter in the parameter list, and the type of the parameter inside the docstring separated by some arbitrary number of lines of code. -- Steven From steve at pearwood.info Sat Aug 16 07:16:40 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 16 Aug 2014 15:16:40 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140815184909.61c4ee53@anarchist.localdomain> References: <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us> <20140815184909.61c4ee53@anarchist.localdomain> Message-ID: <20140816051640.GF4525@ando> On Fri, Aug 15, 2014 at 06:49:09PM -0400, Barry Warsaw wrote: > Docstring annotations almost by definition can contain more information useful > to the human reader than type annotations can, especially if you carefully use > the reST-ish epydoc convention of both :param: and :type:. The latter > contains the type annotation (which an automated system could utilize) while > the former contains the exposition (for the benefit of the human reader). > It's the explanations that are missing from any type annotations. > > I suppose you could intersperse comments with your type annotations, resulting > in a long multiline function signature. I doubt that would be a readability > improvement. I'm sorry, I missed the part of Guido's proposal where he said that docstrings would cease to be used for documentation :-) I don't think that it is a serious risk that docstrings will disappear, or that people will try to shove usage comments inside the parameter list: def frobnicate( # this is the thing to be frobnicated obj:int, # pass a truthy object to use the blue frob instead of red frob blue:object=False, # an extra helping of spam yes_please_more_spam:list[Spam] ): any more than they already do. (I think I've written a comment inside a parameter list maybe twice in the last decade.) I don't think there's much risk of that changing. A more exciting outcome would be for documentation tools to start using the type annotations directly, without needing the writer to include the type annotation in two places (the parameter list and the docstring). But before the doc tools can do that, there needs to be a standard for annotations. -- Steven From donald at stufft.io Sat Aug 16 07:22:05 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 16 Aug 2014 01:22:05 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140816051640.GF4525@ando> References: <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us> <20140815184909.61c4ee53@anarchist.localdomain> <20140816051640.GF4525@ando> Message-ID: > On Aug 16, 2014, at 1:16 AM, Steven D'Aprano wrote: > > On Fri, Aug 15, 2014 at 06:49:09PM -0400, Barry Warsaw wrote: > >> Docstring annotations almost by definition can contain more information useful >> to the human reader than type annotations can, especially if you carefully use >> the reST-ish epydoc convention of both :param: and :type:. The latter >> contains the type annotation (which an automated system could utilize) while >> the former contains the exposition (for the benefit of the human reader). >> It's the explanations that are missing from any type annotations. >> >> I suppose you could intersperse comments with your type annotations, resulting >> in a long multiline function signature. I doubt that would be a readability >> improvement. > > I'm sorry, I missed the part of Guido's proposal where he said that > docstrings would cease to be used for documentation :-) > > I don't think that it is a serious risk that docstrings will disappear, > or that people will try to shove usage comments inside the parameter > list: > > def frobnicate( > # this is the thing to be frobnicated > obj:int, > # pass a truthy object to use the blue frob instead of red frob > blue:object=False, > # an extra helping of spam > yes_please_more_spam:list[Spam] > ): > > any more than they already do. (I think I've written a comment inside a > parameter list maybe twice in the last decade.) I don't think there's > much risk of that changing. > > A more exciting outcome would be for documentation tools to start using > the type annotations directly, without needing the writer to include the > type annotation in two places (the parameter list and the docstring). > But before the doc tools can do that, there needs to be a standard for > annotations. > Couldn?t the documentation tools also just pull that information from the annotations? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Aug 16 07:59:05 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 15 Aug 2014 22:59:05 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140816050431.GE4525@ando> References: <20140814181554.GR4525@ando> <20140815184044.2aaeca8e@anarchist.localdomain> <20140816050431.GE4525@ando> Message-ID: <53EEF329.6080402@stoneleaf.us> On 08/15/2014 10:04 PM, Steven D'Aprano wrote: > > Using docstring annotations splits the information about parameters into > two places. Those two places might be close, but there are still two > sources of ultimate truth instead of one. You have the name of the > parameter in the parameter list, and the type of the parameter inside > the docstring separated by some arbitrary number of lines of code. You say that like it's a bad thing. Not having everything crammed into one spot can be good. Too dense is just as bad as too sparse. -- ~Ethan~ From wichert at wiggy.net Sat Aug 16 08:39:56 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Sat, 16 Aug 2014 08:39:56 +0200 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: > On 15 Aug 2014, at 20:01, Guido van Rossum wrote: > > Please do write about non-toy examples! The entire Zope stack has been build around multiple-dispatch for the last 8 years or so. Of course that uses zope.interface/zope.component, which predate singledispatch by about 10 years as far as I can tell. Wichert. From brakhane at googlemail.com Sat Aug 16 12:09:10 2014 From: brakhane at googlemail.com (Dennis Brakhane) Date: Sat, 16 Aug 2014 12:09:10 +0200 Subject: [Python-ideas] generic code and dependent types In-Reply-To: References: Message-ID: <53EF2DC6.7040703@gmail.com> Am 16.08.2014 06:24, schrieb Antoine Pitrou: > > What is a metafunction? In C++, roughly speaking a function that is evaluated at compile time not runtime, used for template metaprogramming. They accept compile time constants and types as parameters and return constants or types. So as a silly example, if we were to have metafunctions in Python's type system, it would allow me to define a metafunction NestedList, and if I wrote def (x: NestedList[3, int]) it could be evaluated to def(x: List[List[List[int]]]) or IntOrString which would evaluate to Int if the parameter was true, or String otherwise. From steve at pearwood.info Sat Aug 16 12:47:00 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 16 Aug 2014 20:47:00 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <20140816104700.GH4525@ando> On Fri, Aug 15, 2014 at 05:18:11PM -0400, Terry Reedy wrote: [...] > "Depending on the checker" alludes to the fact that there are two types > of annotation consumers: those that read the source or parsed source > (ast) and the annotations therein and those that look at the function > .__annotations__ attribute after compilation. Inspect.signature is an > example of the latter. (If there were none, there would be no purpose to > .__annotations__!) Agreed. > Suppose an integer square root function were annotated with with type > and non-type information. > > >>> def nsqrt(n:(int, 'random non-type info'))->(int, 'more non-type > info'): pass I don't think it is reasonable to expect arbitrary annotation tools to interoperate, unless they are specifically designed to interoperate. If tool A expects annotations to be a certain thing, and tool B expects them to be a different thing, they are going to confuse each other. We need a convention so that tools which expect to work with annotations can identify which annotations are aimed at them. E.g. tool A might decorate the function with a marker that says "A", so that tool B knows to skip those functions. And vice versa. In the absence of any such marker, annotations can be assumed to be standard mypy-style type annotations. (The nature of this marker probably should be standardized. I suggest a key/item in __annotations__, where the key cannot clash with parameter names.) The easiest way to apply that marker is with a decorator: perhaps the typing module could provide a standard decorator that all annotation tools can recognise at compile-time: @register_annotations(A.marker) # for example def function(x:"something understandable by A"): ... # inside module A marker = object() def introspect(func): if magic(func) is marker: # okay to operate on func > To me, inspect.signature already assumes that annotations are about > type, which means that this horse has already left barn. In 3.4.1: I don't see how that follows. Your example demonstrates that inspect treats annotations as arbitrary Python expressions (which is what they are). You annotate the n parameter with the expression (int, "random non-type info"), which is a tuple of two objects. And signature dutifully reports that: > >>> from inspect import signature as sig > >>> str(sig(nsqrt)) > "(n:(, 'random non-type info')) -> (, 'more > non-type info')" > > Typing 'nsqrt(' in Idle and pausing a fraction of a second brings up a > calltip with the same string (without the outer quotes). To me, having > random non-type info in the signature and calltip is noise and therefore > wrong. Then don't put random info in the annotations :-) If Idle has documented that the calltip is *always* type information, then Idle is wrong. Currently, there is no standard interpretation of function annotations, and if Idle documentation says otherwise, it is the documentation which is wrong. In the future, I can see that Idle might want to only display calltips that it knows contain type information, or perhaps show them slightly differently if they are not type annotations. Or perhaps not... see below. > So I agree that the non-standard annotation should be signaled by > a decorator I agree whole-heartedly to this. > *and* suggest that the decorator should remove the > non-standard annotation, which is easily done, so that the signature > string for the above would be But I disagree equally as strongly to this. Other uses of annotations are just as valid as static typing, and may equally want to be available for runtime introspection. All we need is some sort of standardised runtime marker whereby tools can decide whether or not they should use the annotations. > The future relative proportion of pre- and post-compile annotation > consumers is not relevant to my argument. The stdlib already has a > important post-compile consumer that is already somewhat broken by > non-type info remaining in .__annotations__. I think it is only broken if you treat Idle calltips as displaying *types*. If you treat Idle calltips as displaying *annotations* no matter what the nature of those annotations, then it is not broken in the least. You yourself call them *call* tips, not "type tips", so there could be useful information provided other than the types of arguments. Consider two (imaginary) annotations in a graphics library: def move(x: int, y:int): ... def move(x: "distance along the left-right axis", y: "distance along the up-down axis"): ... I think that the second would be far more useful in a library aimed at beginners. [...] > Guido has also suggested 'deprecating' non-type annotations. That would > literally mean raising an error either when source is compiled or when > def statements are executed. Not necessarily. It could mean just documenting that we shouldn't use function annotations for anything other than specifying types. The usual procedure for deprecations is: * for at least one release, tell people not to do this, but don't raise a warning or an exception; * for at least one release, raise a warning but not an exception; * for at least one release, raise an exception "For at least one release" might mean "until Python 5000". > I think deprecation in this sense is both > unwise, since non-type annotation were explicitly invited, and > unnecessary for the purpose of favoring type annotations. I whole-heartedly agree with this part! -- Steven From stefan at bytereef.org Sat Aug 16 13:01:56 2014 From: stefan at bytereef.org (Stefan Krah) Date: Sat, 16 Aug 2014 13:01:56 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53EEF329.6080402@stoneleaf.us> References: <20140814181554.GR4525@ando> <20140815184044.2aaeca8e@anarchist.localdomain> <20140816050431.GE4525@ando> <53EEF329.6080402@stoneleaf.us> Message-ID: <20140816110156.GA15938@sleipnir.bytereef.org> Ethan Furman wrote: > On 08/15/2014 10:04 PM, Steven D'Aprano wrote: > >Using docstring annotations splits the information about parameters into > >two places. Those two places might be close, but there are still two > >sources of ultimate truth instead of one. You have the name of the > >parameter in the parameter list, and the type of the parameter inside > >the docstring separated by some arbitrary number of lines of code. > > You say that like it's a bad thing. Not having everything crammed > into one spot can be good. Too dense is just as bad as too sparse. It *is* a bad thing. ;) Humans can recognize patterns easily, and Steven's example can be understood at a single glance. The information is declarative and the density isn't that high. Density may become a problem if the information is functional and dense, like in APL. Even that can be overcome by training. Stefan Krah From davidhalter88 at gmail.com Sat Aug 16 15:00:15 2014 From: davidhalter88 at gmail.com (Dave Halter) Date: Sat, 16 Aug 2014 15:00:15 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: I'm late, but it's also quite hard to read 200 mails in two days (BlaBlaOverflow) :) I've read most of it. I'm also top-posting. Sorry :) This mail is going to be way shorter this way. *EuroPython* I'm a core-dev of Jedi (the autocompletion library). We were having discussions about using annotations for a longer time, but we couldn't decide on a good approach. This has changed quite drastically at EuroPython were I realized that Jedi is not alone in wanting annotations to become a way of specifying types. Most prominently the some of the pylint guys and Andrey of PyCharm seemed interested to standardize type annotations. This would have led me to eventually write a mail here on Python ideas (probably together with those mentioned), describing our needs. Furthermore there were other interested parties like Cython and Numba devs, because they might eventually use annotations to improve their type knowledge. Conclusion of the conference is that Python's static analysis community wants type annotations. We want them standardized to be actually able to use them. A third party library would be useful only if it had the same opportunities like tulip: Being a third party library with the clear goal of inclusion in the standard library. *The current proposal* While I don't fully support mypy's type system, I think it's a step into the right direction, but I don't think we need it now (BTW: Mypy's ``Protocol`` class is genius and should be adapted into the stdlib anyway). In the Appendix below, I've described what I came up with. I think my solution inferior in capabilities, but way easier to understand and without clutter. Something like Mypy's `typing` should probably be adopted and standardized in a separate PEP, possibly for 3.6, once we have a few first experiences. (Maybe if we start early it could still make 3.5). I think a resulting PEP of this discussion should contain a deprecation note for all usages other than type checking in Python 3.5. The current ambiguous nature of annotations is the fact why no static analysis tool ever checked them. One thing you have to remember at this point, *that the two biggest issues for static analysis are builtins and functions that are never called. Function annotations would solve both partially.* *Argument Clinic* One notable side effect of Guido's proposal would be that Argument Clinic could use annotations in its signatures. This would be a big benefit for static analysis. because it would finally reveal the types of builtin functions (input and output!). I tried to convince Larry Hastings to implement that, but he refused, citing PEP 3107 which states that "This work will be left to third-party libraries.". Argument Clinic in combination with type annotations would be a huge win for the static analysis community. *Issues* By far my biggest concern is the fact that type annotations in CPython don't have any effect on run-time or compile-time. This is really an issue, because people from other languages will actually think that this is a type checker. This could be fixed partially by checking annotations at run-time (and raising warnings [or exceptions with a command line switch]). This way annotations wouldn't be without semantic meaning, they would actually be some kind of pre/post conditions. However, this should also not be the goal of a current PEP. I just wanted to mention it, so that we can keep it in mind. *Appendix (My Proposal)* My proposal (as discussed and evolved with a few good people at EuroPython) for containers would look something like this: def foo(index: int, x: [float], y: {int: str}) -> (float, str): return x[index], y[index] The "default" containers (set, list, dict and tuple) would just serve as a way of specifying containers. This makes a lot of things less complicated: - It's easier to understand (everybody knows builtin types). It also feels natural to me. The example above covers also almost all variations. A tuple could be expanded by using the ellipsis: `(int, object, ..., float)`. - No imports needed. People are more likely to use it, if they don't have to import typing all the time. This is important for static analysis, people are only likely to use it if it's easier than writing docstrings with type information. - Argument clinic could use the same. The standard library quite often doesn't accept abstract data types, btw. - It's what people sometimes use in docstrings: ``@rtype: (float, str)`` or ``:rtype: (float, str)``. I know this doesn't solve the duck typing issue, but if you look at real-life Python code bases, there are very few instances of actually implementing a ``Mapping``, etc. For all of that we should write a separate proposal (with abstract type classes). My proposal is also missing a Union type and a few other things, but I'd rather start type annotations really simple and soon than to wait for good libraries to emerge. I think the typing module (without my additions) would be counterproductive, because it's just too complicated too understand. Most people that are using Python don't know ABCs and would have a hard time dealing with such optional typing. In the end most people would just ignore it entirely. Godspeed! ~ Dave 2014-08-15 1:56 GMT+02:00 Guido van Rossum : > I have read pretty much the entire thread up and down, and I don't think I > can keep up with responding to every individual piece of feedback. (Also, a > lot of responses cancel each other out. :-) > > I think there are three broad categories of questions to think about next. > > (A) Do we even need this? > > (B) What syntax to use? > > (C) Does/should it support ? > > Taking these in turn: > > (A) Do we even need a standard for optional static typing? > > Many people have shown either support for the idea, or pointed to some > other system that addresses the same issue. On the other hand, several > people have claimed that they don't need it, or that they worry it will > make Python less useful for them. (However, many of the detractors seem to > have their own alternative proposal. :-) > > In the end I don't think we can ever know for sure -- but my intuition > tells me that as long as we keep it optional, there is a real demand. In > any case, if we don't start building something we'll never know whether > it'll be useful, so I am going to take a leap of faith and continue to > promote this idea. > > I am going to make one additional assumption: the main use cases will be > linting, IDEs, and doc generation. These all have one thing in common: it > should be possible to run a program even though it fails to type check. > Also, adding types to a program should not hinder its performance (nor will > it help :-). > > (B) What syntax should a standard system for optional static typing use? > > There are many interesting questions here, but at the highest level there > are a few choices that constrain the rest of the discussion, and I'd like > to start with these. I see three or four "families" of approaches, and I > think the first order is to pick a family. > > (1) The mypy family. (http://mypy-lang.org/) This is characterized by its > use of PEP 3107 function annotations and the constraint that its syntax > must be valid (current) Python syntax that can be evaluated without errors > at function definition time. However, mypy also supports collecting > annotations in separate "stub" files; this is how it handles annotations > for the stdlib and C extensions. When mypy annotations occur inline (not in > a stub file) they are used to type check the body of the annotated function > as well as input for type checking its callers. > > (2) The pytypedecl family. (https://github.com/google/pytypedecl) This is > a custom syntax that can only be used in separate stub files. Because it is > not constrained by Python's current syntax, its syntax is slightly more > elegant than mypy. > > (3) The PyCharm family. ( > http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html) > This is a custom syntax that lives entirely in docstrings. There is also a > way to use stub files with this. (In fact, every viable approach has to > support some form of stub files, if only to describe signatures for C > extensions.) > > (I suppose we could add a 4th family that puts everything in comments, but > I don't think anyone is seriously working on such a thing, and I don't see > any benefits.) > > There's also a variant of (1) that ?ukasz Langa would like to see -- use > the syntactic position of function annotations but using a custom syntax > (e.g. one similar to the pytypedecl syntax) that isn't evaluated at > function-definition time. This would have to use "from __future__ import > " for backward compatibility. I'm skeptical about this though; > it is only slightly more elegant than mypy, and it would open the > floodgates of unconstrained language design. > > So how to choose? I've read passionate attacks and defenses of each > approach. I've got a feeling that the three projects aren't all that > different in maturity (all are well beyond the toy stage, none are quite > ready for prime time). In terms of specific type system features (e.g. > forward references, generic types, duck typing) I expect they are all > acceptable, and all probably need some work (and there's no reason to > assume that work can't be done). All support stubs so you can specify > signatures for code you can't edit (whether C extension, stdlib or just > opaque 3rd party code). > > To me there is no doubt that (1) is the most Pythonic approach. When we > discussed PEP 3107 (function annotations) it was always my goal that these > would eventually be used for type annotations. There was no consensus at > the time on what the rules for type checking should be, but their syntactic > position was never in doubt. So we decided to introduce "annotations" in > Python 3 in the hope that 3rd party experiments would eventually produce > something satisfactory. Mypy is one such experiment. One of the important > lessons I draw from mypy is that type annotations are most useful to > linters, and should (normally) not be used to enforce types at run time. > They are also not useful for code generation. None of that was obvious when > we were discussing PEP 3107! > > I don't buy the argument that PEP 3107 promises that annotations are > completely free of inherent semantics. It promises compatibility, and I > take that very seriously, but I think it is reasonable to eventually > deprecate other uses of annotations -- there aren't enough significant > other uses for them to warrant crippling type annotations forever. In the > meantime, we won't be breaking existing use of annotations -- but they may > confuse a type checker, whether a stand-alone linter like mypy or built > into an IDE like PyCharm, and that may serve as an encouragement to look > for a different solution. > > Most of the thornier issues brought up against mypy wouldn't go away if we > adopted another approach: whether to use concrete or abstract types, the > use of type variables, how to define type equivalence, the relationship > between a list of ints and a list of objects, how to spell "something that > implements the buffer interface", what to do about JSON, binary vs. text > I/O and the signature of open(), how to check code that uses isinstance(), > how to shut up the type checker when you know better... The list goes on. > There will be methods whose type signature can't be spelled (yet). There > will be code distributed with too narrowly defined types. Some programmers > will uglify their code to please the type checker. > > There are questions about what to do for older versions of Python. I find > mypy's story here actually pretty good -- the mypy codec may be a hack, but > so is any other approach. Only the __future__ approach really loses out > here, because you can't add a new __future__ import to an old version. > > So there you have it. I am picking the mypy family and I hope we can start > focusing on specific improvements to mypy. I also hope that somebody will > write converters from pytypedecl and PyCharm stubs into mypy stubs, so that > we can reuse the work already put into stub definitions for those two > systems. And of course I hope that PyCharm and pytypedecl will adopt mypy's > syntax (initially in addition to their native syntax, eventually as their > sole syntax). > > PS. I realize I didn't discuss question (C) much. That's intentional -- we > can now start discussing specific mypy features in separate threads (or in > this one :-). > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Sat Aug 16 15:34:51 2014 From: antoine at python.org (Antoine Pitrou) Date: Sat, 16 Aug 2014 09:34:51 -0400 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: Le 15/08/2014 14:01, Guido van Rossum a ?crit : > Please do write about non-toy examples! Are you looking for examples using the multipledispatch library, or multiple dispatch in general? As for multiple dispatch in general, Numba uses something which is morally one in order to select the right specialization of, say, an operator (for example to choose amongst '+ between int and int', '+ between numpy.datetime64 and numpy.timedelta64', '+ between numpy.timedelta64 and numpy.timedelta64', etc.). Regards Antoine. From mrocklin at gmail.com Sat Aug 16 15:58:43 2014 From: mrocklin at gmail.com (Matthew Rocklin) Date: Sat, 16 Aug 2014 06:58:43 -0700 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: Here is a non-trivial example of multiple dispatch. I want to convert data between container types, i.e. given into(a, b) I want to return something with the information content of b in a container like a, e.g. In [24]: into([], (1, 2, 3)) Out[24]: [1, 2, 3] We use this abstraction pretty heavily in Blaze, a project that tries to map relational algebra onto a variety of projects that might possibly be used to do relational-algebra-like tasks. Projects in this scope include sqlalchemy, pandas, numpy, pyspark, pytables, etc.. In [26]: from blaze import into A dataframe with some test data In [25]: df = DataFrame([[1, 'Alice', 100], [2, 'Bob', -200], [3, 'Charlie', 300], [4, 'Dennis', 400], [5, 'Edith', -500]], columns=['id', 'name', 'amount']) migrate list <- DataFrame In [27]: into([], df) Out[27]: [[1, 'Alice', 100], [2, 'Bob', -200], [3, 'Charlie', 300], [4, 'Dennis', 400], [5, 'Edith', -500]] migrate numpy array <- DataFrame In [28]: into(np.ndarray(0), df) Out[28]: rec.array([(1, 'Alice', 100), (2, 'Bob', -200), (3, 'Charlie', 300), (4, 'Dennis', 400), (5, 'Edith', -500)], dtype=[('id', ' wrote: > Le 15/08/2014 14:01, Guido van Rossum a ?crit : > > Please do write about non-toy examples! >> > > Are you looking for examples using the multipledispatch library, or > multiple dispatch in general? > > As for multiple dispatch in general, Numba uses something which is morally > one in order to select the right specialization of, say, an operator (for > example to choose amongst '+ between int and int', '+ between > numpy.datetime64 and numpy.timedelta64', '+ between numpy.timedelta64 and > numpy.timedelta64', etc.). > > Regards > > Antoine. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Aug 16 16:30:47 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 16 Aug 2014 07:30:47 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: On Aug 16, 2014, at 6:00, Dave Halter wrote: > Appendix (My Proposal) > > My proposal (as discussed and evolved with a few good people at EuroPython) for containers would look something like this: > > def foo(index: int, x: [float], y: {int: str}) -> (float, str): > return x[index], y[index] > > The "default" containers (set, list, dict and tuple) would just serve as a way of specifying containers. This makes a lot of things less complicated: > > - It's easier to understand (everybody knows builtin types). It also feels natural to me. The example above covers also almost all variations. A tuple could be expanded by using the ellipsis: `(int, object, ..., float)`. > - No imports needed. People are more likely to use it, if they don't have to import typing all the time. This is important for static analysis, people are only likely to use it if it's easier than writing docstrings with type information. > - Argument clinic could use the same. The standard library quite often doesn't accept abstract data types, btw. > - It's what people sometimes use in docstrings: ``@rtype: (float, str)`` or ``:rtype: (float, str)``. > > I know this doesn't solve the duck typing issue, but if you look at real-life Python code bases, there are very few instances of actually implementing a ``Mapping``, etc. For "Mapping", maybe (although anyone who uses some form of tree-based mapping, either to avoid hash-collision attacks or because he needs sorting, may disagree). But for "etc.", people implement them all the time. Especially "Iterable" and "Callable". And, even some of the ones that people don't implement often, they use often, implicitly or otherwise, like TextIOBase. Anything that encourages people to restrict their code to only working on lists instead of iterables would be a huge step backward to Python 2.2. And your argument for it ("everybody knows builtin types") implies that's exactly what you're expecting with this proposal. However, there's an easy way around this: just let [spam] in your syntax mean what Iterable[spam] means in MyPy's. If someone really needs to declare that they will only accept a list or whatever, that's the uncommon case, and can be written more verbosely. And I don't see any reason why this can't be added on top of genericizing the ABCs a la MyPy. Meanwhile, your tuple doesn't fit the same pattern as the others, because it's explicitly fixed-size and heterogeneous. And I think this is a good thing. And I think it's pretty close to what MyPy does with an expression list of types already; if not, it seems like what MyPy _should_ do. If I loop over a zip of a [str] and an [int], the loop variable is a (str, int), not a Tuple[str or int]. So, making it easier to specify generic builtins is a bad idea, but using builtins to make it easier to specify common types is a great idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Aug 16 16:46:47 2014 From: brett at python.org (Brett Cannon) Date: Sat, 16 Aug 2014 14:46:47 +0000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: <20140814181554.GR4525@ando> <20140815184044.2aaeca8e@anarchist.localdomain> <20140816050431.GE4525@ando> Message-ID: On Sat Aug 16 2014 at 1:05:16 AM Steven D'Aprano wrote: > Some people like the epydoc-style convention of putting type annotations > in docstrings: > > [...] > > >> def demo(self, a, b, c): > > >> """ > > >> This function returns the product of a, b and c > > >> @type self: SimpleEquation > > >> :param a: int - The first number > > >> :param b: int > > >> :param c: int - The third number should not be zero and > should > > >> also > > >> only be -1 if you enjoy carrots (this comment spans 2 > lines) > > >> :return: int > > >> """ > > One issue I haven't see raised is that annotations are available at > runtime, whereas docstrings may not be. (The -OO switch removes > docstrings.) A linter may be able to parse the docstrings at compile > time before the docstrings are discarded, or it may not, but using > docstrings means the information is not always available for > introspection at runtime. I think that's a major disadvantage. > > Although I admit I don't always remember to test my code using -O and > -OO, I do try very hard to do this and I have found bugs in my code from > doing so. I think anything which makes testing -O and -OO modes harder > is a bad thing. > > [Quoting Barry Warsaw] > > docstrings > > naturally live right after the function signature (indeed, or it > wouldn't get > > stuffed into __doc__), so it's always close to the source. That makes it > > quite easy for the third party human reader, but also for the author to > keep > > up-to-date. > > *Close to the source* is not the same as *part of the source*. In the > example above, the difference is as high as eight lines, compared to > zero: > > # Function annotations > def demo(self, a:int, b:int, c:int)->int: > > > Using docstring annotations splits the information about parameters into > two places. Those two places might be close, but there are still two > sources of ultimate truth instead of one. You have the name of the > parameter in the parameter list, and the type of the parameter inside > the docstring separated by some arbitrary number of lines of code. > I'm with Steven on this. I actively hate docstrings that list every parameter, their expected interface, etc. The parameter list exists for a reason and a majority of the time I don't need an explanation of what a parameter does. In those rare instances where I need clarification I can write a quick sentence in the docstring explaining the special case. """Returns the product of a, b, and c. The 'c' parameter should not be zero. If you like carrots, set it to -1 (this comment spans two lines). """ That's 4 lines compared to 7 (which was missing a blank line to begin with so it really should be 8). We're all adults and properly worded parameter names tell you a lot. What we are trying to do here is help programmatic tools know things that we know to be true. Now that is not to say whatever comes out of typing.py shouldn't be legible and not noisy. I'm sure the reason it uses CapWords for e.g. Dict is so you can do `from typing import *` which makes `def demo(a: Int, b: Int, c: Int) -> Int` read just as cleanly as if you used 'int' itself (this might become the one time I promote using import * so enjoy it while you can =). And as others have pointed out, if you really like the docstring approach you can always set up a decorator to do the translation for you, but you can't go the other way from annotation to docstring when examining source. So while you can promote and use your docstring approach and even argue for the inclusion of such a decorator in typing.py, you can't promote docstrings exclusively without completely cutting off the function annotation approach. And I think enough of us like the function annotation approach that cutting it off entirely isn't acceptable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Aug 16 17:02:44 2014 From: brett at python.org (Brett Cannon) Date: Sat, 16 Aug 2014 15:02:44 +0000 Subject: [Python-ideas] Optional static typing -- the crossroads References: Message-ID: On Sat Aug 16 2014 at 10:38:34 AM Andrew Barnert wrote: > On Aug 16, 2014, at 6:00, Dave Halter wrote: > > *Appendix (My Proposal)* > > My proposal (as discussed and evolved with a few good people at > EuroPython) for containers would look something like this: > > def foo(index: int, x: [float], y: {int: str}) -> (float, str): > return x[index], y[index] > > The "default" containers (set, list, dict and tuple) would just serve as a > way of specifying containers. This makes a lot of things less complicated: > > - It's easier to understand (everybody knows builtin types). It also feels > natural to me. The example above covers also almost all variations. A tuple > could be expanded by using the ellipsis: `(int, object, ..., float)`. > - No imports needed. People are more likely to use it, if they don't have > to import typing all the time. This is important for static analysis, > people are only likely to use it if it's easier than writing docstrings > with type information. > - Argument clinic could use the same. The standard library quite often > doesn't accept abstract data types, btw. > - It's what people sometimes use in docstrings: ``@rtype: (float, str)`` > or ``:rtype: (float, str)``. > > I know this doesn't solve the duck typing issue, but if you look at > real-life Python code bases, there are very few instances of actually > implementing a ``Mapping``, etc. > > > For "Mapping", maybe (although anyone who uses some form of tree-based > mapping, either to avoid hash-collision attacks or because he needs > sorting, may disagree). > > But for "etc.", people implement them all the time. Especially "Iterable" > and "Callable". And, even some of the ones that people don't implement > often, they use often, implicitly or otherwise, like TextIOBase. > > Anything that encourages people to restrict their code to only working on > lists instead of iterables would be a huge step backward to Python 2.2. And > your argument for it ("everybody knows builtin types") implies that's > exactly what you're expecting with this proposal. > > However, there's an easy way around this: just let [spam] in your syntax > mean what Iterable[spam] means in MyPy's. If someone really needs to > declare that they will only accept a list or whatever, that's the uncommon > case, and can be written more verbosely. And I don't see any reason why > this can't be added on top of genericizing the ABCs a la MyPy. > > Meanwhile, your tuple doesn't fit the same pattern as the others, because > it's explicitly fixed-size and heterogeneous. And I think this is a good > thing. And I think it's pretty close to what MyPy does with an expression > list of types already; if not, it seems like what MyPy _should_ do. If I > loop over a zip of a [str] and an [int], the loop variable is a (str, int), > not a Tuple[str or int]. > > So, making it easier to specify generic builtins is a bad idea, but using > builtins to make it easier to specify common types is a great idea. > The trick in all of this is making sure people instinctively know what the builtin types represent in terms of an interface w/o necessarily over-specifying. For instance, a list could be viewed as MutableSequence when all that is really necessary is Sequence or Iterable. Just think of those situations where a list or tuple both work as an argument; how do you specify that without assuming mutability? It's tricky to figure out what a proper assumption of what the built-ins represent should be. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Aug 16 19:15:09 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 03:15:09 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <20140816171502.GK4525@ando> On Sat, Aug 16, 2014 at 03:00:15PM +0200, Dave Halter wrote: > I think a resulting PEP of this discussion should contain a deprecation > note for all usages other than type checking in Python 3.5. Can you explain your reasoning for this? I understand that unless they are deliberately built to cooperate, two users of annotations are likely to interfere with each other. The jedi tool wants annotations to be type information; the sith tool wants annotations to be X-Face pictures of kittens. They can't both get what they want. You are asking for alternative uses of annotations, such as pictures of kittens, to be deprecated. But it seems to me that all you really need is some standard way for jedi to look at the function and cheaply see that it shouldn't try interpreting the annotations as types. (And, mutatis mutandis, the same applies to sith.) That allows jedi and sith to co-exist, although we can't use both on the same function at the same time. I think it is fair for Python to standardise on type checking as the default semantics of annotations, but I would like to see a way to opt-out and still use annotations for other purposes. > The current > ambiguous nature of annotations is the fact why no static analysis tool > ever checked them. mypy does. Hence this proposal. So does PyCharm: http://www.jetbrains.com/pycharm/webhelp/type-hinting-in-pycharm.html On the other hand, in this thread we've heard from two others who use function annotations for purposes other than types. So it seems to me that usage of annotations is split right down the middle between typing and non-typing. -- Steven From ethan at stoneleaf.us Sat Aug 16 22:22:48 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 16 Aug 2014 13:22:48 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <53EFBD98.2090802@stoneleaf.us> As a test case for what code may soon look like, here's a bit from one of my code bases: -------------------------------------------------------------------- class ACHPayment(object): """A single payment from company to a vendor.""" def __init__(self, description, sec_code, vendor_name, vendor_inv_num, vendor_rtng, vendor_acct, transaction_code, vendor_acct_type, amount, payment_date): """ description: 10 chars sec_code: 'CCD' or 'CTX' vendor_name: 22 chars vendor_inv_num: 15 chars vendor_rtng: 9 chars vendor_acct: 17 chars transaction_code: ACH_ETC code (enum) vendor_acct_type: 'domestic' or 'foreign' amount: 10 digits (pennies) payment_date: date payment should occur on (datetime.date type class) """ -------------------------------------------------------------------- The question: what would this look like with type annotations? As a point of interest, the last parameter, payment_date, can be /anything/ that quacks like a datetime.date -- I tend to use my own dbf.Date class, which subclasses object, not datetime.date itself. -- ~Ethan~ From rymg19 at gmail.com Sat Aug 16 22:48:10 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sat, 16 Aug 2014 15:48:10 -0500 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EFBD98.2090802@stoneleaf.us> References: <53EFBD98.2090802@stoneleaf.us> Message-ID: Maybe that point of interest could be solved by using some kind of type class/interface(in the Obj C/Java sense). That way, external types that the user has no control of can be added to the interface. On Sat, Aug 16, 2014 at 3:22 PM, Ethan Furman wrote: > As a test case for what code may soon look like, here's a bit from one of > my code bases: > > > -------------------------------------------------------------------- > class ACHPayment(object): > """A single payment from company to a vendor.""" > > def __init__(self, > description, sec_code, > vendor_name, vendor_inv_num, vendor_rtng, vendor_acct, > transaction_code, vendor_acct_type, amount, payment_date): > """ > description: 10 chars > sec_code: 'CCD' or 'CTX' > vendor_name: 22 chars > vendor_inv_num: 15 chars > vendor_rtng: 9 chars > vendor_acct: 17 chars > transaction_code: ACH_ETC code (enum) > vendor_acct_type: 'domestic' or 'foreign' > amount: 10 digits (pennies) > payment_date: date payment should occur on (datetime.date type > class) > """ > -------------------------------------------------------------------- > > > The question: what would this look like with type annotations? As a > point of interest, the last parameter, payment_date, can be /anything/ that > quacks like a datetime.date -- I tend to use my own dbf.Date class, which > subclasses object, not datetime.date itself. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sat Aug 16 23:46:42 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 16 Aug 2014 14:46:42 -0700 (PDT) Subject: [Python-ideas] keyword for introducing generators Message-ID: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com> I'm sure this has been suggested before, but I just spent two days trying to figure out why a method wasn't being called only to find that I'd accidentally pasted a yield into the function. What is the argument against a different keyword for introducing generator functions/methods? If it's backward compatibility, then my suggestion to have a from __future__ and then make it real in Python 4. Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Aug 16 23:55:36 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 17 Aug 2014 07:55:36 +1000 Subject: [Python-ideas] keyword for introducing generators In-Reply-To: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com> References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com> Message-ID: On Sun, Aug 17, 2014 at 7:46 AM, Neil Girdhar wrote: > I'm sure this has been suggested before, but I just spent two days trying to > figure out why a method wasn't being called only to find that I'd > accidentally pasted a yield into the function. What is the argument against > a different keyword for introducing generator functions/methods? There are quite a few changes to a function based on its body, like how the presence of assignment causes a name to be local unless explicitly declared otherwise. It's not necessary to predeclare everything. But if you're having trouble with a function like that, maybe a little decorator would help: def announce(f): def inner(*a,**kw): print("Calling:",f.__name__) ret=f(*a,**kw) print("Return value is a",type(ret)) return ret return inner Decorate a function @announce, and it'll tell you (a) that it's being called, and (b) what type its return value is. If that's a generator, well, there's your answer. ChrisA From mistersheik at gmail.com Sun Aug 17 00:04:45 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 16 Aug 2014 18:04:45 -0400 Subject: [Python-ideas] keyword for introducing generators In-Reply-To: References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com> Message-ID: This is only a good solution once I realize that the method isn't being called after which, I now know what to look for. The whole point is to defensively design a language so that I don't get into this problem in the first place. Since I always know when I'm writing a function whether I want it to be automatically generator-returning using the yield keyword or not, then why not let me specify that? That way if the function gets long and I forget yield or accidentally include it, I get a reasonable error. You're right that it's not necessary to predeclare everything, but a different keyword is hardly more work. (And with respect to local, nonlocal and global variables, we often do predeclare those.) Best, Neil On Sat, Aug 16, 2014 at 5:55 PM, Chris Angelico wrote: > On Sun, Aug 17, 2014 at 7:46 AM, Neil Girdhar > wrote: > > I'm sure this has been suggested before, but I just spent two days > trying to > > figure out why a method wasn't being called only to find that I'd > > accidentally pasted a yield into the function. What is the argument > against > > a different keyword for introducing generator functions/methods? > > There are quite a few changes to a function based on its body, like > how the presence of assignment causes a name to be local unless > explicitly declared otherwise. It's not necessary to predeclare > everything. > > But if you're having trouble with a function like that, maybe a little > decorator would help: > > def announce(f): > def inner(*a,**kw): > print("Calling:",f.__name__) > ret=f(*a,**kw) > print("Return value is a",type(ret)) > return ret > return inner > > Decorate a function @announce, and it'll tell you (a) that it's being > called, and (b) what type its return value is. If that's a generator, > well, there's your answer. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/5-Qm2od4xQ8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Aug 17 00:16:10 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 16 Aug 2014 15:16:10 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: <4C70181D-C011-4B20-9CB9-C1A6BE40DFD5@yahoo.com> On Aug 16, 2014, at 8:02, Brett Cannon wrote: > On Sat Aug 16 2014 at 10:38:34 AM Andrew Barnert wrote: >> On Aug 16, 2014, at 6:00, Dave Halter wrote: >> >>> Appendix (My Proposal) >>> >>> My proposal (as discussed and evolved with a few good people at EuroPython) for containers would look something like this: >>> >>> def foo(index: int, x: [float], y: {int: str}) -> (float, str): >>> return x[index], y[index] >>> >>> The "default" containers (set, list, dict and tuple) would just serve as a way of specifying containers. This makes a lot of things less complicated: >>> >>> - It's easier to understand (everybody knows builtin types). It also feels natural to me. The example above covers also almost all variations. A tuple could be expanded by using the ellipsis: `(int, object, ..., float)`. >>> - No imports needed. People are more likely to use it, if they don't have to import typing all the time. This is important for static analysis, people are only likely to use it if it's easier than writing docstrings with type information. >>> - Argument clinic could use the same. The standard library quite often doesn't accept abstract data types, btw. >>> - It's what people sometimes use in docstrings: ``@rtype: (float, str)`` or ``:rtype: (float, str)``. >>> >>> I know this doesn't solve the duck typing issue, but if you look at real-life Python code bases, there are very few instances of actually implementing a ``Mapping``, etc. >> >> For "Mapping", maybe (although anyone who uses some form of tree-based mapping, either to avoid hash-collision attacks or because he needs sorting, may disagree). >> >> But for "etc.", people implement them all the time. Especially "Iterable" and "Callable". And, even some of the ones that people don't implement often, they use often, implicitly or otherwise, like TextIOBase. >> >> Anything that encourages people to restrict their code to only working on lists instead of iterables would be a huge step backward to Python 2.2. And your argument for it ("everybody knows builtin types") implies that's exactly what you're expecting with this proposal. >> >> However, there's an easy way around this: just let [spam] in your syntax mean what Iterable[spam] means in MyPy's. If someone really needs to declare that they will only accept a list or whatever, that's the uncommon case, and can be written more verbosely. And I don't see any reason why this can't be added on top of genericizing the ABCs a la MyPy. >> >> Meanwhile, your tuple doesn't fit the same pattern as the others, because it's explicitly fixed-size and heterogeneous. And I think this is a good thing. And I think it's pretty close to what MyPy does with an expression list of types already; if not, it seems like what MyPy _should_ do. If I loop over a zip of a [str] and an [int], the loop variable is a (str, int), not a Tuple[str or int]. >> >> So, making it easier to specify generic builtins is a bad idea, but using builtins to make it easier to specify common types is a great idea. > > The trick in all of this is making sure people instinctively know what the builtin types represent in terms of an interface w/o necessarily over-specifying. For instance, a list could be viewed as MutableSequence when all that is really necessary is Sequence or Iterable. Just think of those situations where a list or tuple both work as an argument; how do you specify that without assuming mutability? It's tricky to figure out what a proper assumption of what the built-ins represent should be. Honestly, I think [str] is the only case where this is an important question, so let's think about that rather than trying to think about a more general and abstract problem. I'm not sure whether, when people say they need a list of strings, they more often mean Iterable[str] rather than Sequence[str] or MutableSequence[str]. I suspect it's the former, but I can't prove it, and I also suspect it's more of a 70/10/20 case than a 98/1/1 case. So maybe that argues that [str] just shouldn't be allowed. But if it meant Iterable[str], someone who used it when they needed a Sequence or MutableSequence would get an error when they tried to MyPy their library, app, whatever. And it wouldn't be that hard to make that error explain the problem to them--the same way clang tries to explain template errors in C++ and suggest fixes, except that it would be orders of magnitude easier. "spam is an iterable, so you can't assign to its indexes. Did you mean to declare it as MutableSequence[str]?" On the other hand, if [str] meant MutableSequence[str] (or list[str]), the author who used it on a function that did nothing but loop over spam would not get an error; he'd have to wait until he published his code and someone filed a bug report saying "You declared spam as a MutableSequence, even though all you do is loop over it, and now my code that worked correctly with your spam-1.7, and that still works correctly with spam-1.8 if I don't use static checking, fails the linter. Did you mean to declare it as Iterable[str]?" This obviously isn't a slam-sunk argument. If MutableSequence were used far more often than Iterable, or were more pythonic in some way, then it would make sense for [str] to mean MutableSequence despite the fact that it puts the errors in the less convenient place. But if they're both reasonably common, I think this argues for making it mean Iterable. For the last part: > Just think of those situations where a list or tuple both work as an argument; how do you specify that without assuming mutability? That one's easy: you write Sequence. A lot of similar questions were already answered when abc, Number, collections.abc, and io were designed, and they did a great job answering some tricky questions--which is exactly why I think static typing should use those already-worked-out cases rather than trying to answer all those questions again with a parallel type hierarchy. (And if use of static typing leads people to realize that one of those ABCs got something wrong, better to fix that bug in the ABC than to leave it incorrect and make the typing type different.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.wielicki at sotecware.net Sun Aug 17 00:17:53 2014 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Sun, 17 Aug 2014 00:17:53 +0200 Subject: [Python-ideas] keyword for introducing generators In-Reply-To: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com> References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com> Message-ID: <53EFD891.5030602@sotecware.net> On 16.08.2014 23:46, Neil Girdhar wrote: > I'm sure this has been suggested before, but I just spent two days trying > to figure out why a method wasn't being called only to find that I'd > accidentally pasted a yield into the function. What is the argument > against a different keyword for introducing generator functions/methods? > > If it's backward compatibility, then my suggestion to have a from > __future__ and then make it real in Python 4. For what it?s worth, I know this problem very well, and it can take hours to figure out whats wrong. regards, jwi > > Neil > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From abarnert at yahoo.com Sun Aug 17 00:30:03 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 16 Aug 2014 15:30:03 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> Message-ID: <0B37B957-01A6-41BC-A416-94EF81D4FBA2@yahoo.com> On Aug 16, 2014, at 13:48, Ryan Gonzalez wrote: > Maybe that point of interest could be solved by using some kind of type class/interface(in the Obj C/Java sense). That way, external types that the user has no control of can be added to the interface. We already have that. ABCs are enough like Java interfaces, ObjC mandatory protocols, C++ (non-auto) concepts, etc. to do everything we need here (except genericity, which it seems like everyone agrees with adding) if you want to do it nominatively. You just need to write a Date ABC. Or argue that there should be a datetime.abc library in the stdlib that does it for you. And that's simple. And if you want to do it structurally, like Go protocols, C++ auto concepts, ObjC optional protocols, etc., ABCs can also do that. It's not _quite_ as simple today, but it's not hard, and there are a half dozen libraries that make it easy (I wrote one in a couple hours, and didn't bother publishing it because a quick search turned up so many pre-existing alternatives, and at least three classes in the stdlib that just did it manually without help...). Of course the existing implementations don't give you a way to statically declare the types of method arguments, attribute/properties, etc., but MyPy.Protocol does, and that can easily be adopted into the stdlib as part of this proposal. (In fact, I think it's already on the list.) So, for Ethan's case, the last argument is just "payment_date: datetime.abc.Date", except that nobody has added that to the stdlib yet, so instead he has to write it himself and use it. From abarnert at yahoo.com Sun Aug 17 00:36:26 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 16 Aug 2014 15:36:26 -0700 Subject: [Python-ideas] keyword for introducing generators In-Reply-To: References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com> Message-ID: <17013F47-88B8-4A1C-8EC9-8841DBD7BB1D@yahoo.com> On Aug 16, 2014, at 14:55, Chris Angelico wrote: > On Sun, Aug 17, 2014 at 7:46 AM, Neil Girdhar wrote: >> I'm sure this has been suggested before, but I just spent two days trying to >> figure out why a method wasn't being called only to find that I'd >> accidentally pasted a yield into the function. What is the argument against >> a different keyword for introducing generator functions/methods? > > There are quite a few changes to a function based on its body, like > how the presence of assignment causes a name to be local unless > explicitly declared otherwise. It's not necessary to predeclare > everything. > > But if you're having trouble with a function like that, maybe a little > decorator would help: > > def announce(f): > def inner(*a,**kw): > print("Calling:",f.__name__) > ret=f(*a,**kw) > print("Return value is a",type(ret)) > return ret > return inner I think it would be both simpler and more useful for him to write: def generator(f): assert inspect.isgeneratorfunction(f) return f def function(f): asset not inspect.isgeneratorfunction(f) return f Then he can just declare his functions as @generator or @function as appropriate and get an error at definition time if he accidentally got something wrong. From barry at python.org Sun Aug 17 01:26:46 2014 From: barry at python.org (Barry Warsaw) Date: Sat, 16 Aug 2014 19:26:46 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: <20140815181737.1e1279b4@anarchist.localdomain> <87egwhgmj2.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140816192646.78d2c76e@anarchist.wooz.org> On Aug 16, 2014, at 12:07 PM, Stephen J. Turnbull wrote: >Barry Warsaw writes: > > > OTOH there *is* a cost that could negatively impact readability. > > It's more text visually assaulting you. :) > >Not if you have a syntax-highlighting editor that has a no-see-um >face or text property. Feel free to share your font-lock definitions -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From lukasz at langa.pl Sun Aug 17 01:28:24 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sat, 16 Aug 2014 16:28:24 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EEA521.7020406@canterbury.ac.nz> References: <53EEA521.7020406@canterbury.ac.nz> Message-ID: On Aug 15, 2014, at 5:26 PM, Greg Ewing wrote: > ?ukasz Langa wrote: > >> class C: >> cls_member: str = ?on the class? >> def __init__(self): >> self.obj_member: str = ?on the instance' >> self.cls_member = 2 # that?s the real question: type error or an instance member? > > I think not treating it as an error would make it hard > to reason about the type of x.cls_member for an instance > x of C. Its type would depend on whether del x.cls_member > had been performed on x. > > Code which relied on them being different types would > be rather confusing to a human reader too, so it's > probably fine to discourage that. That was my reasoning exactly. +1 -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Aug 17 01:42:36 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 16 Aug 2014 16:42:36 -0700 Subject: [Python-ideas] keyword for introducing generators In-Reply-To: <53EFD891.5030602@sotecware.net> References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com> <53EFD891.5030602@sotecware.net> Message-ID: On Sat, Aug 16, 2014 at 3:17 PM, Jonas Wielicki wrote: > On 16.08.2014 23:46, Neil Girdhar wrote: > > I'm sure this has been suggested before, but I just spent two days trying > > to figure out why a method wasn't being called only to find that I'd > > accidentally pasted a yield into the function. What is the argument > > against a different keyword for introducing generator functions/methods? > > > > If it's backward compatibility, then my suggestion to have a from > > __future__ and then make it real in Python 4. > > For what it?s worth, I know this problem very well, and it can take > hours to figure out whats wrong. > A linter should be able to figure this out. For example, mypy will insist that a generator has a return type of Iterable[...]. So maybe you won't have to wait for Python 4; if the mypy proposal goes forward you will be able to use type annotations to distinguish generators. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Aug 17 04:08:51 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 12:08:51 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53EFBD98.2090802@stoneleaf.us> References: <53EFBD98.2090802@stoneleaf.us> Message-ID: <20140817020851.GL4525@ando> On Sat, Aug 16, 2014 at 01:22:48PM -0700, Ethan Furman wrote: > As a test case for what code may soon look like, here's a bit from one of > my code bases: > > > -------------------------------------------------------------------- > class ACHPayment(object): > """A single payment from company to a vendor.""" > > def __init__(self, > description, sec_code, > vendor_name, vendor_inv_num, vendor_rtng, vendor_acct, > transaction_code, vendor_acct_type, amount, payment_date): > """ > description: 10 chars > sec_code: 'CCD' or 'CTX' > vendor_name: 22 chars > vendor_inv_num: 15 chars > vendor_rtng: 9 chars > vendor_acct: 17 chars > transaction_code: ACH_ETC code (enum) > vendor_acct_type: 'domestic' or 'foreign' > amount: 10 digits (pennies) > payment_date: date payment should occur on (datetime.date type > class) > """ > -------------------------------------------------------------------- > > > The question: what would this look like with type annotations? As a point > of interest, the last parameter, payment_date, can be /anything/ that > quacks like a datetime.date -- I tend to use my own dbf.Date class, which > subclasses object, not datetime.date itself. I don't think this is a shining example of the value of static typing, at least not by default. As I see it, you would get something like this: def __init__(self, description:str, sec_code:str, vendor_name:str, vendor_inv_num:str, vendor_rtng:str, vendor_acct:str, transaction_code:str, vendor_acct_type:str, amount:int, payment_date:Any)->None: which may not give you much additional value. In this case, I think that the static checks will add nothing except (perhaps) allow you to forgo writing a few isinstance checks. You still have to check that the strings are the right length, and so on. But if you're willing to invest some time creating individual str subclasses, you can push the length checks into the subclass constructor, and write something like this: def __init__(self, description:Str10, sec_code:SecurityCode, vendor_name:Str22, vendor_inv_num:Str15, vendor_rtng:Str9, vendor_acct:Str17, transaction_code:ACH_ETC, vendor_acct_type:VendorAcctType, amount:Pennies, payment_date:DateABC)->None: Without knowing your application in detail, it is difficult to know how much work you should hand over to the type system, and how much you should continue to do in Python. If all you're doing is pushing strings from one place to another, you might not care exactly how long the string is, say because they're truncated when you print them. -- Steven From guido at python.org Sun Aug 17 07:03:48 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 16 Aug 2014 22:03:48 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <20140817020851.GL4525@ando> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: I'd like to summarize the main issues that have come up. As an experiment, I'm not changing the subject, but I am still not quoting anything in particular. Only two issues (or issue clusters) really seem contentious: (1) Should the function annotation syntax (eventually) be reserved for type annotations in a standard syntax? Or can multiple different uses of annotations coexist? And if they can, how should a specific use be indicated? (Also, some questions about compile-time vs. run-time use.) (2) For type annotations, should we adopt (roughly) the mypy syntax or the alternative proposed by Dave Halter? This uses built-in container notations as a shorthand, e.g. {str: int} instead of Dict[str, int]. This also touches on the issue of abstract vs. concrete types (e.g. iterable vs. list). Regarding (1), I continue to believe that we should eventually reserve annotations for types, to avoid confusing both humans and tools, but I think there's nothing we have to do in 3.5 -- 3.5 must preserve backward compatibility, and we're not proposing to give annotations any new semantics anyway -- the actual changes to CPython are limited to a new stdlib module (typing) and some documentation. Perhaps a thornier issue is how mypy should handle decorators that manipulate the signature or annotations of the function they wrap. But I think the only reasonable answer here can be that mypy must understand what decorators do if it wants to have any chance at type-checking decorated functions. I don't actually know how sophisticated mypy's understanding of decorators is, currently, but I don't think there's anything fundamentally more difficult than all the other things it must understand. Moving on to (2), the proposal is elegant enough by itself, and indeed has the advantage of being clear and concise: [T] instead of List[T], {T: U} instead of Dict[T, U], and so on. However, there are a few concerns. My first concern is that these expressions are only unambiguous in the context of function annotations. I want to promote the use of type aliases, and I think in general a type alias should behave similarly to an ABC. In particular, I think that any object used to represent a type in an annotation should itself be a type object (though you may not be able to instantiate it), and e.g. [int] doesn't satisfy that requirement. Without this, it would be difficult to implement isinstance() and issubclass() for type aliases -- and while we could special-case lists, sets and dicts, using a tuple *already* has a meaning! The second concern is that the proposal seems to steer users in the direction of using concrete types. A lot of Python's power stems from concepts like iterable and mapping and their variants (e.g. iterable, container, sequence, mutable sequence). There are justified concerns that users will unnecessarily constrain the argument types more than necessary (e.g. specifying a sequence where any iterable would do), and this proposal lacks the subtlety to express the difference. A third (minor) concern reflects issue (1): until we have agreement that annotations should only be used as type annotations, a type checker cannot assume that the presence of annotations means that types should be checked. Using e.g. Iterable[int] is pretty unambiguous (especially when Iterable is imported from typing.py), whereas just [int] is somewhat ambiguous. I call this only a minor issue because it still occurs for simple types like int or str, so if we can live with it for those we could presumably live with [int] and {str: float}. All in all I prefer the mypy syntax, despite being somewhat more verbose and requiring an import, with one caveat: I agree that it would be nicer if the mypy abstract collection types were the same objects as the ABCs exported by collections.abc. I'm not quite sure whether we should also change the concrete collection types from List, Dict, Set, Tuple to list, dict, set, tuple; the concrete types are so ubiquitous that I worry that there may be working code out there that somehow relies on the type objects themselves not being subscriptable. A mostly unrelated issue: there are two different uses of tuples, and we need a notation for both. One is a tuple of fixed length with heterogeneous, specific types for the elements; for example Tuple[int, float]. But I think we also need a way to indicate that a function expects (or returns) a variable-length tuple with a homogeneous element type. Perhaps we should call this type frozenlist, analogous to frozenset (and it seems there's a proposal for frozendict making the rounds as well). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Aug 17 08:02:01 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 16 Aug 2014 23:02:01 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> On Aug 16, 2014, at 22:03, Guido van Rossum wrote: > Moving on to (2), the proposal is elegant enough by itself, and indeed has the advantage of being clear and concise: [T] instead of List[T], {T: U} instead of Dict[T, U], and so on. However, there are a few concerns. > > My first concern is that these expressions are only unambiguous in the context of function annotations. Good point. Together with your third point (that [str] could be meaningful as a different type of annotation, while Iterable[str] is incredibly unlikely to mean anything other than a static type check--except maybe a runtime type check, but I think it's reasonable to assume they can share annotations), I think this kills the idea. Pity, because I like the way it looked. > All in all I prefer the mypy syntax, despite being somewhat more verbose and requiring an import, with one caveat: I agree that it would be nicer if the mypy abstract collection types were the same objects as the ABCs exported by collections.abc. I'm not quite sure whether we should also change the concrete collection types from List, Dict, Set, Tuple to list, dict, set, tuple; the concrete types are so ubiquitous that I worry that there may be working code out there that somehow relies on the type objects themselves not being subscriptable. I won't belabor the point, but again: I don't think we need a generic list type object, and without it, this entire problem--your only remaining problem that isn't a mere stylistic choice--vanishes. > A mostly unrelated issue: there are two different uses of tuples, and we need a notation for both. One is a tuple of fixed length with heterogeneous, specific types for the elements; for example Tuple[int, float]. But I think we also need a way to indicate that a function expects (or returns) a variable-length tuple with a homogeneous element type. Perhaps we should call this type frozenlist, analogous to frozenset (and it seems there's a proposal for frozendict making the rounds as well). Even if you drop the idea for [str] and {int: str}, which I agree seems unavoidable, I think it may still make sense for (int, str) to mean a heterogeneous iterable. Python already has target lists, argument lists, parameter lists, and expression lists that all have the same syntax as tuples or a superset thereof, but don't define tuples. In (a, b) = zip(c, d), neither (a, b) nor (c, d) is a tuple, and I don't think anyone is confused by that. So, why can't def foo(spam: (int, str)) mean that spam is an iterable of an int and a str, in exactly the same way that the assignment statement means that a and b are assigned the result of unpacking the iterable returned by zip when called with c and d? And this leaves Tuple[str] or tuple[str] free to mean a homogenous tuple (although, again, I don't think we even want or need that...). From steve at pearwood.info Sun Aug 17 09:26:02 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 17:26:02 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> Message-ID: <20140817072602.GN4525@ando> On Sat, Aug 16, 2014 at 11:02:01PM -0700, Andrew Barnert wrote: > On Aug 16, 2014, at 22:03, Guido van Rossum wrote: > > > Moving on to (2), the proposal is elegant enough by itself, and indeed has the advantage of being clear and concise: [T] instead of List[T], {T: U} instead of Dict[T, U], and so on. However, there are a few concerns. > > > > My first concern is that these expressions are only unambiguous in the context of function annotations. > > Good point. Together with your third point (that [str] could be > meaningful as a different type of annotation, while Iterable[str] is > incredibly unlikely to mean anything other than a static type > check--except maybe a runtime type check, but I think it's reasonable > to assume they can share annotations), I think this kills the idea. > Pity, because I like the way it looked. [str] looks nice, but it looks like a list of str, or possibly an optional str, e.g. from the docstring of int: int(x[, base]) -> integer What the [str] syntax doesn't look like is an Iterable of str. Or should that be Sequence of str? MutableSequence perhaps? If [str] means something other than list of str, it is going to be some arbitrary special case to be memorized. Have pity on people teaching Python. I don't want to have to try to explain to beginners why [str] sometimes means a list and sometimes an arbitrary Iterable (or whatever). This is just downright confusing: def func(arg:[str]): x = [str] assert isinstance(x, list) # Always passes. assert isinstance(arg, list) # Sometimes fails. func(iter("abc")) # Fails. [Guido] > > All in all I prefer the mypy syntax, despite being somewhat more > > verbose and requiring an import, with one caveat: I agree that it > > would be nicer if the mypy abstract collection types were the same > > objects as the ABCs exported by collections.abc. I'm not quite sure > > whether we should also change the concrete collection types from > > List, Dict, Set, Tuple to list, dict, set, tuple; We can start with typing.List, Dict, etc., and later on consider using builtins. I worry that if we use builtins, people will declare x:list[int] not because they *need* a list of int, but because it saves typing over from typing import Sequence, Integer def func(x:Sequence[Integer]): So even though I suggested earlier that the builtins grow appropriate __getitem__ methods, on second thoughts I would be very cautious about introducing that. > > the concrete types > > are so ubiquitous that I worry that there may be working code out > > there that somehow relies on the type objects themselves not being > > subscriptable. [Andrew] > I won't belabor the point, but again: I don't think we need a generic > list type object, and without it, this entire problem--your only > remaining problem that isn't a mere stylistic choice--vanishes. I don't understand. If there's no list typing object, how do you declare a variable must be a list and nothing but a list? Or that it returns a list? [Guido] > > A mostly unrelated issue: there are two different uses of tuples, > > and we need a notation for both. One is a tuple of fixed length with > > heterogeneous, specific types for the elements; for example > > Tuple[int, float]. But I think we also need a way to indicate that a > > function expects (or returns) a variable-length tuple with a > > homogeneous element type. Throwing this idea out to be shot down: use some sort of slice notation. Tuple[int, float, str] # Like (23, 1.5, "spam") Tuple[::int, float, str] # Like (1, 2, 3, 4) or (1.5,) or ("x", "y") That is, if the argument to __getitem__ is a slice (None, None, T), T is either a type or a tuple of types. Any other kind of slice is reserved for the future, or an error. > > Perhaps we should call this type > > frozenlist, analogous to frozenset (and it seems there's a proposal > > for frozendict making the rounds as well). [Andrew] > Even if you drop the idea for [str] and {int: str}, which I agree > seems unavoidable, I think it may still make sense for (int, str) to > mean a heterogeneous iterable. That makes no sense to me. It looks like a tuple, not a generic iterable object. Your interpretation has the same problems I discussed above for [str] notation: it is an arbitrary choice whether (...) means Iterable, Sequence or ImmutableSequence, and it doesn't fit nicely with other common uses of parens. See below. > Python already has target lists, argument lists, parameter lists, and > expression lists that all have the same syntax as tuples or a superset > thereof, but don't define tuples. In (a, b) = zip(c, d), neither (a, > b) nor (c, d) is a tuple, and I don't think anyone is confused by > that. Ha , you've obviously stopped reading the "Multi-line with statement" thread on Python-Dev :-) > So, why can't def foo(spam: (int, str)) mean that spam is an > iterable of an int and a str, in exactly the same way that the > assignment statement means that a and b are assigned the result of > unpacking the iterable returned by zip when called with c and d? But a, b = zip(c, d) requires that there be exactly two elements, not some unspecified number. To me, spam:(int, str) has a natural interpretation that spam can be either an int or a str, not an Iterable or Sequence or even a tuple. > And this leaves Tuple[str] or tuple[str] free to mean a homogenous > tuple (although, again, I don't think we even want or need that...). We do. Consider the isinstance() function. Here's the signature according to its docstring: isinstance(object, class-or-type-or-tuple) -> bool The second argument can be a single type, or a tuple of an arbitrary number of types. I'd write it with annotations like: def isinstance(object:Any, class_or_type_or_tuple:(Type, Tuple[::Type]) )->Bool: assuming (a,b) means "either a or b" and Tuple[::a] means a homogenous tuple of a. (With shorter argument names, it even fits on one line.) And, here's issubclass: def issubclass(C:Type, D:(Type, Tuple[::Type]))->Bool: -- Steven From stefan_ml at behnel.de Sun Aug 17 09:28:56 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 17 Aug 2014 09:28:56 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: Guido van Rossum schrieb am 17.08.2014 um 07:03: > I'd like to summarize the main issues that have come up. As an experiment, > I'm not changing the subject, but I am still not quoting anything in > particular. Only two issues (or issue clusters) really seem contentious: > > (1) Should the function annotation syntax (eventually) be reserved for type > annotations in a standard syntax? Or can multiple different uses of > annotations coexist? And if they can, how should a specific use be > indicated? (Also, some questions about compile-time vs. run-time use.) > > Regarding (1), I continue to believe that we should eventually reserve > annotations for types, to avoid confusing both humans and tools, but I > think there's nothing we have to do in 3.5 -- 3.5 must preserve > backward compatibility, and we're not proposing to give annotations any > new semantics anyway -- the actual changes to CPython are limited to a > new stdlib module (typing) and some documentation. As I mentioned before, there is more than one kind of type, even if we stick to reserving annotations for type declarations. That's why Cython currently supports these four ways of type annotations (in addition to its own non-Python way with "cdef"): x: dict x: {"type": dict} x: {"type": "dict"} x: {"ctype": "long double"} The latter three can also be combined, so you could declare a C type for Cython compilation and a Python type for your IDE and other static Python analysis tools, e.g. x: {"type": int, "ctype": "size_t"} Note that this also helps at a documentation level. The expected input is a Python int, but in fact it's restricted to a C size_t by the native implementation. I'd still vote for allowing the simpler "x: dict" as well for cases where it's the only annotation. It's easy enough to switch to the explicit notation when you want to add a second (potentially non-type) annotation. So, rather than "reserving" annotations for type declarations, I vote for making type annotations the default, but allowing other annotations by putting them into a dict that gives each annotation a string name. That name could be a module name in the stdlib or on PyPI, for example. > (2) For type annotations, should we adopt (roughly) the mypy syntax or the > alternative proposed by Dave Halter? This uses built-in container notations > as a shorthand, e.g. {str: int} instead of Dict[str, int]. This also > touches on the issue of abstract vs. concrete types (e.g. iterable vs. > list). I talked to him at EP14 and we agreed that the simpler syntax looks tempting. However, it does not support protocols, so it still needs something that allows us to say Iterable(int) in some way. I always thought that the ABCs were made for that, but so far everyone seemed to think that we need something different again. I'm happy to see that your preference also goes in that direction now. Having yet another typing module seems like an unnecessary duplication of the type system. Stefan From nicholas.cole at gmail.com Sun Aug 17 09:41:33 2014 From: nicholas.cole at gmail.com (Nicholas Cole) Date: Sun, 17 Aug 2014 08:41:33 +0100 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <20140817020851.GL4525@ando> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano wrote: > I don't think this is a shining example of the value of static typing, > at least not by default. As I see it, you would get something like this: > > def __init__(self, > description:str, sec_code:str, > vendor_name:str, vendor_inv_num:str, > vendor_rtng:str, vendor_acct:str, > transaction_code:str, vendor_acct_type:str, > amount:int, payment_date:Any)->None: > > which may not give you much additional value. In this case, I think that > the static checks will add nothing except (perhaps) allow you to forgo > writing a few isinstance checks. You still have to check that the > strings are the right length, and so on. > > But if you're willing to invest some time creating individual str > subclasses, you can push the length checks into the subclass > constructor, and write something like this: > > def __init__(self, > description:Str10, sec_code:SecurityCode, > vendor_name:Str22, vendor_inv_num:Str15, > vendor_rtng:Str9, vendor_acct:Str17, > transaction_code:ACH_ETC, vendor_acct_type:VendorAcctType, > amount:Pennies, payment_date:DateABC)->None: I know that the BDFL has spoken on this issue and said that he finds all of this readable and "pythonic", but these examples perfectly capture what I am going to dislike about this syntax as it becomes popular. I suppose it will be better when I am reading it in an editor that has syntax highlighting but as it stands I had to stare at that block of code for a long time to see how many and what type of arguments it called. At first I thought you had one per line, then I thought you had variable numbers per line. On about the fourth or fifth reading, I saw you had two per line. The problem with the syntax (I think) is that it relies on readers spotting characters like ":" and "[", characters which change how the eye should parse the line (assuming this is going to be optional). I find that those characters get lost very easily in long function definitions, leaving the reading having to read and re-read the block to answer questions like, 1. how many arguments are there? 2. Are any of them keyword arguments? Are they all the same type? 3. What are their names? In the example above, my eye keeps wanting to tell me that one of them is called SecurityCode, for example, even though I know that is the name of a class. This all seems unpythonic to me. Most of python's syntax is expressed in words rather than compact symbols. My fear with all of this is that it turns python into a language that is harder for humans to read. I much prefer the PyCharm docstring approach, because the eye can scan the function signature quickly and then the brain can say, "Ah - 10 arguments, oh, and I see that they have to be particular types and the return code is specified." To put it another way, current python function signatures are immediately intuitive even to someone who is unfamiliar with the language. There is nothing intuitive about this. It is more like looking at ObjectiveC or similar. At root, I don't totally understand what is "Pythonic" about function signatures. On the other hand, more expert people than me seem to like the above, and so I am sure that I am missing something. Perhaps it is simply the DRY principle. On the other hand, I am sure that readability issues are not simply a matter of personal taste. I've just re-read Tog on Inerterface, and perhaps that is colouring my thought! Anyone who is at all dyslexic is, I think, going to struggle! PEP8 should probably specify one argument per line if this kind of syntax is going to be at all re-readable. However, I do accept that the BDFL has spoken, and I'll "get with the program"! I'm sure I'll get used to it. N. From sf at fermigier.com Sun Aug 17 10:07:22 2014 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Sun, 17 Aug 2014 10:07:22 +0200 Subject: [Python-ideas] RFC: Multiple Dispatch Message-ID: GvR wrote: > Are there good uses of singledispatch in the wild even? Martijn Faassen wrote a whole Web framework (Morepath = http://morepath.readthedocs.org/en/latest/) on top of it: http://blog.startifact.com/posts/reg-now-with-more-generic.html S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Founder, Nuxeo - Enterprise Content Management Platform - http://www.nuxeo.com/ Co-Founder and Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Founder & Vice-President, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Co-Founder & Co-organiser, Open World Forum - http://www.openworldforum.paris/fr/ --- "No problem is too small or too trivial if we can really do something about it." - Richard P. Feynman "Well done is better than well said." - Benjamin Franklin "There's no such thing as can't. You always have a choice." - Ken Gor -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sun Aug 17 10:23:52 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 17 Aug 2014 18:23:52 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <854mxb5xt3.fsf_-_@benfinney.id.au> Guido van Rossum writes: > A mostly unrelated issue: there are two different uses of tuples, and > we need a notation for both. One is a tuple of fixed length with > heterogeneous, specific types for the elements; for example Tuple[int, > float]. That's the meaning of a tuple data structure, to me. > But I think we also need a way to indicate that a function expects (or > returns) a variable-length tuple with a homogeneous element type. Why? What real-world uses are there, where a list won't do the job adequately? I have encountered many uses of ?homogeneous, variable-length sequence? and every time a Python tuple is used for that, I perceive a Python list would be better precisely *because* it better indicates that semantic meaning. I'd like to know how you think that's not true, and what real-world code makes you think so. -- \ ?Contentment is a pearl of great price, and whosoever procures | `\ it at the expense of ten thousand desires makes a wise and | _o__) happy purchase.? ?J. Balguy | Ben Finney From steve at pearwood.info Sun Aug 17 10:34:53 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 18:34:53 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <20140817083453.GO4525@ando> On Sun, Aug 17, 2014 at 08:41:33AM +0100, Nicholas Cole wrote: > On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano wrote: [...] > > def __init__(self, > > description:Str10, sec_code:SecurityCode, > > vendor_name:Str22, vendor_inv_num:Str15, > > vendor_rtng:Str9, vendor_acct:Str17, > > transaction_code:ACH_ETC, vendor_acct_type:VendorAcctType, > > amount:Pennies, payment_date:DateABC)->None: > > I know that the BDFL has spoken on this issue and said that he finds > all of this readable and "pythonic", but these examples perfectly > capture what I am going to dislike about this syntax as it becomes > popular. Even though I am in favour of the proposal, I do sympathise, and I see what you mean. But, I think it is important to realise that a method with ten arguments (plus self) is not going to be exactly readable at the best of times. > I suppose it will be better when I am reading it in an editor that has > syntax highlighting but as it stands I had to stare at that block of > code for a long time to see how many and what type of arguments it > called. At first I thought you had one per line, then I thought you > had variable numbers per line. On about the fourth or fifth reading, > I saw you had two per line. Look for the commas :-) But yes, as given that makes a big wall of text. I suppose it will take some time for people to decide what formatting works best for them. It might help to align the arguments in columns (even though that goes against PEP-8): def __init__(self, description:Str10, sec_code:SecurityCode, vendor_name:Str22, vendor_inv_num:Str15, vendor_rtng:Str9, vendor_acct:Str17, transaction_code:ACH_ETC, vendor_acct_type:VendorAcctType, amount:Pennies, payment_date:DateABC, ) -> None: That works for me. > My fear with all of this is that it turns python into a language that > is harder for humans to read. Declaring types in the function parameter list is very common, in many languages. If it were *that* much harder to read, languages wouldn't keep using it. (Not many languages follow Forth or APL syntax.) Perhaps because I learned to program in Pascal, I find the annotation syntax very easy to read, but, yes, anything which increases the density of information per line risks hurting readability a little. It's a tradeoff, and of course all of this is optional. Syntax highlighting will help, and I expect that in a few years time emacs and vim will have some way to hide annotations when editing code :-) -- Steven From ncoghlan at gmail.com Sun Aug 17 10:50:08 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Aug 2014 18:50:08 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <20140817083453.GO4525@ando> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <20140817083453.GO4525@ando> Message-ID: On 17 August 2014 18:34, Steven D'Aprano wrote: > > Declaring types in the function parameter list is very common, in many > languages. If it were *that* much harder to read, languages wouldn't > keep using it. (Not many languages follow Forth or APL syntax.) Perhaps > because I learned to program in Pascal, I find the annotation syntax > very easy to read, but, yes, anything which increases the density of > information per line risks hurting readability a little. I once had the "pleasure" of inheriting some code written in K&R style C, where the parameter type declarations were separate from the signature line: void foo(a, b, c) double a; char b; { ... } ANSI C, with inline typing, is far more readable :) When it comes to the readability of function headers with lots and lots of parameters... I'm in the "those are inherently unreadable, even if sometimes an unfortunate necessity" camp :) Reorganising-the-subprocess-module-docs-was-interesting'ly, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Sun Aug 17 10:51:46 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 17 Aug 2014 04:51:46 -0400 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? In-Reply-To: <854mxb5xt3.fsf_-_@benfinney.id.au> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> Message-ID: On 8/17/2014 4:23 AM, Ben Finney wrote: > Guido van Rossum writes: > >> A mostly unrelated issue: there are two different uses of tuples, and >> we need a notation for both. One is a tuple of fixed length with >> heterogeneous, specific types for the elements; for example Tuple[int, >> float]. > > That's the meaning of a tuple data structure, to me. > >> But I think we also need a way to indicate that a function expects (or >> returns) a variable-length tuple with a homogeneous element type. There are also fixed-length homogeneous structures, like points. > Why? What real-world uses are there, where a list won't do the job > adequately? Variable-length homogenous tuples are part of python syntax in multiple places. Tuples can be hashed and put in sets an used as dict keys, lists cannot. Tuple contants are calculated just once when the code is compiled (and typically saved as .pyc). -- Terry Jan Reedy From abarnert at yahoo.com Sun Aug 17 10:52:21 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 01:52:21 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <20140817072602.GN4525@ando> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> <20140817072602.GN4525@ando> Message-ID: <828F3377-EEF0-4AC6-B721-3DB65FBFA062@yahoo.com> On Aug 17, 2014, at 0:26, Steven D'Aprano wrote: > On Sat, Aug 16, 2014 at 11:02:01PM -0700, Andrew Barnert wrote: >> > >> I won't belabor the point, but again: I don't think we need a generic >> list type object, and without it, this entire problem--your only >> remaining problem that isn't a mere stylistic choice--vanishes. > > I don't understand. If there's no list typing object, how do you declare > a variable must be a list and nothing but a list? Or that it returns a > list? You think about it and make sure you really do need a list and nothing but a list. Most of the time (as in all three of the examples given in this thread) this is a mistake. If it's not, then you use List. (Or, if the stdlib doesn't provide that, you have to write one line of code: List = TypeAlias(list), and then you can use it.) If having list[T] is going to be more of an attractive nuisance than a useful feature, and it will be especially attractive and nuisanceful for exactly the same novices who are unlikely to know how to TypeAlias it themselves, why is it a problem to leave it out? >> Even if you drop the idea for [str] and {int: str}, which I agree >> seems unavoidable, I think it may still make sense for (int, str) to >> mean a heterogeneous iterable. > > That makes no sense to me. It looks like a tuple, not a generic iterable > object. Your interpretation has the same problems I discussed above for > [str] notation: it is an arbitrary choice whether (...) means Iterable, > Sequence or ImmutableSequence, and it doesn't fit nicely with other > common uses of parens. See below. > >> Python already has target lists, argument lists, parameter lists, and >> expression lists that all have the same syntax as tuples or a superset >> thereof, but don't define tuples. In (a, b) = zip(c, d), neither (a, >> b) nor (c, d) is a tuple, and I don't think anyone is confused by >> that. > > Ha , you've obviously stopped reading the "Multi-line with statement" > thread on Python-Dev :-) OK, granted, only 4 of the 5 attempts to reuse the comma-separated lists in Python have been 100% successful. Still not a bad batting average. >> So, why can't def foo(spam: (int, str)) mean that spam is an >> iterable of an int and a str, in exactly the same way that the >> assignment statement means that a and b are assigned the result of >> unpacking the iterable returned by zip when called with c and d? > > But a, b = zip(c, d) requires that there be exactly two elements, not > some unspecified number. And spam:(int, str) requires that there be exactly two elements (and that the first be an int and the second a str), not some unspecified number. How is that any different? > To me, spam:(int, str) has a natural interpretation that spam can be > either an int or a str, not an Iterable or Sequence or even a tuple. OK, I see the parallel there with exception statements now that you mention it. But almost anywhere else in Python, a comma-separated list is a sequence of values, targets, parameters, etc., not a disjunction. The obvious way to spell what you want here is "int | str" (and the fact that it was independently suggested three times on this thread and no other alternatives have been suggested until now makes me feel pretty confident that it really is the obvious way). Of course there is a _different_ alternative that could be borrowed from some of the typed functional languages: int * str. But I don't think that's at all obvious to a Python reader. From greg.ewing at canterbury.ac.nz Sun Aug 17 11:22:45 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 17 Aug 2014 21:22:45 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <53F07465.1070103@canterbury.ac.nz> Guido van Rossum wrote: > Perhaps a thornier issue is how mypy should handle decorators that > manipulate the signature or annotations of the function they wrap. But I > think the only reasonable answer here can be that mypy must understand > what decorators do if it wants to have any chance at type-checking > decorated functions. Seems to me the only way to do that in general is to execute the decorators. That means importing everything the decorators depend on and probably running at least the top-level module code. Is executing arbitrary code at type-checking time really desirable? -- Greg From greg.ewing at canterbury.ac.nz Sun Aug 17 11:33:13 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 17 Aug 2014 21:33:13 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <53F076D9.4020807@canterbury.ac.nz> Stefan Behnel wrote: > However, it does not support protocols, so it still needs > something that allows us to say Iterable(int) in some way. Just had a thought -- does mypy provide a way to express a type that supports more than one protocol? E.g. can you say that something must be both Iterable and Hashable? -- Greg From lukasz at langa.pl Sun Aug 17 11:41:17 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 02:41:17 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl> On Aug 16, 2014, at 10:03 PM, Guido van Rossum wrote: > I'd like to summarize the main issues that have come up. As an experiment, I'm not changing the subject, but I am still not quoting anything in particular. Only two issues (or issue clusters) really seem contentious: > > (1) Should the function annotation syntax (eventually) be reserved for type annotations in a standard syntax? Or can multiple different uses of annotations coexist? And if they can, how should a specific use be indicated? (Also, some questions about compile-time vs. run-time use.) Consider what Stefan Behnel is proposing: using function annotations for types by default, but in the presence of a dictionary, search for type in the 'type' key. This is very nice, provides a way to be concise if possible, and generic, if needed. My suggestion: we should support that. > All in all I prefer the mypy syntax, despite being somewhat more verbose and requiring an import, with one caveat: I agree that it would be nicer if the mypy abstract collection types were the same objects as the ABCs exported by collections.abc. Good :) If the functionality will be implemented in the ABCs, what is the purpose of the typing module? My suggestion: if the functionality will be implemented in the ABCs, there's no need to introduce the "typing" module. We can back-port the new ABCs, for sure, but for Python 3.5 `collections` is enough (already has aliases to collections.abc0. > I'm not quite sure whether we should also change the concrete collection types from List, Dict, Set, Tuple to list, dict, set, tuple; the concrete types are so ubiquitous that I worry that there may be working code out there that somehow relies on the type objects themselves not being subscriptable. While unlikely, such code can exist in the wild. That being said, I think builtins should support the one-obvious-way-to-do-it syntax for generics, if only for uniformity. Please note that there also can be code in the future that relies on type objects not to support binary-or. We will still need to add this, though, a type union of (int | str) or (str | None) will be a common thing. My suggestion: add __getitem__ and __or__/__ror__ to both builtins and ABCs. Steven D'Aprano touches on an interesting point that list[int] will be tempting for users because Iterable[int] is both longer and requires an import. We could extend PEP 8 to talk about typing and how people should think about introducing hints, but I think Steven is generally right: list[int] will sadly win. The real reason that we'll see list[T] everywhere is that Iterable[str] == str and Sequence[str] == str. Whoever will try and fail to use those abstract types to specify a collection of strings, but *not* a single string, will migrate to using concrete data types. And that's such a common use case! AFAIK, there is no good workaround at the moment. The solution for that, which we sadly can't implement, would be to make strings non-iterable. My suggestion: two new ABCs, let me temporarily call them StrictIterable and StrictSequence. Those would return False for issubclass(str, ...). My other suggestion: deprecate iterating over strings (and possibly bytes, too?). I'm not saying "remove", but just officially say "this was a bad idea, don't use it". > A mostly unrelated issue: there are two different uses of tuples, and we need a notation for both. One is a tuple of fixed length with heterogeneous, specific types for the elements; for example Tuple[int, float]. But I think we also need a way to indicate that a function expects (or returns) a variable-length tuple with a homogeneous element type. Perhaps we should call this type frozenlist, analogous to frozenset (and it seems there's a proposal for frozendict making the rounds as well). On one hand, not being able to represent variable-length homogeneous tuples in the type hints will be a strong signal that this usage of tuples is disputable. On the other hand, we might *need* to support this to be compatible with existing framework code in the wild. frozenlists would be a nice, explicit solution for that but sadly they'd be a new collection, so no today's code returning/accepting var-length tuples would use that. My suggestion: tuple[int, ...] For uniformity, we would also accept this form for other iterables, I suppose. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Aug 17 11:41:52 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 19:41:52 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <828F3377-EEF0-4AC6-B721-3DB65FBFA062@yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> <20140817072602.GN4525@ando> <828F3377-EEF0-4AC6-B721-3DB65FBFA062@yahoo.com> Message-ID: <20140817094152.GP4525@ando> On Sun, Aug 17, 2014 at 01:52:21AM -0700, Andrew Barnert wrote: > On Aug 17, 2014, at 0:26, Steven D'Aprano wrote: > > > On Sat, Aug 16, 2014 at 11:02:01PM -0700, Andrew Barnert wrote: > >> > > > >> I won't belabor the point, but again: I don't think we need a generic > >> list type object, and without it, this entire problem--your only > >> remaining problem that isn't a mere stylistic choice--vanishes. > > > > I don't understand. If there's no list typing object, how do you declare > > a variable must be a list and nothing but a list? Or that it returns a > > list? > > You think about it and make sure you really do need a list and nothing > but a list. Most of the time (as in all three of the examples given in > this thread) this is a mistake. If it's not, then you use List. (Or, > if the stdlib doesn't provide that, you have to write one line of > code: List = TypeAlias(list), and then you can use it.) Ah, that is the point I missed. You think that the stdlib shouldn't provide a standard typing object for lists, but that people should just create their own if they need it. Okay, but I don't understand why you're singling out lists. If you want to propose providing only abstract classes (Sequence, Mapping, etc.) and not concrete classes (list, dict, etc.) by default, that makes some sense to me. But I don't understand including typing.Dict as an alias for dict (say) but not List. > If having list[T] is going to be more of an attractive nuisance than a > useful feature, and it will be especially attractive and nuisanceful > for exactly the same novices who are unlikely to know how to TypeAlias > it themselves, why is it a problem to leave it out? There are at least three scenarios: (1) Built-ins can be used directly in static type annotations: x:list[dict] This has the advantage of not needing special names, but the disadvantage of encouraging lazy programmers to specify concrete types when they should be using abstract Sequence[Mapping]. (2) Built-ins *cannot* be used, you have to import them from typing: from typing import List, Dict x:List[Dict] The advantage is that since you have to do an import anyway, it is not much more effort to Do The Right Thing: from typing import Sequence, Mapping x:Sequence[Mapping] (3) And the final scenario, the one which confuses me, but seems to be what you are suggesting: you can use the built-ins, *but not list*, and there is no List to import either: from typing import Sequence x:Sequence[dict] I don't understand the advantage of this. [...] > >> So, why can't def foo(spam: (int, str)) mean that spam is an > >> iterable of an int and a str, in exactly the same way that the > >> assignment statement means that a and b are assigned the result of > >> unpacking the iterable returned by zip when called with c and d? > > > > But a, b = zip(c, d) requires that there be exactly two elements, not > > some unspecified number. > > And spam:(int, str) requires that there be exactly two elements (and > that the first be an int and the second a str), not some unspecified > number. How is that any different? Ah, that's what I didn't understand. I thought you meant an iterable of either ints or strs, without requiring a fixed number of them. I must admit, I just assumed that (based on the example of isinstance, and general Python practice), unions of types would be represented as a tuple, but I see that mypy uses Union[int, str]. In other words, I was thinking: Iterable[Union[int, str]] == Iterable[(int, str)] and thought you wanted to drop the Iterable[ ] and just be left with the (int, str). > > To me, spam:(int, str) has a natural interpretation that spam can be > > either an int or a str, not an Iterable or Sequence or even a tuple. > > OK, I see the parallel there with exception statements now that you > mention it. > > But almost anywhere else in Python, a comma-separated list is a > sequence of values, targets, parameters, etc., not a disjunction. The > obvious way to spell what you want here is "int | str" Which mypy spells as Union[ ]. http://mypy-lang.org/tutorial.html -- Steven From lukasz at langa.pl Sun Aug 17 11:44:32 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 02:44:32 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F076D9.4020807@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> Message-ID: <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl> On Aug 17, 2014, at 2:33 AM, Greg Ewing wrote: > Stefan Behnel wrote: >> However, it does not support protocols, so it still needs >> something that allows us to say Iterable(int) in some way. > > Just had a thought -- does mypy provide a way to express > a type that supports more than one protocol? E.g. can you > say that something must be both Iterable and Hashable? That would be union types. Current syntax: Union[Iterable[int], Hashable] Proposed more concise syntax: Iterable[int] | Hashable -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Aug 17 11:45:36 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 19:45:36 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F076D9.4020807@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> Message-ID: <20140817094536.GQ4525@ando> On Sun, Aug 17, 2014 at 09:33:13PM +1200, Greg Ewing wrote: > Stefan Behnel wrote: > >However, it does not support protocols, so it still needs > >something that allows us to say Iterable(int) in some way. > > Just had a thought -- does mypy provide a way to express > a type that supports more than one protocol? E.g. can you > say that something must be both Iterable and Hashable? mypy has Union[Iterable, Hashable], but that would mean anything iterable, or anything hashable, but not necessarily both at the same time. I don't see anything that says it must support both, but I've only gone through the tutorial: http://mypy-lang.org/tutorial.html -- Steven From lukasz at langa.pl Sun Aug 17 11:49:59 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 02:49:59 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl> Message-ID: On Aug 17, 2014, at 2:44 AM, ?ukasz Langa wrote: > On Aug 17, 2014, at 2:33 AM, Greg Ewing wrote: > >> Stefan Behnel wrote: >>> However, it does not support protocols, so it still needs >>> something that allows us to say Iterable(int) in some way. >> >> Just had a thought -- does mypy provide a way to express >> a type that supports more than one protocol? E.g. can you >> say that something must be both Iterable and Hashable? > > That would be union types. Current syntax: > Union[Iterable[int], Hashable] > > Proposed more concise syntax: > Iterable[int] | Hashable Ah, scratch that. What Greg asked about would be Iterable[int] & Hashable ;-) Don't know if this is supported but makes sense. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Aug 17 11:51:52 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 19:51:52 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl> Message-ID: <20140817095152.GR4525@ando> On Sun, Aug 17, 2014 at 02:44:32AM -0700, ?ukasz Langa wrote: > On Aug 17, 2014, at 2:33 AM, Greg Ewing wrote: > > > Stefan Behnel wrote: > >> However, it does not support protocols, so it still needs > >> something that allows us to say Iterable(int) in some way. > > > > Just had a thought -- does mypy provide a way to express > > a type that supports more than one protocol? E.g. can you > > say that something must be both Iterable and Hashable? > > That would be union types. Current syntax: > Union[Iterable[int], Hashable] I don't think so. The mypy tutorial says: Use the Union[...] type constructor to construct a union type. For example, the type Union[int, str] is compatible with both integers and strings. You can use an isinstance check to narrow down the type to a specific type ... which implies that x:Union[a, b] means: isinstance(x, (a, b)) rather than: isinstance(x, a) and isinstance(x, b) which I think is what Greg is asking for. -- Steven From nicholas.cole at gmail.com Sun Aug 17 11:52:52 2014 From: nicholas.cole at gmail.com (Nicholas Cole) Date: Sun, 17 Aug 2014 10:52:52 +0100 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <20140817083453.GO4525@ando> Message-ID: On Sun, Aug 17, 2014 at 9:50 AM, Nick Coghlan wrote: > On 17 August 2014 18:34, Steven D'Aprano wrote: >> >> Declaring types in the function parameter list is very common, in many >> languages. If it were *that* much harder to read, languages wouldn't >> keep using it. (Not many languages follow Forth or APL syntax.) Perhaps >> because I learned to program in Pascal, I find the annotation syntax >> very easy to read, but, yes, anything which increases the density of >> information per line risks hurting readability a little. > > I once had the "pleasure" of inheriting some code written in K&R style > C, where the parameter type declarations were separate from the > signature line: > > void foo(a, b, c) > double a; > char b; > { > ... > } > > ANSI C, with inline typing, is far more readable :) I think you've put your finger on it. It comes down to a disagreement over density of information. You'd like everything in "one pass" as it were. The way my brain is wired, I read your example here as: "This function takes three (non-named) parameters, the a is a double, the b is a char." I find that faster to process than the inline alternative, especially when the alternative is optional. With meaningful parameter names it would be even easier. If I re-write your example: > void foo(double a, void b, c) My brain takes an extra fraction of a second to count the number of arguments. Syntax highlighting would help, of course. Some of this can be very subtle. For example, it's important for readability that in Python positional parameters are all specified, then keyword ones, so the brain doesn't have to keep switching backwards and forwards. In C, of course, everything has to be typed, and so I can see it makes sense to put it all inline. But what if you are mixing the two, and some are typed and some not? I think it is all going to get very dense and hard to read. The same tension occurs in natural languages. I tend to write quite dense English prose, myself. My editors always want me to write less densely, and over time I've come to see that they are right! Dense prose is fine for the specialist, but it doesn't help the student or the casual reader. N. From willvarfar at gmail.com Sun Aug 17 12:13:24 2014 From: willvarfar at gmail.com (willvarfar at gmail.com) Date: Sun, 17 Aug 2014 12:13:24 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: When discussed on reddit programming ("proggit" as its called), there were plenty of people saying they really didn't like the mypy syntax: http://www.reddit.com/r/programming/comments/2disob/proposal_for_python_type_annotations_from_guido/ (Declaration: I am the author of obiwan https://pypi.python.org/pypi/obiwan and I got a lot of upvotes on that proggit thread when I promoted the obiwan style instead) On 14 Aug 2014 13:16, "willvarfar at gmail.com" wrote: > I fully support formalizing Python 3's annotations for type checking. > > I wrote - and use daily - my own type checker called obiwan > https://pypi.python.org/pypi/obiwan > > Its a runtime type checker, and if enabled will check and enforce > types on every call. > > I support the wider adoption and standardization of static type > checkers, but runtime checkers are still wanted for very dynamic code > and for checking external data e.g. I use obiwan for validating JSON. > > One small detail is that I feel the obiwan annotations are more > pythonic than the mypy examples given. > > E.g. instead of: > > from typing import List, Dict > > def word_count(input: List[str]) -> Dict[str, int]: > ... > > It would be: > > def word_count(input: [str]) -> {str, int}: > ... > > Obiwan does not check types within functions; I was unwilling to try > and overload comments! You can invoke obiwan to check things > explicitly, but these are more as assertions. > > Anyway, when I look at the mypy in-function annotations (where > comments are overloaded) I am cautious. It would be far nicer if we > had annotations as part of the language instead, e.g. instead of: > > result = {} #type: Dict[str, int] > > It would be: > > result = {} -> {str, int} > > where we use the -> arrow again. I can see pros and cons for any > implementation (as we'd want the annotation to be both to declare a > type and to check a type, and want the annotation to be attached to > the variable forever etc) so this would need a full PEP treatment and > possibly be configurable as asserts are. > > But proper annotation support in the language rather than overloading > comments would definitely be my preference. > > /Will > > /Will > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Aug 17 12:20:59 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 20:20:59 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <20140817102059.GS4525@ando> On Sat, Aug 16, 2014 at 10:03:48PM -0700, Guido van Rossum wrote: > I'd like to summarize the main issues that have come up. As an experiment, > I'm not changing the subject, but I am still not quoting anything in > particular. Only two issues (or issue clusters) really seem contentious: > > (1) Should the function annotation syntax (eventually) be reserved for type > annotations in a standard syntax? Reserved, as in alternatives are prohibited? No. But assumed to be types by default? I think so. > Or can multiple different uses of > annotations coexist? And if they can, how should a specific use be > indicated? (Also, some questions about compile-time vs. run-time use.) Determining whether annotations should be interpreted as type annotations or not needs to be possible both at compile-time and run-time. I suggest a special decorator, imported from the typing module: from typing import skip_types # or some better name? @skip_types(marker) # see below for the purpose of marker @another_decorator # the order of decorators doesn't matter def function(x:"this is not a type")->[int]: ... At compile-time, static typing tools should treat any function decorated by skip_types as if it were dynamic. The skip_types decorator also writes marker to the __annotations__ dict, using a key which cannot be used as an identifier. Say, "+mark+". The purpose of the marker is so that other tools can determine whether the annotations are aimed at them. def handle_annotations(func): if func.__annotations__.get("+mark+", None) is MyMarker: do_stuff(func.__annotations__) else: pass Any time a function is not decorated by skip_types (or whatever name it has), or doesn't have that "+mark+" key in the __annotations__ dict, static type checking tools are entitled to assume the annotations are used for typing. > (2) For type annotations, should we adopt (roughly) the mypy syntax or the > alternative proposed by Dave Halter? This uses built-in container notations > as a shorthand, e.g. {str: int} instead of Dict[str, int]. This also > touches on the issue of abstract vs. concrete types (e.g. iterable vs. > list). I prefer the mypy syntax. > Regarding (1), I continue to believe that we should eventually reserve > annotations for types, to avoid confusing both humans and tools, but I > think there's nothing we have to do in 3.5 -- 3.5 must preserve backward > compatibility, and we're not proposing to give annotations any new > semantics anyway -- the actual changes to CPython are limited to a new > stdlib module (typing) and some documentation. > > Perhaps a thornier issue is how mypy should handle decorators that > manipulate the signature or annotations of the function they wrap. But I > think the only reasonable answer here can be that mypy must understand what > decorators do if it wants to have any chance at type-checking decorated > functions. Hmmm. Anything the decorators do to the annotations will be at runtime, so is it reasonable to say that the static typing tool will only operate on the annotations available at compile time? That is, given: @mangle_annotations def spam(x:int)->List[str]: ... the type checker is expected to use the annotations seen at compile-time, no matter what the mangle_annotations decorator happens to do at run-time. Otherwise, the type-checker needs to be a full Python interpreter, in order to see what mangle_annotations does. And that could be an intractible problem: def mangle_annotations(func): if random.random() < 0.5: func.__annotations__['x'] = List[str] else: func.__annotations__['x'] = float I don't see how any type-checker is supposed to take that into account. (1) Compile-time type checkers should ignore the decorator and just use the annotations available at compile-time; (2) Run-time analysis tools should use the annotations available at run-time; (3) If they happen to be different, oh well, consenting adults. -- Steven From davidhalter88 at gmail.com Sun Aug 17 12:26:35 2014 From: davidhalter88 at gmail.com (Dave Halter) Date: Sun, 17 Aug 2014 12:26:35 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: Message-ID: 2014-08-16 16:30 GMT+02:00 Andrew Barnert : > On Aug 16, 2014, at 6:00, Dave Halter wrote: > > *Appendix (My Proposal)* > > My proposal (as discussed and evolved with a few good people at > EuroPython) for containers would look something like this: > > def foo(index: int, x: [float], y: {int: str}) -> (float, str): > return x[index], y[index] > > The "default" containers (set, list, dict and tuple) would just serve as a > way of specifying containers. This makes a lot of things less complicated: > > - It's easier to understand (everybody knows builtin types). It also feels > natural to me. The example above covers also almost all variations. A tuple > could be expanded by using the ellipsis: `(int, object, ..., float)`. > - No imports needed. People are more likely to use it, if they don't have > to import typing all the time. This is important for static analysis, > people are only likely to use it if it's easier than writing docstrings > with type information. > - Argument clinic could use the same. The standard library quite often > doesn't accept abstract data types, btw. > - It's what people sometimes use in docstrings: ``@rtype: (float, str)`` > or ``:rtype: (float, str)``. > > I know this doesn't solve the duck typing issue, but if you look at > real-life Python code bases, there are very few instances of actually > implementing a ``Mapping``, etc. > > > For "Mapping", maybe (although anyone who uses some form of tree-based > mapping, either to avoid hash-collision attacks or because he needs > sorting, may disagree). > > But for "etc.", people implement them all the time. Especially "Iterable" > and "Callable". And, even some of the ones that people don't implement > often, they use often, implicitly or otherwise, like TextIOBase. > > Anything that encourages people to restrict their code to only working on > lists instead of iterables would be a huge step backward to Python 2.2. And > your argument for it ("everybody knows builtin types") implies that's > exactly what you're expecting with this proposal. > > However, there's an easy way around this: just let [spam] in your syntax > mean what Iterable[spam] means in MyPy's. If someone really needs to > declare that they will only accept a list or whatever, that's the uncommon > case, and can be written more verbosely. And I don't see any reason why > this can't be added on top of genericizing the ABCs a la MyPy. > > Meanwhile, your tuple doesn't fit the same pattern as the others, because > it's explicitly fixed-size and heterogeneous. And I think this is a good > thing. And I think it's pretty close to what MyPy does with an expression > list of types already; if not, it seems like what MyPy _should_ do. If I > loop over a zip of a [str] and an [int], the loop variable is a (str, int), > not a Tuple[str or int]. > > So, making it easier to specify generic builtins is a bad idea, but using > builtins to make it easier to specify common types is a great idea. > Very interesting ideas. I'm not sure if this is the right way to go, but it's definitely an option. It could confuse people, if they cannot use `isinstance(x, list)` anymore. But then again it would just be a really neat way of expressing abstract types. One other confusion might come from the fact that `-> list` would not be an abstract data type, but `[str]` will. It's ok, but also not very obvious. I've also thought about adding something like an `-> abstract([abstract(str)])` which would return the abstract data types for builtins. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Aug 17 12:31:28 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 17 Aug 2014 22:31:28 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <53F08480.6060305@canterbury.ac.nz> Nicholas Cole wrote: > On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano wrote: > >> def __init__(self, >> description:str, sec_code:str, >> vendor_name:str, vendor_inv_num:str, >> vendor_rtng:str, vendor_acct:str, >> transaction_code:str, vendor_acct_type:str, >> amount:int, payment_date:Any)->None: > >I had to stare at that block of > code for a long time to see how many and what type of arguments it > called. Pascal's function signature syntax had a nice feature that everyone else seems to have forgotten about. If you had multiple parameters of the same type, you only had to write the type once: procedure Init(description, sec_code, vendor_name, vendor_inv_num, vendor_rtng, vendor_acct, transaction_code, vendor_acct_type, amount: str; payment_date: Any) Disappointingly, Python's annotations make the same blunder as C, and most other languages since, in requiring each parameter to have its own individual annotation. -- Greg From andrey.vlasovskikh at gmail.com Sun Aug 17 12:35:57 2014 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Sun, 17 Aug 2014 14:35:57 +0400 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl> Message-ID: <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com> 2014-08-17, 13:41, ?ukasz Langa wrote: > On Aug 16, 2014, at 10:03 PM, Guido van Rossum wrote: > >> All in all I prefer the mypy syntax, despite being somewhat more verbose and requiring an import, with one caveat: I agree that it would be nicer if the mypy abstract collection types were the same objects as the ABCs exported by collections.abc. > > Good :) If the functionality will be implemented in the ABCs, what is the purpose of the typing module? > > My suggestion: if the functionality will be implemented in the ABCs, there's no need to introduce the "typing" module. We can back-port the new ABCs, for sure, but for Python 3.5 `collections` is enough (already has aliases to collections.abc0. -1 for collections.abc classes, +1 for mypy's typing classes. There is a problem in static analysis of current types that are instances of abc.ABCMeta or types that just define their own __instancecheck__ / __subclasscheck__. Static analyzers cannot infer in general case what attributes of an instance / subclass do these methods check, because their body can be arbitrarily complex. Mypy's typing.Protocol subclasses are much easier to analyze statically, since they are required to explicitly define abstract methods as function defintions inside the class body. Current collectoins.abc classes do define their methods explicitly too, so it seems that at least these classes are fine. But their inheritors don't have to do it, they may just override __subclasshook__. And promoting abc.ABCMeta-based ABCs would mean that not all ABCs can be used as static type annotations. -- Andrey Vlasovskikh Web: http://pirx.ru/ From andrey.vlasovskikh at gmail.com Sun Aug 17 12:36:07 2014 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Sun, 17 Aug 2014 14:36:07 +0400 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: <84F83885-BD67-45E0-9521-7742546E8279@gmail.com> 2014-08-17, 9:03, Guido van Rossum wrote: > My first concern is that these expressions are only unambiguous in the context of function annotations. I want to promote the use of type aliases, and I think in general a type alias should behave similarly to an ABC. In particular, I think that any object used to represent a type in an annotation should itself be a type object (though you may not be able to instantiate it), and e.g. [int] doesn't satisfy that requirement. Without this, it would be difficult to implement isinstance() and issubclass() for type aliases -- and while we could special-case lists, sets and dicts, using a tuple *already* has a meaning! Having type annotations as type objects sounds good. The fact that we can use isinstance() and issubclass() for all type annotations would provide some level of compatibility between static type checking and potential dynamic type checking: if "x: " then "isinstance(x, )". Note, that not all type annotations of mypy are currently type objects. Probably this should be fixed. -- Andrey Vlasovskikh Web: http://pirx.ru/ From abarnert at yahoo.com Sun Aug 17 12:41:17 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 03:41:17 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <20140817094152.GP4525@ando> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> <20140817072602.GN4525@ando> <828F3377-EEF0-4AC6-B721-3DB65FBFA062@yahoo.com> <20140817094152.GP4525@ando> Message-ID: <7BC3BC4C-F209-4509-90E3-559BE6D15FE1@yahoo.com> On Aug 17, 2014, at 2:41, Steven D'Aprano wrote: > On Sun, Aug 17, 2014 at 01:52:21AM -0700, Andrew Barnert wrote: >> On Aug 17, 2014, at 0:26, Steven D'Aprano wrote: >> >>> On Sat, Aug 16, 2014 at 11:02:01PM -0700, Andrew Barnert wrote: >>> >>>> I won't belabor the point, but again: I don't think we need a generic >>>> list type object, and without it, this entire problem--your only >>>> remaining problem that isn't a mere stylistic choice--vanishes. >>> >>> I don't understand. If there's no list typing object, how do you declare >>> a variable must be a list and nothing but a list? Or that it returns a >>> list? >> >> You think about it and make sure you really do need a list and nothing >> but a list. Most of the time (as in all three of the examples given in >> this thread) this is a mistake. If it's not, then you use List. (Or, >> if the stdlib doesn't provide that, you have to write one line of >> code: List = TypeAlias(list), and then you can use it.) > > Ah, that is the point I missed. You think that the stdlib shouldn't > provide a standard typing object for lists, but that people should just > create their own if they need it. > > Okay, but I don't understand why you're singling out lists. If you want > to propose providing only abstract classes (Sequence, Mapping, etc.) > and not concrete classes (list, dict, etc.) by default, that makes > some sense to me. But I don't understand including typing.Dict as > an alias for dict (say) but not List. I'm not singling out lists. I referred to tuple in exactly the same way in the same message. I didn't mention dict, frozenset, etc. because I'd already given the full argument a few hundred messages ago and I didn't think anyone wanted to read it again. But the "if" sentence in your paragraph is a good summary of the whole thing. >> If having list[T] is going to be more of an attractive nuisance than a >> useful feature, and it will be especially attractive and nuisanceful >> for exactly the same novices who are unlikely to know how to TypeAlias >> it themselves, why is it a problem to leave it out? > > There are at least three scenarios: > > (1) Built-ins can be used directly in static type annotations: > > x:list[dict] > > This has the advantage of not needing special names, but the > disadvantage of encouraging lazy programmers to specify concrete > types when they should be using abstract Sequence[Mapping]. > > (2) Built-ins *cannot* be used, you have to import them from typing: > > from typing import List, Dict > x:List[Dict] > > The advantage is that since you have to do an import anyway, > it is not much more effort to Do The Right Thing: > > from typing import Sequence, Mapping > x:Sequence[Mapping] > > (3) And the final scenario, the one which confuses me, but seems > to be what you are suggesting: you can use the built-ins, > *but not list*, and there is no List to import either: > > from typing import Sequence > x:Sequence[dict] > > I don't understand the advantage of this. As explained above, I'm suggesting (2), not (3). > [...] >>>> So, why can't def foo(spam: (int, str)) mean that spam is an >>>> iterable of an int and a str, in exactly the same way that the >>>> assignment statement means that a and b are assigned the result of >>>> unpacking the iterable returned by zip when called with c and d? >>> >>> But a, b = zip(c, d) requires that there be exactly two elements, not >>> some unspecified number. >> >> And spam:(int, str) requires that there be exactly two elements (and >> that the first be an int and the second a str), not some unspecified >> number. How is that any different? > > Ah, that's what I didn't understand. I thought you meant an iterable of > either ints or strs, without requiring a fixed number of them. Right. The "paradigm case" for tuples is as fixed-length, heterogeneous collections, where each index has a specific semantic meaning (and therefore type). The problem is that, at least in Python, tuples are _also_ used as general-purpose immutable sequences--and, in a few cases, that's specifically enshrined in syntax (e.g., the exception types in an except statement) or builtins (e.g., the arguments to str.__mod__), so we can't just ignore that. My suggestion is that (int, str) means the first (what you'd call int * str if you wanted your language to look more like type theory than something a normal human would write), so Tuple[int] works exactly like every other generic type: it's a homogenous collection of ints. >>> To me, spam:(int, str) has a natural interpretation that spam can be >>> either an int or a str, not an Iterable or Sequence or even a tuple. >> >> OK, I see the parallel there with exception statements now that you >> mention it. >> >> But almost anywhere else in Python, a comma-separated list is a >> sequence of values, targets, parameters, etc., not a disjunction. The >> obvious way to spell what you want here is "int | str" > > Which mypy spells as Union[ ]. > > http://mypy-lang.org/tutorial.html Yes, but multiple people in this thread have suggested spelling it as int | str, nobody's objected, and both Jukka and Guido have given at least tentative assent. (I realize there's a whole lot of messages to read through and remember. And I could easily have missed an argument against this syntax just as you missed the suggestions for it.) From ben+python at benfinney.id.au Sun Aug 17 12:53:18 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 17 Aug 2014 20:53:18 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> Message-ID: <85zjf34cbl.fsf@benfinney.id.au> Terry Reedy writes: > On 8/17/2014 4:23 AM, Ben Finney wrote: > > Guido van Rossum writes: > > > >> A mostly unrelated issue: there are two different uses of tuples, and > >> we need a notation for both. One is a tuple of fixed length with > >> heterogeneous, specific types for the elements; for example Tuple[int, > >> float]. > > > > That's the meaning of a tuple data structure, to me. > > > >> But I think we also need a way to indicate that a function expects > >> (or returns) a variable-length tuple with a homogeneous element > >> type. > > There are also fixed-length homogeneous structures, like points. I assume you mean where the sequence is something like ?(x, y, z)? where each position has a meaning specific to that position. That's not homogeneous as I understood this usage, because the psitions are not homogeneous in meaning. The number 7.03 has very different meaning in the first, second, or third positions. A homogeneous sequence would imply there are deliberately *no* specific meanings to each position. The value 7.03 would have the same semantic value at any position in the sequence. So a fixed-length sequence where each position implies a special meaning is a heterogeneous sequence, and I agree that's an excellent use for a tuple. > > Why? What real-world uses are there, where a list won't do the job > > adequately? > > Variable-length homogenous tuples are part of python syntax in > multiple places. > > Tuples can be hashed and put in sets an used as dict keys, lists > cannot. Tuple contants are calculated just once when the code is > compiled (and typically saved as .pyc). Okay, so these are not for the semantic purpose a tuple is for. I think a putative ?frozenlist? type is best for that, to keep the heterogeneous implication of a tuple separate from the homogeneous implication of a list. -- \ ?I prayed for twenty years but received no answer until I | `\ prayed with my legs.? ?Frederick Douglass, escaped slave | _o__) | Ben Finney From abarnert at yahoo.com Sun Aug 17 12:53:47 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 03:53:47 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F076D9.4020807@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> Message-ID: <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com> On Aug 17, 2014, at 2:33, Greg Ewing wrote: > Stefan Behnel wrote: >> However, it does not support protocols, so it still needs >> something that allows us to say Iterable(int) in some way. > > Just had a thought -- does mypy provide a way to express > a type that supports more than one protocol? E.g. can you > say that something must be both Iterable and Hashable? The obvious way that already works (with MyPy, and also with ABCs for isinstance checking): class HashableIterable(Iterable, Hashable): pass def spam(a: HashableIterable): pass But, there's no reason typing.py couldn't add a wrapper that does this automatically: class Multiple: @staticmethod def __getitem__(*types); return type(types[0])( '_'.join(t.__name__ for t in types), types, {}) def spam(a: Multiple[Iterable, Hashable]): pass And, on analogy with the proposal for | as a shortcut for Union, the base class could add: def __and__(self, other): return Multiple[self, other] def spam(a: Iterable & Hashable): pass From skip at pobox.com Sun Aug 17 13:31:55 2014 From: skip at pobox.com (Skip Montanaro) Date: Sun, 17 Aug 2014 06:31:55 -0500 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <854mxb5xt3.fsf_-_@benfinney.id.au> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> Message-ID: On Sun, Aug 17, 2014 at 3:23 AM, Ben Finney wrote: > I have encountered many uses of ?homogeneous, variable-length sequence? > and every time a Python tuple is used for that, I perceive a Python list > would be better precisely *because* it better indicates that semantic > meaning. Agreed. While "variable-length" doesn't imply mutability, they often seem to go hand-in-hand. Skip From jeanpierreda at gmail.com Sun Aug 17 13:44:31 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 17 Aug 2014 04:44:31 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <854mxb5xt3.fsf_-_@benfinney.id.au> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> Message-ID: On Sun, Aug 17, 2014 at 1:23 AM, Ben Finney wrote: > I have encountered many uses of ?homogeneous, variable-length sequence? > and every time a Python tuple is used for that, I perceive a Python list > would be better precisely *because* it better indicates that semantic > meaning. > > I'd like to know how you think that's not true, and what real-world code > makes you think so. isinstance is real world code that for the second parameter accepts types and (recursively) tuples of any length of things it accepts. It doesn't accept lists, because then it would need to check for cycles. -- Devin From jeanpierreda at gmail.com Sun Aug 17 14:05:24 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 17 Aug 2014 05:05:24 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> Message-ID: On Sun, Aug 17, 2014 at 4:44 AM, Devin Jeanpierre wrote: > On Sun, Aug 17, 2014 at 1:23 AM, Ben Finney wrote: >> I have encountered many uses of ?homogeneous, variable-length sequence? >> and every time a Python tuple is used for that, I perceive a Python list >> would be better precisely *because* it better indicates that semantic >> meaning. >> >> I'd like to know how you think that's not true, and what real-world code >> makes you think so. > > isinstance is real world code that for the second parameter accepts > types and (recursively) tuples of any length of things it accepts. > > It doesn't accept lists, because then it would need to check for cycles. I was going to leave it at that -- the fact that a builtin works this way is an argument in favor of any standardized typing syntax supporting it -- but maybe it deserves justification, too. The key question is, given two values A and B that are valid second parameters to isinstance, how do you combine them into one valid second parameter, which forms the union of A and B? if A and B were always tuples, you could do A + B (isinstance predates sets). But we want to be able to do isinstance(x, SomeType), so A and B can't always be tuples. So Python adopts the convention that you can do (A, B) and, no matter what they are, isinstance(x, (A, B)) holds if and only if (isinstance(x, A) or isinstance(x, B)). As it happens we could use frozensets instead, but, oops. Given that frozensets are out of the question (in this case, because of dating, in other cases, because of unhashability or whatever) using sequences seems reasonable, and specifically using tuples so that we can avoid cycle checks (a messy algorithm) also seems entirely reasonable. I've also seen people use tuples to emphasize that the sequence is not mutated, but there's no reason to require that from a type signature in that case. -- Devin From stefan_ml at behnel.de Sun Aug 17 14:10:27 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 17 Aug 2014 14:10:27 +0200 Subject: [Python-ideas] RFC: Multiple Dispatch In-Reply-To: References: Message-ID: <53F09BB3.5090508@behnel.de> Antoine Pitrou schrieb am 16.08.2014 um 15:34: > Le 15/08/2014 14:01, Guido van Rossum a ?crit : >> Please do write about non-toy examples! > > Are you looking for examples using the multipledispatch library, or > multiple dispatch in general? > > As for multiple dispatch in general, Numba uses something which is morally > one in order to select the right specialization of, say, an operator (for > example to choose amongst '+ between int and int', '+ between > numpy.datetime64 and numpy.timedelta64', '+ between numpy.timedelta64 and > numpy.timedelta64', etc.). Similar for Cython's "fused types", i.e. compile time generics. It generates specialised function implementations for a cross product of the type sets used in the signature and then calls the right specialisation based on the input types at runtime, when called from Python code. Understands input buffer types and C types in signatures, though, so it's a bit more complex than the "average Python generic function dispatch" (same applies to Numba, I guess). Stefan From steve at pearwood.info Sun Aug 17 14:31:27 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 17 Aug 2014 22:31:27 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <20140817123127.GT4525@ando> On Sun, Aug 17, 2014 at 12:13:24PM +0200, willvarfar at gmail.com wrote: > When discussed on reddit programming ("proggit" as its called), there were > plenty of people saying they really didn't like the mypy syntax: > > http://www.reddit.com/r/programming/comments/2disob/proposal_for_python_type_annotations_from_guido/ > > (Declaration: I am the author of obiwan https://pypi.python.org/pypi/obiwan > and I got a lot of upvotes on that proggit thread when I promoted the > obiwan style instead) My thoughts on the obiwan style? Quicker, easier, more seductive. Easily they flow. But once you start down the dark path, forever will it dominate your destiny. *wink* Your example: [mypy] def word_count(input: List[str]) -> Dict[str, int]: [obiwan] def word_count(input: [str]) -> {str, int}: Now consider that annotations are just expressions. Inside or outside of a function parameter definition, Dict[str, int] is exactly the same thing: it's an abstract type. I should be able to use it at runtime: isinstance(x, word_count.__annotations__['return']) But obiwan's style (like the old Jedi himself) twists the truth around. The return annotation is only a type from a certain perspective, but in reality it is an instance of a concrete class, not an abstract class. It's not even a dict. {str, int} is a set, but obiwan uses it to represent a dict. (Although maybe that's a typo, and you meant to write {str: int}.) -- Steven From ncoghlan at gmail.com Sun Aug 17 14:43:45 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Aug 2014 22:43:45 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> Message-ID: On 17 August 2014 21:44, Devin Jeanpierre wrote: > On Sun, Aug 17, 2014 at 1:23 AM, Ben Finney wrote: >> I have encountered many uses of ?homogeneous, variable-length sequence? >> and every time a Python tuple is used for that, I perceive a Python list >> would be better precisely *because* it better indicates that semantic >> meaning. >> >> I'd like to know how you think that's not true, and what real-world code >> makes you think so. > > isinstance is real world code that for the second parameter accepts > types and (recursively) tuples of any length of things it accepts. There are a few other cases where tuples are special cased as arguments: - str.__mod__ - str.startswith (ditto for binary sequences) - str.endswith (ditto for binary sequences) >>> "aa".endswith(['a', 'b', 'c']) Traceback (most recent call last): File "", line 1, in TypeError: endswith first arg must be str or a tuple of str, not list Searching the C files for "tuple of" turned up a couple more: * N-dimensional indexing also relies specifically on tuples-of-integers, rather than arbitrary iterators. * dynamic type creation expects to receive the bases as a tuple * the decimal module uses tuples of digits for internal data representations And that inspired recollection of several other cases where mutability would be wrong, because the tuple represents cached information rather than dynamic state: * various "*args" related APIs use "tuple of object" or "tuple of thing" (e.g. attributes of partial objects, internal storage in contextlib.ExitStack when used with arbitrary callbacks) * other introspection related APIs use tuples to report information about inspected objects * namedtuple _fields attributes are a tuple of strings * BaseException.args publishes the full args tuple passed to the constructor str.startswith, str.endswith, isinstance and issubclass use the "implied or" interpretation, everything else does not. In most cases, the immutability conveys relevant semantic information (usually indicating that it's a read-only API). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rymg19 at gmail.com Sun Aug 17 16:09:07 2014 From: rymg19 at gmail.com (Ryan) Date: Sun, 17 Aug 2014 09:09:07 -0500 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F08480.6060305@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F08480.6060305@canterbury.ac.nz> Message-ID: Nimrod has that feature, too, which makes type lists easier on the eyes. Greg Ewing wrote: >Nicholas Cole wrote: >> On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano > wrote: >> >>> def __init__(self, >>> description:str, sec_code:str, >>> vendor_name:str, vendor_inv_num:str, >>> vendor_rtng:str, vendor_acct:str, >>> transaction_code:str, vendor_acct_type:str, >>> amount:int, payment_date:Any)->None: >> >>I had to stare at that block of >> code for a long time to see how many and what type of arguments it >> called. > >Pascal's function signature syntax had a nice feature >that everyone else seems to have forgotten about. If you >had multiple parameters of the same type, you only had >to write the type once: > > procedure Init(description, sec_code, vendor_name, > vendor_inv_num, vendor_rtng, vendor_acct, > transaction_code, vendor_acct_type, amount: str; > payment_date: Any) > >Disappointingly, Python's annotations make the same >blunder as C, and most other languages since, in >requiring each parameter to have its own individual >annotation. > >-- >Greg >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Aug 17 16:30:49 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 18 Aug 2014 00:30:49 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <20140817102059.GS4525@ando> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <20140817102059.GS4525@ando> Message-ID: On Sun, Aug 17, 2014 at 8:20 PM, Steven D'Aprano wrote: > Hmmm. Anything the decorators do to the annotations will be at runtime, > so is it reasonable to say that the static typing tool will only operate > on the annotations available at compile time? > > That is, given: > > @mangle_annotations > def spam(x:int)->List[str]: ... > > the type checker is expected to use the annotations seen at > compile-time, no matter what the mangle_annotations decorator happens to > do at run-time. > > Otherwise, the type-checker needs to be a full Python interpreter, in > order to see what mangle_annotations does. And that could be an > intractible problem: > > def mangle_annotations(func): > if random.random() < 0.5: > func.__annotations__['x'] = List[str] > else: > func.__annotations__['x'] = float > > > I don't see how any type-checker is supposed to take that into account. You give an example of a malicious mangling, but more significant is the naive mangling - wrapping the decorated function in a non-annotated outer function, without using functools.wraps() or equivalent (I'm sure it'd be possible to propagate the annotations through wraps(), so that would take care of a lot of cases). IMO the right handling here is to completely ignore all unrecognized decorators, on the assumption that most decorators should be returning an "equivalently usable" function. I don't, for instance, see real-world examples of decorators that add extra parameters to a function, even though it would be plausible (maybe you have a whole bunch of functions that all take an optional mode parameter, which causes other arguments to be translated automatically by the decorator?). If you're annotating the function, the type checker can assume that that's intended to be correct. ChrisA From stefan_ml at behnel.de Sun Aug 17 16:37:13 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 17 Aug 2014 16:37:13 +0200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F08480.6060305@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F08480.6060305@canterbury.ac.nz> Message-ID: Greg Ewing schrieb am 17.08.2014 um 12:31: > Nicholas Cole wrote: >> On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano wrote: >> >>> def __init__(self, >>> description:str, sec_code:str, >>> vendor_name:str, vendor_inv_num:str, >>> vendor_rtng:str, vendor_acct:str, >>> transaction_code:str, vendor_acct_type:str, >>> amount:int, payment_date:Any)->None: >> >> I had to stare at that block of >> code for a long time to see how many and what type of arguments it >> called. > > Pascal's function signature syntax had a nice feature > that everyone else seems to have forgotten about. If you > had multiple parameters of the same type, you only had > to write the type once: > > procedure Init(description, sec_code, vendor_name, > vendor_inv_num, vendor_rtng, vendor_acct, > transaction_code, vendor_acct_type, amount: str; > payment_date: Any) > > Disappointingly, Python's annotations make the same > blunder as C, and most other languages since, in > requiring each parameter to have its own individual > annotation. The difference is that Pascal requires type declarations whereas they are purely optional in Python. That makes the case that they are "missing" the right thing to optimise for, i.e. they should be explicit where they are and not take away space where they are not. Allowing argument sequences under a single type annotation would require some kind of marker for either that list or for the other arguments that are not typed. If you have a long argument list of, say, three positional arguments and ten optional keyword arguments, and you only want to annotate the first two positional arguments with types and leave the rest free, that's a lot nicer to express with two explicit type annotations than with grouped annotations and (potentially) explicit non-annotations. Stefan From rosuav at gmail.com Sun Aug 17 16:37:46 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 18 Aug 2014 00:37:46 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F08480.6060305@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F08480.6060305@canterbury.ac.nz> Message-ID: On Sun, Aug 17, 2014 at 8:31 PM, Greg Ewing wrote: > Disappointingly, Python's annotations make the same > blunder as C, and most other languages since, in > requiring each parameter to have its own individual > annotation. Python doesn't really have any option here, because a non-annotated parameter already has meaning. But even in C-family languages, I'm not sure that it's all that advantageous; it makes editing less clear, so it's really only beneficial when you have sets of related arguments (eg "int r,g,b" to specify a color). With variable declarations, there's a difference between "int r,g,b; double x,y,z;", where the block of integers is terminated by a semicolon (and, conventionally, a line ending); in argument lists, you don't get that, so it's not as clear where one starts and one stops. (Imagine you misspell a type name. It's no longer a keyword. How will your mistake be reported?) ChrisA From guido at python.org Sun Aug 17 17:03:29 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 17 Aug 2014 08:03:29 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F07465.1070103@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F07465.1070103@canterbury.ac.nz> Message-ID: On Sun, Aug 17, 2014 at 2:22 AM, Greg Ewing wrote: > Guido van Rossum wrote: > > Perhaps a thornier issue is how mypy should handle decorators that >> manipulate the signature or annotations of the function they wrap. But I >> think the only reasonable answer here can be that mypy must understand what >> decorators do if it wants to have any chance at type-checking decorated >> functions. >> > > Seems to me the only way to do that in general is to > execute the decorators. That means importing everything > the decorators depend on and probably running at least > the top-level module code. Is executing arbitrary > code at type-checking time really desirable? Nah, you just have to type-check the decorators. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sun Aug 17 17:27:36 2014 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 17 Aug 2014 16:27:36 +0100 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F08480.6060305@canterbury.ac.nz> Message-ID: <53F0C9E8.1090803@mrabarnett.plus.com> On 2014-08-17 15:37, Stefan Behnel wrote: > Greg Ewing schrieb am 17.08.2014 um 12:31: >> Nicholas Cole wrote: >>> On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano wrote: >>> >>>> def __init__(self, >>>> description:str, sec_code:str, >>>> vendor_name:str, vendor_inv_num:str, >>>> vendor_rtng:str, vendor_acct:str, >>>> transaction_code:str, vendor_acct_type:str, >>>> amount:int, payment_date:Any)->None: >>> >>> I had to stare at that block of >>> code for a long time to see how many and what type of arguments it >>> called. >> >> Pascal's function signature syntax had a nice feature >> that everyone else seems to have forgotten about. If you >> had multiple parameters of the same type, you only had >> to write the type once: >> >> procedure Init(description, sec_code, vendor_name, >> vendor_inv_num, vendor_rtng, vendor_acct, >> transaction_code, vendor_acct_type, amount: str; >> payment_date: Any) >> >> Disappointingly, Python's annotations make the same >> blunder as C, and most other languages since, in >> requiring each parameter to have its own individual >> annotation. > > The difference is that Pascal requires type declarations whereas they are > purely optional in Python. That makes the case that they are "missing" the > right thing to optimise for, i.e. they should be explicit where they are > and not take away space where they are not. Allowing argument sequences > under a single type annotation would require some kind of marker for either > that list or for the other arguments that are not typed. If you have a long > argument list of, say, three positional arguments and ten optional keyword > arguments, and you only want to annotate the first two positional arguments > with types and leave the rest free, that's a lot nicer to express with two > explicit type annotations than with grouped annotations and (potentially) > explicit non-annotations. > I wonder whether you could include the colon but omit the type if it's the same as that of the following parameter: def __init__(self, description:, sec_code:, vendor_name:, vendor_inv_num:, vendor_rtng:, vendor_acct:, transaction_code:, vendor_acct_type:str, amount:int, payment_date:Any)->None: From brett at python.org Sun Aug 17 17:37:31 2014 From: brett at python.org (Brett Cannon) Date: Sun, 17 Aug 2014 15:37:31 +0000 Subject: [Python-ideas] Optional static typing -- the crossroads References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl> <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com> Message-ID: On Sun Aug 17 2014 at 6:36:43 AM Andrey Vlasovskikh < andrey.vlasovskikh at gmail.com> wrote: > 2014-08-17, 13:41, ?ukasz Langa wrote: > > > On Aug 16, 2014, at 10:03 PM, Guido van Rossum wrote: > > > >> All in all I prefer the mypy syntax, despite being somewhat more > verbose and requiring an import, with one caveat: I agree that it would be > nicer if the mypy abstract collection types were the same objects as the > ABCs exported by collections.abc. > > > > Good :) If the functionality will be implemented in the ABCs, what is > the purpose of the typing module? > > > > My suggestion: if the functionality will be implemented in the ABCs, > there's no need to introduce the "typing" module. We can back-port the new > ABCs, for sure, but for Python 3.5 `collections` is enough (already has > aliases to collections.abc0. > > -1 for collections.abc classes, +1 for mypy's typing classes. > > There is a problem in static analysis of current types that are instances > of abc.ABCMeta or types that just define their own __instancecheck__ / > __subclasscheck__. Static analyzers cannot infer in general case what > attributes of an instance / subclass do these methods check, because their > body can be arbitrarily complex. > That's only an issue if the type-checking code chooses to care about __instancecheck__/__subclasscheck__. The tool could choose to simply ignore those methods and treat them as a run-time only benefit for isinstance/issubclass checks but not for type checking. This is especially true if the check is being done on the AST instead of imported code. People can simply be told that their linter tool will not pick up magical __instancecheck__/__subclasscheck__ implementations. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Sun Aug 17 17:55:11 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 08:55:11 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl> <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com> Message-ID: <94148077-5CFD-4D85-B867-6563B78A4450@langa.pl> On Aug 17, 2014, at 3:35 AM, Andrey Vlasovskikh wrote: > There is a problem in static analysis of current types that are instances of abc.ABCMeta or types that just define their own __instancecheck__ / __subclasscheck__. Static analyzers cannot infer in general case what attributes of an instance / subclass do these methods check, because their body can be arbitrarily complex. That's right. Moreover, arbitrary classes can be register()'ed on an ABC to respond to subclass and instance checks during runtime. Meta-classes and __new__ can do surprising things with returned class objects, too. In all cases Mypy would generate a false-positive type error. That's fine, we can improve on that in multiple ways. [1] People will put classes with ABCMeta in function annotations whether we design for it or not. Subclassing a MutableMapping is the easiest way to implement the whole protocol. As you say, __instancecheck__ and friends aren't limited to ABCs. People will put classes implementing those in function annotations, too. People will have classes with elaborate metaclasses as well. Most importantly: often those classes will come from libraries and frameworks that those people didn't write themselves. We can't expect them to open every black box to see if it works with type hinting. Lastly, classes may and will evolve, sometimes to grow more static and sometimes to become more dynamic. What I'm saying is that the typing module does not shield you from any of it. We need to define type hinting as a best-effort solution to set reasonable expectations. You need to understand the additional cognitive burden a new module like 'typing' this would impose on us and our users. > Mypy's typing.Protocol subclasses are much easier to analyze statically, since they are required to explicitly define abstract methods as function defintions inside the class body. [1] My suggestions: - ABCMeta is special and we can reasonably improve on the runtime .register() invocations by scanning the source for those - a redefined __instancecheck__ would generate a warning - we can define a way for a Python program to say: this module is safe to be imported independently for executing __subclasscheck__ and __subclasshook__ checks on its classes - failing all of the above, this is a perfectly fine case for optional runtime type checking; I don't think anybody has the expectation that we will statically analyse their highly dynamic codebases All the above suggestions might be incremental. It's totally fine if we just begin with the last one. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Sun Aug 17 17:56:45 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 08:56:45 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl> <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com> Message-ID: On Aug 17, 2014, at 8:37 AM, Brett Cannon wrote: > On Sun Aug 17 2014 at 6:36:43 AM Andrey Vlasovskikh wrote: > There is a problem in static analysis of current types that are instances of abc.ABCMeta or types that just define their own __instancecheck__ / __subclasscheck__. Static analyzers cannot infer in general case what attributes of an instance / subclass do these methods check, because their body can be arbitrarily complex. > > That's only an issue if the type-checking code chooses to care about __instancecheck__/__subclasscheck__. The tool could choose to simply ignore those methods and treat them as a run-time only benefit for isinstance/issubclass checks but not for type checking. This is especially true if the check is being done on the AST instead of imported code. People can simply be told that their linter tool will not pick up magical __instancecheck__/__subclasscheck__ implementations. +1 -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leban.us Sun Aug 17 18:54:23 2014 From: bruce at leban.us (Bruce Leban) Date: Sun, 17 Aug 2014 09:54:23 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? In-Reply-To: <85zjf34cbl.fsf@benfinney.id.au> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <85zjf34cbl.fsf@benfinney.id.au> Message-ID: On Sun, Aug 17, 2014 at 3:53 AM, Ben Finney wrote: > A homogeneous sequence would imply there are deliberately *no* specific > meanings to each position. > That's a set. A sequence implies order matters in some way. > The value 7.03 would have the same semantic > value at any position in the sequence. > Not at all. The 7 implies a units value, the 0 is a tenths value and the 3 is the hundredths. Those are different semantics. In Roman numerals before they invented subtraction, XVI = VIX so order did not matter. Roman numerals were basically sets of numbers to be added. Here's another: (KentuckyDerby, PreaknessStakes, BelmontStakes) -- these three objects are all instances of the class StakesRace and the order is significant. --- Bruce Learn how hackers think: http://j.mp/gruyere-security -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Aug 17 21:31:19 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 12:31:19 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> Message-ID: <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> I think we're conflating multiple problems here. Sometimes we use tuple to mean a homogeneous, arbitrary-length, immutable sequence, and other times we use it to mean a heterogeneous, fixed-length sequence.?Nick's list demonstrates that the former is (a) common enough to worry about, and (b) not always a mistake. But even if such uses were always a mistake, Python is obviously not going to add a frozenlist, with new display syntax, and change over all existing misuses of tuple (which would clearly require a full deprecation cycle) before adding static typing. So, we'd still need a way to statically type both uses. And even if the homogeneous uses had never existed in the first place, a heterogeneous tuple would still not be a parametric type in the same sense as all of the other collections, the io classes, etc. So, we'd still need to distinguish it syntactically from all of the other generics. So, arguing about whether we need to handle heterogeneous tuples specially is pointless; the only question is how we do it. I can think of four possibilities: 1. Use a tuple of types to mean a tuple of types: (int, str). This is exactly what we already do in isinstance, the except statement, etc. And it's how Swift, C++, D, and other languages specify a tuple of types (although Swift can also use a product). This definitely would be potentially confusing if we went with the obiwan-inspired syntax of [str] for lists (or iterables or mutable sequences) of str, but since I think Guido has pretty conclusively argued against that syntax on other grounds, this isn't a problem. This probably implies that function types should be written as Function[(str, int), int] instead of Function[[str, int], int], but I think MyPy already handles that, and I think it makes more sense anyway. 2. Use a product of types to mean a tuple: int*str. Tuples really are just product types. This is how you specify them in type theory (and relational theory, and elsewhere).?This is exactly how ML, Haskell, and other languages that designed their type systems carefully instead of haphazardly represent tuples of types.?It also nicely parallels the suggestions for str|int for union types and str&int for multi-inherited subtypes. The first big problem here is that there's no way to specify a tuple of one type. That's not a problem for theory-inspired languages, because in those languages, a tuple of one value is the same thing as that value, but that's obviously not true for Python. The fact that Python doesn't have an appropriate unit type (None is a value that people frequently want to use in tuples) is also at least a theoretical problem. Also, unless we were going to change isinstance, except, etc. to accept (or require) this syntax instead of a tuple of types, I think it would be confusing to remember that you use a tuple of types in some places, a product of types in others. Unlike subscripting, this looks too similar, and too syntactic, to avoid confusion. 3. Use subscripting, but a different form of subscripting, as Steven suggested: Tuple[::int, str]. This isn't actually right; it doesn't mean Tuple[::(int, str)], but Tuple[(::int), str]. But let's assume it's possible to come up with a readable syntax that doesn't require parentheses and ignore that problem. This implies a more general use of slicing in type subscription: the start is the type parameter, and the step is some other stuff to be used in a special way that's specific to that type. If we have other such uses, that would be a door worth leaving open, but I suspect we don't. (If we had dynamic/implicit named tuples, as people have suggested a few times, would that be relevant here?) Also notice that this is still passing a tuple of types, it's just wrapping it up in a slice and then passing that to Tuple just so it can mark the tuple of types as actually meaning a tuple of types. Is that adding enough additional information to be worth all that additional verbosity and complexity? 4. Just use Tuple[int, str] and note that this is a special case that doesn't mean the same thing as other generic types. It seems to me potentially very confusing to have Tuple[int] mean a homogeneous arbitrary-length immutable sequence of ints, while Tuple[int, str] means exactly one int and one str (and Tuple[int,] means exactly one int). You could argue that this confusion is inherent in Python's use of tuples for those two different cases in the first place, but we're still spreading that confusion further. If there were no better alternatives, I think this might be better than Steven's suggestion (practicality beats purity, and his suggestion really doesn't remove that much confusion), but I think there are better alternatives?namely, the first one. On Sunday, August 17, 2014 5:44 AM, Nick Coghlan wrote: > > >On 17 August 2014 21:44, Devin Jeanpierre wrote: >> On Sun, Aug 17, 2014 at 1:23 AM, Ben Finney wrote: >>> I have encountered many uses of ?homogeneous, variable-length sequence? >>> and every time a Python tuple is used for that, I perceive a Python list >>> would be better precisely *because* it better indicates that semantic >>> meaning. >>> >>> I'd like to know how you think that's not true, and what real-world code >>> makes you think so. >> >> isinstance is real world code that for the second parameter accepts >> types and (recursively) tuples of any length of things it accepts. > >There are a few other cases where tuples are special cased as arguments: > >- str.__mod__ >- str.startswith (ditto for binary sequences) >- str.endswith (ditto for binary sequences) > >>>> "aa".endswith(['a', 'b', 'c']) >Traceback (most recent call last): >? File "", line 1, in >TypeError: endswith first arg must be str or a tuple of str, not list > >Searching the C files for "tuple of" turned up a couple more: > >* N-dimensional indexing also relies specifically on >tuples-of-integers, rather than arbitrary iterators. > >* dynamic type creation expects to receive the bases as a tuple > >* the decimal module uses tuples of digits for internal data representations > >And that inspired recollection of several other cases where mutability >would be wrong, because the tuple represents cached information rather >than dynamic state: > >* various "*args" related APIs use "tuple of object" or "tuple of >thing" (e.g. attributes of partial objects, internal storage in >contextlib.ExitStack when used with arbitrary callbacks) > >* other introspection related APIs use tuples to report information >about inspected objects > >* namedtuple _fields attributes are a tuple of strings > >* BaseException.args publishes the full args tuple passed to the constructor > >str.startswith, str.endswith, isinstance and issubclass use the >"implied or" interpretation, everything else does not. In most cases, >the immutability conveys relevant semantic information (usually >indicating that it's a read-only API). > >Cheers, >Nick. > >-- >Nick Coghlan? |? ncoghlan at gmail.com? |? Brisbane, Australia > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Aug 17 22:34:36 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 17 Aug 2014 13:34:36 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: Where is it said that Tuple[int] is a homogeneous variable size list? -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Aug 18 00:46:49 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 15:46:49 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> On Sunday, August 17, 2014 1:34 PM, Guido van Rossum wrote: >Where is it said that Tuple[int] is a homogeneous variable size list? (I'm assuming you're referring to the homogeneity and arbitrary length here, not the fact that someone presumably said "list" when they meant "tuple", because otherwise the answer is trivial?) First, that's how the current typing.py interprets it: Tuple[str] is a homogeneous, arbitrary-length (although of course unchanging, because it's immutable) tuple of strings. Second, what else _would_ it mean? If List[str] and Set[str] mean homogeneous arbitrary-length lists and sets of strs, and the same goes for Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and every other example in typing.py, it would be pretty surprising if it weren't the same for Tuple[str]. Third, if it didn't mean that, how would you define the argument types to any of Nick's examples? For example: ? ? def isinstance(obj: object, types: type | Tuple[type]) -> bool: That had better mean a homogeneous arbitrary-length tuple of types; if not, there doesn't seem to be any other way to declare its type. From guido at python.org Mon Aug 18 01:15:27 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 17 Aug 2014 16:15:27 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: I still think you are mistaken. I don't think mypy has a way to spell a homogeneous arbitrary-length tuple. All uses of Tuple[...] refer to "anonymous struct" tuples. I tried this: from typing import Tuple def f(a: Tuple[int]) -> None: pass def main() -> None: f((1,)) f((1, 2)) and I get an error for the second call, f((1, 2)): a.py: In function "main": a.py, line 8: Argument 1 to "f" has incompatible type "Tuple[int, int]"; expected "Tuple[int]" On Sun, Aug 17, 2014 at 3:46 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Sunday, August 17, 2014 1:34 PM, Guido van Rossum > wrote: > > >Where is it said that Tuple[int] is a homogeneous variable size list? > > > (I'm assuming you're referring to the homogeneity and arbitrary length > here, not the fact that someone presumably said "list" when they meant > "tuple", because otherwise the answer is trivial?) > > First, that's how the current typing.py interprets it: Tuple[str] is a > homogeneous, arbitrary-length (although of course unchanging, because it's > immutable) tuple of strings. > > > Second, what else _would_ it mean? If List[str] and Set[str] mean > homogeneous arbitrary-length lists and sets of strs, and the same goes for > Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and > every other example in typing.py, it would be pretty surprising if it > weren't the same for Tuple[str]. > > Third, if it didn't mean that, how would you define the argument types to > any of Nick's examples? For example: > > def isinstance(obj: object, types: type | Tuple[type]) -> bool: > > That had better mean a homogeneous arbitrary-length tuple of types; if > not, there doesn't seem to be any other way to declare its type. > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Aug 18 01:16:39 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 18 Aug 2014 09:16:39 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <20140817231638.GC25957@ando> On Sun, Aug 17, 2014 at 01:34:36PM -0700, Guido van Rossum wrote: > Where is it said that Tuple[int] is a homogeneous variable size list? Did you mean variable sized tuple rather than list? The mypy tutorial says the opposite: http://www.mypy-lang.org/tutorial.html#tuples which implies that Tuple[int] would accept (23,) but not (23, 42). There's two, or three, cases to consider: Variable sized tuple of some homogenous type - e.g. (1,), (1, 2) - Tuple[int] for consistency with other types - mypy doesn't appear to support this Fixed size tuple of given hetrogeneous types - e.g. (23, "abc"), (42, "xyz") - mypy uses Tuple[int, str] which is a special case Some way to specify two or more types, e.g. - (str, int) for consistency with isinstance, issubclass - Union[str, int] as used by mypy - str|int obvious short-cut for Union - str*int will make type theorists happy Making Tuple[] a special case troubles me, I strongly prefer Tuple[int] to mean a homogenous tuple of ints. mypy already has Union[str, int] for union types (giving str|int as the obvious short-cut), which leaves (str, int) for the hetrogenous 2-tuple case. Perhaps in 3.6 we can consider allowing isinstance and issubclass to accept Union types as well as tuples of types. -- Steven From guido at python.org Mon Aug 18 01:17:39 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 17 Aug 2014 16:17:39 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: I sent that a little too soon; I should add that I think this is the right way; and that's why I keep suggesting a different way to spell a homogeneous variable-length tuple. 1-tuples should not be special. On Sun, Aug 17, 2014 at 4:15 PM, Guido van Rossum wrote: > I still think you are mistaken. I don't think mypy has a way to spell a > homogeneous arbitrary-length tuple. All uses of Tuple[...] refer to > "anonymous struct" tuples. > > I tried this: > > from typing import Tuple > > def f(a: Tuple[int]) -> None: > pass > > def main() -> None: > f((1,)) > f((1, 2)) > > and I get an error for the second call, f((1, 2)): > > a.py: In function "main": > a.py, line 8: Argument 1 to "f" has incompatible type "Tuple[int, int]"; > expected "Tuple[int]" > > > > On Sun, Aug 17, 2014 at 3:46 PM, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > >> On Sunday, August 17, 2014 1:34 PM, Guido van Rossum >> wrote: >> >> >Where is it said that Tuple[int] is a homogeneous variable size list? >> >> >> (I'm assuming you're referring to the homogeneity and arbitrary length >> here, not the fact that someone presumably said "list" when they meant >> "tuple", because otherwise the answer is trivial?) >> >> First, that's how the current typing.py interprets it: Tuple[str] is a >> homogeneous, arbitrary-length (although of course unchanging, because it's >> immutable) tuple of strings. >> >> >> Second, what else _would_ it mean? If List[str] and Set[str] mean >> homogeneous arbitrary-length lists and sets of strs, and the same goes for >> Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and >> every other example in typing.py, it would be pretty surprising if it >> weren't the same for Tuple[str]. >> >> Third, if it didn't mean that, how would you define the argument types to >> any of Nick's examples? For example: >> >> def isinstance(obj: object, types: type | Tuple[type]) -> bool: >> >> That had better mean a homogeneous arbitrary-length tuple of types; if >> not, there doesn't seem to be any other way to declare its type. >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Aug 18 01:19:01 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 18 Aug 2014 11:19:01 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl> <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com> Message-ID: <53F13865.4040302@canterbury.ac.nz> Andrey Vlasovskikh wrote: > There is a problem in static analysis of current types that are instances of > abc.ABCMeta or types that just define their own __instancecheck__ / > __subclasscheck__. Static analyzers cannot infer in general case what > attributes of an instance / subclass do these methods check, because their > body can be arbitrarily complex. However, I'd be worried about having two very similar but subtly different sets of type objects floating around. That just seems like a recipe for massive confusion. -- Greg From lukasz at langa.pl Mon Aug 18 01:19:16 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 16:19:16 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <20140817231638.GC25957@ando> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <20140817231638.GC25957@ando> Message-ID: <8AAD22FE-7366-4BC6-9ABE-8126C9BE7924@langa.pl> On Aug 17, 2014, at 4:16 PM, Steven D'Aprano wrote: > Perhaps in 3.6 we can consider allowing isinstance and issubclass to > accept Union types as well as tuples of types. Why not 3.5? -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Aug 18 01:24:29 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 18 Aug 2014 09:24:29 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <8AAD22FE-7366-4BC6-9ABE-8126C9BE7924@langa.pl> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <20140817231638.GC25957@ando> <8AAD22FE-7366-4BC6-9ABE-8126C9BE7924@langa.pl> Message-ID: <20140817232429.GD25957@ando> On Sun, Aug 17, 2014 at 04:19:16PM -0700, ?ukasz Langa wrote: > On Aug 17, 2014, at 4:16 PM, Steven D'Aprano wrote: > > > Perhaps in 3.6 we can consider allowing isinstance and issubclass to > > accept Union types as well as tuples of types. > > > Why not 3.5? In case Guido changes his mind :-) -- Steven From lukasz at langa.pl Mon Aug 18 01:27:56 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 16:27:56 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com> Message-ID: On Aug 17, 2014, at 3:53 AM, Andrew Barnert wrote: > The obvious way that already works (with MyPy, and also with ABCs for isinstance checking): > > class HashableIterable(Iterable, Hashable): pass > > def spam(a: HashableIterable): > pass No. Specifying a subclass of Iterable and Hashable cannot possibly mean that *any* type that is both Iterable and Hashable is behaving like your subclass. >>> from collections import * >>> class IterableHashable(Iterable, Hashable): pass ... >>> isinstance("str", Iterable) True >>> isinstance("str", Hashable) True >>> isinstance("str", IterableHashable) False We would need a new construct like Union, maybe named AnySubclass, that checks True when all its base classes are in a given type's MRO. Some ABCs don't explicitly appear in a type's MRO but I fixed this problem with singledispatch. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Mon Aug 18 01:35:11 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 16:35:11 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: On Aug 17, 2014, at 3:46 PM, Andrew Barnert wrote: > Second, what else _would_ it mean? If List[str] and Set[str] mean homogeneous arbitrary-length lists and sets of strs, and the same goes for Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and every other example in typing.py, it would be pretty surprising if it weren't the same for Tuple[str]. You're playing the uniformity card and usually I would agree with you. In this case, there is no uniformity between different *data structures*. Nobody expects to be able to say: int[int], dict[str], set[str: int]. Tuples were meant to be heterogenic, Raymond draws that distinction in many classes and talks he gives. On the other hand, some types are hard to be typed heterogenically, e.g. you can't reasonably specify an Iterable that yield strings except for the third yield, which is an int. All in all, I think it's not at all confusing to say that tuple[int, int] means a point, for instance (1, 1). That being said, we will need support for homogenous tuples, too, simply because they are already in the wild. I proposed tuple[int, ...], which is explicit and obvious (if you're Polish, that is). -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Aug 18 01:40:53 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 18 Aug 2014 09:40:53 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: On Mon, Aug 18, 2014 at 9:35 AM, ?ukasz Langa wrote: > That being said, we will need support for homogenous tuples, too, simply > because they are already in the wild. I proposed tuple[int, ...], which is > explicit and obvious (if you're Polish, that is). Conceptually, these kinds of constructs are sometimes called frozenlists. Why not actually create that type? frozenlist = tuple; Et voila. Now just define that frozenlist[int] is like list[int] rather than like tuple[int], and there you are, out of your difficulty at once! ChrisA From greg.ewing at canterbury.ac.nz Mon Aug 18 01:43:51 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 18 Aug 2014 11:43:51 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <20140817102059.GS4525@ando> Message-ID: <53F13E37.5090807@canterbury.ac.nz> Chris Angelico wrote: > I don't, for instance, see > real-world examples of decorators that add extra parameters to a > function, even though it would be plausible Some decorators, such as property(), don't return a function at all. Ignoring the decorator in that case would give completely the wrong idea. I'm now thinking the right thing to do with decorators is to analyse them statically if possible, otherwise treat the result as untyped. If a decorator is well-behaved, type inference should be able to propagate the types of the decorated function through to the result. If not, all bets are off. -- Greg From lukasz at langa.pl Mon Aug 18 01:47:49 2014 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 17 Aug 2014 16:47:49 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: On Aug 17, 2014, at 4:40 PM, Chris Angelico wrote: > Et voila. Now just define that frozenlist[int] is like list[int] > rather than like tuple[int], and there you are, out of your difficulty > at once! That's a neat trick and I think Guido suggested this naming before. It feels a little icky though, consider: - issubclass(tuple[int, int, int], frozenlist[int]) ? I think True. - issubclass(frozenlist[int], tuple[int, int, int]) ? I would think False. But because that's technically the same tuple underneath, *sometimes* instances of frozenlist[int] will respond True to isinstance(tuple[int, int, int]). Saying explicitly tuple[int, ...] takes that riddle away. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Aug 18 01:59:24 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 18 Aug 2014 09:59:24 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: On Mon, Aug 18, 2014 at 9:47 AM, ?ukasz Langa wrote: > - issubclass(tuple[int, int, int], frozenlist[int]) ? I think True. > - issubclass(frozenlist[int], tuple[int, int, int]) ? I would think False. > But because that's technically the same tuple underneath, *sometimes* > instances of frozenlist[int] will respond True to isinstance(tuple[int, int, > int]). I would say False to both of those, because they have different implications of intent. But it is arguable, so I'd be happy with the first one being True if it's easier to code that way. ChrisA From greg.ewing at canterbury.ac.nz Mon Aug 18 02:04:20 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 18 Aug 2014 12:04:20 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com> Message-ID: <53F14304.6040705@canterbury.ac.nz> Andrew Barnert wrote: > The obvious way that already works (with MyPy, and also with ABCs for isinstance checking): > > class HashableIterable(Iterable, Hashable): pass > > def spam(a: HashableIterable): > pass That way might be obvious, but it's wrong. It says that spam() only takes an instance of the specific class HashableIterable or a subclass thereof. It won't accept anything else, even if it implements Iterable and Hashable perfectly well. > > But, there's no reason typing.py couldn't add a wrapper that does this automatically: > > class Multiple: > @staticmethod > def __getitem__(*types); > return type(types[0])( > '_'.join(t.__name__ for t in types), types, {}) Automating it won't make it any more right. There needs to be a distinct concept such as Intersection() as a counterpart to Union(). -- Greg From guido at python.org Mon Aug 18 02:30:37 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 17 Aug 2014 17:30:37 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: On Sun, Aug 17, 2014 at 4:47 PM, ?ukasz Langa wrote: > On Aug 17, 2014, at 4:40 PM, Chris Angelico wrote: > > Et voila. Now just define that frozenlist[int] is like list[int] > rather than like tuple[int], and there you are, out of your difficulty > at once! > > That's a neat trick and I think Guido suggested this naming before. It > feels a little icky though, consider: > > - issubclass(tuple[int, int, int], frozenlist[int]) ? I think True. > - issubclass(frozenlist[int], tuple[int, int, int]) ? I would think > False. But because that's technically the same tuple underneath, > *sometimes* instances of frozenlist[int] will respond True to > isinstance(tuple[int, int, int]). > > Saying explicitly tuple[int, ...] takes that riddle away. > Hm. That looks just as [tr]icky. Plus, I expect both pedants and ignoramuses may wonder about the empty tuple. :-) I think it deserves a proper name, not magic syntax. I do agree that frozentuple sounds odd, but perhaps we just need to get used to it. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlehtosalo at gmail.com Mon Aug 18 03:01:47 2014 From: jlehtosalo at gmail.com (Jukka Lehtosalo) Date: Sun, 17 Aug 2014 18:01:47 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: On Sun, Aug 17, 2014 at 5:30 PM, Guido van Rossum wrote: > On Sun, Aug 17, 2014 at 4:47 PM, ?ukasz Langa wrote: > >> That's a neat trick and I think Guido suggested this naming before. It >> feels a little icky though, consider: >> >> - issubclass(tuple[int, int, int], frozenlist[int]) ? I think True. >> - issubclass(frozenlist[int], tuple[int, int, int]) ? I would think >> False. But because that's technically the same tuple underneath, >> *sometimes* instances of frozenlist[int] will respond True to >> isinstance(tuple[int, int, int]). >> >> Saying explicitly tuple[int, ...] takes that riddle away. >> > > Hm. That looks just as [tr]icky. Plus, I expect both pedants and > ignoramuses may wonder about the empty tuple. :-) I think it deserves a > proper name, not magic syntax. > > I do agree that frozentuple sounds odd, but perhaps we just need to get > used to it. > I've considered adding a mypy type for arbitrary-length tuples [1]. My original idea was to call it TupleSequence[T], since it's basically a concrete tuple that is used like a sequence. Fixed-length tuples can already be used as abstract iterables and sequences: def f(x: Iterable[int]) -> None: pass f((1, 2, 3)) # Ok [1] https://github.com/JukkaL/mypy/issues/184 Jukka > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Aug 18 03:05:02 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 18:05:02 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <20140817231638.GC25957@ando> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <20140817231638.GC25957@ando> Message-ID: <1408323902.81771.YahooMailNeo@web181001.mail.ne1.yahoo.com> On Sunday, August 17, 2014 4:17 PM, Steven D'Aprano wrote: > There's two, or three, cases to consider: > > Variable sized tuple of some homogenous type > - e.g. (1,), (1, 2) > - Tuple[int] for consistency with other types > - mypy doesn't appear to support this Are you sure? In [typing.py](https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py#L17), Tuple is defined as TypeAlias(tuple), exactly the same way List is defined as TypeAlias(list). And I don't see any code anywhere else in that module that adds the special-casing to it. So, isn't this what MyPy is already doing? Or is there some hidden functionality outside of typing.py that changes it? Anyway, I agree with you that, whatever it _currently_ means in MyPy, it _should_ mean a tuple of an arbitrary number of int values. > Fixed size tuple of given hetrogeneous types > - e.g. (23, "abc"), (42, "xyz") > - mypy uses Tuple[int, str] which is a special case This is the one I'd like to write as (int, str). > Some way to specify two or more types, e.g. > - (str, int) for consistency with isinstance, issubclass > - Union[str, int] as used by mypy > - str|int obvious short-cut for Union > - str*int will make type theorists happy That last one will not make type theorists happy. A product of two types is a tuple (your case #2, not #3). What you want is a sum or union of two types, which you'd write str|int, str U int, or maybe str+int. > Making Tuple[] a special case troubles me, I strongly prefer Tuple[int]? > to mean a homogenous tuple of ints. > > mypy already has Union[str, int] for union types (giving str|int as the > obvious short-cut), which leaves (str, int) for the hetrogenous 2-tuple > case. > > Perhaps in 3.6 we can consider allowing isinstance and issubclass to > accept Union types as well as tuples of types. > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From rosuav at gmail.com Mon Aug 18 03:11:23 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 18 Aug 2014 11:11:23 +1000 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F13E37.5090807@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <20140817102059.GS4525@ando> <53F13E37.5090807@canterbury.ac.nz> Message-ID: On Mon, Aug 18, 2014 at 9:43 AM, Greg Ewing wrote: > Chris Angelico wrote: >> >> I don't, for instance, see >> real-world examples of decorators that add extra parameters to a >> function, even though it would be plausible > > > Some decorators, such as property(), don't return a > function at all. Ignoring the decorator in that case > would give completely the wrong idea. So you don't annotate the function that handles the property. Simple! There may need to be some other handling of it (telling mypy that this property will always be a str, for instance), but the function's arguments aren't significant to the type system, so they don't need annotations. Same is true of anything else that decorates the function away altogether. ChrisA From abarnert at yahoo.com Mon Aug 18 03:12:58 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 18:12:58 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <53F14304.6040705@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com> <53F14304.6040705@canterbury.ac.nz> Message-ID: <1408324378.30223.YahooMailNeo@web181003.mail.ne1.yahoo.com> > On Sunday, August 17, 2014 5:04 PM, Greg Ewing wrote: > > Andrew Barnert wrote: >> The obvious way that already works (with MyPy, and also with ABCs for > isinstance checking): >> >> ? ? class HashableIterable(Iterable, Hashable): pass >> >> ? ? def spam(a: HashableIterable): >> ? ? ? ? pass > > That way might be obvious, but it's wrong. It says that spam() > only takes an instance of the specific class HashableIterable > or a subclass thereof. It won't accept anything else, even if > it implements Iterable and Hashable perfectly well. typing.Iterable and typing.Hashable are both typing.Protocols, so HashableIterable is also a typing.Protocol, so, if I understand Protocol correctly, isinstance will return True iff every method and attribute in HashableIterable (that is, every method and attribute in Iterable, plus every method and attribute in Hashable) is implemented by the type. Which is what we want here, right? That obviously doesn't work for nominal rather than structural ABCs, or for ad-hoc structural ABCs like the ones in collections.abc that don't automatically compose, but isn't that the whole point of Protocol, to provide structural subtyping that follows the rules of structural rather than nominative subtyping? > >> >> But, there's no reason typing.py couldn't add a wrapper that does > this automatically: >> >> ? ? class Multiple: >> ? ? ? ? @staticmethod >> ? ? ? ? def __getitem__(*types); >> ? ? ? ? ? ? return type(types[0])( >> ? ? ? ? ? ? ? ? '_'.join(t.__name__ for t in types), types, > {}) > > Automating it won't make it any more right. There needs to > be a distinct concept such as Intersection() as a counterpart > to Union(). > > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From schesis at gmail.com Mon Aug 18 03:21:21 2014 From: schesis at gmail.com (Zero Piraeus) Date: Sun, 17 Aug 2014 21:21:21 -0400 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <20140818012121.GA10658@piedra> : On Sun, Aug 17, 2014 at 12:31:19PM -0700, Andrew Barnert wrote: > I think we're conflating multiple problems here. > > Sometimes we use tuple to mean a homogeneous, arbitrary-length, > immutable sequence, and other times we use it to mean a heterogeneous, > fixed-length sequence.?Nick's list demonstrates that the former is (a) > common enough to worry about, and (b) not always a mistake. > > [...] > > I can think of four possibilities: Since we're already repurposing a number of operators - TypeA[TypeB], TypeA|TypeB and possibly TypeA&TypeB, here's a fifth possibility: Leave Tuple[int] with the same semantic meaning as List[int], Set[int] etc., and use e.g. Tuple%(int, str, float) to declare the signature of a fixed-length, heterogenous sequence (borrowing from the use of % in string interpolation). This has the advantages that it a) doesn't make tuple a special case, and so allows other container types to use it (including user-defined types which aren't tuples, but are intended to behave similarly to them), and b) is both reasonably compact, and easily distinguished from Tuple[int]. The specific operator used doesn't really matter - when this idea came to me I thought of @ (since it's shiny and new in 3.5), but % seems a better conceptual fit. -[]z. -- Zero Piraeus: unus multorum http://etiol.net/pubkey.asc From ryan at ryanhiebert.com Mon Aug 18 03:25:35 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Sun, 17 Aug 2014 20:25:35 -0500 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com> > On Aug 17, 2014, at 7:30 PM, Guido van Rossum wrote: > > I do agree that frozentuple sounds odd, but perhaps we just need to get used to it. I?d expect that most uses of Tuple[int] to mean an arbitrary length tuple of integers would be better served with Sequence[int] anyway, so I?d definitely find it odd for Tuple[int] to mean anything other than a tuple with exactly one integer in it. I could imagine somebody wanting to say that it could be a Sequence but NOT a MutableSequence. Perhaps that could be spelled: Sequence[int] - MutableSequence[int] and/or (Sequence - MutableSequence)[int] From abarnert at yahoo.com Mon Aug 18 03:41:08 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 18:41:08 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <1408326068.7864.YahooMailNeo@web181006.mail.ne1.yahoo.com> On Sunday, August 17, 2014 4:36 PM, ?ukasz Langa wrote: >On Aug 17, 2014, at 3:46 PM, Andrew Barnert wrote: > >>Second, what else _would_ it mean? If List[str] and Set[str] mean homogeneous arbitrary-length lists and sets of strs, and the same goes for Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and every other example in typing.py, it would be pretty surprising if it weren't the same for Tuple[str]. >> > > >You're playing the uniformity card and usually I would agree with you. In this case, there is no uniformity between different *data structures*. There is uniformity between every single generic data structure defined by MyPy except Tuple. That makes Tuple an exceedingly special special case. And its specialness is almost invisible. I went through typing.py in more detail, and I'm pretty sure there is nothing there to indicate that the Tuple = TypeAlias(tuple) line is going to do anything different than all of the other TypeAlias calls. Whatever difference there is must be in the private, implementation-specific code. > Nobody expects to be able to say: int[int], dict[str], set[str: int]. But int and dict are not sequences; tuple is. > Tuples were meant to be heterogenic, Raymond draws that distinction in many classes and talks he gives. Sure. But?as Nick pointed out, there are places all over even the core language and the stdlib where they're used homogeneously. You can argue that was a mistake, but you can't just wish it away. >All in all, I think it's not at all confusing to say that tuple[int, int] means a point, for instance (1, 1).? I think it's far less confusing to say that (int, int) is the type of (1, 1). And other languages agree: ? ? $ ghci ? ? Prelude> set +t ? ? Prelude> (1, 1) ? ? (1,1) ? ? it :: (Integer, Integer) ? ? Prelude> ^D ? ? Leaving GHCi. ? ? $ xcrun swift ? ? ? 1> (1, 1) ? ? $R0: (Int, Int) = { ? ? ? 0 = 1 ? ? ? 1 = 1 ? ? } ? ? ? 2> ^D Those languages don't try to make heterogeneous tuples look like parametric collection types. Why should Python? From guido at python.org Mon Aug 18 03:49:59 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 17 Aug 2014 18:49:59 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <1408323902.81771.YahooMailNeo@web181001.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <20140817231638.GC25957@ando> <1408323902.81771.YahooMailNeo@web181001.mail.ne1.yahoo.com> Message-ID: On Sun, Aug 17, 2014 at 6:05 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Sunday, August 17, 2014 4:17 PM, Steven D'Aprano > wrote: > > > There's two, or three, cases to consider: > > > > Variable sized tuple of some homogenous type > > - e.g. (1,), (1, 2) > > - Tuple[int] for consistency with other types > > - mypy doesn't appear to support this > > Are you sure? > > In [typing.py]( > https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py#L17), > Tuple is defined as TypeAlias(tuple), exactly the same way List is defined > as TypeAlias(list). And I don't see any code anywhere else in that module > that adds the special-casing to it. > > So, isn't this what MyPy is already doing? Or is there some hidden > functionality outside of typing.py that changes it? > There is -- typing.py is not the typechecker, it's just a bunch of dummies crafted so that your code can also be executed by vanilla Python 3.2. (If you look at the definition of TypeAlias a few lines earlier, it has almost no semantics, and you can see that it is also used for wildly other types, e.g. Function and Union. > Anyway, I agree with you that, whatever it _currently_ means in MyPy, it > _should_ mean a tuple of an arbitrary number of int values. > I disagree. (I've said that before, but there seems to be a large delay between our exchanges.) The Foo[bar, ...] notation has no inherent semantics -- compare for example Dict[str, int], Tuple[str, int], and Union[str, int]. The most useful way to define Tuple[T1, T2, ..., Tn] is to use it for an n-tuple whose element types are T1 etc. > > Fixed size tuple of given hetrogeneous types > > - e.g. (23, "abc"), (42, "xyz") > > - mypy uses Tuple[int, str] which is a special case > > This is the one I'd like to write as (int, str). > I know that's your proposal, but it seems to blind you for the other position. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Aug 18 03:58:00 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 17 Aug 2014 18:58:00 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com> Message-ID: <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com> On Sunday, August 17, 2014 6:26 PM, Ryan Hiebert wrote: ? > I?d expect that most uses of Tuple[int] to mean an arbitrary length tuple of > integers would be better served with Sequence[int] anyway It would be nice if that were true, but unfortunately, Python has a long history of using tuple, and only tuple, both in the core language and in the stdlib, specifically to mean an arbitrary-length homogeneous tuple in APIs or syntax that take a single value or tuple of values. I don't want to repeat the whole list that Nick Coghlan provided, but I would like to add that any third-party code that interacts with those parts of the syntax or stdlib has to do the same thing (e.g., an exception-logging decorator has to taken an exception type or tuple of exception types to use in an except statement), and there's probably some third-party code that's done the same thing completely independently, just because it's endorsed by the stdlib. From ryan at ryanhiebert.com Mon Aug 18 04:18:57 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Sun, 17 Aug 2014 21:18:57 -0500 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com> <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com> Message-ID: > On Aug 17, 2014, at 8:58 PM, Andrew Barnert wrote: > > On Sunday, August 17, 2014 6:26 PM, Ryan Hiebert wrote: > >> I?d expect that most uses of Tuple[int] to mean an arbitrary length tuple of >> integers would be better served with Sequence[int] anyway > > It would be nice if that were true, but unfortunately, Python has a long history of using tuple, and only tuple, both in the core language and in the stdlib, specifically to mean an arbitrary-length homogeneous tuple in APIs or syntax that take a single value or tuple of values. > > I don't want to repeat the whole list that Nick Coghlan provided, but I would like to add that any third-party code that interacts with those parts of the syntax or stdlib has to do the same thing (e.g., an exception-logging decorator has to taken an exception type or tuple of exception types to use in an except statement), and there's probably some third-party code that's done the same thing completely independently, just because it's endorsed by the stdlib. Thanks. As I read more, I?m recalling some apis that use tuples as special markers for iterables rather than, say, strings, so that there can be polymorphism based on the arguments, allowing both a string and a tuple. Iterable strings bite again ;-) From alexander.belopolsky at gmail.com Mon Aug 18 07:02:36 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 18 Aug 2014 01:02:36 -0400 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? (was: Optional static typing -- the crossroads) In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: > On Aug 17, 2014, at 8:30 PM, Guido van Rossum wrote: > > Plus, I expect both pedants and ignoramuses may wonder about the empty tuple. :-) I think it deserves a proper name, not magic syntax. I agree, even though covering all cases in a consistent syntax does not seem hard: Tuple[int] - variable length homogeneous tuple. Tuple[int,float] - length 2 Tuple[int,] - length 1 Tuple[()] - length 0 Since length 1 and more so length 0 cases don't seem to get much use, a somewhat non-obvious syntax should be fine. From greg.ewing at canterbury.ac.nz Mon Aug 18 08:06:47 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 18 Aug 2014 18:06:47 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F08480.6060305@canterbury.ac.nz> Message-ID: <53F197F7.4090105@canterbury.ac.nz> Chris Angelico wrote: > With variable declarations, > there's a difference between "int r,g,b; double x,y,z;", where the > block of integers is terminated by a semicolon ... > in argument lists, you don't get that, Python's argument annotation syntax *could* have been defined to work the Pascal way, with semicolons separating the groups. But it wasn't, and it's too late to change now. > (Imagine you misspell a type > name. It's no longer a keyword. How will your mistake be reported?) I don't understand that. Type names are always clearly separated from keywords in either style (they come after a colon), so there's no danger of confusing them. -- Greg From greg.ewing at canterbury.ac.nz Mon Aug 18 08:19:36 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 18 Aug 2014 18:19:36 +1200 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? In-Reply-To: <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <53F19AF8.1050204@canterbury.ac.nz> Andrew Barnert wrote: > First, that's how the current typing.py interprets it: Tuple[str] is a > homogeneous, arbitrary-length (although of course unchanging, because it's > immutable) tuple of strings. So how do you spell a heterogeneous tuple of length 1 containing a string? -- Greg From jlehtosalo at gmail.com Mon Aug 18 08:29:01 2014 From: jlehtosalo at gmail.com (Jukka Lehtosalo) Date: Sun, 17 Aug 2014 23:29:01 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? In-Reply-To: <53F19AF8.1050204@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> <53F19AF8.1050204@canterbury.ac.nz> Message-ID: On Sun, Aug 17, 2014 at 11:19 PM, Greg Ewing wrote: > Andrew Barnert wrote: > >> First, that's how the current typing.py interprets it: Tuple[str] is a >> homogeneous, arbitrary-length (although of course unchanging, because it's >> immutable) tuple of strings. >> > > So how do you spell a heterogeneous tuple of length 1 > containing a string? > Tuple[str] :-) The only arbitrary-length tuple that mypy knows about is just 'tuple' and it isn't generic or necessarily homogeneous (it's dynamically typed, basically a catch-all for a completely arbitrary tuple). However, arbitrary-length, homogeneous tuples would be a nice addition, once we figure out how the type should be named. Jukka > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Aug 18 08:33:11 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 18 Aug 2014 18:33:11 +1200 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: <1408324378.30223.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <53F076D9.4020807@canterbury.ac.nz> <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com> <53F14304.6040705@canterbury.ac.nz> <1408324378.30223.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <53F19E27.6010807@canterbury.ac.nz> Andrew Barnert wrote: > typing.Iterable and typing.Hashable are both typing.Protocols, so > HashableIterable is also a typing.Protocol, so, if I understand Protocol > correctly, isinstance will return True iff every method and attribute in > HashableIterable (that is, every method and attribute in Iterable, plus every > method and attribute in Hashable) is implemented by the type. Urg. In that case, how do you spell a type that *is* a concrete class that implements Hashable and Iterable, rather than a new protocol? There's too much magic going on here for my liking. -- Greg From greg.ewing at canterbury.ac.nz Mon Aug 18 08:41:07 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 18 Aug 2014 18:41:07 +1200 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? In-Reply-To: <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com> <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com> Message-ID: <53F1A003.1010001@canterbury.ac.nz> So we need a name for a type that's a tuple used as a sequence. How about Tuppence? -- Greg From ben+python at benfinney.id.au Mon Aug 18 08:53:04 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 18 Aug 2014 16:53:04 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com> <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com> <53F1A003.1010001@canterbury.ac.nz> Message-ID: <85a97247cf.fsf@benfinney.id.au> Greg Ewing writes: > So we need a name for a type that's a tuple used > as a sequence. Um. Isn't every tuple used as a sequence? So, the name ?tuple? fits. Maybe you mean ?used as a *homogeneous* sequence?. Or maybe you mean ?used as a *variable-length* sequence?. Or something else? -- \ ?Too many Indians spoil the golden egg.? ?Sir Joh | `\ Bjelke-Petersen | _o__) | Ben Finney From rosuav at gmail.com Mon Aug 18 08:53:53 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 18 Aug 2014 16:53:53 +1000 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? In-Reply-To: <53F1A003.1010001@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com> <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com> <53F1A003.1010001@canterbury.ac.nz> Message-ID: On Mon, Aug 18, 2014 at 4:41 PM, Greg Ewing wrote: > So we need a name for a type that's a tuple used > as a sequence. > > How about Tuppence? King Tuppence - or - A Good Deal Less Than Half a Sovereign? ChrisA From daviesk24 at yahoo.com Mon Aug 18 17:24:58 2014 From: daviesk24 at yahoo.com (Kevin Davies) Date: Mon, 18 Aug 2014 05:24:58 -1000 Subject: [Python-ideas] float comparison in doctest In-Reply-To: <53E9B914.1000702@yahoo.com> References: <53E8EE7B.50408@yahoo.com> <53E9B914.1000702@yahoo.com> Message-ID: <53F21ACA.9080206@yahoo.com> An HTML attachment was scrubbed... URL: From daviesk24 at yahoo.com Mon Aug 18 17:24:45 2014 From: daviesk24 at yahoo.com (Kevin Davies) Date: Mon, 18 Aug 2014 05:24:45 -1000 Subject: [Python-ideas] float comparison in doctest In-Reply-To: <53E9B914.1000702@yahoo.com> References: <53E8EE7B.50408@yahoo.com> <53E9B914.1000702@yahoo.com> Message-ID: <53F21ABD.5000107@yahoo.com> An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Aug 18 19:00:38 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 18 Aug 2014 10:00:38 -0700 Subject: [Python-ideas] Variable-length, homogeneous tuple: why? In-Reply-To: <53F1A003.1010001@canterbury.ac.nz> References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> <854mxb5xt3.fsf_-_@benfinney.id.au> <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com> <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com> <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com> <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com> <53F1A003.1010001@canterbury.ac.nz> Message-ID: <53F23136.2060200@stoneleaf.us> On 08/17/2014 11:41 PM, Greg Ewing wrote: > So we need a name for a type that's a tuple used > as a sequence. > > How about Tuppence? Only if we also get Pidgeons! ;) -- ~Ethan~ From ethan at stoneleaf.us Mon Aug 18 22:15:36 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 18 Aug 2014 13:15:36 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way Message-ID: <53F25EE8.8000900@stoneleaf.us> After reading through the OST threads, and learning a bit more about what static typing means, and the difference between nominal and structural subclassing, I think I have a better appreciation for the knee-jerk "no way!" reaction, and also how to make what seems like a foreign-to-Python concept pythonistic. The most-feared problem seems to be over-specification (it is certainly mine). I think the answer has been alluded to a couple times already, but just to hopefully bring it front and center: Instead of either 'nominal' or 'structural', use our own 'dynamical' subclassing scheme. In other words: def spamify(current: datetime.date, moved:int) -> bool: # ONE_DAY is a one day time delta, previously declared days = ONE_DAY * moved proposed_date = current + days return it_works(proposed_date) The type checker will not look to see if 'current' is a datetime.date, but rather will check that what datetime.date's __add__ works with (datetime.delta) and then check that what is passed in for current has an __add__ that also works with datetime.delta. I think this would be far more valuable than just nominal as it follows duck-typing, and hopefully less work than structural as it's only checking the methods and attributes actually used. On the minus side, there are times when we need *exactly* a datetime.date, so we would need a way to specify exact vs dynamic, but dynamic should be the default. -- ~Ethan~ From ncoghlan at gmail.com Tue Aug 19 15:27:08 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Aug 2014 23:27:08 +1000 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: <53F25EE8.8000900@stoneleaf.us> References: <53F25EE8.8000900@stoneleaf.us> Message-ID: On 19 August 2014 06:15, Ethan Furman wrote: > After reading through the OST threads, and learning a bit more about what > static typing means, and the difference between nominal and structural > subclassing, I think I have a better appreciation for the knee-jerk "no > way!" reaction, and also how to make what seems like a foreign-to-Python > concept pythonistic. > > The most-feared problem seems to be over-specification (it is certainly > mine). I think the answer has been alluded to a couple times already, but > just to hopefully bring it front and center: The exact same fear was raised when ABCs were added in PEP 3119 - that by formalising ABCs, we'd see a plethora of additional isinstance checks as people tried to make things "fail fast". That fear turned out to be spurious then, and I think it's spurious now (unless we allow builtins to be used for type annotations - we shouldn't do that, as I think it *would* cause problems when developers migrate to Python from languages like C, Java and C#). Most Python code should continue to be written without type annotations - the use of annotations (and type checkers) should be limited to projects (and organisations) that are large enough for the additional complexity to be worth the hassle. Anecdotally, my experience is that the split between proponents of static and dynamic typing tends to divide along "team size" lines. If someone thinks a team of 20 working on one project is "a lot of people", then they're likely to favour dynamic typing for the additional flexibility it offers. If they think such a team is small, then they're more likely to want compiler enforced assistance in ensuring that everyone is playing by the rules of the API. Myself, I think the kind of structural typechecking offered by "pylint -E" and the "optional type declarations" proposal starts looking far more attractive the first time you're stuck at work after hours debugging a production failure that happened due to a typo in an error handling path that wasn't covered by your test suite. However, if your code isn't in the category of "if this doesn't work, my company/project/organisation stops until it is fixed", then type assertions are unlikely to be worth the investment needed to write them in the first place :) Ultimately, my perspective is that Guido's proposal boils down to having a nice syntax where: def myfunc(a : KindX, b: KindY, c: KindZ): ... is the moral equivalent of: def myfunc(a, b, c): assert isinstance(a, KindX) assert isinstance(b, KindY) assert isinstance(c, KindZ) except done in a way that is more amenable to static analysis by looking at the compiled AST, rather than actually executing the code. Calling it "optional static typing" is probably a bad idea, since the proposal isn't to change the type system itself - that will be just as dynamic as it always has been. "optional type assertions" is probably the most accurate, but then people may think it is *actually* translated to runtime checks as I show above, which it won't be. "optional type hinting" is probably the most reassuring term that could be used, while still remaining accurate. > Instead of either 'nominal' or 'structural', use our own 'dynamical' > subclassing scheme. > > In other words: > > def spamify(current: datetime.date, moved:int) -> bool: > # ONE_DAY is a one day time delta, previously declared > days = ONE_DAY * moved > proposed_date = current + days > return it_works(proposed_date) > > The type checker will not look to see if 'current' is a datetime.date, but > rather will check that what datetime.date's __add__ works with > (datetime.delta) and then check that what is passed in for current has an > __add__ that also works with datetime.delta. > > I think this would be far more valuable than just nominal as it follows > duck-typing, and hopefully less work than structural as it's only checking > the methods and attributes actually used. ABCs already support ducktyping, and explicit registration, etc, etc. We don't need a completely new type system, we can just file the rough edges off the categories we have already defined, and fill in some of the missing pieces (like allowing union types, which I seem to recall being discussed back in the PEP 3119 time frame, but postponed until more concrete use cases presented themselves). We may even need to finally add String and BytesLike ABCs, but we've been muttering about doing that for years. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Tue Aug 19 19:01:12 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 19 Aug 2014 10:01:12 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: On Tue, Aug 19, 2014 at 6:27 AM, Nick Coghlan wrote: [Agreeable musings about team size...] Ultimately, my perspective is that Guido's proposal boils down to > having a nice syntax where: > > def myfunc(a : KindX, b: KindY, c: KindZ): > ... > > is the moral equivalent of: > > def myfunc(a, b, c): > assert isinstance(a, KindX) > assert isinstance(b, KindY) > assert isinstance(c, KindZ) > Please no. The asserts affect runtime. The type declarations are for linting, IDEs and docs. I don't want the difference to be swept under the rug. I want it to be the key feature of the proposal. > except done in a way that is more amenable to static analysis by > looking at the compiled AST, rather than actually executing the code. > Ironically, mypy also understands isinstance() checks, using them as a kind of implied "typecase" (though I don't know if it understands assert -- it definitely understands "if isinstance(x, int): x = x**2". > Calling it "optional static typing" is probably a bad idea, since the > proposal isn't to change the type system itself - that will be just as > dynamic as it always has been. "optional type assertions" is probably > the most accurate, but then people may think it is *actually* > translated to runtime checks as I show above, which it won't be. > I don't know what you mean by "accurate". > "optional type hinting" is probably the most reassuring term that > could be used, while still remaining accurate. > I like that. > > Instead of either 'nominal' or 'structural', use our own 'dynamical' > > subclassing scheme. > > > > In other words: > > > > def spamify(current: datetime.date, moved:int) -> bool: > > # ONE_DAY is a one day time delta, previously declared > > days = ONE_DAY * moved > > proposed_date = current + days > > return it_works(proposed_date) > > > > The type checker will not look to see if 'current' is a datetime.date, > but > > rather will check that what datetime.date's __add__ works with > > (datetime.delta) and then check that what is passed in for current has an > > __add__ that also works with datetime.delta. > > > > I think this would be far more valuable than just nominal as it follows > > duck-typing, and hopefully less work than structural as it's only > checking > > the methods and attributes actually used. > > ABCs already support ducktyping, and explicit registration, etc, etc. > We don't need a completely new type system, we can just file the rough > edges off the categories we have already defined, and fill in some of > the missing pieces (like allowing union types, which I seem to recall > being discussed back in the PEP 3119 time frame, but postponed until > more concrete use cases presented themselves). > Yeah, my response to Ethan's use case is that either he shouldn't bother with type annotations, or he should probably care enough to also declare his duck type for datetime.date as such, and an ABC regsitration sounds perfect for that. > We may even need to finally add String and BytesLike ABCs, but we've > been muttering about doing that for years. > Mypy has some ideas here. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From censi at mit.edu Tue Aug 19 19:49:37 2014 From: censi at mit.edu (Andrea Censi) Date: Tue, 19 Aug 2014 13:49:37 -0400 Subject: [Python-ideas] type checking using PyContracts; was: Proposal: Use mypy syntax for function annotations Message-ID: Dear all, (Apologies for not replying to the original thread, I only signed up to the mailing list today) I wrote PyContracts[1], which implements type-checking functionality using decorators for Python 2.7+ and annotations in Python 3. [1] http://andreacensi.github.com/contracts/ > def word_count(input: List[str]) -> Dict[str, int]: Here's that example done using PyContracts and Python 3, and note the extra constraint '>=1' that PyContract can express: from contracts import contract @contract def word_count(input:'list(str)') -> 'dict(str:int,>=1): ... Here's that example done using PyContracts and Python 2.7, using a decorator instead of annotations: from contracts import contract @contract(input='list(str)', returns='dict(str:int,>=1)') def word_count(input): ... I would like to make a few points from my experience of writing PyContracts regarding the general issues that one encounters in adding type checking to Python. 1) Once one starts to think about it, "types" are very broad and if you design a system from scratch, make sure it is extensible. For example, you might want to have "positive integer" in addition to "integer" ('int', and 'int,>=0' in PyContracts). Also, let the users add their own types. 2) Dependent types are a must and they must have a simple syntax. For example, this is a PyContracts annotation that says the function takes a list of strings and returns a list of integers of the same size N: @contract(x='list[N](str)', returns='list[N](int)') def f(x): return list(map(len, x)) PyContract's convention is that and a,b,c,...,y,z bind to any variables and A,B,C,...,Z bind to integer variables. This makes numpy contracts very nice looking: @contract def gray2rgb(gray:'array[HxW](uint8)') -> 'array[HxWx3](uint8)': ... 3) There is an argument for using regular strings to describe annotations rather than Python constructs such as MyPy. One advantage is that the system is backwards compatible. It also makes the code much cleaner because you don't need to import the various symbols. Here's a simple example in MyPy: from typing typevar, Sequence T = typevar('T') # Declare type variable def first(seq: Sequence[T]) -> T: return seq[0] Here's the same example in PyContracts (using Python 3's annotations): from contracts import contract @contract def first(s: 'seq(type(t))') -> 'type(t)': return s[0] Same example in Python 2.7+ (using decorators): from contracts import contract @contract(s='seq(type(t))', returns='type(t)') def first(s): return s[0] (By the way, after a while, I actually like the decorator version better than Python 3's annotations, because you can see very easily the function's name and parameters, and only if you want you can then read the function annotations above.) cheers, A. -- Andrea Censi | http://censi.mit.edu | "Not all those who wander are lost." research scientist @ LIDS / Massachusetts Institute of Technology From apalala at gmail.com Tue Aug 19 23:18:07 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Tue, 19 Aug 2014 16:48:07 -0430 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: On Sun, Aug 17, 2014 at 12:33 AM, Guido van Rossum wrote: > > (2) For type annotations, should we adopt (roughly) the mypy syntax or the > alternative proposed by Dave Halter? This uses built-in container notations > as a shorthand, e.g. {str: int} instead of Dict[str, int]. This also > touches on the issue of abstract vs. concrete types (e.g. iterable vs. > list). Maybe we've been missing that in most programming languages the sub-language for describing types is separate from the sub-language for describing algorithms. It is so in Haskell, and in C, and in LISP, and many others. It should be OK if similar constructs mean different things in each sub-language, as the necessary symbol reuse (if one omits APL) come mostly from the IBM keyboard and ASCII. Personally, I'd like to type less, as is most often the case with current Python. Beyond that, I agree it is important that the adopted syntax does not mislead. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Aug 20 00:50:13 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 19 Aug 2014 18:50:13 -0400 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: On 8/19/2014 9:27 AM, Nick Coghlan wrote: What I like: 'optional type hints' based on a fleshed-out ABC systems that collects all ABCs together on one module via import from current files. It seems that we have been slowly groping towards this for years. What I also like: shadow skeleton files (for the stdlib) written by people other than core developers, who obviously have the energy to do so, having already started with various syntaxes, and who just need a standard to combine their efforts. There should be one set that, as much as possible, fully allows duck-typing, which is to say, does not reject valid arguments. What I want to add: the advantage of separate skeleton files, aside from parallel development by other people, is that there can be more than one skeleton file for a given stdlib module. Linters typically allow users to set local standards by selecting options. A group could also set local standards by restrictive type hints in their local skeletons. A realistic example would be sum(it: Iterable[Number]). -- Terry Jan Reedy From antoine at python.org Wed Aug 20 01:02:50 2014 From: antoine at python.org (Antoine Pitrou) Date: Tue, 19 Aug 2014 19:02:50 -0400 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: Le 19/08/2014 18:50, Terry Reedy a ?crit : > On 8/19/2014 9:27 AM, Nick Coghlan wrote: > > What I like: 'optional type hints' based on a fleshed-out ABC systems > that collects all ABCs together on one module via import from current > files. It seems that we have been slowly groping towards this for years. Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are). Besides, the fact that type descriptions must be parametrable, and therefore end up being used as instances, sounds like it kills the idea of making them regular (runtime) types as well. Unless we make them metaclasses? Regards Antoine. From guido at python.org Wed Aug 20 01:45:27 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 19 Aug 2014 16:45:27 -0700 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: On Tue, Aug 19, 2014 at 2:18 PM, Juancarlo A?ez wrote: > > > On Sun, Aug 17, 2014 at 12:33 AM, Guido van Rossum > wrote: > >> >> (2) For type annotations, should we adopt (roughly) the mypy syntax or >> the alternative proposed by Dave Halter? This uses built-in container >> notations as a shorthand, e.g. {str: int} instead of Dict[str, int]. This >> also touches on the issue of abstract vs. concrete types (e.g. iterable vs. >> list). > > > Maybe we've been missing that in most programming languages the > sub-language for describing types is separate from the sub-language for > describing algorithms. It is so in Haskell, and in C, and in LISP, and many > others. > > It should be OK if similar constructs mean different things in each > sub-language, as the necessary symbol reuse (if one omits APL) come mostly > from the IBM keyboard and ASCII. > > Personally, I'd like to type less, as is most often the case with current > Python. Beyond that, I agree it is important that the adopted syntax does > not mislead. > Actually, Python has a long tradition of reusing the same sub-language for things that are different sub-languages in other languages. Starting with 'int' being callable, in fact. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Aug 20 02:12:54 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 19 Aug 2014 17:12:54 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou wrote: > Le 19/08/2014 18:50, Terry Reedy a ?crit : > > On 8/19/2014 9:27 AM, Nick Coghlan wrote: >> >> What I like: 'optional type hints' based on a fleshed-out ABC systems >> that collects all ABCs together on one module via import from current >> files. It seems that we have been slowly groping towards this for years. >> > > Hmm, I've been saying this already, but my intuition is that it's a bad > idea to conflate *type descriptions* (what this proposal is about) and > actual *runtime types* (what ABCs are). > But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it. > Besides, the fact that type descriptions must be parametrable, and > therefore end up being used as instances, sounds like it kills the idea of > making them regular (runtime) types as well. Unless we make them > metaclasses? > No, that doesn't follow at all. The concept of metaclasses means that classes *are* instances (of the metaclass) and it lets us define operations (in particular __getitem__ :-) on the classes. (I was going to show a working example, but I ran out of time, and I'm taking a vacation the rest of this week.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Wed Aug 20 03:50:20 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Tue, 19 Aug 2014 21:20:20 -0430 Subject: [Python-ideas] Optional static typing -- the crossroads In-Reply-To: References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando> Message-ID: On Tue, Aug 19, 2014 at 7:15 PM, Guido van Rossum wrote: > Actually, Python has a long tradition of reusing the same sub-language for > things that are different sub-languages in other languages. Starting with > 'int' being callable, in fact. :-) I love the consistency of Python's type system. It's intuitive (difficult to get wrong). Maybe that's why I'm one of the fearful ones regarding this move towards formalizing type annotations. BTW, today I took the time to read Mr. C.D. Smith's (old) article about type systems, and I found it relevant, illustrative, smart, and entertaining. May I remind the list about it? I will: http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Aug 20 13:27:34 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Aug 2014 21:27:34 +1000 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: On 20 Aug 2014 10:14, "Guido van Rossum" wrote: > > On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou wrote: >> >> Le 19/08/2014 18:50, Terry Reedy a ?crit : >> >>> On 8/19/2014 9:27 AM, Nick Coghlan wrote: >>> >>> What I like: 'optional type hints' based on a fleshed-out ABC systems >>> that collects all ABCs together on one module via import from current >>> files. It seems that we have been slowly groping towards this for years. >> >> >> Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are). > > > But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it. One of the ways I describe the explicit registration mechanism to people is as giving you the option of lying to the interpreter. If you use explicit registration, your registered type basically walks around with a sign saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and assumes it will be able to quack without actually checking in advance. It's really just a constrained version of normal duck typing, where the interpreter just assumes that *everything* may exhibit duck-like tendencies. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tleeuwenburg at gmail.com Wed Aug 20 14:08:48 2014 From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg) Date: Wed, 20 Aug 2014 22:08:48 +1000 Subject: [Python-ideas] Optional static typing -- late to the party Message-ID: Hi all, I apologise in advance for being across only that proportion of the previous correspondence that I could read in about half an hour. Too much has been said to fully integrate it all. I have some responses to what was written, and one of those I feel has been largely missed. Please accept this as "just some stuff I think" and not any kind of strident criticism or positioning. The primary goal should be a syntax which enhances human readability. While I agree that having good static code analysis tools is very useful for the write --> test --> fix cycle, actually the more bugs you can catch during the human read/understand/write cycle, the better. The earlier you can communicate useful information about variable types, the better, and the earliest stage of that is code comprehension at the human level. I mostly agree with those who are preferring docstring embedding over signature decoration, particularly for complex cases, due to the desire to separate the complexity into structured sections. The signature is like the heading, and the annotation is like additional detail. I was led to a conclusion: what is wrong with allowing both? Indeed, clearly both are actually already supported by various tools. So perhaps there is actually a higher-level common concept -- what are the actual assertion which are going to be supported? Can I declare a variable only to be a list, or can it be a list-of-ints? Further, is it possible to standardise between some of the syntax/terminology, and some of the assertion types, such that they are consistent between function annotation syntaxes and docstring embedding syntaxes? Could I use function annotation for simple cases, but seamlessly move to a docstring-embedded approach for a more complex example? e.g. def frobnicate(x: int, y: int) -> float ''' Applies some kind of standard function ''' return x**2 / y is great. But def handle_complex_case(name, context, objective, datacube): ''' @param name: str, contains a "Firstname Lastname" string @param context: dict, contains a data record @param objective: int, contains a coded directive @param datacube: numpy.array, contains a field of continuous spatial data @return: instructionset: list(callable), a list of callable instructions ''' allows me to describe what's going on in that function in a way which supports both typing and static analysis. Apologies for inventing the docstring syntax rather than correctly using an existing syntax. However, my point is that in the first example, an annotation approach is *more* readable. In the second example, you really want to spend more time describing and annotating the function. If there could be some agreement about the details and consistency, I see no reason that a dual style could not be preferable, and no more complex to understand and use. Regards, -Tennessee Leeuwenburg -- -------------------------------------------------- Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think" -------------- next part -------------- An HTML attachment was scrubbed... URL: From tleeuwenburg at gmail.com Wed Aug 20 14:14:34 2014 From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg) Date: Wed, 20 Aug 2014 22:14:34 +1000 Subject: [Python-ideas] type checking using PyContracts; was: Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: I like your comments about making the type system easily extensible. I can well imagine this being used, particularly for data types with an inherent structure such as tuples, lists, dicts and namedtuples. On Wed, Aug 20, 2014 at 3:49 AM, Andrea Censi wrote: > Dear all, > > (Apologies for not replying to the original thread, I only signed up > to the mailing list today) > > I wrote PyContracts[1], which implements type-checking functionality > using decorators for Python 2.7+ and annotations in Python 3. > > [1] http://andreacensi.github.com/contracts/ > > > def word_count(input: List[str]) -> Dict[str, int]: > > Here's that example done using PyContracts and Python 3, > and note the extra constraint '>=1' that PyContract can express: > > from contracts import contract > > @contract > def word_count(input:'list(str)') -> 'dict(str:int,>=1): > ... > > Here's that example done using PyContracts and Python 2.7, using > a decorator instead of annotations: > > from contracts import contract > > @contract(input='list(str)', returns='dict(str:int,>=1)') > def word_count(input): > ... > > I would like to make a few points from my experience of writing > PyContracts regarding the general issues that one encounters in adding > type checking to Python. > > 1) Once one starts to think about it, "types" are very broad and if you > design a system from scratch, make sure it is extensible. For example, > you might want to have "positive integer" in addition to "integer" > ('int', and 'int,>=0' in PyContracts). Also, let the users add > their own types. > > 2) Dependent types are a must and they must have a simple syntax. > For example, this is a PyContracts annotation that says the function > takes > a list of strings and returns a list of integers of the same size N: > > @contract(x='list[N](str)', returns='list[N](int)') > def f(x): > return list(map(len, x)) > > PyContract's convention is that and a,b,c,...,y,z bind to any variables > and A,B,C,...,Z bind to integer variables. This makes numpy contracts > very nice looking: > > @contract > def gray2rgb(gray:'array[HxW](uint8)') -> 'array[HxWx3](uint8)': > ... > > 3) There is an argument for using regular strings to describe annotations > rather than Python constructs such as MyPy. One advantage > is that the system is backwards compatible. It also makes the code > much cleaner because you don't need to import the various symbols. > > Here's a simple example in MyPy: > > from typing typevar, Sequence > > T = typevar('T') # Declare type variable > > def first(seq: Sequence[T]) -> T: > return seq[0] > > Here's the same example in PyContracts (using Python 3's annotations): > > from contracts import contract > > @contract > def first(s: 'seq(type(t))') -> 'type(t)': > return s[0] > > Same example in Python 2.7+ (using decorators): > > from contracts import contract > > @contract(s='seq(type(t))', returns='type(t)') > def first(s): > return s[0] > > (By the way, after a while, I actually like the decorator version > better than Python 3's annotations, because you can see very > easily the function's name and parameters, and only if you want > you can then read the function annotations above.) > > cheers, > A. > > -- > Andrea Censi | http://censi.mit.edu | "Not all those who wander are lost." > research scientist @ LIDS / Massachusetts Institute of Technology > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- -------------------------------------------------- Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think" -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Aug 20 14:35:09 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 20 Aug 2014 22:35:09 +1000 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: <20140820123508.GK25957@ando> On Wed, Aug 20, 2014 at 10:08:48PM +1000, Tennessee Leeuwenburg wrote: [...] > I was led to a conclusion: what is wrong with allowing both? "Both" being function annotations and docstring annotations. Docstring annotations will not suddenly go away if (when) Guido's proposal is accepted. But remember that any annotations in docstrings may not be available at runtime, whereas function annotations will be unless you deliberately delete them. Also, docstring annotations have the disadvantage of being text only, instead of expressions which evaluate to arbitrarily powerful or useful objects. > Indeed, > clearly both are actually already supported by various tools. So perhaps > there is actually a higher-level common concept -- what are the actual > assertion which are going to be supported? Can I declare a variable only to > be a list, or can it be a list-of-ints? Yes. Using Guido's suggestion of mypy's syntax: from typing import List def spam(x:List)->List[int]: ... declares that spam accepts as argument a list of anything, and returns a list of ints. But of course you're more likely to want to accept anything which quacks like a list: from typing import Sequence def spam(x:Sequence)->List[int]: ... The precise details of the syntax aren't yet finalised, but this is suggestive of what it will likely be (I hope). > Further, is it possible to standardise between some of the > syntax/terminology, and some of the assertion types, such that they are > consistent between function annotation syntaxes and docstring embedding > syntaxes? Could I use function annotation for simple cases, but seamlessly > move to a docstring-embedded approach for a more complex example? That entirely depends on the tool you are using for static type analysis and/or linting. -- Steven From ben+python at benfinney.id.au Wed Aug 20 14:53:24 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 20 Aug 2014 22:53:24 +1000 Subject: [Python-ideas] Optional static typing -- late to the party References: <20140820123508.GK25957@ando> Message-ID: <85sikr1fwb.fsf@benfinney.id.au> Steven D'Aprano writes: > But remember that any annotations in docstrings may not be available > at runtime How so? What conformant Python implementation is discarding docstrings from code objects? > Also, docstring annotations have the disadvantage of being text only, > instead of expressions which evaluate to arbitrarily powerful or > useful objects. True. This is why I'm a fan of reStructuredText for docstrings, and the conventions that have arisen for using reST field lists for code annotations . The structure in a docstring isn't available without parsing the reST, but that seems no worse than requiring new syntax and new tools to process function annotations. I'm not arguing against function annotations, merely addressing these claims about docstrings. -- \ ?The cost of a thing is the amount of what I call life which is | `\ required to be exchanged for it, immediately or in the long | _o__) run.? ?Henry David Thoreau | Ben Finney From rosuav at gmail.com Wed Aug 20 14:56:32 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 20 Aug 2014 22:56:32 +1000 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: <85sikr1fwb.fsf@benfinney.id.au> References: <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au> Message-ID: On Wed, Aug 20, 2014 at 10:53 PM, Ben Finney wrote: > Steven D'Aprano writes: > >> But remember that any annotations in docstrings may not be available >> at runtime > > How so? What conformant Python implementation is discarding docstrings > from code objects? CPython with the -OO flag. Or is that non-conformant? ChrisA From censi at mit.edu Wed Aug 20 14:56:38 2014 From: censi at mit.edu (Andrea Censi) Date: Wed, 20 Aug 2014 08:56:38 -0400 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: On Wed, Aug 20, 2014 at 5:08 AM, Tennessee Leeuwenburg wrote: > > I was led to a conclusion: what is wrong with allowing both? Indeed, clearly > both are actually already supported by various tools. So perhaps there is > actually a higher-level common concept -- what are the actual assertion > which are going to be supported? Can I declare a variable only to be a list, > or can it be a list-of-ints? Yes - why not both? or all three ways? Here's the "nonempty list of positive int" example done using PyContracts [1] in 3 ways: 1) Using decorators; 2) Using annotations (Python 3 only); 3) Using docstrings. The type "list[length-type](entry-type)" means a list where the length satisfies length-type and the entries satisfy entry-type. So "list[>=1](int,>0)" means a list of length at least 1 whose entries are positive integers. 1) Using decorators: @contract(l='list[>=1](int,>0)', returns='int,>0') def mysum(l): ... 2) Using annotations: @contract def mysum(l: 'list[>=1](int,>0)') -> 'int,>0': ... 3) Using docstrings, with the :type: and :rtype: tags: @contract def mysum(l): """ :type l: list[>=1](int,>0) :rtype: int,>0 """ ... If using (1) and (2), a docstring like in (3) is automatically generated. [1]: https://pypi.python.org/pypi/PyContracts Steven D'Aprano says: > docstring annotations have the disadvantage of being text only, instead of expressions which evaluate to arbitrarily powerful or useful objects. I disagree, it's actually the other way around: Python's general purpose syntax is actually cumbersome with respect to a custom language for types. Once you have the freedom of using a custom language (e.g. using PyParsing) then you can create very compact and at the same time powerful contracts. For example, expressing a type such as "list[>=1](int,>0)" using a valid Python expression will take much more space. And you have the ability of creating special syntax where it makes sense, for example in Numpy --- look at this definition of matrix multiplication with compatible dimensions: @contract def matrix_multiply(a, b): ''' Multiplies two matrices together. :param a: The first matrix. Must be a 2D array. :type a: array[MxN],M>0,N>0 :param b: The second matrix. Must be of compatible dimensions. :type b: array[NxP],P>0 :rtype: array[MxP] ''' return numpy.dot(a, b) If the contract is violated, there will be a nice message saying exactly what's wrong Example: a = numpy.zeros((2,2)) b = numpy.zeros((3,2)) matrix_multiply(a,b) Exception: Breach for argument 'b' to matrix_multiply(). Expected value for 'N' was: Instance of int: 2 instead I received: Instance of int: 3 checking: N for value: Instance of int: 3 checking: NxP for value: Instance of tuple: (3, 2) checking: array[NxP] for value: array['3x2'](float64) array([[ 0., 0.], [ 0., 0.], ... [clip] checking: array[NxP],P>0 for value: array['3x2'](float64) array([[ 0., 0.], [ 0., 0.], ... [clip] best, A. -- Andrea Censi | http://censi.mit.edu | "Not all those who wander are lost." research scientist @ LIDS / Massachusetts Institute of Technology From p.f.moore at gmail.com Wed Aug 20 15:28:55 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 20 Aug 2014 14:28:55 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On 14 August 2014 00:30, Guido van Rossum wrote: > We certainly *could* do that. However, I haven't seen sufficient other uses > of annotations. If there is only one use for annotations (going forward), > annotations would be unambiguous. If we allow different types of > annotations, there would have to be a way to tell whether a particular > annotation is intended as a type annotation or not. Currently mypy ignores > all modules that don't import typing.py (using any form of import > statement), and we could continue this convention. But it would mean that > something like this would still require the typing import in order to be > checked by mypy: > > import typing > > def gcd(int a, int b) -> int: > Sorry, I'm slowly going through this thread, so my apologies if this has been covered later, but it seems to me that projects (and in particular libraries) that want to target 2.7 as well as 3.x will be forced to avoid this feature. And avoid any dependencies that use it. Specifically, the annotation syntax is purely Python 3.x, so without some form of translation or import hook, 2.7 won't accept it. And adding a dependency (even if it's only a very lightweight typing.py plus a hook installed somewhere/somehow) for something that only has value as part of a developer-level type check doesn't seem like a good solution. So, I'd like someone to explain (maybe by pointing me to relevant mypy docs, I haven't investigated them yet) how I should modify my existing code that supports 2.7 and 3.x so that it uses the new functionality *without* extra runtime dependencies that only deliver build/test-time benefits (that's the problem I always had with setuptools/pkg_resources, and I'd not like to see it repeated here). Paul From p.f.moore at gmail.com Wed Aug 20 16:06:09 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 20 Aug 2014 15:06:09 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140814173103.GO4525@ando> References: <20140814173103.GO4525@ando> Message-ID: On 14 August 2014 18:31, Steven D'Aprano wrote: > opt-out and use something else: > > @use_spam_annotations > def frobnicate(x: spam, y: eggs)->breakfast: ... Which reminds me. Why is mypy exempt from the general recommendation to only use annotations on functions that are tagged with a decorator? Paul. From brett at python.org Wed Aug 20 16:11:21 2014 From: brett at python.org (Brett Cannon) Date: Wed, 20 Aug 2014 14:11:21 +0000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations References: Message-ID: On Wed Aug 20 2014 at 9:29:34 AM Paul Moore wrote: > On 14 August 2014 00:30, Guido van Rossum wrote: > > We certainly *could* do that. However, I haven't seen sufficient other > uses > > of annotations. If there is only one use for annotations (going forward), > > annotations would be unambiguous. If we allow different types of > > annotations, there would have to be a way to tell whether a particular > > annotation is intended as a type annotation or not. Currently mypy > ignores > > all modules that don't import typing.py (using any form of import > > statement), and we could continue this convention. But it would mean that > > something like this would still require the typing import in order to be > > checked by mypy: > > > > import typing > > > > def gcd(int a, int b) -> int: > > > > Sorry, I'm slowly going through this thread, so my apologies if this > has been covered later, but it seems to me that projects (and in > particular libraries) that want to target 2.7 as well as 3.x will be > forced to avoid this feature. And avoid any dependencies that use it. > > Specifically, the annotation syntax is purely Python 3.x, so without > some form of translation or import hook, 2.7 won't accept it. And > adding a dependency (even if it's only a very lightweight typing.py > plus a hook installed somewhere/somehow) for something that only has > value as part of a developer-level type check doesn't seem like a good > solution. > > So, I'd like someone to explain (maybe by pointing me to relevant mypy > docs, I haven't investigated them yet) how I should modify my existing > code that supports 2.7 and 3.x so that it uses the new functionality > *without* extra runtime dependencies that only deliver build/test-time > benefits (that's the problem I always had with > setuptools/pkg_resources, and I'd not like to see it repeated here). > I suspect the answer is "you don't". Just like everything else that is syntactically exclusive to Python 3, it's a perk you get with Python 3-only code that you simply can't get if you want to maintain backwards-compatibility. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at ryanhiebert.com Wed Aug 20 16:27:47 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Wed, 20 Aug 2014 09:27:47 -0500 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <51F3C4F3-A6A9-4F97-A287-B3CC7608E130@ryanhiebert.com> > On Aug 20, 2014, at 8:28 AM, Paul Moore wrote: > > Specifically, the annotation syntax is purely Python 3.x, so without > some form of translation or import hook, 2.7 won't accept it. And > adding a dependency (even if it's only a very lightweight typing.py > plus a hook installed somewhere/somehow) for something that only has > value as part of a developer-level type check doesn't seem like a good > solution. > > So, I'd like someone to explain (maybe by pointing me to relevant mypy > docs, I haven't investigated them yet) how I should modify my existing > code that supports 2.7 and 3.x so that it uses the new functionality > *without* extra runtime dependencies that only deliver build/test-time > benefits (that's the problem I always had with > setuptools/pkg_resources, and I'd not like to see it repeated here). As you?ve suggested, using annotations won?t work on Python 2 where there are no annotations. This feature would not available for applications that wish to support Python 2. However, that stipulation isn?t unique to this feature. If you want be able to run on older versions of the language, you must forgo the features that are available in newer versions of the language. You can?t use with_statement unless you?re willing to drop Python 2.4 support, you can?t use print_function unless you?re willing to drop 2.5 support, and you can?t use annotations unless you?re willing to drop all Python 2 support. This is the nature of language improvements. From jean-charles.douet at laposte.net Wed Aug 20 16:13:38 2014 From: jean-charles.douet at laposte.net (Jean-Charles Douet) Date: Wed, 20 Aug 2014 16:13:38 +0200 Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 118 In-Reply-To: References: Message-ID: <53F4AD12.8070209@laposte.net> Hi All, Just a few remarks : - Here is what is said about literate programming : http://www.literateprogramming.com/quotes_sd.html. Isn't that Pythonic ? - PyContracts, annotations, etc? look like ? literate programming ?. Doc is optional, type hint too, why not put the former with the latter ? - Now about PyContracts: I cannot prevent myself from thinking of it as a ? CHECK CONSTRAINT ? in PostgreSQL and co. - PyContracts aims at describing constraints, including them in docstrings. Thus documentation has to be more structured, more formatted and more informative about what's going on in the software being interpreted by Python automata. Perhaps structured doc becomes a solution to the Rice's theorem ? - Moreover, the ? doctest ? module could be very useful too. Whenever you write a docstring, you can have a test too; this module could test annotations and contracts too. Why not include it in standard library ? - Two systems of structured documentation : - CWEB : http://www-cs-faculty.stanford.edu/~uno/cweb.html - FunnelWeb - Are there collisions with these formats : ReSt, GRASS GIS' ? boilerplating ? in comments, #FIXME and #TODO for the IDEs, etc? Best regards, JCD. From donald at stufft.io Wed Aug 20 16:25:31 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 20 Aug 2014 10:25:31 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> > On Aug 20, 2014, at 10:11 AM, Brett Cannon wrote: > > > > On Wed Aug 20 2014 at 9:29:34 AM Paul Moore > wrote: > On 14 August 2014 00:30, Guido van Rossum > wrote: > > We certainly *could* do that. However, I haven't seen sufficient other uses > > of annotations. If there is only one use for annotations (going forward), > > annotations would be unambiguous. If we allow different types of > > annotations, there would have to be a way to tell whether a particular > > annotation is intended as a type annotation or not. Currently mypy ignores > > all modules that don't import typing.py (using any form of import > > statement), and we could continue this convention. But it would mean that > > something like this would still require the typing import in order to be > > checked by mypy: > > > > import typing > > > > def gcd(int a, int b) -> int: > > > > Sorry, I'm slowly going through this thread, so my apologies if this > has been covered later, but it seems to me that projects (and in > particular libraries) that want to target 2.7 as well as 3.x will be > forced to avoid this feature. And avoid any dependencies that use it. > > Specifically, the annotation syntax is purely Python 3.x, so without > some form of translation or import hook, 2.7 won't accept it. And > adding a dependency (even if it's only a very lightweight typing.py > plus a hook installed somewhere/somehow) for something that only has > value as part of a developer-level type check doesn't seem like a good > solution. > > So, I'd like someone to explain (maybe by pointing me to relevant mypy > docs, I haven't investigated them yet) how I should modify my existing > code that supports 2.7 and 3.x so that it uses the new functionality > *without* extra runtime dependencies that only deliver build/test-time > benefits (that's the problem I always had with > setuptools/pkg_resources, and I'd not like to see it repeated here). > > I suspect the answer is "you don't". Just like everything else that is syntactically exclusive to Python 3, it's a perk you get with Python 3-only code that you simply can't get if you want to maintain backwards-compatibility. > mypy does have a codec that will ignore annotations on 2.x. But other than that the answer is you don?t. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From apieum at gmail.com Wed Aug 20 16:34:06 2014 From: apieum at gmail.com (Gregory Salvan) Date: Wed, 20 Aug 2014 16:34:06 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: In mypy FAQ: http://www.mypy-lang.org/faq.html "All of my code is still in Python 2. What are my options? Mypy currently supports Python 3 syntax. Python 2 support is still in early stages of development. However, Python 2 support will be improving." I don't know how. Does someone know if a decorator that add "__annotations__" attribute to the function can do the job ? with something like that: @mypy(int, int, returns=int) def gcd(a, b): 2014-08-20 16:11 GMT+02:00 Brett Cannon : > > > On Wed Aug 20 2014 at 9:29:34 AM Paul Moore wrote: > >> On 14 August 2014 00:30, Guido van Rossum wrote: >> > We certainly *could* do that. However, I haven't seen sufficient other >> uses >> > of annotations. If there is only one use for annotations (going >> forward), >> > annotations would be unambiguous. If we allow different types of >> > annotations, there would have to be a way to tell whether a particular >> > annotation is intended as a type annotation or not. Currently mypy >> ignores >> > all modules that don't import typing.py (using any form of import >> > statement), and we could continue this convention. But it would mean >> that >> > something like this would still require the typing import in order to be >> > checked by mypy: >> > >> > import typing >> > >> > def gcd(int a, int b) -> int: >> > >> >> Sorry, I'm slowly going through this thread, so my apologies if this >> has been covered later, but it seems to me that projects (and in >> particular libraries) that want to target 2.7 as well as 3.x will be >> forced to avoid this feature. And avoid any dependencies that use it. >> >> Specifically, the annotation syntax is purely Python 3.x, so without >> some form of translation or import hook, 2.7 won't accept it. And >> adding a dependency (even if it's only a very lightweight typing.py >> plus a hook installed somewhere/somehow) for something that only has >> value as part of a developer-level type check doesn't seem like a good >> solution. >> >> So, I'd like someone to explain (maybe by pointing me to relevant mypy >> docs, I haven't investigated them yet) how I should modify my existing >> code that supports 2.7 and 3.x so that it uses the new functionality >> *without* extra runtime dependencies that only deliver build/test-time >> benefits (that's the problem I always had with >> setuptools/pkg_resources, and I'd not like to see it repeated here). >> > > I suspect the answer is "you don't". Just like everything else that is > syntactically exclusive to Python 3, it's a perk you get with Python 3-only > code that you simply can't get if you want to maintain > backwards-compatibility. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Aug 20 16:47:34 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 21 Aug 2014 00:47:34 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140814173103.GO4525@ando> Message-ID: <20140820144734.GL25957@ando> On Wed, Aug 20, 2014 at 03:06:09PM +0100, Paul Moore wrote: > On 14 August 2014 18:31, Steven D'Aprano wrote: > > opt-out and use something else: > > > > @use_spam_annotations > > def frobnicate(x: spam, y: eggs)->breakfast: > ... > Which reminds me. Why is mypy exempt from the general recommendation > to only use annotations on functions that are tagged with a decorator? That's the whole point of Guido's proposal: he wants to bless, not mypy precisely since that's an alternative implementation of Python, but mypy's syntax as the standard use of annotations. Since it will be the official standard, there's no need to flag it with a decorator, even if CPython will not itself do anything with those annotations. If Guido's suggestion is accepted, any linter, editor, IDE, Python compiler, or other static tool will be entitled to assume that annotations are type hints, using the mypy-derived syntax, at least unless explicitly told not to. -- Steven From abarnert at yahoo.com Wed Aug 20 16:55:00 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 20 Aug 2014 07:55:00 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> Message-ID: <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> On Aug 20, 2014, at 7:25, Donald Stufft wrote: > >> On Aug 20, 2014, at 10:11 AM, Brett Cannon wrote: >> >> >> >> On Wed Aug 20 2014 at 9:29:34 AM Paul Moore wrote: >>> On 14 August 2014 00:30, Guido van Rossum wrote: >>> > We certainly *could* do that. However, I haven't seen sufficient other uses >>> > of annotations. If there is only one use for annotations (going forward), >>> > annotations would be unambiguous. If we allow different types of >>> > annotations, there would have to be a way to tell whether a particular >>> > annotation is intended as a type annotation or not. Currently mypy ignores >>> > all modules that don't import typing.py (using any form of import >>> > statement), and we could continue this convention. But it would mean that >>> > something like this would still require the typing import in order to be >>> > checked by mypy: >>> > >>> > import typing >>> > >>> > def gcd(int a, int b) -> int: >>> > >>> >>> Sorry, I'm slowly going through this thread, so my apologies if this >>> has been covered later, but it seems to me that projects (and in >>> particular libraries) that want to target 2.7 as well as 3.x will be >>> forced to avoid this feature. And avoid any dependencies that use it. >>> >>> Specifically, the annotation syntax is purely Python 3.x, so without >>> some form of translation or import hook, 2.7 won't accept it. And >>> adding a dependency (even if it's only a very lightweight typing.py >>> plus a hook installed somewhere/somehow) for something that only has >>> value as part of a developer-level type check doesn't seem like a good >>> solution. >>> >>> So, I'd like someone to explain (maybe by pointing me to relevant mypy >>> docs, I haven't investigated them yet) how I should modify my existing >>> code that supports 2.7 and 3.x so that it uses the new functionality >>> *without* extra runtime dependencies that only deliver build/test-time >>> benefits (that's the problem I always had with >>> setuptools/pkg_resources, and I'd not like to see it repeated here). >> >> I suspect the answer is "you don't". Just like everything else that is syntactically exclusive to Python 3, it's a perk you get with Python 3-only code that you simply can't get if you want to maintain backwards-compatibility. This seems like much more of an issue for libraries than for applications. It's not a big deal for an app to require Python 3.5, but most libraries out there are supporting, e.g., 2.7/3.3+, and I think that will continue for quite some time. You want your library to be useful to people whose apps are in 2.x and who don't yet have a compelling reason to port them, unless you're confident that your library in itself is compelling enough to be worth the effort and risk. > > mypy does have a codec that will ignore annotations on 2.x. But other than that the answer is you don?t. If the issue here is not wanting to create a new runtime dependency, that should be pretty easy to solve by just applying the codec at build time for 2.x installs. If the codec isn't easy enough to use from setup.py, that shouldn't be too hard to fix. And maybe the codec (or equivalent functionality) could be added to setuptools, so it doesn't add an extra build-time dependency to most projects. I realize that at first glance this sounds similar to saying "just run 2to3 or 3to2 at install time", which turned out to be unrealistic for many libraries. But the difference is that this is a very simple, specific transformation that should have no semantic impact on your runtime code, so I don't think it's naive to expect that it would be simple and reliable as a fully automated transformation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Aug 20 17:08:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 20 Aug 2014 16:08:45 +0100 Subject: [Python-ideas] Annotations (and static typing) Message-ID: Sigh. I go away for a week and come back to a mega-thread I can never hope to catch up on :-) TL; DR; - Although mypy looks interesting, I think it's too soon to close the door on all other uses of annotations. Let's find a solution that allows exploration of alternative uses for a while longer. OK, can I just make some points regarding the static typing thread. First of all, I have no issue with the idea of static typing, and in fact I look forward to seeing what benefits it might have (if nothing else, the pointer to mypy, which I'd never heard of before, is appreciated). It won't be something I use soon (see below) but that's fine. But Guido seems to be saying (on a number of occasions) that nobody is really using annotations, so he wants to focus on the static typing use case alone. I think this is a mistake. First of all, I see no reason why functions using typing annotations could not be introduced with a decorator[1]. So why insist that this is the *only* use of annotations, when it's pretty easy to allow others to co-exist? Also, the "nobody is using annotations" argument? Personally, I know of a few other uses: 1. Argument parsers - at least 3 have been mentioned in the thread. 2. Structure unpacking - I think there is a library that uses annotations for this, although I may be wrong. 3. FFI bindings. I know I've seen this discussed, although I can't find a reference just now. There are probably other ideas around as well (GUI bindings, documentation generation, ...) None are particularly mature, and most are just at the "ideas" stage, but typically the ideas I have seen are the sort of thing you'd write a library for, and Python 3 only libraries *really* aren't common yet. The problem for people wanting to experiment with annotations, is that they need to be writing Python 3 only code. While Python 3 adoption is growing rapidly, I suspect that large applications are typically still focused on either going through, or tidying up after, a 2-3 migration. And new projects, while they may be developed from the ground up using Python 3, will typically be using programmers skilled in Python 2, to whom Python 3 features are not yet an "instinctive" part of the toolset. Apart from large standalone applications, there are smaller scripts (which are typically going to be too small to need programming-in-the-large features like annotations) and libraries (which really aren't yet in a position to drop Python 2.x totally, unless they have a fairly small user base). So I don't see it as compelling that usage of annotations in the wild is not yet extensive. Rather than close the door on alternative uses of annotations, can I suggest: 1. By all means bless mypy syntax as the standard static typing notation - this seems like a good thing. 2. Clarify that static typing annotations should be introduced with a decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy declaration in the stdlib, and have a registration protocols for tools to hook their own implementation into it.[2] 3. Leave the door open for other uses of decorators, at least until some of the more major libraries drop Python 2.x support completely (and hence can afford to have a dependency on a Python 3 only module that uses annotations). See at that stage if annotations take off. 4. If we get to a point where even libraries that *could* use annotations don't, then revisit the idea of restricting usage to just type information. Paul [1] Also, a decorator could allow a Python 2 compatible form by using decorator arguments as an alternative to annotations. [2] I've yet to see a clear explanation of how "a tool using type annotations" like an linter, editor, IDE or Python compiler would use them in such a way that precludes decoration as a means of signalling the annotation semantics. From guido at python.org Wed Aug 20 17:53:02 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 20 Aug 2014 08:53:02 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: Was that meant to support Antoine's position (ABCs are not type descriptions) or mine (they are)? On Wed, Aug 20, 2014 at 4:27 AM, Nick Coghlan wrote: > > On 20 Aug 2014 10:14, "Guido van Rossum" wrote: > > > > On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou > wrote: > >> > >> Le 19/08/2014 18:50, Terry Reedy a ?crit : > >> > >>> On 8/19/2014 9:27 AM, Nick Coghlan wrote: > >>> > >>> What I like: 'optional type hints' based on a fleshed-out ABC systems > >>> that collects all ABCs together on one module via import from current > >>> files. It seems that we have been slowly groping towards this for > years. > >> > >> > >> Hmm, I've been saying this already, but my intuition is that it's a bad > idea to conflate *type descriptions* (what this proposal is about) and > actual *runtime types* (what ABCs are). > > > > > > But are they? I think the registration mechanism makes it clear that > they aren't (necessarily) runtime types -- by linking a concrete type with > an ABC through registration you are pretty clearly stating that the ABC > *describes* (an aspect of) the concrete class without automatically adding > any behavior from the ABC to it. > > One of the ways I describe the explicit registration mechanism to people > is as giving you the option of lying to the interpreter. If you use > explicit registration, your registered type basically walks around with a > sign saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and > assumes it will be able to quack without actually checking in advance. It's > really just a constrained version of normal duck typing, where the > interpreter just assumes that *everything* may exhibit duck-like tendencies. > > Cheers, > Nick. > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Aug 20 18:04:31 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 20 Aug 2014 09:04:31 -0700 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: Message-ID: I expect that we'll find a way for type and other annotations to coexist, but I want to correct what I am quoted as having said. I didn't just say there was little use; I also said that my original intent was to have type annotations (see the 2004-2005 Artima blog posts referenced in the threads) and accepted PEP 3107 in the expectation that it would enable 3rd party experiments to determine the right syntax for type annotations. IMO mypy fits right into that category. (Admittedly I had a long discussion with the author at PyCon 2013 where I convinced him to change his syntax. :-) On Wed, Aug 20, 2014 at 8:08 AM, Paul Moore wrote: > Sigh. I go away for a week and come back to a mega-thread I can never > hope to catch up on :-) > > TL; DR; - Although mypy looks interesting, I think it's too soon to > close the door on all other uses of annotations. Let's find a solution > that allows exploration of alternative uses for a while longer. > > OK, can I just make some points regarding the static typing thread. > First of all, I have no issue with the idea of static typing, and in > fact I look forward to seeing what benefits it might have (if nothing > else, the pointer to mypy, which I'd never heard of before, is > appreciated). It won't be something I use soon (see below) but that's > fine. > > But Guido seems to be saying (on a number of occasions) that nobody is > really using annotations, so he wants to focus on the static typing > use case alone. I think this is a mistake. First of all, I see no > reason why functions using typing annotations could not be introduced > with a decorator[1]. So why insist that this is the *only* use of > annotations, when it's pretty easy to allow others to co-exist? > > Also, the "nobody is using annotations" argument? Personally, I know > of a few other uses: > > 1. Argument parsers - at least 3 have been mentioned in the thread. > 2. Structure unpacking - I think there is a library that uses > annotations for this, although I may be wrong. > 3. FFI bindings. I know I've seen this discussed, although I can't > find a reference just now. > > There are probably other ideas around as well (GUI bindings, > documentation generation, ...) None are particularly mature, and most > are just at the "ideas" stage, but typically the ideas I have seen are > the sort of thing you'd write a library for, and Python 3 only > libraries *really* aren't common yet. > > The problem for people wanting to experiment with annotations, is that > they need to be writing Python 3 only code. While Python 3 adoption is > growing rapidly, I suspect that large applications are typically still > focused on either going through, or tidying up after, a 2-3 migration. > And new projects, while they may be developed from the ground up using > Python 3, will typically be using programmers skilled in Python 2, to > whom Python 3 features are not yet an "instinctive" part of the > toolset. Apart from large standalone applications, there are smaller > scripts (which are typically going to be too small to need > programming-in-the-large features like annotations) and libraries > (which really aren't yet in a position to drop Python 2.x totally, > unless they have a fairly small user base). > > So I don't see it as compelling that usage of annotations in the wild > is not yet extensive. > > Rather than close the door on alternative uses of annotations, can I > suggest: > > 1. By all means bless mypy syntax as the standard static typing > notation - this seems like a good thing. > 2. Clarify that static typing annotations should be introduced with a > decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy > declaration in the stdlib, and have a registration protocols for tools > to hook their own implementation into it.[2] > 3. Leave the door open for other uses of decorators, at least until > some of the more major libraries drop Python 2.x support completely > (and hence can afford to have a dependency on a Python 3 only module > that uses annotations). See at that stage if annotations take off. > 4. If we get to a point where even libraries that *could* use > annotations don't, then revisit the idea of restricting usage to just > type information. > > Paul > > [1] Also, a decorator could allow a Python 2 compatible form by using > decorator arguments as an alternative to annotations. > [2] I've yet to see a clear explanation of how "a tool using type > annotations" like an linter, editor, IDE or Python compiler would use > them in such a way that precludes decoration as a means of signalling > the annotation semantics. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From burak.arslan at arskom.com.tr Wed Aug 20 17:58:52 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Wed, 20 Aug 2014 18:58:52 +0300 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: <53F4C5BC.8080008@arskom.com.tr> Chiming in as well (first post to python-ideas, fingers crossed!), because Spyne (I'm the author) will have to interoperate with this sooner or later :) To put my opinions in context: http://spyne.io/docs/2.10/manual/03_types.html On 08/13/14 22:44, Guido van Rossum wrote: > [There is no TL;DR other than the subject line. Please read the whole > thing before replying. I do have an appendix with some motivations for > adding type annotations at the end.] > Not only that, but I went through most of the thread as well :) > > To read up on mypy's annotation syntax, please see the mypy-lang.org > website. Here's just one complete example, to > give a flavor: > > from typing import List, Dict > > def word_count(input: List[str]) -> Dict[str, int]: > result = {} #type: Dict[str, int] > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result > I couldn't agree more with Dennis Brakhane saying: > "I would be very sad to see annotations being limited to convey type > information." It'd be very tedious for us corporate web-service guys having to specify things twice to get the advantages of Python's static typing -- whatever they might end up being. But it's not just that -- str[120] is a kind of string (of length 120) the same way list[str] is a kind of list (that only contains strings). But this gets out of hand fast: Why not say str[6,32,'[A-Fa-f0-9]+'] is a hex string between lengths 6 and 32? So my proposition is to have __getitem__ accept **kwargs. That way the above becomes: HexString = str[min_length=6, max_length=32, pattern='[A-Fa-f0-9]+'] This would elegantly blend in with duck typing as well. object[write=callable] is a file-like object. Or maybe I should say object[write=callable[bytes]] ? Alternatively, having what I call "specializers" in typing.py could work just as well: HexString = str[min_length(6), max_length(32), pattern('[A-Fa-f0-9]+')] IntToStrMapping = dict[key(int), value(str)] But it looks hideous and as a bonus, invalid specializations like dict[pattern('[a-z0-9]+')] will have to be dealt with. Or, it's also possible to do what we're doing with Spyne and leave builtins like str, list etc. alone and introduce a separate set of markers to be used exclusively for type annotation purposes: HexString = Unicode(min_length=6, max_length=32, pattern='[A-Fa-f0-9]+') IntToStrMapping = Dict(key=int, value=str) This has the disadvantage of doing generics with parens, which turned out to be pretty confusing for some new users of Spyne. While we're into the run-time checked types territory, why make do with declarative restrictions and not pass arbitrary callables to validate incoming objects? e.g. PngData = bytes[validator=ensure_png] where ensure_png is a callable[bytes] that ensures that the incoming bytes object is indeed a valid png data. Now you might think this proposal an abomination in the context of a community of "consenting adults". But when you let your functions be called by untrusted sources (which is, I'll argue, pretty much anyone whose code listens to a socket one way or the other) precisely expressing and applying such constraints is very very important. IMO, how this kind of information is passed around needs to be standardized for the same reasons Python got a standard event loop. We already have a high amount of largely incompatible systems of varying levels of sophistication and purposes for enforcing constraints to incoming data. I hope all this makes sense. Best regards, Burak -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Aug 20 18:26:23 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 20 Aug 2014 09:26:23 -0700 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: Message-ID: <53F4CC2F.70308@stoneleaf.us> On 08/20/2014 09:04 AM, Guido van Rossum wrote: > > I expect that we'll find a way for type and other annotations to coexist [...] Another thing to keep in mind, and which my scription utility will probably switch to, is that we are not stuck only with __annotations__ -- we can also add other attributes, such as __scription__ or __plac__ or __my_custom_stuff__ or whatever. It could easily be that coexisting means use a decorator instead of annotations, and store the info in a different attribute. As a bonus this method is even 2.x compatible. -- ~Ethan~ From p.f.moore at gmail.com Wed Aug 20 18:27:58 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 20 Aug 2014 17:27:58 +0100 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: Message-ID: On 20 August 2014 17:04, Guido van Rossum wrote: > I expect that we'll find a way for type and other annotations to coexist, > but I want to correct what I am quoted as having said. I didn't just say > there was little use; I also said that my original intent was to have type > annotations (see the 2004-2005 Artima blog posts referenced in the threads) > and accepted PEP 3107 in the expectation that it would enable 3rd party > experiments to determine the right syntax for type annotations. IMO mypy > fits right into that category. (Admittedly I had a long discussion with the > author at PyCon 2013 where I convinced him to change his syntax. :-) Sorry, yeah. I glossed over the point about having type annotations and converging on the right syntax - as I said, I'm interested in that, but not so much in the short term. Paul From edk141 at gmail.com Wed Aug 20 18:22:13 2014 From: edk141 at gmail.com (Ed Kellett) Date: Wed, 20 Aug 2014 17:22:13 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: This proposal effectively negates any benefit to adding annotations to Python. Annotations' only function was to provide unspecified information about parameters and return values to other Python at runtime. That goes with the acceptance of this proposal: It's still their sole function, but now we're *not allowed to use them* to do that any more. Annotations alone are syntax that corresponds to exactly no semantic meaning, and far from encouraging developers to assign meaning to them, you're banning them from doing so. In its current form, Python code can't even make use of the annotations that are allowed: mypy's type system is not Python's, and without a specification of how the static type checker's type system actually works, anything looking at the annotations from the Python side would have to come up with its own way to interpret them. You could have picked *any* other way to communicate static typing information, and it wouldn't neuter a feature so completely: - use a @typechecked decorator to signify that a decorated function's annotations are for use as type information - use a certain set of annotations to denote type information, without deprecating the use of annotations for other purposes - parse a standardized docstring format for type information - add new syntax for expressing type information without conflicting with annotations: def foo(int bar), for example Static typing is what it is and I won't try to convince you that it's not worth adding to the language, but it's not worth rendering a feature useless when you could *not* render that feature useless and retain all the same benefits. Ed Kellett On 13 August 2014 20:44, Guido van Rossum wrote: > [There is no TL;DR other than the subject line. Please read the whole thing > before replying. I do have an appendix with some motivations for adding type > annotations at the end.] > > Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man > of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: > http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can > learn from Haskell (and other languages); yesterday he gave the same talk at > Dropbox. The talk is online > (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad > strokes comes down to three suggestions: > > (a) Python should adopt mypy's syntax for function annotations > (b) Python's use of mutabe containers by default is wrong > (c) Python should adopt some kind of Abstract Data Types > > Proposals (b) and (c) don't feel particularly actionable (if you disagree > please start a new thread, I'd be happy to discuss these further if there's > interest) but proposal (a) feels right to me. > > So what is mypy? It is a static type checker for Python written by Jukka > for his Ph.D. thesis. The basic idea is that you add type annotations to > your program using some custom syntax, and when running your program using > the mypy interpreter, type errors will be found during compilation (i.e., > before the program starts running). > > The clever thing here is that the custom syntax is actually valid Python 3, > using (mostly) function annotations: your annotated program will still run > with the regular Python 3 interpreter. In the latter case there will be no > type checking, and no runtime overhead, except to evaluate the function > annotations (which are evaluated at function definition time but don't have > any effect when the function is called). > > In fact, it is probably more useful to think of mypy as a heavy-duty linter > than as a compiler or interpreter; leave the type checking to mypy, and the > execution to Python. It is easy to integrate mypy into a continuous > integration setup, for example. > > To read up on mypy's annotation syntax, please see the mypy-lang.org > website. Here's just one complete example, to give a flavor: > > from typing import List, Dict > > def word_count(input: List[str]) -> Dict[str, int]: > result = {} #type: Dict[str, int] > for line in input: > for word in line.split(): > result[word] = result.get(word, 0) + 1 > return result > > Note that the #type: comment is part of the mypy syntax; mypy uses comments > to declare types in situations where no syntax is available -- although this > particular line could also be written as follows: > > result = Dict[str, int]() > > Either way the entire function is syntactically valid Python 3, and a > suitable implementation of typing.py (containing class definitions for List > and Dict, for example) can be written to make the program run correctly. One > is provided as part of the mypy project. > > I should add that many of mypy's syntactic choices aren't actually new. The > basis of many of its ideas go back at least a decade: I blogged about this > topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- > see also the two followup posts linked from the top there). > > I'll emphasize once more that mypy's type checking happens in a separate > pass: no type checking happens at run time (other than what the interpreter > already does, like raising TypeError on expressions like 1+"1"). > > There's a lot to this proposal, but I think it's possible to get a PEP > written, accepted and implemented in time for Python 3.5, if people are > supportive. I'll go briefly over some of the action items. > > (1) A change of direction for function annotations > > PEP 3107, which introduced function annotations, is intentional > non-committal about how function annotations should be used. It lists a > number of use cases, including but not limited to type checking. It also > mentions some rejected proposals that would have standardized either a > syntax for indicating types and/or a way for multiple frameworks to attach > different annotations to the same function. AFAIK in practice there is > little use of function annotations in mainstream code, and I propose a > conscious change of course here by stating that annotations should be used > to indicate types and to propose a standard notation for them. > > (We may have to have some backwards compatibility provision to avoid > breaking code that currently uses annotations for some other purpose. > Fortunately the only issue, at least initially, will be that when running > mypy to type check such code it will produce complaints about the > annotations; it will not affect how such code is executed by the Python > interpreter. Nevertheless, it would be good to deprecate such alternative > uses of annotations.) > > (2) A specification for what to add to Python 3.5 > > There needs to be at least a rough consensus on the syntax for annotations, > and the syntax must cover a large enough set of use cases to be useful. Mypy > is still under development, and some of its features are still evolving > (e.g. unions were only added a few weeks ago). It would be possible to argue > endlessly about details of the notation, e.g. whether to use 'list' or > 'List', what either of those means (is a duck-typed list-like type > acceptable?) or how to declare and use type variables, and what to do with > functions that have no annotations at all (mypy currently skips those > completely). > > I am proposing that we adopt whatever mypy uses here, keeping discussion of > the details (mostly) out of the PEP. The goal is to make it possible to add > type checking annotations to 3rd party modules (and even to the stdlib) > while allowing unaltered execution of the program by the (unmodified) Python > 3.5 interpreter. The actual type checker will not be integrated with the > Python interpreter, and it will not be checked into the CPython repository. > The only thing that needs to be added to the stdlib is a copy of mypy's > typing.py module. This module defines several dozen new classes (and a few > decorators and other helpers) that can be used in expressing argument types. > If you want to type-check your code you have to download and install mypy > and run it separately. > > The curious thing here is that while standardizing a syntax for type > annotations, we technically still won't be adopting standard rules for type > checking. This is intentional. First of all, fully specifying all the type > checking rules would make for a really long and boring PEP (a much better > specification would probably be the mypy source code). Second, I think it's > fine if the type checking algorithm evolves over time, or if variations > emerge. The worst that can happen is that you consider your code correct but > mypy disagrees; your code will still run. > > That said, I don't want to completely leave out any specification. I want > the contents of the typing.py module to be specified in the PEP, so that it > can be used with confidence. But whether mypy will complain about your > particular form of duck typing doesn't have to be specified by the PEP. > Perhaps as mypy evolves it will take options to tell it how to handle > certain edge cases. Forks of mypy (or entirely different implementations of > type checking based on the same annotation syntax) are also a possibility. > Maybe in the distant future a version of Python will take a different > stance, once we have more experience with how this works out in practice, > but for Python 3.5 I want to restrict the scope of the upheaval. > > Appendix -- Why Add Type Annotations? > > The argument between proponents of static typing and dynamic typing has been > going on for many decades. Neither side is all wrong or all right. Python > has traditionally fallen in the camp of extremely dynamic typing, and this > has worked well for most users, but there are definitely some areas where > adding type annotations would help. > > - Editors (IDEs) can benefit from type annotations; they can call out > obvious mistakes (like misspelled method names or inapplicable operations) > and suggest possible method names. Anyone who has used IntelliJ or Xcode > will recognize how powerful these features are, and type annotations will > make such features more useful when editing Python source code. > > - Linters are an important tool for teams developing software. A linter > doesn't replace a unittest, but can find certain types of errors better or > quicker. The kind of type checking offered by mypy works much like a linter, > and has similar benefits; but it can find problems that are beyond the > capabilities of most linters. > > - Type annotations are useful for the human reader as well! Take the above > word_count() example. How long would it have taken you to figure out the > types of the argument and return value without annotations? Currently most > people put the types in their docstrings; developing a standard notation for > type annotations will reduce the amount of documentation that needs to be > written, and running the type checker might find bugs in the documentation, > too. Once a standard type annotation syntax is introduced, it should be > simple to add support for this notation to documentation generators like > Sphinx. > > - Refactoring. Bob's talk has a convincing example of how type annotations > help in (manually) refactoring code. I also expect that certain automatic > refactorings will benefit from type annotations -- imagine a tool like 2to3 > (but used for some other transformation) augmented by type annotations, so > it will know whether e.g. x.keys() is referring to the keys of a dictionary > or not. > > - Optimizers. I believe this is actually the least important application, > certainly initially. Optimizers like PyPy or Pyston wouldn't be able to > fully trust the type annotations, and they are better off using their > current strategy of optimizing code based on the types actually observed at > run time. But it's certainly feasible to imagine a future optimizer also > taking type annotations into account. > > -- > --Guido "I need a new hobby" van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From p.f.moore at gmail.com Wed Aug 20 18:31:50 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 20 Aug 2014 17:31:50 +0100 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: <53F4CC2F.70308@stoneleaf.us> References: <53F4CC2F.70308@stoneleaf.us> Message-ID: On 20 August 2014 17:26, Ethan Furman wrote: > On 08/20/2014 09:04 AM, Guido van Rossum wrote: >> >> I expect that we'll find a way for type and other annotations to coexist >> [...] > > Another thing to keep in mind, and which my scription utility will probably > switch to, is that we are not stuck only with __annotations__ -- we can also > add other attributes, such as __scription__ or __plac__ or > __my_custom_stuff__ or whatever. > > It could easily be that coexisting means use a decorator instead of > annotations, and store the info in a different attribute. As a bonus this > method is even 2.x compatible. The obvious thought is, if a decorator is a sufficiently good notation, is there any need for annotations at all (even for static typing)? But that argument does seem to imply that using annotations for anything *other* than static typing is pointless. (As in, has more disadvantages than advantages). I wonder if that's the real issue here? Paul From ethan at stoneleaf.us Wed Aug 20 18:43:03 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 20 Aug 2014 09:43:03 -0700 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: <53F4CC2F.70308@stoneleaf.us> Message-ID: <53F4D017.4040300@stoneleaf.us> On 08/20/2014 09:31 AM, Paul Moore wrote: > On 20 August 2014 17:26, Ethan Furman wrote: >> On 08/20/2014 09:04 AM, Guido van Rossum wrote: >>> >>> I expect that we'll find a way for type and other annotations to coexist >>> [...] >> >> Another thing to keep in mind, and which my scription utility will probably >> switch to, is that we are not stuck only with __annotations__ -- we can also >> add other attributes, such as __scription__ or __plac__ or >> __my_custom_stuff__ or whatever. >> >> It could easily be that coexisting means use a decorator instead of >> annotations, and store the info in a different attribute. As a bonus this >> method is even 2.x compatible. > > The obvious thought is, if a decorator is a sufficiently good > notation, is there any need for annotations at all (even for static > typing)? What can one do with an annotation that one cannot do with a decorator? -- ~Ethan~ From alexander.belopolsky at gmail.com Wed Aug 20 18:51:39 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 20 Aug 2014 12:51:39 -0400 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: <53F4D017.4040300@stoneleaf.us> References: <53F4CC2F.70308@stoneleaf.us> <53F4D017.4040300@stoneleaf.us> Message-ID: On Wed, Aug 20, 2014 at 12:43 PM, Ethan Furman wrote: > What can one do with an annotation that one cannot do with a decorator? Spell "returns" in two characters. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed Aug 20 19:23:02 2014 From: mertz at gnosis.cx (David Mertz) Date: Wed, 20 Aug 2014 10:23:02 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: I have to say that I find the capabilities in PyContracts--which are absent currently in mypy--to be very compelling. If we are going to add encouragement to use type annotations, adding richer types seems extremely worthwhile. I do not see in any of the discussion a way that mypy/typing.py could express a concept like "A list/sequence of at least N elements, each of which has some property (as well as a basic data type)." In fact, I have trouble imagining how a native syntax could do this, rather than using a custom DSL like PyContracts uses. Well, we could do it by sticking lambdas in for properties, but that would get very ugly quickly. 2) Using annotations: > > @contract > def mysum(l: 'list[>=1](int,>0)') -> 'int,>0': > ... > > 3) Using docstrings, with the :type: and :rtype: tags: > > @contract > def mysum(l): > """ > :type l: list[>=1](int,>0) > :rtype: int,>0 > """ > I don't really care about the annotations vs. docstring issue much. But incorporating pre/post-conditions as part of typing makes a whole lot of sense to me. > @contract > def matrix_multiply(a, b): > ''' Multiplies two matrices together. > > :param a: The first matrix. Must be a 2D array. > :type a: array[MxN],M>0,N>0 > > :param b: The second matrix. Must be of compatible dimensions. > :type b: array[NxP],P>0 > > :rtype: array[MxP] > ''' > return numpy.dot(a, b) > This is also quite lovely and powerful (and not doable in mypy), as is the error report produced. > Example: > > a = numpy.zeros((2,2)) > b = numpy.zeros((3,2)) > matrix_multiply(a,b) > > Exception: > > Breach for argument 'b' to matrix_multiply(). > Expected value for 'N' was: Instance of int: 2 > instead I received: Instance of int: 3 > checking: N for value: Instance of int: 3 > checking: NxP for value: Instance of tuple: (3, 2) > checking: array[NxP] for value: array['3x2'](float64) > array([[ 0., 0.], [ 0., 0.], ... [clip] > checking: array[NxP],P>0 for value: array['3x2'](float64) > array([[ 0., 0.], [ 0., 0.], ... [clip] > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From censi at mit.edu Wed Aug 20 19:37:21 2014 From: censi at mit.edu (Andrea Censi) Date: Wed, 20 Aug 2014 13:37:21 -0400 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: <53F4CC2F.70308@stoneleaf.us> <53F4D017.4040300@stoneleaf.us> Message-ID: On Wed, Aug 20, 2014 at 12:51 PM, Alexander Belopolsky wrote: >> What can one do with an annotation that one cannot do with a decorator? > > > Spell "returns" in two characters. True. (Is there any alternative to using a "returns" argument in the decorator?) We can also compile another list: What can one do with a decorator that cannot be done with annotations? 1) Comment out type checking with one character. (Admittedly, this is thinking short term --- it's a limitation of the editors rather than the language itself. We can imagine that language-aware editors of the future will be more flexible. For example, the editor would have a shortcut to hide or disable type annotations.) 2) Easily parse (with your eyes) the function signature without having to think about types. ... A. From mertz at gnosis.cx Wed Aug 20 19:50:18 2014 From: mertz at gnosis.cx (David Mertz) Date: Wed, 20 Aug 2014 10:50:18 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: Hi Bob, I enjoyed watching your talk (on video, not live), and I certainly see the appeal of a Haskell-like type system. However, what we seem to be discussing here with mypy annotations looks a lot closer to a C type system than to a Haskell type system. All those things that PyContracts get us are easy enough in Haskell--why aim so low if we are thinking of a change in Python? On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito wrote: > On Wed, Aug 20, 2014 at 10:23 AM, David Mertz wrote: > >> I have to say that I find the capabilities in PyContracts--which are >> absent currently in mypy--to be very compelling. If we are going to add >> encouragement to use type annotations, adding richer types seems extremely >> worthwhile. >> > > It's easy to check any expressible condition at runtime, but this is not > the case ahead of time as with a static analysis tool or linter. It'd be > great to have this sort of capability in mypy, but what you're asking for > is at the fringes of current research and will not be practical to add to > Python any time soon. > > -bob > > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob at redivi.com Wed Aug 20 19:42:55 2014 From: bob at redivi.com (Bob Ippolito) Date: Wed, 20 Aug 2014 10:42:55 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: On Wed, Aug 20, 2014 at 10:23 AM, David Mertz wrote: > I have to say that I find the capabilities in PyContracts--which are > absent currently in mypy--to be very compelling. If we are going to add > encouragement to use type annotations, adding richer types seems extremely > worthwhile. > It's easy to check any expressible condition at runtime, but this is not the case ahead of time as with a static analysis tool or linter. It'd be great to have this sort of capability in mypy, but what you're asking for is at the fringes of current research and will not be practical to add to Python any time soon. -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Aug 20 20:13:06 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 20 Aug 2014 19:13:06 +0100 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: <53F4CC2F.70308@stoneleaf.us> <53F4D017.4040300@stoneleaf.us> Message-ID: On 20 August 2014 18:37, Andrea Censi wrote: > What can one do with a decorator that cannot be done with annotations? Python 2 compatibility. Paul PS This discussion is leading me to feel that annotations aren't really that useful compared to a well-written decorator, so why *not* leave them specifically for static type information, as Guido suggested... From bob at redivi.com Wed Aug 20 20:14:31 2014 From: bob at redivi.com (Bob Ippolito) Date: Wed, 20 Aug 2014 11:14:31 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: There's the misunderstanding: PyContracts style contracts are not "easy enough" in Haskell. mypy's types are not like a C type system. There are a few missing features that are being worked on, but it is much better than some here perceive it to be. On Wednesday, August 20, 2014, David Mertz wrote: > Hi Bob, > > I enjoyed watching your talk (on video, not live), and I certainly see the > appeal of a Haskell-like type system. However, what we seem to be > discussing here with mypy annotations looks a lot closer to a C type system > than to a Haskell type system. All those things that PyContracts get us > are easy enough in Haskell--why aim so low if we are thinking of a change > in Python? > > > On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito > wrote: > >> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz > > wrote: >> >>> I have to say that I find the capabilities in PyContracts--which are >>> absent currently in mypy--to be very compelling. If we are going to add >>> encouragement to use type annotations, adding richer types seems extremely >>> worthwhile. >>> >> >> It's easy to check any expressible condition at runtime, but this is not >> the case ahead of time as with a static analysis tool or linter. It'd be >> great to have this sort of capability in mypy, but what you're asking for >> is at the fringes of current research and will not be practical to add to >> Python any time soon. >> >> -bob >> >> > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed Aug 20 20:38:46 2014 From: mertz at gnosis.cx (David Mertz) Date: Wed, 20 Aug 2014 11:38:46 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: On Wed, Aug 20, 2014 at 11:14 AM, Bob Ippolito wrote: > There's the misunderstanding: PyContracts style contracts are not "easy enough" in Haskell. Well, for example, I found something like this for Haskell: newtype Digit = Digit { digitVal :: Int } deriving (Eq, Ord, Show) mkDigit :: Int -> Maybe Digit mkDigit n | n >= 0 && n < 10 = Just (Digit n) | otherwise = Nothing OK, sure it's not just one line. But now we have a predicate-restricted data type right in the type system. If we introduce a static type system in Python, I'd like it to be able to do that. PyContracts--or something close to it--lets us do that in a DSL. But I would be equally happy to use some Python code that could be tucked away but reused. E.g., completely hypothetical syntax: class Digit(mypy.Int): def __predicate__(self): return 0 <= self < 10 def times_modulo(incr: Digit, num: Int) -> Digit: result = 0 for i in range(num): result += incr result %= 10 return result I could just stick Digit and all my other custom types in 'mytypes.py', and the actual annotations would have a richer type system, plus remain readable (if I chose decent names for the types). > mypy's types are not like a C type system. There are a few missing features that are being worked on, but it is much better than some here perceive it to be. > > > On Wednesday, August 20, 2014, David Mertz wrote: >> >> Hi Bob, >> >> I enjoyed watching your talk (on video, not live), and I certainly see the appeal of a Haskell-like type system. However, what we seem to be discussing here with mypy annotations looks a lot closer to a C type system than to a Haskell type system. All those things that PyContracts get us are easy enough in Haskell--why aim so low if we are thinking of a change in Python? >> >> >> On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito wrote: >>> >>> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz wrote: >>>> >>>> I have to say that I find the capabilities in PyContracts--which are absent currently in mypy--to be very compelling. If we are going to add encouragement to use type annotations, adding richer types seems extremely worthwhile. >>> >>> >>> It's easy to check any expressible condition at runtime, but this is not the case ahead of time as with a static analysis tool or linter. It'd be great to have this sort of capability in mypy, but what you're asking for is at the fringes of current research and will not be practical to add to Python any time soon. >>> >>> -bob >>> >> >> >> >> -- >> Keeping medicines from the bloodstreams of the sick; food >> from the bellies of the hungry; books from the hands of the >> uneducated; technology from the underdeveloped; and putting >> advocates of freedom in prisons. Intellectual property is >> to the 21st century what the slave trade was to the 16th. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Aug 20 20:44:59 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 20 Aug 2014 14:44:59 -0400 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: <53F4CC2F.70308@stoneleaf.us> <53F4D017.4040300@stoneleaf.us> Message-ID: On Wed, Aug 20, 2014 at 2:13 PM, Paul Moore wrote: > On 20 August 2014 18:37, Andrea Censi wrote: > > What can one do with a decorator that cannot be done with annotations? > > Python 2 compatibility. Out of the way mark-up. Compare @types(int, float, returns=float) def foo(x, y): .. to def foo(x:int, y:float) -> float: .. The decorator variant is visually closer to plain unannotated Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaiser.yann at gmail.com Wed Aug 20 21:02:32 2014 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Wed, 20 Aug 2014 21:02:32 +0200 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: <53F4CC2F.70308@stoneleaf.us> <53F4D017.4040300@stoneleaf.us> Message-ID: Let's add convenient namespacing to that list: @annotate('typing', one=str, two=int, three=int) @annotate('cli', two=int, three='t') def func(one, two, *, three=5): ... I might add something like this to sigtools.modifiers.annotate and .specifiers.signature to extract it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Aug 20 21:21:38 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Aug 2014 15:21:38 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> Message-ID: On 8/20/2014 10:55 AM, Andrew Barnert wrote: > This seems like much more of an issue for libraries than for > applications. It's not a big deal for an app to require Python 3.5, but > most libraries out there are supporting, e.g., 2.7/3.3+, and I think > that will continue for quite some time. You want your library to be > useful to people whose apps are in 2.x and who don't yet have a > compelling reason to port them, unless you're confident that your > library in itself is compelling enough to be worth the effort and risk. If the annotation for a library are in a separate skeleton file (which would only run on 3.5+), then users of the library (including the library itself), would not see the annotations unless they run on 3.5+ and look for the annotation file. -- Terry Jan Reedy From ethan at stoneleaf.us Wed Aug 20 21:37:55 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 20 Aug 2014 12:37:55 -0700 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: <53F4CC2F.70308@stoneleaf.us> <53F4D017.4040300@stoneleaf.us> Message-ID: <53F4F913.9030807@stoneleaf.us> On 08/20/2014 12:02 PM, Yann Kaiser wrote: > Let's add convenient namespacing to that list: > > @annotate('typing', one=str, two=int, three=int) > @annotate('cli', two=int, three='t') > def func(one, two, *, three=5): > ... For what I was thinking, the above decorator would store the different annotations in, for example, func._typing and func._cli. -- ~Ethan~ From tjreedy at udel.edu Wed Aug 20 21:40:29 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Aug 2014 15:40:29 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <51F3C4F3-A6A9-4F97-A287-B3CC7608E130@ryanhiebert.com> References: <51F3C4F3-A6A9-4F97-A287-B3CC7608E130@ryanhiebert.com> Message-ID: On 8/20/2014 10:27 AM, Ryan Hiebert wrote: > >> On Aug 20, 2014, at 8:28 AM, Paul Moore >> wrote: >> >> Specifically, the annotation syntax is purely Python 3.x, so >> without some form of translation or import hook, 2.7 won't accept >> it. And adding a dependency (even if it's only a very lightweight >> typing.py plus a hook installed somewhere/somehow) for something >> that only has value as part of a developer-level type check doesn't >> seem like a good solution. >> >> So, I'd like someone to explain (maybe by pointing me to relevant >> mypy docs, I haven't investigated them yet) how I should modify my >> existing code that supports 2.7 and 3.x so that it uses the new >> functionality *without* extra runtime dependencies that only >> deliver build/test-time benefits (that's the problem I always had >> with setuptools/pkg_resources, and I'd not like to see it repeated >> here). > > As you?ve suggested, using annotations won?t work on Python 2 where > there are no annotations. Correct. > This feature would not available for applications that wish to support Python 2. Not exactly true, which is to say, true for run-time use, not true for pre-compile call checking. Guido mostly wants type hints for the latter. Call checking mostly just needs to be done once (on 3.5+). To expand on what I said elsewhere: suppose a 2&3 app A uses 2&3 library L, which has a separate annotation file. You run the call checks on 3.5. Assuming that the 2&3 code of each use bytes and unicode and not str, calls should be good on all versions. (They will be unless the same code produces objects of different types on different versions.) -- Terry Jan Reedy From ahammel87 at gmail.com Wed Aug 20 21:42:42 2014 From: ahammel87 at gmail.com (Alex Hammel) Date: Wed, 20 Aug 2014 12:42:42 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: The problem is that the proposal is to use mypy syntax for *static* type checking. It's really hard to prove that values of Digit never go below zero without doing checks at runtime. Your Haskell example doesn't do that either, actually. Sure, the value of Digit can never be a negative number, but neither is it always a positive number (it might be Nothing). Say I've got a function that requires a non-negative number to work properly, so I specify that it takes one of your Digits. If I mess up and pass it a negative number, the Digit class converts it to Nothing, resulting in a runtime error (or some exceptional runtime behaviour, depending on what I do with that Nothing). What I really want if for the compiler to say "sorry, I'm not compiling this because I can't prove that the input to this function is non-negative". You could probably do this with a sufficiently clever Digit type in Haskell, but I don't see how youc could do it with __predicate__. On Wed, Aug 20, 2014 at 11:38 AM, David Mertz wrote: > On Wed, Aug 20, 2014 at 11:14 AM, Bob Ippolito wrote: > > There's the misunderstanding: PyContracts style contracts are not "easy > enough" in Haskell. > > Well, for example, I found something like this for Haskell: > > newtype Digit = Digit { digitVal :: Int } > deriving (Eq, Ord, Show) > mkDigit :: Int -> Maybe Digit > mkDigit n > | n >= 0 && n < 10 = Just (Digit n) > | otherwise = Nothing > > > OK, sure it's not just one line. But now we have a predicate-restricted > data type right in the type system. If we introduce a static type system > in Python, I'd like it to be able to do that. PyContracts--or something > close to it--lets us do that in a DSL. But I would be equally happy to use > some Python code that could be tucked away but reused. E.g., completely > hypothetical syntax: > > class Digit(mypy.Int): > > def __predicate__(self): > > return 0 <= self < 10 > > def times_modulo(incr: Digit, num: Int) -> Digit: > > result = 0 > > for i in range(num): > > result += incr > > result %= 10 > > return result > > > I could just stick Digit and all my other custom types in 'mytypes.py', > and the actual annotations would have a richer type system, plus remain > readable (if I chose decent names for the types). > > > > mypy's types are not like a C type system. There are a few missing > features that are being worked on, but it is much better than some here > perceive it to be. > > > > > > On Wednesday, August 20, 2014, David Mertz wrote: > >> > >> Hi Bob, > >> > >> I enjoyed watching your talk (on video, not live), and I certainly see > the appeal of a Haskell-like type system. However, what we seem to be > discussing here with mypy annotations looks a lot closer to a C type system > than to a Haskell type system. All those things that PyContracts get us > are easy enough in Haskell--why aim so low if we are thinking of a change > in Python? > >> > >> > >> On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito wrote: > >>> > >>> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz wrote: > >>>> > >>>> I have to say that I find the capabilities in PyContracts--which are > absent currently in mypy--to be very compelling. If we are going to add > encouragement to use type annotations, adding richer types seems extremely > worthwhile. > >>> > >>> > >>> It's easy to check any expressible condition at runtime, but this is > not the case ahead of time as with a static analysis tool or linter. It'd > be great to have this sort of capability in mypy, but what you're asking > for is at the fringes of current research and will not be practical to add > to Python any time soon. > >>> > >>> -bob > >>> > >> > >> > >> > >> -- > >> Keeping medicines from the bloodstreams of the sick; food > >> from the bellies of the hungry; books from the hands of the > >> uneducated; technology from the underdeveloped; and putting > >> advocates of freedom in prisons. Intellectual property is > >> to the 21st century what the slave trade was to the 16th. > > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob at redivi.com Wed Aug 20 22:03:25 2014 From: bob at redivi.com (Bob Ippolito) Date: Wed, 20 Aug 2014 13:03:25 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: On Wed, Aug 20, 2014 at 11:38 AM, David Mertz wrote: > On Wed, Aug 20, 2014 at 11:14 AM, Bob Ippolito wrote: > > There's the misunderstanding: PyContracts style contracts are not "easy > enough" in Haskell. > > Well, for example, I found something like this for Haskell: > > newtype Digit = Digit { digitVal :: Int } > deriving (Eq, Ord, Show) > mkDigit :: Int -> Maybe Digit > mkDigit n > | n >= 0 && n < 10 = Just (Digit n) > | otherwise = Nothing > > > OK, sure it's not just one line. But now we have a predicate-restricted > data type right in the type system. If we introduce a static type system > in Python, I'd like it to be able to do that. [?] > This isn't in the type system. This is what is called a smart constructor [1]. These predicates are strictly a runtime construct, not compile time. Guard syntax (a sequence of pipes each followed by a predicate and then an equal sign and some term) is just a convenient way to write if[/else if?]/else. There are ways to do this specific sort of dependent typing [2] in Haskell using an unholy combination of GHC-specific extensions and the absolute latest compiler. This is exciting research, but not so practical today. [1] http://www.haskell.org/haskellwiki/Smart_constructors [2] http://en.wikipedia.org/wiki/Dependent_type -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From castironpi at gmail.com Wed Aug 20 22:12:01 2014 From: castironpi at gmail.com (Aaron Brady) Date: Wed, 20 Aug 2014 15:12:01 -0500 Subject: [Python-ideas] Mutating while iterating In-Reply-To: References: Message-ID: On Wed, Aug 13, 2014 at 7:21 AM, Aaron Brady wrote: > On Sun, Aug 3, 2014 at 12:46 PM, Aaron Brady wrote: >> On Sat, Jul 26, 2014 at 10:39 PM, Nick Coghlan wrote: [snip] >> >> Python is replete with examples of prohibiting structures which are >> likely bugs but aren't segfaults. There are also reciprocal-- though >> not necessarily inverse-- limitations of the unordered collections >> themselves ("set" and "dict"). The new behavior is transparent for >> the programmer; no possible programs can /rely/ on the existing >> behavior. The new behavior introduces no new objects, possibly except >> "IterationError", no new syntax, and no new costs. >> >> I propose we leave this discussion thread open for the time being. I >> also take the issue of /re/-assigning to keys during iteration to be >> settled as permitted. > > Nick, It works by comparing the state of the container, to its state > when the iterator was opened. We're ensuring it will always have a > unique state, up to comparison. A state can be reused once no > iterators refer to it, hence needing the reference count. A full > "object" is not needed for a memo, only the reference count, but the > "object" is easier and only twice the size, as "PyBaseObject Type" is > allocated anyway. > > I'll point out that among the additional costs that there aren't, > garbage collection isn't any slower, as both "tp traverse" and "tp > clear" are empty in the "PyBaseObject Type" definition, on line > 3511-3512 in "typeobject.c" at time of writing [1]. > > [1] http://svn.python.org/view/python/trunk/Objects/typeobject.c?revision=81744&view=markup#l3511 For ordered containers, there are several consistent behaviors if there's an iterator on the element being removed. 1) Always advance 2) Always retreat 3) Always close 4) Specified in iterator / reverse iter's 5) Specified by "remove" caller Iterators could define a 3rd method for it, or 4th counting "prev". Iterators might be invalidated permanently or until the next call to "Next" in some languages, after the mutating call. It raises the issue of when there's an iterator on the first or last item and it's removed (and others are inserted there). The same sentinel value can't be used before it's started and after it's finished. Either the container or the iterator needs extra storage to distinguish. Though the sizes can be kept the same by changing the semantics slightly. From abarnert at yahoo.com Wed Aug 20 22:14:33 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 20 Aug 2014 13:14:33 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: References: Message-ID: <0BE0982A-54A7-4500-B08D-291956DA1EC6@yahoo.com> On Aug 20, 2014, at 12:42, Alex Hammel wrote: > The problem is that the proposal is to use mypy syntax for static type checking. It's really hard to prove that values of Digit never go below zero without doing checks at runtime. > > Your Haskell example doesn't do that either, actually. Sure, the value of Digit can never be a negative number, but neither is it always a positive number (it might be Nothing). > > Say I've got a function that requires a non-negative number to work properly, so I specify that it takes one of your Digits. If I mess up and pass it a negative number, the Digit class converts it to Nothing, resulting in a runtime error (or some exceptional runtime behaviour, depending on what I do with that Nothing). What I really want if for the compiler to say "sorry, I'm not compiling this because I can't prove that the input to this function is non-negative". If the compiler could say, "I am compiling it because I can prove it's either non-negative or from an un-checkable source, but I'll keep track of which so I can propagate that information", that could be useful too. Of course this is only useful if uncheckable sources are rare and contained in your app, the checker makes it easy to read out what did and didn't get infected with permissiveness, and the type system makes it easy to write things so that you don't have to make each type handle permissiveness manually the way you would in Haskell. There are a couple of ML variants aimed at making this easier, and there's no reason PyConstrajnts couldn't do something similar. > You could probably do this with a sufficiently clever Digit type in Haskell, but I don't see how youc could do it with __predicate__. > > > On Wed, Aug 20, 2014 at 11:38 AM, David Mertz wrote: >> On Wed, Aug 20, 2014 at 11:14 AM, Bob Ippolito wrote: >> > There's the misunderstanding: PyContracts style contracts are not "easy enough" in Haskell. >> >> Well, for example, I found something like this for Haskell: >> >> newtype Digit = Digit { digitVal :: Int } >> deriving (Eq, Ord, Show) >> mkDigit :: Int -> Maybe Digit >> mkDigit n >> | n >= 0 && n < 10 = Just (Digit n) >> | otherwise = Nothing >> >> OK, sure it's not just one line. But now we have a predicate-restricted data type right in the type system. If we introduce a static type system in Python, I'd like it to be able to do that. PyContracts--or something close to it--lets us do that in a DSL. But I would be equally happy to use some Python code that could be tucked away but reused. E.g., completely hypothetical syntax: >> >> class Digit(mypy.Int): >> def __predicate__(self): >> return 0 <= self < 10 >> >> def times_modulo(incr: Digit, num: Int) -> Digit: >> result = 0 >> for i in range(num): >> result += incr >> result %= 10 >> return result >> >> I could just stick Digit and all my other custom types in 'mytypes.py', and the actual annotations would have a richer type system, plus remain readable (if I chose decent names for the types). >> >> >> > mypy's types are not like a C type system. There are a few missing features that are being worked on, but it is much better than some here perceive it to be. >> > >> > >> > On Wednesday, August 20, 2014, David Mertz wrote: >> >> >> >> Hi Bob, >> >> >> >> I enjoyed watching your talk (on video, not live), and I certainly see the appeal of a Haskell-like type system. However, what we seem to be discussing here with mypy annotations looks a lot closer to a C type system than to a Haskell type system. All those things that PyContracts get us are easy enough in Haskell--why aim so low if we are thinking of a change in Python? >> >> >> >> >> >> On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito wrote: >> >>> >> >>> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz wrote: >> >>>> >> >>>> I have to say that I find the capabilities in PyContracts--which are absent currently in mypy--to be very compelling. If we are going to add encouragement to use type annotations, adding richer types seems extremely worthwhile. >> >>> >> >>> >> >>> It's easy to check any expressible condition at runtime, but this is not the case ahead of time as with a static analysis tool or linter. It'd be great to have this sort of capability in mypy, but what you're asking for is at the fringes of current research and will not be practical to add to Python any time soon. >> >>> >> >>> -bob >> >>> >> >> >> >> >> >> >> >> -- >> >> Keeping medicines from the bloodstreams of the sick; food >> >> from the bellies of the hungry; books from the hands of the >> >> uneducated; technology from the underdeveloped; and putting >> >> advocates of freedom in prisons. Intellectual property is >> >> to the 21st century what the slave trade was to the 16th. >> >> >> >> >> -- >> Keeping medicines from the bloodstreams of the sick; food >> from the bellies of the hungry; books from the hands of the >> uneducated; technology from the underdeveloped; and putting >> advocates of freedom in prisons. Intellectual property is >> to the 21st century what the slave trade was to the 16th. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Wed Aug 20 22:29:16 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 21 Aug 2014 06:29:16 +1000 Subject: [Python-ideas] Optional static typing -- late to the party References: <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au> Message-ID: <85lhqi29cz.fsf@benfinney.id.au> Chris Angelico writes: > On Wed, Aug 20, 2014 at 10:53 PM, Ben Finney wrote: > > Steven D'Aprano writes: > > > >> But remember that any annotations in docstrings may not be > >> available at runtime > > > > How so? What conformant Python implementation is discarding > > docstrings from code objects? > > CPython with the -OO flag. Or is that non-conformant? Okay, one can deliberately remove docstrings with an option to the interpreter (the ?-OO? option explicitly has that purpose). So a user wanting to check annotations in docstrings wouldn't use that option. So this case doesn't seem relevant to the discussion. Remember that we're talking about type annotations that are for *static code checkers* to inspect. Docstrings will certainly be available there. -- \ ?? no testimony can be admitted which is contrary to reason; | `\ reason is founded on the evidence of our senses.? ?Percy Bysshe | _o__) Shelley, _The Necessity of Atheism_, 1811 | Ben Finney From ncoghlan at gmail.com Wed Aug 20 23:25:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Aug 2014 07:25:24 +1000 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: On 21 Aug 2014 01:53, "Guido van Rossum" wrote: > > Was that meant to support Antoine's position (ABCs are not type descriptions) or mine (they are)? Yours. The ability to register any type with any ABC means that even an explicit isinstance() ABC check at runtime is only advisory - developers are free to force a "true" answer if they choose to do so. You can even disable a check entirely by registering "object" with the relevant ABC (actually doing that would likely be a *bad idea*, but the interpreter allows it). That puts them squarely in the "descriptions of intent" category for me. A bit more rambling, since I've only been skimming (at best) the type hinting threads: The thing I appreciate getting out of linters is picking up the "this is almost certainly wrong" cases (like trying to call "indx" on something I have said I expect to be a sequence). This is especially useful when code is modified over time, but retains the original annotations documenting the interface assumptions - if you expand the function to cover new types, you'll need to change the annotation, which is also a good reminder that the docs will need updating. I'm not particularly worried if there are still cases where linters go "I have no idea what that object is, so I'll assume anything goes". After all, that's the baseline we start from when we don't run a linter at all. Cheers, Nick. > > > On Wed, Aug 20, 2014 at 4:27 AM, Nick Coghlan wrote: >> >> >> On 20 Aug 2014 10:14, "Guido van Rossum" wrote: >> > >> > On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou wrote: >> >> >> >> Le 19/08/2014 18:50, Terry Reedy a ?crit : >> >> >> >>> On 8/19/2014 9:27 AM, Nick Coghlan wrote: >> >>> >> >>> What I like: 'optional type hints' based on a fleshed-out ABC systems >> >>> that collects all ABCs together on one module via import from current >> >>> files. It seems that we have been slowly groping towards this for years. >> >> >> >> >> >> Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are). >> > >> > >> > But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it. >> >> One of the ways I describe the explicit registration mechanism to people is as giving you the option of lying to the interpreter. If you use explicit registration, your registered type basically walks around with a sign saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and assumes it will be able to quack without actually checking in advance. It's really just a constrained version of normal duck typing, where the interpreter just assumes that *everything* may exhibit duck-like tendencies. >> >> Cheers, >> Nick. > > > > > -- > --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Wed Aug 20 23:57:02 2014 From: antoine at python.org (Antoine Pitrou) Date: Wed, 20 Aug 2014 17:57:02 -0400 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: Le 19/08/2014 20:12, Guido van Rossum a ?crit : > Hmm, I've been saying this already, but my intuition is that it's a > bad idea to conflate *type descriptions* (what this proposal is > about) and actual *runtime types* (what ABCs are). > > > But are they? I think the registration mechanism makes it clear that > they aren't (necessarily) runtime types -- by linking a concrete type > with an ABC through registration you are pretty clearly stating that the > ABC *describes* (an aspect of) the concrete class without automatically > adding any behavior from the ABC to it. Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class. Regards Antoine. From steve at pearwood.info Thu Aug 21 02:02:39 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 21 Aug 2014 10:02:39 +1000 Subject: [Python-ideas] Annotations (and static typing) In-Reply-To: References: <53F4CC2F.70308@stoneleaf.us> Message-ID: <20140821000234.GN25957@ando> On Wed, Aug 20, 2014 at 05:31:50PM +0100, Paul Moore wrote: > The obvious thought is, if a decorator is a sufficiently good > notation, is there any need for annotations at all (even for static > typing)? Decorators are *not* sufficiently good notation. Decorators have many nice uses, but they are second class for this purpose since they require you to write the parameters twice: once in the decorator, and once in the actual function declaration. They also visually separate the parameter from its value. If you're thinking about editing code with one or two such functions, with simple parameter declarations, it's no big deal. They're not *very* far away, a line or two maybe, it's easy to cope. But as the number of parameters increases, or the complexity of the type declaration increases, you get code like this: @annotate(fe=Str[blah blah blah blah]) @annotate(fi=List[blah blah blah blah]) @annotate(fo=Dict[blah blah blah blah]) @annotate(fum=Tuple[blah blah blah blah]) @returns(int) def spam(fe=aaa, fi=bbb, fo=ccc, fum=ddd): ... and now you're looking at a solid wall of annotations, and the distance between the first decorator and the parameter list isn't "a line or two" any more. Or you get something like this, which may be a bit better: @annotate(fe=Str[blah blah blah blah], fi=List[blah blah blah blah], fo=Dict[blah blah blah blah], fum=Tuple[blah blah blah blah]) @returns(int) def spam(fe=aaa, fi=bbb, fo=ccc, fum=ddd): ... but still separates the type of fe from the declaration of fe by four lines. With annotations, everything[1] stays together: def spam(fe:Str[blah blah blah blah]=aaa, fi:List[blah blah blah blah]=bbb, fo:Dict[blah blah blah blah]=ccc, fum:Tuple[blah blah blah blah]=ddd, )->int: ... I'm going to take the liberty of quote Nick from an earlier message: [quote] I once had the "pleasure" of inheriting some code written in K&R style C, where the parameter type declarations were separate from the signature line: void foo(a, b, c) double a; char b; { ... } ANSI C, with inline typing, is far more readable :) [end quote] C programmers, hands up if you prefer to use K&R style declarations? I expect you will be in a very small minority. There is a reason why most languages with type declarations normally put them together with the parameter declarations. If anyone wishes to argue for decorator style *in preference* to annotations, a good place to start is with a list of programming languages which use similar syntax for their type declarations. [1] Not quite, you still have to write documentation separately, but since the documentation for a single parameter might be an entire paragraph of text, we don't want to put that in the parameter list. Some problems are just intractable. -- Steven From steve at pearwood.info Thu Aug 21 02:09:54 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 21 Aug 2014 10:09:54 +1000 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: <85lhqi29cz.fsf@benfinney.id.au> References: <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au> <85lhqi29cz.fsf@benfinney.id.au> Message-ID: <20140821000954.GO25957@ando> On Thu, Aug 21, 2014 at 06:29:16AM +1000, Ben Finney wrote: > Chris Angelico writes: > > > On Wed, Aug 20, 2014 at 10:53 PM, Ben Finney wrote: > > > Steven D'Aprano writes: > > > > > >> But remember that any annotations in docstrings may not be > > >> available at runtime > > > > > > How so? What conformant Python implementation is discarding > > > docstrings from code objects? > > > > CPython with the -OO flag. Or is that non-conformant? > > Okay, one can deliberately remove docstrings with an option to the > interpreter (the ?-OO? option explicitly has that purpose). So a user > wanting to check annotations in docstrings wouldn't use that option. So > this case doesn't seem relevant to the discussion. But "the user" can be two different people: the writer of the code using the annotations, and the person running the code that wants to use annotations. It's the *second* user, not the first, who decides whether or not to use -OO. If you're the original developer, you might insist that your code is not compatible with -OO, but (in my opinion) that's a mean thing to do if it can be avoided. Better to not rely on docstrings for anything other than documentation. > Remember that we're talking about type annotations that are for > *static code checkers* to inspect. Docstrings will certainly be > available there. True, but runtime checks may (or may not) be a part of the code checker. At least, we should not rule that out. -- Steven From ethan at stoneleaf.us Thu Aug 21 02:22:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 20 Aug 2014 17:22:45 -0700 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: <20140821000954.GO25957@ando> References: <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au> <85lhqi29cz.fsf@benfinney.id.au> <20140821000954.GO25957@ando> Message-ID: <53F53BD5.5090902@stoneleaf.us> On 08/20/2014 05:09 PM, Steven D'Aprano wrote: > On Thu, Aug 21, 2014 at 06:29:16AM +1000, Ben Finney wrote: >> >> Remember that we're talking about type annotations that are for >> *static code checkers* to inspect. Docstrings will certainly be >> available there. > > True, but runtime checks may (or may not) be a part of the code checker. > At least, we should not rule that out. This proposal is not about "a code checker" but about *static* typing. Seems to be a lot of people forgetting that *static* (at least in this case), means not actually running the program -- hence, no loss of docstrings. -- ~Ethan~ From stephen at xemacs.org Thu Aug 21 03:42:45 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 21 Aug 2014 10:42:45 +0900 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> Message-ID: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > If the annotation for a library are in a separate skeleton file You mean stub file, right? (Stub is the term used so far in this thread.) To develop Terry's idea a bit more explicitly, ISTM that during early development, you use Python 3 with annotations in file. Then when you need to test Python 2, you use a tool (as yet to be developed) which generates a Python file that has had all annotations stripped, and a stub file[1]. Throw away the stub file, test the Python file, and return to development. For distribution (to *both* Python 3 and Python 2 users) you ignore the original annotated code and distribute the generated Python file and the stubs file. Probably there would be a tool for combining the Python and stub files back into an annotated Python file. The two tools (stripper and annotator) can be very accurate (although the annotator may not produce precisely the style desired by users, the tools probably can be designed so that the re-annotated file accurately reflects the style used by the authors). Is the implied workflow really so burdensome? Especially to downstream? Note that Terry works on IDLE, so it seems likely that at least one Python IDE would support this workflow very well. :-) Footnotes: [1] Yes, I know, the stub file is actually Python, too. You know what I mean. From jim.baker at python.org Thu Aug 21 05:28:34 2014 From: jim.baker at python.org (Jim Baker) Date: Wed, 20 Aug 2014 21:28:34 -0600 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: (My apologies in advance that I have not been able to read all the emails on this set of threads.) Mypy?s approach to parameterizing types using slice notation/__getitem__ is a rather neat hack, given how it does not require any new notation, matches rather similar syntax in languages like Scala and Haskell, and can be readily added to existing types, whether builtins like list, collections like deque, or ABC collection types like Iterator andSequence. Adding such parameterization support to builtin/stdlib types is a compelling reason for including in 3.5. An enthusiastic +1! Such support is also a very good reason for Jython 3.x to target 3.5. However, let's not restrict Python?s support for static typing, or related schemes like gradual typing, to just what Mypy can do now. This should be about a preferred syntax, especially since there?s other active work, some of which I?m involved in like Reticulated ( http://wphomes.soic.indiana.edu/jsiek/), which has the possibility of helping with such issues as better Java integration in Jython, such as with the Clamp project (https://github.com/jimbaker/clamped). Instead, I like what one message stated - that such type expressions in function annotations can be arbitrarily complicated expressions. We can have simple parameterization, but also disjunctive types, and more. In certain simple cases, linters will be able to meaningfully work with such type expressions without having to evaluate the defining module (if not actually run it); in other cases, they won?t. But this is just what linters cope with now. We see something similar in Clamp, since we need to evaluate class definitions ahead-of-time to generate the bytecode for corresponding Java proxy classes - otherwise, we would not have a jar that Java/JVM code could use for linkage. Using type variables is another aspect that can be addressed in this proposed syntax, but it is readily doable without further modification. Consider a type expression like Sequence[T] - the only thing that we need to do is assign the type variable T in the scope of the annotation usage. I?m pretty certain we can do some pretty nice things with such type variable assignments to describe type bounds, variance, and what not, for those so inclined and type checkers that can handle. Again, this keeps things nicely open without undue restrictions. One last thing: let?s not add List, TList, etc. Just use list and other existing types to generate new types. I think this is the main reason why 3.5 support is needed. Being able to use types as objects in this fashion is a wonderful pleasure of working with Python as of new style classes (so I will assume now that we are talking about Python 3.5!). The same goes for putting such support in docstrings (although static type checkers could potentially attempt to cross validate) and in comments. Using stub files in contrast is a great idea, and something that can also be standardized on. Thanks again for a fantastic, if somewhat overwhelming conversation. - Jim On Wed, Aug 20, 2014 at 7:42 PM, Stephen J. Turnbull wrote: > Terry Reedy writes: > > > If the annotation for a library are in a separate skeleton file > > You mean stub file, right? (Stub is the term used so far in this > thread.) > > To develop Terry's idea a bit more explicitly, ISTM that during early > development, you use Python 3 with annotations in file. Then when you > need to test Python 2, you use a tool (as yet to be developed) which > generates a Python file that has had all annotations stripped, and a > stub file[1]. Throw away the stub file, test the Python file, and > return to development. > > For distribution (to *both* Python 3 and Python 2 users) you ignore > the original annotated code and distribute the generated Python file > and the stubs file. Probably there would be a tool for combining the > Python and stub files back into an annotated Python file. > > The two tools (stripper and annotator) can be very accurate (although > the annotator may not produce precisely the style desired by users, > the tools probably can be designed so that the re-annotated file > accurately reflects the style used by the authors). > > Is the implied workflow really so burdensome? Especially to > downstream? > > Note that Terry works on IDLE, so it seems likely that at least one > Python IDE would support this workflow very well. :-) > > Footnotes: > [1] Yes, I know, the stub file is actually Python, too. You know > what I mean. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- - Jim jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com} twitter.com/jimbaker github.com/jimbaker bitbucket.com/jimbaker -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Aug 21 08:03:46 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 21 Aug 2014 02:03:46 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 8/20/2014 9:42 PM, Stephen J. Turnbull wrote: > Terry Reedy writes: > > > If the annotation for a library are in a separate skeleton file > > You mean stub file, right? (Stub is the term used so far in this > thread.) Then yes, that I what i mean ;-). > To develop Terry's idea a bit more explicitly, ISTM that during early > development, you use Python 3 with annotations in file. Then when you > need to test Python 2, you use a tool (as yet to be developed) which > generates a Python file that has had all annotations stripped, and a > stub file[1]. Throw away the stub file, test the Python file, and > return to development. An alternative is to develop code and stub separately. There are people (not core developers) already working on stub files for the stdlib and other libraries. In fact there are multiple groups using different syntaxes. Presumably some will join forces if there is a blessed, standard, syntax. There are no plans to add annotations to the stdlib code itself. Even if one does not add annotations to one own code, which many projects, especially smaller ones, will not, there will be the possibility of having calls into the stdlib and other annotated or stubbed libraries statically checked on 3.5 (or later). One will be able to at least partially benefit without writing annotations oneself. > For distribution (to *both* Python 3 and Python 2 users) you ignore > the original annotated code and distribute the generated Python file > and the stubs file. Probably there would be a tool for combining the > Python and stub files back into an annotated Python file. > > The two tools (stripper and annotator) can be very accurate (although > the annotator may not produce precisely the style desired by users, > the tools probably can be designed so that the re-annotated file > accurately reflects the style used by the authors). I imagine both tools will emerge. > Is the implied workflow really so burdensome? Especially to > downstream? > > Note that Terry works on IDLE, I have been watching this thread precisely from that viewpoint. > so it seems likely that at least one > Python IDE would support this workflow very well. :-) Part this summer's Idle GSOC project was developing a framework to run 3rd party code checkers on code in an Idle editor from the editor. I just need to review and possibly revise the code Saimadhav wrote and decide among the alternatives tried. Acesss to existing pyflakes, pylint, or anything else, once installed (easy now with pip), will soon be possible. Anything in the stdlib, rather than at PyPI or wherever, would likely get even more direct support. For instance, one can check syntax without running the edit buffer, and I believe that just imports and runs tokenize.tokenize. I can imagine doing something similar with stripper, splitter, and joiner functions. (This might be in an extension module so the feature can to disabled, as might be appropriate for a beginner class.) -- Terry Jan Reedy From antoine at python.org Thu Aug 21 15:23:39 2014 From: antoine at python.org (Antoine Pitrou) Date: Thu, 21 Aug 2014 09:23:39 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Le 20/08/2014 23:28, Jim Baker a ?crit : > > Mypy?s approach to parameterizing types using slice > notation/|__getitem__| is a rather neat hack, given how it does not > require any new notation, matches rather similar syntax in languages > like Scala and Haskell, and can be readily added to existing types, > whether builtins like |list|, collections like |deque|, or ABC > collection types like |Iterator| and|Sequence|. Adding such > parameterization support to builtin/stdlib types is a compelling reason > for including in 3.5. An enthusiastic +1! I'm -1 on using __getitem__ for parametering types. It is indeed a hack and its limitations will bite us harshly when wanting to refine the type descriptions (for example to describe a constraint on a collection's size). > One last thing: let?s not add |List|, |TList|, etc. Just use |list| and > other existing types to generate new types. I'm -1 on that, for exactly the same reason as above. Regards Antoine. From brett at python.org Thu Aug 21 16:30:03 2014 From: brett at python.org (Brett Cannon) Date: Thu, 21 Aug 2014 14:30:03 +0000 Subject: [Python-ideas] Optional Static Typing -- the Python Way References: <53F25EE8.8000900@stoneleaf.us> Message-ID: On Wed Aug 20 2014 at 5:58:31 PM Antoine Pitrou wrote: > > Le 19/08/2014 20:12, Guido van Rossum a ?crit : > > Hmm, I've been saying this already, but my intuition is that it's a > > bad idea to conflate *type descriptions* (what this proposal is > > about) and actual *runtime types* (what ABCs are). > > > > > > But are they? I think the registration mechanism makes it clear that > > they aren't (necessarily) runtime types -- by linking a concrete type > > with an ABC through registration you are pretty clearly stating that the > > ABC *describes* (an aspect of) the concrete class without automatically > > adding any behavior from the ABC to it. > > Hmm... well, they are usable at runtime (if only for isinstance calls > :-)). I admit my wording was a bit vague here. But our type descriptions > should be able to express more information than ABCs currently do. For > example, I don't how you'd express the idea of a "mapping from str to > int" using the current Mapping ABC, while retaining the runtime > properties of the Mapping class. > Isn't that the magic of __instancecheck__/__subclasscheck__? If typing.py does the right thing here then it shouldn't matter. You could use __init__ or __getitem__ to construct an object who upon introspection provides the details you want and do what you expect, and then override the appropriate methods so that isinstance() checks check against the ABC instead of the type class itself (heck we could make the base type class require that you inherit from an ABC to promote working with ABCs instead of concrete classes). I have to also say that I like using ABCs because `from collections import abc as a` leads to reading parameters like "x is a.Mapping" which is linguistically quaint. =) P.S.: Off-topic for this email but something I don't think has been mentioned more than once, type hinting like we are proposing is exactly what Dart does and it is nice. Built-in API documentation of interfaces along with IDE support at the API makes all of this worth it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Thu Aug 21 16:54:17 2014 From: antoine at python.org (Antoine Pitrou) Date: Thu, 21 Aug 2014 10:54:17 -0400 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: Le 21/08/2014 10:30, Brett Cannon a ?crit : > > Isn't that the magic of __instancecheck__/__subclasscheck__? If > typing.py does the right thing here then it shouldn't matter. What is "the right thing"? > You could > use __init__ or __getitem__ to construct an object who upon > introspection provides the details you want and do what you expect, and > then override the appropriate methods so that isinstance() checks check > against the ABC instead of the type class itself (heck we could make the > base type class require that you inherit from an ABC to promote working > with ABCs instead of concrete classes). So we agree that type descriptions should be separate things from ABCs, or not? > I have to also say that I like using ABCs because `from collections > import abc as a` leads to reading parameters like "x is a.Mapping" which > is linguistically quaint. =) Except that "x is a.Mapping" is False with a normal Python. Regards Antoine. From abarnert at yahoo.com Thu Aug 21 17:46:46 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 21 Aug 2014 08:46:46 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: <1B33C3C0-A9E9-4380-BAD8-7B35A91FF367@yahoo.com> On Aug 21, 2014, at 7:30, Brett Cannon wrote: > I have to also say that I like using ABCs because `from collections import abc as a` leads to reading parameters like "x is a.Mapping" which is linguistically quaint. =) Cute, but it seems like it actually impedes the reading of "x is an Iterable". (Although I guess as compensation you get "x is an Integer" if you import numbers as n, and you get to yodel like Adam Ant when reading code about file types.) More seriously: I argued along the same lines as you as to the advantages of using the ABCs directly, but I understand the opposing viewpoint: although static type hints and runtime type constraints are _normally_ the same things, they're not _necessarily_ so, and forcing them to share a representation may preclude some of the fancy things people want to do with types beyond the simple generic model (e.g.; the PyConstraints sub-thread). From ncoghlan at gmail.com Thu Aug 21 18:15:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 22 Aug 2014 02:15:53 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 21 August 2014 23:23, Antoine Pitrou wrote: > Le 20/08/2014 23:28, Jim Baker a ?crit : >> >> >> Mypy?s approach to parameterizing types using slice >> notation/|__getitem__| is a rather neat hack, given how it does not >> >> require any new notation, matches rather similar syntax in languages >> like Scala and Haskell, and can be readily added to existing types, >> whether builtins like |list|, collections like |deque|, or ABC >> collection types like |Iterator| and|Sequence|. Adding such >> >> parameterization support to builtin/stdlib types is a compelling reason >> for including in 3.5. An enthusiastic +1! > > > I'm -1 on using __getitem__ for parametering types. It is indeed a hack and > its limitations will bite us harshly when wanting to refine the type > descriptions (for example to describe a constraint on a collection's size). Given: Sequence[Number] Sequence[Number|None] There's still at least three options for extending the syntax even further: Sequence[Number](EXPR) Sequence[Number:EXPR] Sequence[Number,EXPR] You'd probably want to use strings at that point, for the same reasons PyContracts does. For example: Number,"N>0" # Positive number Sequence[Number,"N>0"] # Sequence of positive numbers Sequence[Number],"N>0" # Non-empty sequence I doubt we'll ever get to that point, though. Type hinting and runtime assertions aren't the same thing. >> One last thing: let?s not add |List|, |TList|, etc. Just use |list| and >> >> other existing types to generate new types. > > I'm -1 on that, for exactly the same reason as above. I'm also -1, on the grounds that concrete types and ABCs are different, and I think type hinting should be heavily biased towards ABCs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Thu Aug 21 18:16:18 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 21 Aug 2014 09:16:18 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com> On Aug 20, 2014, at 14:57, Antoine Pitrou wrote: > Le 19/08/2014 20:12, Guido van Rossum a ?crit : >> Hmm, I've been saying this already, but my intuition is that it's a >> bad idea to conflate *type descriptions* (what this proposal is >> about) and actual *runtime types* (what ABCs are). >> >> >> But are they? I think the registration mechanism makes it clear that >> they aren't (necessarily) runtime types -- by linking a concrete type >> with an ABC through registration you are pretty clearly stating that the >> ABC *describes* (an aspect of) the concrete class without automatically >> adding any behavior from the ABC to it. > > Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class. Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping. This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong). Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6. Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime. I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one). From rosuav at gmail.com Thu Aug 21 18:21:17 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 22 Aug 2014 02:21:17 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Aug 22, 2014 at 2:15 AM, Nick Coghlan wrote: > I'm also -1, on the grounds that concrete types and ABCs are > different, and I think type hinting should be heavily biased towards > ABCs. We keep seeing this line of argument coming up, and while I've only been skimming, I haven't seen anyone comment that this applies to parameters but not so much to return values. A function might take a Sequence and iterate over it, and then return a concrete integer. Or it might take a file-like object and a string, and yield a series of strings. Or it might take some set of parameters, and return None unconditionally. There's nothing wrong with being concrete about return values. ChrisA From ncoghlan at gmail.com Thu Aug 21 18:26:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 22 Aug 2014 02:26:24 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 22 August 2014 02:21, Chris Angelico wrote: > On Fri, Aug 22, 2014 at 2:15 AM, Nick Coghlan wrote: >> I'm also -1, on the grounds that concrete types and ABCs are >> different, and I think type hinting should be heavily biased towards >> ABCs. > > We keep seeing this line of argument coming up, and while I've only > been skimming, I haven't seen anyone comment that this applies to > parameters but not so much to return values. A function might take a > Sequence and iterate over it, and then return a concrete integer. Or > it might take a file-like object and a string, and yield a series of > strings. Or it might take some set of parameters, and return None > unconditionally. There's nothing wrong with being concrete about > return values. For standalone functions, I agree. For methods and generics, I'd still be somewhat wary of encouraging it. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From apieum at gmail.com Thu Aug 21 18:43:00 2014 From: apieum at gmail.com (Gregory Salvan) Date: Thu, 21 Aug 2014 18:43:00 +0200 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com> References: <53F25EE8.8000900@stoneleaf.us> <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com> Message-ID: > I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some > realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one). The use case I see is functions composition. I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid. 2014-08-21 18:16 GMT+02:00 Andrew Barnert : > On Aug 20, 2014, at 14:57, Antoine Pitrou wrote: > > > Le 19/08/2014 20:12, Guido van Rossum a ?crit : > >> Hmm, I've been saying this already, but my intuition is that it's a > >> bad idea to conflate *type descriptions* (what this proposal is > >> about) and actual *runtime types* (what ABCs are). > >> > >> > >> But are they? I think the registration mechanism makes it clear that > >> they aren't (necessarily) runtime types -- by linking a concrete type > >> with an ABC through registration you are pretty clearly stating that the > >> ABC *describes* (an aspect of) the concrete class without automatically > >> adding any behavior from the ABC to it. > > > > Hmm... well, they are usable at runtime (if only for isinstance calls > :-)). I admit my wording was a bit vague here. But our type descriptions > should be able to express more information than ABCs currently do. For > example, I don't how you'd express the idea of a "mapping from str to int" > using the current Mapping ABC, while retaining the runtime properties of > the Mapping class. > > Someone (I think Guido, but I can't find it) gave a concrete proposal here > (which happens to match how MyPy already works): you make the collections > ABCs implement subscripting behavior exactly the way the typing classes > do--that is, they just return self. This means that at compile time > Mapping[str, int] can be used as a static type hint that a function returns > mappings from str to int, while a runtime isinstance check only verifies > that the thing in question is a Mapping. > > This avoids thorny questions like distinguishing covariant from > contravariant contexts at runtime (which I don't think is solvable without > a bunch of separate isinstance functions, and a more complicated > __subclasshook__ protocol, but I could be wrong). > > Also, there's no reason this couldn't be added in 3.5, and then an > enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once > someone works out those issues. (So isinstance still ignores the > parameters, but iscovariant checks them covariantly, etc.) That wouldn't > risk breaking 3.5 code in 3.6. > > Also, keep in mind that, just because you _can_ write isinstance(d, > Mapping[str, int]) and that might be confusing, that doesn't mean there's > any reason you _would_ write that. It will only come up in metaprogramming > contexts (e.g., where you're generating a function at runtime to match some > type argument). Maybe you could argue that this means the compile-time > machinery should be available at runtime, but I don't think you could argue > that this means the compile-time types shouldn't be available at runtime. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Thu Aug 21 19:24:15 2014 From: antoine at python.org (Antoine Pitrou) Date: Thu, 21 Aug 2014 13:24:15 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Le 21/08/2014 12:15, Nick Coghlan a ?crit : > > Given: > > Sequence[Number] > Sequence[Number|None] > > There's still at least three options for extending the syntax even further: > > Sequence[Number](EXPR) > Sequence[Number:EXPR] > Sequence[Number,EXPR] > > You'd probably want to use strings at that point, for the same reasons > PyContracts does. For example: > > Number,"N>0" # Positive number > Sequence[Number,"N>0"] # Sequence of positive numbers > Sequence[Number],"N>0" # Non-empty sequence All of those are horrible and un-Pythonic, though. Python has one of the most powerful and user-friendly function call syntaxes around, why reinvent something clearly inferior and alien? > I doubt we'll ever get to that point, though. Type hinting and runtime > assertions aren't the same thing. It's less about defining all those use cases up front and more about not painting ourselves in a corner, though. Regards Antoine. From abarnert at yahoo.com Thu Aug 21 20:36:44 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 21 Aug 2014 11:36:44 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com> Message-ID: On Aug 21, 2014, at 9:43, Gregory Salvan wrote: > > > I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some > > realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one). > > The use case I see is functions composition. > I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid. OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin, and how often do you miss it?) that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that takes Iterable[Number] and try to validate those compile-time type annotations at runtime. For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable? > 2014-08-21 18:16 GMT+02:00 Andrew Barnert : >> On Aug 20, 2014, at 14:57, Antoine Pitrou wrote: >> >> > Le 19/08/2014 20:12, Guido van Rossum a ?crit : >> >> Hmm, I've been saying this already, but my intuition is that it's a >> >> bad idea to conflate *type descriptions* (what this proposal is >> >> about) and actual *runtime types* (what ABCs are). >> >> >> >> >> >> But are they? I think the registration mechanism makes it clear that >> >> they aren't (necessarily) runtime types -- by linking a concrete type >> >> with an ABC through registration you are pretty clearly stating that the >> >> ABC *describes* (an aspect of) the concrete class without automatically >> >> adding any behavior from the ABC to it. >> > >> > Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class. >> >> Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping. >> >> This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong). >> >> Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6. >> >> Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From scottyblair at gmail.com Thu Aug 21 21:23:16 2014 From: scottyblair at gmail.com (Scott Blair) Date: Thu, 21 Aug 2014 14:23:16 -0500 Subject: [Python-ideas] Idea Message-ID: Add continue to a try, except statement to continue where the code threw an error. Or a function of exceptions to get the line number at which it was thrown along with a way to go back to that line. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Aug 21 21:38:07 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 21 Aug 2014 20:38:07 +0100 Subject: [Python-ideas] Idea In-Reply-To: References: Message-ID: On 21 August 2014 20:23, Scott Blair wrote: > Add continue to a try, except statement to continue where the code threw an > error. Or a function of exceptions to get the line number at which it was > thrown along with a way to go back to that line. I'm pretty sure this has been suggested before. The big problem is that an exception can exit many levels of function invocation before being caught. Restarting function call frames that have been exited is a non-trivial exercise in general, and for all practical purposes not possible in CPython. If you're interested in details, Stackless Python does something similar, and for a more theoretical discussion you can start at http://en.wikipedia.org/wiki/Continuation Paul From tjreedy at udel.edu Thu Aug 21 22:19:33 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 21 Aug 2014 16:19:33 -0400 Subject: [Python-ideas] Idea In-Reply-To: References: Message-ID: On 8/21/2014 3:23 PM, Scott Blair wrote: > Add continue to a try, except statement to continue where the code threw > an error. As a practical matter, the way to do this now is try: except SomeError: pass Anything more has technical difficulties discussed by Paul More. -- Terry Jan Reedy From apieum at gmail.com Thu Aug 21 22:58:50 2014 From: apieum at gmail.com (Gregory Salvan) Date: Thu, 21 Aug 2014 22:58:50 +0200 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com> Message-ID: > OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin,... True, explicit composition is actually uncommon practice in python, whereas there are lots of libs: toolz, funcy, fn.py, Pymonads.... Note that composition can be implicit: f(g(x)) Using mypy syntax was originally pushed with a presentation about what python can learn from FP. Personnally, I consider FP types have a different meaning than OO types, I feel them more as a way to describe programs flows (by func de/composition) and that's something I find really usefull in some cases (mainly for parsing and parallel computing). That's what I expect python would learn from FP and why I'm delighted about this proposition. (my opinion doesn't matter but +1000 for the proposition :) > and, how often do you miss it ? sufficiently to have written my own tool, which I started to extract from a project few days before Guido announces his mypy proposal. (in development but usable: https://github.com/apieum/lawvere ) > that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that > takes Iterable[Number] and try to validate those compile-time type annotations at runtime. I've not such cases in mind, I thought more of these ones: from functools import singledispatch @singledispatch def div1000(a: intexcept0) -> int: return 1000 / a @div1000.register def divby0(a: Just(0)): log('something')... In this case it's interesting to know if for ex. div1000 can be composed with sub500... when compiling and if it's 0 or not at runtime. (OK the example is simple and you can use a condition inside div1000 but you can have more complex cases of this type or want to use divby0 elsewhere) > For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable? I'm not sure to understand what you are asking. I see Iterable[Number] as a kind of type constructor (like Just(0)) even if it's cached. I would probably check it at runtime with isinstance. 2014-08-21 20:36 GMT+02:00 Andrew Barnert : > On Aug 21, 2014, at 9:43, Gregory Salvan wrote: > > > > I'll admit that I could be off on some of this. It's hard to discuss > anything beyond the basics without either getting formal (which I don't > think anyone wants) or providing some > > realistic examples of where you might want to runtime-check against an > annotation (which I think would be very useful, if anyone has one). > > The use case I see is functions composition. > I think, it's interesting to check at compile time if 2 functions can be > composed and at runtime if given arguments are valid. > > > OK, how about a _concrete_ use case? Function composition is rare enough > in Python (we don't even have a compose in functools, much less as an > operator or builtin, and how often do you miss it?) that I'm finding it > hard to imagine where you'd ever want to, e.g., accept a function that > returns Sequence[int] and rcompose it with a function that takes > Iterable[Number] and try to validate those compile-time type annotations at > runtime. > > For that matter, even if you wanted to, how _would_ you validate that > something matches Iterable[Number] at runtime, beyond validating that it's > iterable? > > 2014-08-21 18:16 GMT+02:00 Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid>: > >> On Aug 20, 2014, at 14:57, Antoine Pitrou wrote: >> >> > Le 19/08/2014 20:12, Guido van Rossum a ?crit : >> >> Hmm, I've been saying this already, but my intuition is that it's a >> >> bad idea to conflate *type descriptions* (what this proposal is >> >> about) and actual *runtime types* (what ABCs are). >> >> >> >> >> >> But are they? I think the registration mechanism makes it clear that >> >> they aren't (necessarily) runtime types -- by linking a concrete type >> >> with an ABC through registration you are pretty clearly stating that >> the >> >> ABC *describes* (an aspect of) the concrete class without automatically >> >> adding any behavior from the ABC to it. >> > >> > Hmm... well, they are usable at runtime (if only for isinstance calls >> :-)). I admit my wording was a bit vague here. But our type descriptions >> should be able to express more information than ABCs currently do. For >> example, I don't how you'd express the idea of a "mapping from str to int" >> using the current Mapping ABC, while retaining the runtime properties of >> the Mapping class. >> >> Someone (I think Guido, but I can't find it) gave a concrete proposal >> here (which happens to match how MyPy already works): you make the >> collections ABCs implement subscripting behavior exactly the way the typing >> classes do--that is, they just return self. This means that at compile time >> Mapping[str, int] can be used as a static type hint that a function returns >> mappings from str to int, while a runtime isinstance check only verifies >> that the thing in question is a Mapping. >> >> This avoids thorny questions like distinguishing covariant from >> contravariant contexts at runtime (which I don't think is solvable without >> a bunch of separate isinstance functions, and a more complicated >> __subclasshook__ protocol, but I could be wrong). >> >> Also, there's no reason this couldn't be added in 3.5, and then an >> enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once >> someone works out those issues. (So isinstance still ignores the >> parameters, but iscovariant checks them covariantly, etc.) That wouldn't >> risk breaking 3.5 code in 3.6. >> >> Also, keep in mind that, just because you _can_ write isinstance(d, >> Mapping[str, int]) and that might be confusing, that doesn't mean there's >> any reason you _would_ write that. It will only come up in metaprogramming >> contexts (e.g., where you're generating a function at runtime to match some >> type argument). Maybe you could argue that this means the compile-time >> machinery should be available at runtime, but I don't think you could argue >> that this means the compile-time types shouldn't be available at runtime. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Aug 22 02:37:31 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 22 Aug 2014 10:37:31 +1000 Subject: [Python-ideas] Idea In-Reply-To: References: Message-ID: On Fri, Aug 22, 2014 at 5:23 AM, Scott Blair wrote: > Add continue to a try, except statement to continue where the code threw an > error. Or a function of exceptions to get the line number at which it was > thrown along with a way to go back to that line. You can get the line number from the traceback. However, I don't think it's a good idea *in the general case* to jump back to the line that started a problem, because you have no way of undoing anything that got half done (Python is not transactional). But if you have a specific situation in which you want to do this, chances are you can wrap stuff up into a function. For instance, here's something that would reasonably want a "RESUME" statement (like in BASIC): try: install_file("base.py") install_file("subdir/foo.py") install_file("subdir/bar.py") install_file("otherdir/spam.py") install_file("otherdir/ham.py") except PathNotFoundError: create_directory(whichever_one_is_missing) resume But this is better served by a simple wrapper: def install_and_create_dir(dir,fn): while "infinite retry" try: install_file(dir+"/"+fn) return except PathNotFoundError: create_directory(dir) install_file("base.py") install_and_create_dir("subdir","foo.py") install_and_create_dir("subdir","bar.py") install_and_create_dir("otherdir","spam.py") install_and_create_dir("otherdir","ham.py") Do you have a concrete use-case that begs for a RESUME statement and can't easily be refactored into a (possibly nested) function? ChrisA From ncoghlan at gmail.com Fri Aug 22 02:40:37 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 22 Aug 2014 10:40:37 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 22 Aug 2014 03:27, "Antoine Pitrou" wrote: > > > Le 21/08/2014 12:15, Nick Coghlan a ?crit : > >> >> Given: >> >> Sequence[Number] >> Sequence[Number|None] >> >> There's still at least three options for extending the syntax even further: >> >> Sequence[Number](EXPR) >> Sequence[Number:EXPR] >> Sequence[Number,EXPR] >> >> You'd probably want to use strings at that point, for the same reasons >> PyContracts does. For example: >> >> Number,"N>0" # Positive number >> Sequence[Number,"N>0"] # Sequence of positive numbers >> Sequence[Number],"N>0" # Non-empty sequence > > > All of those are horrible and un-Pythonic, though. > Python has one of the most powerful and user-friendly function call syntaxes around, why reinvent something clearly inferior and alien? I don't think it's possible to have "Pythonic" design by contract (PyContracts makes a laudable effort, but I think its decorator based syntax is cleaner than its annotation based one) Decorators aren't going anywhere, so I'm comfortable with the idea of complexity constraints on the type hint notation - anything that doesn't fit into the type hint syntax can still go in a decorator, and get a handy decorator name for readers to look up in the docs or on Google. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Fri Aug 22 03:24:17 2014 From: antoine at python.org (Antoine Pitrou) Date: Thu, 21 Aug 2014 21:24:17 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53F69BC1.30205@python.org> Le 21/08/2014 20:40, Nick Coghlan a ?crit : > On 22 Aug 2014 03:27, "Antoine Pitrou" > > wrote: > > > > > > Le 21/08/2014 12:15, Nick Coghlan a ?crit : > > > >> > >> Given: > >> > >> Sequence[Number] > >> Sequence[Number|None] > >> > >> There's still at least three options for extending the syntax even > further: > >> > >> Sequence[Number](EXPR) > >> Sequence[Number:EXPR] > >> Sequence[Number,EXPR] > >> > >> You'd probably want to use strings at that point, for the same reasons > >> PyContracts does. For example: > >> > >> Number,"N>0" # Positive number > >> Sequence[Number,"N>0"] # Sequence of positive numbers > >> Sequence[Number],"N>0" # Non-empty sequence > > > > > > All of those are horrible and un-Pythonic, though. > > Python has one of the most powerful and user-friendly function call > syntaxes around, why reinvent something clearly inferior and alien? > > I don't think it's possible to have "Pythonic" design by contract We're not talking about design by contract, we're talking about type annotations. "A list of X integers" isn't exactly an extremely complicated notion. We shouldn't need weird convoluted syntaxes to express it. Regards Antoine. From ethan at stoneleaf.us Fri Aug 22 03:31:57 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 21 Aug 2014 18:31:57 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53F69BC1.30205@python.org> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F69BC1.30205@python.org> Message-ID: <53F69D8D.5030803@stoneleaf.us> On 08/21/2014 06:24 PM, Antoine Pitrou wrote: > Le 21/08/2014 20:40, Nick Coghlan a ?crit : >> On 22 Aug 2014 03:27, "Antoine Pitrou" >> > > wrote: >> > >> > >> > Le 21/08/2014 12:15, Nick Coghlan a ?crit : >> > >> >> >> >> Given: >> >> >> >> Sequence[Number] >> >> Sequence[Number|None] >> >> >> >> There's still at least three options for extending the syntax even >> further: >> >> >> >> Sequence[Number](EXPR) >> >> Sequence[Number:EXPR] >> >> Sequence[Number,EXPR] >> >> >> >> You'd probably want to use strings at that point, for the same reasons >> >> PyContracts does. For example: >> >> >> >> Number,"N>0" # Positive number >> >> Sequence[Number,"N>0"] # Sequence of positive numbers >> >> Sequence[Number],"N>0" # Non-empty sequence >> > >> > >> > All of those are horrible and un-Pythonic, though. >> > Python has one of the most powerful and user-friendly function call >> syntaxes around, why reinvent something clearly inferior and alien? >> >> I don't think it's possible to have "Pythonic" design by contract > > We're not talking about design by contract, we're talking about type annotations. "A list of X integers" isn't exactly > an extremely complicated notion. We shouldn't need weird convoluted syntaxes to express it. I don't know for whom I am arguing and against whom I am arguing, I just want to remind us that a type can be quite a bit more complicated than int or str: class Counting(int): def __new__(cls, value): if value < 1: raise ValueError('Counting can only be positive') ... -- ~Ethan~ From abarnert at yahoo.com Fri Aug 22 05:11:15 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 21 Aug 2014 20:11:15 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com> Message-ID: <1408677075.31030.YahooMailNeo@web181002.mail.ne1.yahoo.com> On Thursday, August 21, 2014 1:59 PM, Gregory Salvan wrote: >> OK, how about a _concrete_ use case?? > > >I've not such cases in mind, I thought more of these ones: > > >from functools import singledispatch >@singledispatch >def div1000(a: intexcept0) -> int: > >return 1000 / a > > >@div1000.register >def divby0(a: Just(0)): > >log('something')... >In this case it's interesting to know if for ex. div1000 can be composed with sub500... when compiling >and if it's 0 or not at runtime.? Antoine was against using the same types for runtime ABCs and compile-time type-checking, because?"I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class." You want something that's more powerful than existing ABCs at runtime, and more powerful than existing typing.py classes at compile-time, but I don't see any reason why you can't write them as the _same_ something. In fact, your example seems to be doing exactly that. So, it seems like you're arguing for the exact same position as the one you disagreed with? What am I missing? >> For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable? >I'm not sure to understand what you are asking. >I see Iterable[Number] as a kind of type constructor (like Just(0)) even if it's cached. I would probably check it at runtime with isinstance. But how would a runtime isinstance check on Iterable[Number] work??Either it has to ignore the generic part and just test Iterable (which is exactly what Guido's suggestion entails, which Antoine, and I thought you, didn't like), or it has to iterate all of the values and check them all, leaving you an empty iterable after checking (which may take infinite time, to boot). >2014-08-21 20:36 GMT+02:00 Andrew Barnert : > >On Aug 21, 2014, at 9:43, Gregory Salvan wrote: >> >> >> >>>> I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some >>>> realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one). >>> >>> >>>The use case I see is functions composition. >>>I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid. >>> >> >>OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin, and how often do you miss it?)?that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that takes Iterable[Number] and try to validate those compile-time type annotations at runtime. >> >> >>For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable? >> >> >>2014-08-21 18:16 GMT+02:00 Andrew Barnert : >>> >>>On Aug 20, 2014, at 14:57, Antoine Pitrou wrote: >>>> >>>>> Le 19/08/2014 20:12, Guido van Rossum a ?crit : >>>>>>? ? Hmm, I've been saying this already, but my intuition is that it's a >>>>>>? ? bad idea to conflate *type descriptions* (what this proposal is >>>>>>? ? about) and actual *runtime types* (what ABCs are). >>>>>> >>>>>> >>>>>> But are they? I think the registration mechanism makes it clear that >>>>>> they aren't (necessarily) runtime types -- by linking a concrete type >>>>>> with an ABC through registration you are pretty clearly stating that the >>>>>> ABC *describes* (an aspect of) the concrete class without automatically >>>>>> adding any behavior from the ABC to it. >>>>> >>>>> Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class. >>>> >>>>Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping. >>>> >>>>This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong). >>>> >>>>Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6. >>>> >>>>Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime. >>>> >>>> >>>>_______________________________________________ >>>>Python-ideas mailing list >>>>Python-ideas at python.org >>>>https://mail.python.org/mailman/listinfo/python-ideas >>>>Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>_______________________________________________ >>>Python-ideas mailing list >>>Python-ideas at python.org >>>https://mail.python.org/mailman/listinfo/python-ideas >>>Code of Conduct: http://python.org/psf/codeofconduct/ > > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ > > From greg at krypto.org Fri Aug 22 08:42:59 2014 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 21 Aug 2014 23:42:59 -0700 Subject: [Python-ideas] Decorators that execute once rather than every import (was: codec and decorator hacks) Message-ID: On Wed, Aug 20, 2014 at 7:25 AM, Donald Stufft wrote: > > mypy does have a codec that will ignore annotations on 2.x. > Note that all codec hacks and decorator hacks have a down side other than being hacks: They significantly slow down program import and start time. At least the results of the codec hack are cached when a .pyc is generated. Decorator executions are not. hmm.. could a flavor of decorator that is cacheable in .pyc's be created so the cost is only paid once rather than for each time the module is imported? -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Aug 22 09:41:04 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 22 Aug 2014 17:41:04 +1000 Subject: [Python-ideas] Decorators that execute once rather than every import (was: codec and decorator hacks) In-Reply-To: References: Message-ID: <20140822074103.GR25957@ando> On Thu, Aug 21, 2014 at 11:42:59PM -0700, Gregory P. Smith wrote: > On Wed, Aug 20, 2014 at 7:25 AM, Donald Stufft wrote: > > > > > mypy does have a codec that will ignore annotations on 2.x. > > > > Note that all codec hacks and decorator hacks have a down side other than > being hacks: They significantly slow down program import and start time. At > least the results of the codec hack are cached when a .pyc is generated. > Decorator executions are not. Decorator executions are no different from any other function call. They occur when you call them. If you only call the decorator once, at the top level of the module, they don't get called again when you import the module again. That's no different from any other code at the top-level: [steve at ando ~]$ cat test_deco.py def decorate(func): print("calling an expensive decorator...") func.__name__ = "decorated_%s" % func.__name__ return func @decorate def spam(n): return "spam"*n print("%s: %s" % (spam.__name__, spam(3))) [steve at ando ~]$ python3.3 Python 3.3.0rc3 (default, Sep 27 2012, 18:44:58) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux Type "help", "copyright", "credits" or "license" for more information. py> import test_deco as d calling an expensive decorator... decorated_spam: spamspamspam py> import test_deco # importing again doesn't re-call the decorator py> test_deco.spam.__name__ 'decorated_spam' > hmm.. could a flavor of decorator that is cacheable in .pyc's be created so > the cost is only paid once rather than for each time the module is imported? I'm not sure that you're trying to solve a real problem. Can you give an example? -- Steven From miguelglafuente at gmail.com Fri Aug 22 09:47:34 2014 From: miguelglafuente at gmail.com (Rock Neurotiko) Date: Fri, 22 Aug 2014 09:47:34 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53F69D8D.5030803@stoneleaf.us> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F69BC1.30205@python.org> <53F69D8D.5030803@stoneleaf.us> Message-ID: First, sorry for my (probably) bad english :) I think that everyone who wants to learn about types, type system, what is static type, what is not, why function annotations (or some other, but I prefer it) refering to types is being proposed, should learn a little bit (you won't regret!) 1) This article (What to know before debating type systems) has been sent before in this thread (I think) but is a must read: http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ 2) Bob's talk in Europython last month "What can python learn from haskell" at least until minute 20, that talks about this things. http://pyvideo.org/video/3046/what-can-python-learn-from-haskell 3) If you love maths, some book of Benjamin C. Pierce (about type systems, of course!), this three are so good (but hard). Ordered from low to high level. - Type Systems for programming languages. - Types for programming languages. - Advanced topics in Types and programming languages. 4) If want to learn more, search in google, there are so much books/articles about types, and if you can learn about haskell types (or Idris if you are crazy), better ;) As last word, I want to give my opinion about the proposal. I like the proposal, and I would love to see it in Python 3.x, I don't think that function notations can have a better purpose. Of course, what is being proposed won't be needed to run a python program, and you will still can write your function notations whatever you want, but have a standard proposal will help a lot to build tools that read the function notations (like IDEs, linters, static type checkers, ...) About the syntax, I love the mypy's syntax, and remember me a lot to other languages like scala. def primes_to_n(n: Int) => List[Int] def primes_to_n(n: int) -> List[int] This two function definitions are a function that get an int and return a list of integers. First definition is Scala, second one is Python with mypy's syntax. I like much more the Haskell syntax, but is another world and I think that python shouldn't have a syntax like that: primes_to_n :: Int -> [Int] About other syntax that I had seen in this thread: - decorators: I really don't like that one, is compatible with python 2, ok, but looks so bad (to me) - docs: That's not bad at all, and can still exist with the typing syntax to declare the types, but we have function annotations where the types are much close to the definition, and that's definitively good. - PyContracts: That's not bad at all, but just for check types in runtime, much things (the main diferences with mypy) can't be done in static, for example, how you know if this will work *before* execute it: @contract def f(x: 'list[3>](str)') -> 'list(str)' f(input().split(" ")) That's a function that take a list of three or more strings, and is called with the user input. You can't know if that list will be over three elements if you don't run it and get the input, and even those, you can't say that will be true all the times you execute it. That's just my opinion :) Cheers! -- Miguel Garc?a Lafuente - Rock Neurotiko Do it, the devil is in the details. The quieter you are, the more you are able to hear. Happy Coding. Code with Passion, Decode with Patience. If we make consistent effort, based on proper education, we can change the world. El contenido de este e-mail es privado, no se permite la revelacion del contenido de este e-mail a gente ajena a ?l. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Aug 22 10:10:22 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 22 Aug 2014 10:10:22 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53F69BC1.30205@python.org> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F69BC1.30205@python.org> Message-ID: <53F6FAEE.9010705@egenix.com> On 22.08.2014 03:24, Antoine Pitrou wrote: > Le 21/08/2014 20:40, Nick Coghlan a ?crit : >> On 22 Aug 2014 03:27, "Antoine Pitrou" >> > > wrote: >> > >> > >> > Le 21/08/2014 12:15, Nick Coghlan a ?crit : >> > >> >> >> >> Given: >> >> >> >> Sequence[Number] >> >> Sequence[Number|None] >> >> >> >> There's still at least three options for extending the syntax even >> further: >> >> >> >> Sequence[Number](EXPR) >> >> Sequence[Number:EXPR] >> >> Sequence[Number,EXPR] >> >> >> >> You'd probably want to use strings at that point, for the same reasons >> >> PyContracts does. For example: >> >> >> >> Number,"N>0" # Positive number >> >> Sequence[Number,"N>0"] # Sequence of positive numbers >> >> Sequence[Number],"N>0" # Non-empty sequence >> > >> > >> > All of those are horrible and un-Pythonic, though. >> > Python has one of the most powerful and user-friendly function call >> syntaxes around, why reinvent something clearly inferior and alien? >> >> I don't think it's possible to have "Pythonic" design by contract > > We're not talking about design by contract, we're talking about type annotations. "A list of X > integers" isn't exactly an extremely complicated notion. We shouldn't need weird convoluted syntaxes > to express it. I agree with Antoine. Those EXPRs should not be needed for type annotations. The above is mixing type information with runtime information. All a linter needs to know is that the result of operation1(type1) is of type2, i.e. the information about certain features of those types is implicit in the type name and the information about whether types can be adapted to each other. If the linter then finds that the result of operation1() is being used as input for operation2(), which only supports type1 arguments, and it doesn't have information about how to adapt type2 to type1, it comes back with an error message. It's all abstract. Trying to conflate arbitrary type properties into the syntax is not needed - after all, type annotations are not meant for runtime checks. Regarding the whole idea of inline type annotations, I'm at most +0 on those. They add more complexity to the language and make it less readable for little added value. Making the type annotations external is much better in this respect and I'm sure IDEs could even help with this by providing nice interfaces for adding the type information. They could semi- automate the process and make suggestions based on the functions used inside the functions/method body. Now, since these tools would have to read and write the annotations programmatically, the syntax does not have to be nicely readable or Pythonic. It could just as well be a YAML/JSON/XML file. But hey, I don't want to spoil all the fun in defining a little language for type annotations, so please do continue to have fun defining Python .pyh header files ;-) Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 22 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-09-19: PyCon UK 2014, Coventry, UK ... 28 days to go 2014-09-30: Python Meeting Duesseldorf ... 39 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ram.rachum at gmail.com Fri Aug 22 13:40:49 2014 From: ram.rachum at gmail.com (Ram Rachum) Date: Fri, 22 Aug 2014 04:40:49 -0700 (PDT) Subject: [Python-ideas] Allow getting all the items out of a `threading.local` Message-ID: <72de8dd9-4ce3-48a9-a22c-7c6dcf4b6d2a@googlegroups.com> I propose adding a method to `threading.local` that allows to check the values stored for any and all threads. I asked on Stack Overflow how to do that: http://stackoverflow.com/questions/25435908/python-getting-all-the-items-out-of-a-threading-local But I got the answer that it's only possible on the pure Python implementation, not the C one. And it's quite ugly too. I propose a method be added to both the Python implementation and the C implementation of `threading.local` that allows checking the values stored for all threads. Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Fri Aug 22 13:50:52 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 22 Aug 2014 20:50:52 +0900 Subject: [Python-ideas] Decorators that execute once rather than every import (was: codec and decorator hacks) In-Reply-To: <20140822074103.GR25957@ando> References: <20140822074103.GR25957@ando> Message-ID: Longer import time makes CLI applications slower. It's a real problem for people writing command line application. For example, both of mercurial and Bazaar has lazy import system to minimize import. On Fri, Aug 22, 2014 at 4:41 PM, Steven D'Aprano wrote: > On Thu, Aug 21, 2014 at 11:42:59PM -0700, Gregory P. Smith wrote: >> On Wed, Aug 20, 2014 at 7:25 AM, Donald Stufft wrote: >> >> > >> > mypy does have a codec that will ignore annotations on 2.x. >> > >> >> Note that all codec hacks and decorator hacks have a down side other than >> being hacks: They significantly slow down program import and start time. At >> least the results of the codec hack are cached when a .pyc is generated. >> Decorator executions are not. > > Decorator executions are no different from any other function call. They > occur when you call them. If you only call the decorator once, at the > top level of the module, they don't get called again when you import the > module again. That's no different from any other code at the top-level: > > > [steve at ando ~]$ cat test_deco.py > def decorate(func): > print("calling an expensive decorator...") > func.__name__ = "decorated_%s" % func.__name__ > return func > > @decorate > def spam(n): > return "spam"*n > > print("%s: %s" % (spam.__name__, spam(3))) > > [steve at ando ~]$ python3.3 > Python 3.3.0rc3 (default, Sep 27 2012, 18:44:58) > [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux > Type "help", "copyright", "credits" or "license" for more information. > py> import test_deco as d > calling an expensive decorator... > decorated_spam: spamspamspam > py> import test_deco # importing again doesn't re-call the decorator > py> test_deco.spam.__name__ > 'decorated_spam' > > >> hmm.. could a flavor of decorator that is cacheable in .pyc's be created so >> the cost is only paid once rather than for each time the module is imported? > > I'm not sure that you're trying to solve a real problem. Can you give > an example? > > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- INADA Naoki From brett at python.org Fri Aug 22 14:53:25 2014 From: brett at python.org (Brett Cannon) Date: Fri, 22 Aug 2014 12:53:25 +0000 Subject: [Python-ideas] Optional Static Typing -- the Python Way References: <53F25EE8.8000900@stoneleaf.us> Message-ID: On Thu Aug 21 2014 at 10:55:47 AM Antoine Pitrou wrote: > > Le 21/08/2014 10:30, Brett Cannon a ?crit : > > > > Isn't that the magic of __instancecheck__/__subclasscheck__? If > > typing.py does the right thing here then it shouldn't matter. > > What is "the right thing"? > I don't feel like scrolling all the way back through this thread to find your original email, but "the right thing" is "whatever it takes". =) My point is that I don't think there is any technical reason why we can't make ABCs work. > > > You could > > use __init__ or __getitem__ to construct an object who upon > > introspection provides the details you want and do what you expect, and > > then override the appropriate methods so that isinstance() checks check > > against the ABC instead of the type class itself (heck we could make the > > base type class require that you inherit from an ABC to promote working > > with ABCs instead of concrete classes). > > So we agree that type descriptions should be separate things from ABCs, > or not? > We don't agree. I'm fine with reusing ABCs. My only point is that if you really don't want to tweak ABCs to work directly, typing.py can still use ABCs as a subclass and then add whatever is necessary on top of them. > > > I have to also say that I like using ABCs because `from collections > > import abc as a` leads to reading parameters like "x is a.Mapping" which > > is linguistically quaint. =) > > Except that "x is a.Mapping" is False with a normal Python. > You're reading that too literally as code and not as a descriptive sentence. What I meant to convey is that code `def foo(x: a.Mapping)` can be read in your inner voice as "function foo has a parameter x which is a.Mapping", where "a.Mapping" somewhat reads like "a Mapping". -Brett > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Fri Aug 22 15:42:00 2014 From: skip at pobox.com (Skip Montanaro) Date: Fri, 22 Aug 2014 08:42:00 -0500 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: There's been a lot to read in this and related threads over the past nine days. (According to Gmail. It qualitatively seems much longer to me.) I think I've followed along reasonably well. Forgive me if I missed the answers to these questions. The proposal on the table is to adopt MyPy's type annotations for Python 3.mumble. I presume MyPy already supports them. 1. Can MyPy be used today as a standalone static type checker for Python 3.x code without actually compiling anything? That is, can I just sprinkle type annotations into my code and run a front-end pass of MyPy much the same way I'd run pylint, vulture, flake8, or other lint-ish program? 2. Assuming the answer to #1 is "yes," if you start sprinkling type annotations into your code and running "mypy *.py", will it tell you when it needs a missing type annotation to more fully check things, or will it silently process code without annotations and not let you know that it's not really checking much? 3. Will we get into a phase like the early days of "const" in ANSI C where addition of "const" in one location in your existing code base forced you into a never-ending iterations of adding const all over the place? I forget what that was called ("const propagation"?), but I recall that it generally wasn't a fun activity. 4. If the answer to #1 is "no," are there plans to produce a front-end-only MyPy which can be used as a linter for Python code? Thx, Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Fri Aug 22 16:36:26 2014 From: antoine at python.org (Antoine Pitrou) Date: Fri, 22 Aug 2014 10:36:26 -0400 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: <53F7556A.205@python.org> Le 22/08/2014 08:53, Brett Cannon a ?crit : > > I have to also say that I like using ABCs because `from collections > > import abc as a` leads to reading parameters like "x is > a.Mapping" which > > is linguistically quaint. =) > > Except that "x is a.Mapping" is False with a normal Python. > > > You're reading that too literally as code and not as a descriptive > sentence. What I meant to convey is that code `def foo(x: a.Mapping)` > can be read in your inner voice as "function foo has a parameter x which > is a.Mapping", where "a.Mapping" somewhat reads like "a Mapping". So? That would be true with anything else that isn't an ABC, as long as you import the relevant module with the name "a" (which is silly, but hey :-)). def foo(x: a.List(a.Number)) -> a.Number Regards Antoine. From antoine at python.org Fri Aug 22 16:38:58 2014 From: antoine at python.org (Antoine Pitrou) Date: Fri, 22 Aug 2014 10:38:58 -0400 Subject: [Python-ideas] Allow getting all the items out of a `threading.local` In-Reply-To: <72de8dd9-4ce3-48a9-a22c-7c6dcf4b6d2a@googlegroups.com> References: <72de8dd9-4ce3-48a9-a22c-7c6dcf4b6d2a@googlegroups.com> Message-ID: Le 22/08/2014 07:40, Ram Rachum a ?crit : > I propose adding a method to `threading.local` that allows to check the > values stored for any and all threads. > > I asked on Stack Overflow how to do > that: http://stackoverflow.com/questions/25435908/python-getting-all-the-items-out-of-a-threading-local > > But I got the answer that it's only possible on the pure Python > implementation, not the C one. And it's quite ugly too. > > I propose a method be added to both the Python implementation and the C > implementation of `threading.local` that allows checking the values > stored for all threads. Do other languages have such a functionality? As for feasibility, the best way to check is to try to write a patch :-) Regards Antoine. From bob at redivi.com Fri Aug 22 16:30:36 2014 From: bob at redivi.com (Bob Ippolito) Date: Fri, 22 Aug 2014 07:30:36 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Friday, August 22, 2014, Skip Montanaro wrote: > There's been a lot to read in this and related threads over the past nine > days. (According to Gmail. It qualitatively seems much longer to me.) I > think I've followed along reasonably well. Forgive me if I missed the > answers to these questions. The proposal on the table is to adopt MyPy's > type annotations for Python 3.mumble. I presume MyPy already supports them. > > 1. Can MyPy be used today as a standalone static type checker for Python > 3.x code without actually compiling anything? That is, can I just sprinkle > type annotations into my code and run a front-end pass of MyPy much the > same way I'd run pylint, vulture, flake8, or other lint-ish program? > Yes. mypy -S performs checking without execution. 2. Assuming the answer to #1 is "yes," if you start sprinkling type > annotations into your code and running "mypy *.py", will it tell you when > it needs a missing type annotation to more fully check things, or will it > silently process code without annotations and not let you know that it's > not really checking much? > No. It happily and silently accepts dynamic code with no annotations. It can't do much for you with that code, but it doesn't complain. 3. Will we get into a phase like the early days of "const" in ANSI C where > addition of "const" in one location in your existing code base forced you > into a never-ending iterations of adding const all over the place? I forget > what that was called ("const propagation"?), but I recall that it generally > wasn't a fun activity. > Nope. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Aug 22 18:15:01 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 23 Aug 2014 02:15:01 +1000 Subject: [Python-ideas] Optional static typing -- late to the party In-Reply-To: <53F53BD5.5090902@stoneleaf.us> References: <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au> <85lhqi29cz.fsf@benfinney.id.au> <20140821000954.GO25957@ando> <53F53BD5.5090902@stoneleaf.us> Message-ID: <20140822161501.GU25957@ando> On Wed, Aug 20, 2014 at 05:22:45PM -0700, Ethan Furman wrote: > On 08/20/2014 05:09 PM, Steven D'Aprano wrote: > >On Thu, Aug 21, 2014 at 06:29:16AM +1000, Ben Finney wrote: > >> > >>Remember that we're talking about type annotations that are for > >>*static code checkers* to inspect. Docstrings will certainly be > >>available there. > > > >True, but runtime checks may (or may not) be a part of the code checker. > >At least, we should not rule that out. > > This proposal is not about "a code checker" but about *static* typing. > > Seems to be a lot of people forgetting that *static* (at least in this > case), means not actually running the program -- hence, no loss of > docstrings. I'm aware of that, but I'm just saying that we shouldn't rule out the possibility of tools which operate at runtime using those same annotations. If the annotations are in the docstring, then they cannot be used at runtime with -OO. Function annotations can be used both statically and dynamically, while docstring annotations cannot be relied on to be present at runtime. -- Steven From tjreedy at udel.edu Sat Aug 23 01:36:53 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 22 Aug 2014 19:36:53 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F69BC1.30205@python.org> <53F69D8D.5030803@stoneleaf.us> Message-ID: On 8/22/2014 3:47 AM, Rock Neurotiko wrote: > About the syntax, I love the mypy's syntax, and remember me a lot to > other languages like scala. > > def primes_to_n(n: Int) => List[Int] > > def primes_to_n(n: int) -> List[int] The Python version need ':' at the end, after the return type. I don't think you are the first to omit it. -- Terry Jan Reedy From miguelglafuente at gmail.com Sat Aug 23 01:50:49 2014 From: miguelglafuente at gmail.com (Rock Neurotiko) Date: Sat, 23 Aug 2014 01:50:49 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F69BC1.30205@python.org> <53F69D8D.5030803@stoneleaf.us> Message-ID: Yes, and the Scala version need a " = {}", I was talking just about how the syntax of the types and annotations are, I omitted the ':' because it was irrelevant. 2014-08-23 1:36 GMT+02:00 Terry Reedy : > On 8/22/2014 3:47 AM, Rock Neurotiko wrote: > > About the syntax, I love the mypy's syntax, and remember me a lot to >> other languages like scala. >> >> def primes_to_n(n: Int) => List[Int] >> >> def primes_to_n(n: int) -> List[int] >> > > The Python version need ':' at the end, after the return type. > I don't think you are the first to omit it. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Miguel Garc?a Lafuente - Rock Neurotiko Do it, the devil is in the details. The quieter you are, the more you are able to hear. Happy Coding. Code with Passion, Decode with Patience. If we make consistent effort, based on proper education, we can change the world. El contenido de este e-mail es privado, no se permite la revelacion del contenido de este e-mail a gente ajena a ?l. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Aug 23 03:36:47 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 23 Aug 2014 11:36:47 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140823013646.GV25957@ando> On Thu, Aug 21, 2014 at 09:23:39AM -0400, Antoine Pitrou wrote: > I'm -1 on using __getitem__ for parametering types. It is indeed a hack > and its limitations will bite us harshly when wanting to refine the type > descriptions (for example to describe a constraint on a collection's size). > >One last thing: let?s not add |List|, |TList|, etc. Just use |list| and > >other existing types to generate new types. > > I'm -1 on that, for exactly the same reason as above. It's not 100% clear to me whether you're -1 on re-using |list| or -1 or adding |List|. I'm going to assume that you mean -1 on re-using list (see below). In a later message, Antoine wrote: > Python has one of the most powerful and user-friendly function call > syntaxes around, why reinvent something clearly inferior and alien? I wouldn't say it is alien. abc[xyz] is clearly Python syntax. Nor would I necessarily say it is inferior, although it is more limited. Deliberately so, I think. How complex will the static type declarations need to be for us to want to accept arbitrary keyword arguments, for example? I think limiting the syntax possible is a good thing, it will discourage people from trying to declare arbitrarily complex information in the type notations. But another reason for disliking call syntax for this is that it gets confusing: def function(param:Sequence(Spam, Eggs))->Mapping(Cheese, Toast): suffers compared to this: def function(param:Sequence[Spam, Eggs])->Mapping[Cheese, Toast]: In the second case, the type annotations stand out more easily because they use square brackets instead of round. As far as the question of list versus List, I don't believe we can use built-ins if we follow Antoine's suggestion of call syntax. def function(param:list(int))->list(str): for example, intends to declare param is a list of ints. But list(x) already has a meaning, and if x is an int, it raises TypeError. param:List(int) at least can work, but now consider int. Perhaps we might like to specify a range of ints, from 0 to 9 (say). The obvious syntax would be: param:List(int(0, 10)) # inclusive lower bound, exclusive upper bound but again that already has meaning. If we want the flexibility of providing arguments to scalar types like int, I believe we need to use subscripting syntax: param:List[int[0, 10]] -1 on re-using call syntax: it is more confusing to read inside a parameter list, too flexible, and clashes with existing use calling types. +1 on subscripting syntax: it is more easily distinguishable from calling, just flexible enough, and doesn't clash with existing functionality. -- Steven From antoine at python.org Sat Aug 23 04:31:18 2014 From: antoine at python.org (Antoine Pitrou) Date: Fri, 22 Aug 2014 22:31:18 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140823013646.GV25957@ando> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> Message-ID: Le 22/08/2014 21:36, Steven D'Aprano a ?crit : > > It's not 100% clear to me whether you're -1 on re-using |list| or -1 or > adding |List|. I'm going to assume that you mean -1 on re-using list > (see below). Indeed. > In a later message, Antoine wrote: > >> Python has one of the most powerful and user-friendly function call >> syntaxes around, why reinvent something clearly inferior and alien? > > I wouldn't say it is alien. abc[xyz] is clearly Python syntax. But it is completely uncommon for anything else than subscripting and indexing containers. Python isn't known for reusing operators for completely unrelated things, e.g. it doesn't overload ">>" for I/O (thank goodness). > How complex will the static type declarations > need to be for us to want to accept arbitrary keyword arguments, for > example? I think limiting the syntax possible is a good thing, it will > discourage people from trying to declare arbitrarily complex information > in the type notations. I think this is an incredibly weak argument, and the proof is that it hasn't been brought for anything else in the stdlib. Why the typing module should be any different and use a deliberately repressive parameter-passing syntax is beyond me - even though it's an advanced, optional, feature and will be put in the hands of consenting adults. > But another reason for disliking call syntax for this is that it gets > confusing: > > def function(param:Sequence(Spam, Eggs))->Mapping(Cheese, Toast): > > suffers compared to this: > > def function(param:Sequence[Spam, Eggs])->Mapping[Cheese, Toast]: If you find it confusing (which may simply be a matter of habit, I don't know), you can simply alias the types: spam_eggs_sequence = Sequence(Spam, Eggs) cheese_toast_mapping = Mapping(Cheese, Toast) def function(param: spam_eggs_sequence) -> cheese_toast_mapping: ... Regardless of readability, I would expect people to create such aliases anyway, the same way people tend to create "typedefs" in other languages, because it helps condense and clarify which actual types are in use in a module. > As far as the question of list versus List, I don't believe we can use > built-ins if we follow Antoine's suggestion of call syntax. > > def function(param:list(int))->list(str): > > for example, intends to declare param is a list of ints. But list(x) > already has a meaning, and if x is an int, it raises TypeError. And indeed this is exactly why I said I was against using built-ins for this (or the existing ABCs) :-)) Really, for it to be powerful enough, the type description system has to use its own set of classes. It can't try to hack away existing builtins and ABCs in the hope of expressing different things with them, or it *will* hit a wall some day (and, IMO, sooner rather than later). Regards Antoine. From steve at pearwood.info Sat Aug 23 07:13:57 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 23 Aug 2014 15:13:57 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> Message-ID: <20140823051357.GX25957@ando> On Fri, Aug 22, 2014 at 10:31:18PM -0400, Antoine Pitrou wrote: > >>Python has one of the most powerful and user-friendly function call > >>syntaxes around, why reinvent something clearly inferior and alien? > > > >I wouldn't say it is alien. abc[xyz] is clearly Python syntax. > > But it is completely uncommon for anything else than subscripting and > indexing containers. And function call syntax is completely uncommon for anything else than calling functions (including types and methods). One way or the other, we're adding a new use, type declarations. > Python isn't known for reusing operators for > completely unrelated things, e.g. it doesn't overload ">>" for I/O > (thank goodness). I see you've forgotten Python 2's print >>sys.stderr, x, y, z :-) > >How complex will the static type declarations > >need to be for us to want to accept arbitrary keyword arguments, for > >example? I think limiting the syntax possible is a good thing, it will > >discourage people from trying to declare arbitrarily complex information > >in the type notations. > > I think this is an incredibly weak argument, and the proof is that it > hasn't been brought for anything else in the stdlib. We frequently reject syntax because it will be (allegedly) confusing or hard to read. E.g. try...except expressions, anonymous code blocks, multiline lambda. Especially multiline lambda. Even something which I think is a no-brainer, allowing the with statement to use parentheses around the context managers: with (this_manager() as m1, that_manager() as m2): ... was criticised by Nick on the basis that if you can't fit the context managers all on one line, you ought to refactor your code. I'm sure if you check the rejected PEPs and Python-Ideas archives, you'll find many examples of my argument applied to the standard library. I don't think it's unPythonic to argue that type hinting syntax should by default use an annotation style which is not arbitrarily complex. The annotations themselves will remain arbitrary Python expressions, so if you really need a complex type declaration, you can create a helper and call it with whatever signature you like.[1] > Why the typing > module should be any different and use a deliberately repressive > parameter-passing syntax is beyond me - even though it's an advanced, > optional, feature and will be put in the hands of consenting adults. I can't speak for the author of mypy, Jukka Lehtosalo, but for me, I think the answer is that type declarations make a *mini-language*, not full-blown Python. It is unclear to me just how powerful the type language will be, but surely we don't expect it to evaluate arbitrarily complex Python expressions at compile time? Do we? I find it unlikely that we expect a static linter to make sense of this: def bizarre(param:int if random.random() > 0.5 else str)->float: except to say it returns a float and takes who-knows-what as argument. An extreme case, I grant, but I expect that there are going to be limits to what can be checked at compile time based on static analysis. I admit that I don't fully understand all the implications of static versus runtime type checking, but it doesn't seem unreasonable to expect static declarations to be considerably simpler than what can be expressed at runtime. Hence, a simpler mini-language is appropriate. > Really, for it to be powerful enough, the type description system has to > use its own set of classes. It can't try to hack away existing builtins > and ABCs in the hope of expressing different things with them, or it > *will* hit a wall some day (and, IMO, sooner rather than later). How powerful is "powerful enough"? Powerful enough for what? You've probably heard of the famous "the compiler found my infinite loop", where the ML type checker was able to detect that code would never terminate. I find that remarkable, and even more astonishing that, according to some, solving the halting problem is "nothing special" for type systems.[2] But many languages make do with less powerful type systems, and tools such as IDEs and editors surely don't need something that powerful. So: - how powerful do we expect the type system to be? - and how much of that power needs to be expressed using function annotations? The second question is critical, because there are alternatives to function annotations: decorators, docstring annotations, and external stub files. [1] Although if you do so, it is not clear to me how much the static checker will be able to use it. [2] http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ -- Steven From jeanpierreda at gmail.com Sat Aug 23 10:09:56 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 23 Aug 2014 01:09:56 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140823051357.GX25957@ando> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> Message-ID: On Fri, Aug 22, 2014 at 10:13 PM, Steven D'Aprano wrote: > You've probably heard of the famous "the compiler found my infinite > loop", where the ML type checker was able to detect that code would > never terminate. I find that remarkable, and even more astonishing that, > according to some, solving the halting problem is "nothing special" for > type systems.[2] But many languages make do with less powerful type systems, > and tools such as IDEs and editors surely don't need something that > powerful. --snip-- > [2] http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ You seem to be mischaracterizing ML's type system as something magically powerful. The ML type checker can be implemented in a few dozen lines of Python code, and resolved efficiently. ML does type inference by looking at the constraints on arguments inside the body of a function. Infinite recursion often accidentally results in you omitting constraints that you didn't meant to. For example, "def foo(): return foo()" will be straight up marked as not returning at all, because no additional constraints are placed on the return value, which can only mean that it recurses infinitely. Many infinite loops will not be marked, though. It's a type system, not magic. (The ML type system is actually substantially simpler than anything one is likely to come up with to describe Python, because ML didn't have an object system or structural subtyping.) As for the comment that this is "nothing special", taken literally, they said something wrong -- solving the halting problem for Turing complete languages is impossible, static types or no. The only way static types can help is by allowing you to define a (sub-)language that is not Turing-complete. I think they were alluding to the fact that, counter to many peoples' intuition, creating a type system that allows you to detect some of the code that halts isn't a matter of creating a powerful type system. If you add even the simplest type system to the lambda calculus, poof, now every well-typed expression can only be "executed" finitely many steps! As for whether IDEs need to make something complicated enough that it can positively identify code that does or doesn't halt, I'd leave that to the IDE folks. C and Java tools do this sort of thing all the time, for things like detecting dead code. Why shouldn't Python tools? (This isn't a feature of the type system so much as control flow analysis under a given type system, so you're right that it doesn't belong in the types themselves.) -- Devin From g.brandl at gmx.net Sat Aug 23 11:21:58 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 23 Aug 2014 11:21:58 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 08/21/2014 03:23 PM, Antoine Pitrou wrote: >> One last thing: let?s not add |List|, |TList|, etc. Just use |list| and >> other existing types to generate new types. > > I'm -1 on that, for exactly the same reason as above. Another reason is that it makes it even easier for people to use the usually too narrow and therefore annoying list[int] instead of the correct Iterable[int] or Sequence[int] Another thought is whether, assuming class MyList(list): ... the type "MyList[int]" would do the expected thing. (And if not, how you would write the correct type generation code.) Georg From antoine at python.org Sat Aug 23 15:44:02 2014 From: antoine at python.org (Antoine Pitrou) Date: Sat, 23 Aug 2014 09:44:02 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140823051357.GX25957@ando> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> Message-ID: Le 23/08/2014 01:13, Steven D'Aprano a ?crit : > On Fri, Aug 22, 2014 at 10:31:18PM -0400, Antoine Pitrou wrote: > >>>> Python has one of the most powerful and user-friendly function call >>>> syntaxes around, why reinvent something clearly inferior and alien? >>> >>> I wouldn't say it is alien. abc[xyz] is clearly Python syntax. >> >> But it is completely uncommon for anything else than subscripting and >> indexing containers. > > And function call syntax is completely uncommon for anything else than > calling functions (including types and methods). One way or the other, > we're adding a new use, type declarations. It's not a new use. A type class is a class, and calling it is just instantiating that class. There's nothing new here. If you think that's a bit "meta", it's no different than e.g. higher-order functions. >> Python isn't known for reusing operators for >> completely unrelated things, e.g. it doesn't overload ">>" for I/O >> (thank goodness). > > I see you've forgotten Python 2's > > print >>sys.stderr, x, y, z > > :-) I always forget about it, but the fact that we finally made print() a function in Python 3 actually supports my point: we had a special case, and we had to remove it - breaking compatibility in the process - because it was impeding on extensibility (adding keyword-only arguments) and flexibility (swapping out print() with another function). "The brackets looks better" is a misguided argument, the same way the idea that "print" was cuter as a non-parenthetical statement (and that we wouldn't need the power of a regular function call; eventually we *did* need it) was a misguided argument. > The > annotations themselves will remain arbitrary Python expressions, so if > you really need a complex type declaration, you can create a helper and > call it with whatever signature you like.[1] I don't think you understand what this discussion is about. It isn't only about annotations (they are only the channel used to convey the information), it is about standardizing a typing system in the stdlib - and, therefore, accross the Python community. I want the *default* typing system to be extensible (not by me, but by the community, because use cases for that *will* arise). I would want the same thing even if the proposal was to use something else than annotation to convey type descriptions. > I can't speak for the author of mypy, Jukka Lehtosalo, but for me, I > think the answer is that type declarations make a *mini-language*, not > full-blown Python. No, the answer is that mypy wants to support runtime instantiations of its type descriptions. It wants to allow you to write e.g. "List[Int]()" to instantiate a regular Python list. While that's one of mypy's design points, though (because mypy ultimately seems to aim at becoming a separate Python dialect), it's not desired for regular Python. > It is unclear to me just how powerful the type > language will be, but surely we don't expect it to evaluate arbitrarily > complex Python expressions at compile time? Do we? Why wouldn't it? Ideally, "compile time" is just the import of the module. (yes, some libraries are ill-behaved on import; they break pydoc and the like; I'd say it's their problem) > I find it unlikely that we expect a static linter to make sense of this: > > def bizarre(param:int if random.random() > 0.5 else str)->float: Yeah... so what, exactly? Do you also criticize Python because it allows you to write absurd code? Does it make Python "too powerful"? > So: > > - how powerful do we expect the type system to be? This is a bad question to ask. That's like asking "how powerful does a function or decorator need to be?" The entire point of devising new language or library tools is to enable use cases that we *don't know about yet*. I'm not the only one pointing this out. Other people have also come with examples where they'd like to make the type system more powerful. And that's right now. The future will see more interesting propositions, the same way decorators grew to a mechanism powering very diverse and sophisticated use cases. Saying "I don't want to make this functionality powerful because I can't think of powerful use cases" is the mentality that leads to things such as PHP (closures? functions as objects? "why would I need that to write Web pages?"). > - and how much of that power needs to be expressed using > function annotations? > > The second question is critical, because there are alternatives to > function annotations: decorators, docstring annotations, and external > stub files. Why would those other channels use a type description syntax different from the one used in function annotations? That would be crazy. You seem to think that a type system should simply be some kind of textual description. It's not. It's a set of objects (or classes) with some behaviour attached to them; that's why it uses Python syntax. Because it *is* Python code. Regards Antoine. From warren.weckesser at gmail.com Sat Aug 23 17:30:47 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sat, 23 Aug 2014 11:30:47 -0400 Subject: [Python-ideas] Add nullifier argument to functools.reduce? Message-ID: I'd like to add an additional optional argument to functools.reduce. The argument is the "nullifier" of the reducing operation. It is a value such that function(nullifier, anything) returns nullifier. For example, if function(x, y) computes x*y, the nullifier is 0. If function(x, y) is the intersection of the sets x and y, the nullifier is the empty set. The argument would allow reduce to "short circuit" its calculation. When reduce encounters the nullifier, it can return immediately. This can provide a significant improvement in performance in some cases. The change is simple. Here, for example, is the "rough equivalent" for functools.reduce from the docs: def reduce(function, iterable, initializer=None): it = iter(iterable) if initializer is None: try: initializer = next(it) except StopIteration: raise TypeError('reduce() of empty sequence with no initial value') accum_value = initializer for x in it: accum_value = function(accum_value, x) return accum_value Here's how it looks with the optional nullifier argument; the only change is the new argument and an 'if' statement in the 'for' loop. def reduce(function, iterable, initializer=None, nullifier=None): it = iter(iterable) if initializer is None: try: initializer = next(it) except StopIteration: raise TypeError('reduce() of empty sequence with no initial value') accum_value = initializer for x in it: if nullifier is not None and accum_value == nullifier: break accum_value = function(accum_value, x) return accum_value (It might be better to use a distinct singleton for the default value of nullifier, to allow None to be a valid nullifier.) The actual implementation is in the extension module _functoolsmodule.c. It looks like the changes to the C code should be straightforward. Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Sat Aug 23 17:47:22 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 23 Aug 2014 08:47:22 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On Sat, Aug 23, 2014 at 8:30 AM, Warren Weckesser wrote: > The argument would allow reduce to "short circuit" its calculation. When > reduce encounters the nullifier, it can return immediately. This can > provide > a significant improvement in performance in some cases. If you want something like this you should probably use an explicit loop. And I say that as someone who *likes* reduce. Loops are far more flexible. That said, I guess I am +0. Does this feature exist in any other programming languages? -- Devin From joshua at landau.ws Sat Aug 23 18:01:20 2014 From: joshua at landau.ws (Joshua Landau) Date: Sat, 23 Aug 2014 17:01:20 +0100 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On 23 August 2014 16:30, Warren Weckesser wrote: > I'd like to add an additional optional argument to functools.reduce. > The argument is the "nullifier" of the reducing operation. It is a value > such that function(nullifier, anything) returns nullifier. For example, if > function(x, y) computes x*y, the nullifier is 0. If function(x, y) is > the intersection of the sets x and y, the nullifier is the empty set. > > The argument would allow reduce to "short circuit" its calculation. When > reduce encounters the nullifier, it can return immediately. This can > provide > a significant improvement in performance in some cases. This hasn't been given a use-case and seems like needless complexity. It also seems far too magical. -1 for those reasons. If its only purpose is to speed up "reduce" it seems like a very bad trade-off, as *everywhere else* has a new now-sanctioned behaviour to worry about. A better answer in my opinion would be something more like "reduce(..., stop_on=sentinel)". This could even allow more optimisations than your idea: reduce(operator.mul, iterable, stop_on=0) reduce(operator.add, iterable, stop_on=float("nan")) etc. I would be -0 on that, because there hasn't been a mentioned use-case. From jeanpierreda at gmail.com Sat Aug 23 18:15:02 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 23 Aug 2014 09:15:02 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On Sat, Aug 23, 2014 at 9:01 AM, Joshua Landau wrote: > A better answer in my opinion would be something more like > "reduce(..., stop_on=sentinel)". This could even allow more > optimisations than your idea: > > reduce(operator.mul, iterable, stop_on=0) > reduce(operator.add, iterable, stop_on=float("nan")) > > etc. How is this any different? -- Devin From warren.weckesser at gmail.com Sat Aug 23 18:27:12 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sat, 23 Aug 2014 12:27:12 -0400 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On Sat, Aug 23, 2014 at 12:01 PM, Joshua Landau wrote: > On 23 August 2014 16:30, Warren Weckesser > wrote: > > I'd like to add an additional optional argument to functools.reduce. > > The argument is the "nullifier" of the reducing operation. It is a value > > such that function(nullifier, anything) returns nullifier. For example, > if > > function(x, y) computes x*y, the nullifier is 0. If function(x, y) is > > the intersection of the sets x and y, the nullifier is the empty set. > > > > The argument would allow reduce to "short circuit" its calculation. > When > > reduce encounters the nullifier, it can return immediately. This can > > provide > > a significant improvement in performance in some cases. > > This hasn't been given a use-case and seems like needless complexity. > It also seems far too magical. -1 for those reasons. > I guess "magical" is a matter of perspective ("Any sufficiently advanced technology..." and all that). > > If its only purpose is to speed up "reduce" it seems like a very bad > trade-off, as *everywhere else* has a new now-sanctioned behaviour to > worry about. > I don't understand what you mean. Could you elaborate? > > A better answer in my opinion would be something more like > "reduce(..., stop_on=sentinel)". This could even allow more > optimisations than your idea: > > reduce(operator.mul, iterable, stop_on=0) > reduce(operator.add, iterable, stop_on=float("nan")) > > Do you mean it would stop when the sentinel was encountered in iterable? That wouldn't help in an example such as reduce(lambda x, y: x & y, [{1,2}, {3, 4}, {5, 6}], nullifier={}) The nullifier is the empty set, but the empty set does not occur in the iterable. (Joshua, sorry for sending this to you again. Forgot to "Reply all" the first time.) etc. > > I would be -0 on that, because there hasn't been a mentioned use-case. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Aug 23 19:25:06 2014 From: mertz at gnosis.cx (David Mertz) Date: Sat, 23 Aug 2014 10:25:06 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: This "nullifier" is mathematically called an "absorbing element", but saying an "attractor" might be a little more general. I.e. think of a local optimization problem, where multiple local min/max points might occur. If you reach one, further iteration won't budge from that point, even if it's not the "global absorbing element." However, any such argument--including the much more useful sentinel/'stop_on' idea--significantly changes the semantics of reduce. In particular, as is reduce() always consumes its iterator. Under these changes, it may or may not consume the iterator, depending on what elements occur. Given that one can easily write one's own three line wrapper 'reduce_with_attractor()' for this special semantics which hasn't been given a use case, I can't see a point of including the argument in the stdlib. -1 on proposal. On Sat, Aug 23, 2014 at 8:30 AM, Warren Weckesser < warren.weckesser at gmail.com> wrote: > I'd like to add an additional optional argument to functools.reduce. > The argument is the "nullifier" of the reducing operation. It is a value > such that function(nullifier, anything) returns nullifier. For example, if > function(x, y) computes x*y, the nullifier is 0. If function(x, y) is > the intersection of the sets x and y, the nullifier is the empty set. > > The argument would allow reduce to "short circuit" its calculation. When > reduce encounters the nullifier, it can return immediately. This can > provide > a significant improvement in performance in some cases. > > The change is simple. Here, for example, is the "rough equivalent" for > functools.reduce from the docs: > > def reduce(function, iterable, initializer=None): > it = iter(iterable) > if initializer is None: > try: > initializer = next(it) > except StopIteration: > raise TypeError('reduce() of empty sequence with no > initial value') > accum_value = initializer > for x in it: > accum_value = function(accum_value, x) > return accum_value > > Here's how it looks with the optional nullifier argument; the only > change is the new argument and an 'if' statement in the 'for' loop. > > def reduce(function, iterable, initializer=None, nullifier=None): > it = iter(iterable) > if initializer is None: > try: > initializer = next(it) > except StopIteration: > raise TypeError('reduce() of empty sequence with no > initial value') > accum_value = initializer > for x in it: > if nullifier is not None and accum_value == nullifier: > break > accum_value = function(accum_value, x) > return accum_value > > (It might be better to use a distinct singleton for the default > value of nullifier, to allow None to be a valid nullifier.) > > The actual implementation is in the extension module _functoolsmodule.c. > It looks like the changes to the C code should be straightforward. > > > Warren > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Aug 23 19:25:41 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Aug 2014 03:25:41 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> Message-ID: <20140823172536.GZ25957@ando> On Sat, Aug 23, 2014 at 09:44:02AM -0400, Antoine Pitrou wrote: > Le 23/08/2014 01:13, Steven D'Aprano a ?crit : > >On Fri, Aug 22, 2014 at 10:31:18PM -0400, Antoine Pitrou wrote: > > > >>>>Python has one of the most powerful and user-friendly function call > >>>>syntaxes around, why reinvent something clearly inferior and alien? > >>> > >>>I wouldn't say it is alien. abc[xyz] is clearly Python syntax. > >> > >>But it is completely uncommon for anything else than subscripting and > >>indexing containers. > > > >And function call syntax is completely uncommon for anything else than > >calling functions (including types and methods). One way or the other, > >we're adding a new use, type declarations. > > It's not a new use. A type class is a class, and calling it is just > instantiating that class. There's nothing new here. If you think that's > a bit "meta", it's no different than e.g. higher-order functions. There's no instantiation during *static* analysis, because the code hasn't run yet. I don't think it's productive to argue about what's new and what isn't. I think it is more important to discuss what appears to me to be a problem with your suggestion that annotations use call syntax rather than subscript syntax. Here's an annotation: param:Sequence[int] or as you prefer: param:Sequence(int) There's two problems with the later, as I see it. Of course, during static analysis, we can use any syntax we like, but Python annotations are also evaluated at runtime. (1) Sequence is an ABC, and like all abstract classes, you're not supposed to be able to instantiate it: py> Sequence() Traceback (most recent call last): File "", line 1, in TypeError: Can't instantiate abstract class Sequence with abstract methods __getitem__, __len__ For your proposal to be accepted, that would have to change. I think that is a problem. At the very least, it's a backwards-incompatibility, even within Python 3.x. (2) Even if we allow instantiating Sequence, and returning a type (rather than an instance of Sequence), I don't think that Sequence should accept arguments that concrete subclasses would accept: py> list(int) Traceback (most recent call last): File "", line 1, in TypeError: 'type' object is not iterable So we would have to allow list() to accept type arguments, and return a type, as well as the usual iterable arguments. That strikes me as messy, ugly and confusing. I would much prefer keeping call syntax on a type to instantiate the type, so that list(obj) either fails or returns a list. list[int] can return anything needed, it doesn't have to be an instance of list. > "The brackets looks better" is a misguided argument, the same way the > idea that "print" was cuter as a non-parenthetical statement (and that > we wouldn't need the power of a regular function call; eventually we > *did* need it) was a misguided argument. You're entitled to your opinion, but in my opinion, trying to avoid confusing, hard to read syntax is never misguided. Overloading call syntax to do two things (instantiate types, and type annotations), and placing such calls in the function parameter list which already has parens, risks creating hard to read, ugly code. I think that using [] helps the annotations stand out, and I think that allowing the full function call syntax complete with keyword-only arguments inside type annotations is a case of YAGNI as well as confusing and hard to read. That's my opinion; yours may differ. In any case, I think this is a minor point, and would rather not get into arguments about aesthetics. Ultimately, Guido will decide which one he prefers the look of. > >The > >annotations themselves will remain arbitrary Python expressions, so if > >you really need a complex type declaration, you can create a helper and > >call it with whatever signature you like.[1] > > I don't think you understand what this discussion is about. It isn't > only about annotations (they are only the channel used to convey the > information), it is about standardizing a typing system in the stdlib - > and, therefore, accross the Python community. I think you've got that exactly backwards. Guido's original post seemed clear to me that this proposal was *not* about adding a typing system to CPython, at least not now, but *only* about standardizing on the syntax used. A couple of quotes from his original post: [GvR] The actual type checker will not be integrated with the Python interpreter, and it will not be checked into the CPython repository. The only thing that needs to be added to the stdlib is a copy of mypy's typing.py module. [...] The curious thing here is that while standardizing a syntax for type annotations, we technically still won't be adopting standard rules for type checking. This is intentional. > >It is unclear to me just how powerful the type > >language will be, but surely we don't expect it to evaluate arbitrarily > >complex Python expressions at compile time? Do we? > > Why wouldn't it? Ideally, "compile time" is just the import of the module. > (yes, some libraries are ill-behaved on import; they break pydoc and the > like; I'd say it's their problem) When I say compile-time, I mean when the module is compiled to byte code, before it is executed or imported. > >- how powerful do we expect the type system to be? > > This is a bad question to ask. That's like asking "how powerful does a > function or decorator need to be?" The entire point of devising new > language or library tools is to enable use cases that we *don't know > about yet*. Decorators are an excellent example. Have you noticed how limited the syntax for decorators are? Probably not, because you can do a lot with decorators even though the syntax is deliberately limited. But limited it is. For instance, you can't use decorator syntax with an index lookup: py> @decorators[0] File "", line 1 @decorators[0] ^ SyntaxError: invalid syntax or more than one call: py> @decorate(x)(y) File "", line 1 @decorate(x)(y) ^ SyntaxError: invalid syntax (Both examples in 3.3; 3.4 may differ.) Does this limit the power of decorators? No, of course not. The sorts of things decorators can do is only lightly constrained by the restrictions on syntax, and I believe that the same will apply to type annotations. [...] > >- and how much of that power needs to be expressed using > > function annotations? > > > >The second question is critical, because there are alternatives to > >function annotations: decorators, docstring annotations, and external > >stub files. > > Why would those other channels use a type description syntax different > from the one used in function annotations? That would be crazy. Because function annotations are limited to a single expression, while the others are not. You're talking about me wanting to limit the power of the static type system, but I'm not really. I'm just wanting to limit how much clutter ends up inside the function parameter list. Not every arbitrarily complex type description needs to be written as a single expression inside the parameter list. Static analysis tools can make use of: * function annotations; * class definitions outside of the function (e.g. ABCs); * decorators; * docstring annotations; * stub files; * type inference on variables; * even inline comments (as mypy does) Guido's proposal is only about the first, and possibly by implication the second. The rest aren't going away, nor are they being standardised yet. There's plenty of opportunity for future tools to develop. > You seem to think that a type system should simply be some kind of > textual description. It's not. It's a set of objects (or classes) with > some behaviour attached to them; that's why it uses Python syntax. > Because it *is* Python code. That may be true about *runtime* typing, but it is not true for *static* typing since that occurs before the Python code runs. You're absolutely correct that there are other uses for type annotations apart from static type checking, and if you go back over my previous posts you'll see I've made a number of references to dynamic runtime behaviour. Ethan even (politely) told me off for forgetting the static part of the proposal, when he thought I was emphasising the runtime implications too much. For you now to think I've forgotten the runtime implications is ironic :-) -- Steven From steve at pearwood.info Sat Aug 23 19:43:58 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Aug 2014 03:43:58 +1000 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: <20140823174358.GA25957@ando> On Sat, Aug 23, 2014 at 10:25:06AM -0700, David Mertz wrote: > This "nullifier" is mathematically called an "absorbing element", but > saying an "attractor" might be a little more general. I.e. think of a > local optimization problem, where multiple local min/max points might > occur. If you reach one, further iteration won't budge from that point, > even if it's not the "global absorbing element." Yes. Note that arbitrary systems may have more than one attractors, including cycles (a -> b -> c -> a) and even "strange attractors" of infinite complexity. It's probably too much to expect reduce to deal with cycles, but a really general solution should at least deal with multiple attractors. > Given that one can easily write one's own three line wrapper > 'reduce_with_attractor()' for this special semantics I don't think you can. Although it's 3:30am here and it's past my bed time, so perhaps I'm wrong. The problem is that the wrapper cannot see the reduced value until reduce() returns, and we want to short-circuit the call once the reduced value is the attractor. Still, easy or not, I think the semantics are too specialised to justify in the standard library, especially given that Guido doesn't like reduce and it almost got removed. A better solution, I think, would be to stick this reduce_with_attractor() in some third-party functional tool library. -- Steven From bob at redivi.com Sat Aug 23 19:44:28 2014 From: bob at redivi.com (Bob Ippolito) Date: Sat, 23 Aug 2014 10:44:28 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: Yes, such a 'reduce_with_attractor()' can be easily implemented with itertools.accumulate. Maybe this would be good to add to the "Itertools Recipes" section of the docs? On Sat, Aug 23, 2014 at 10:25 AM, David Mertz wrote: > This "nullifier" is mathematically called an "absorbing element", but > saying an "attractor" might be a little more general. I.e. think of a > local optimization problem, where multiple local min/max points might > occur. If you reach one, further iteration won't budge from that point, > even if it's not the "global absorbing element." > > However, any such argument--including the much more useful > sentinel/'stop_on' idea--significantly changes the semantics of reduce. In > particular, as is reduce() always consumes its iterator. Under these > changes, it may or may not consume the iterator, depending on what elements > occur. > > Given that one can easily write one's own three line wrapper > 'reduce_with_attractor()' for this special semantics which hasn't been > given a use case, I can't see a point of including the argument in the > stdlib. > > -1 on proposal. > > > > On Sat, Aug 23, 2014 at 8:30 AM, Warren Weckesser < > warren.weckesser at gmail.com> wrote: > >> I'd like to add an additional optional argument to functools.reduce. >> The argument is the "nullifier" of the reducing operation. It is a value >> such that function(nullifier, anything) returns nullifier. For example, >> if >> function(x, y) computes x*y, the nullifier is 0. If function(x, y) is >> the intersection of the sets x and y, the nullifier is the empty set. >> >> The argument would allow reduce to "short circuit" its calculation. When >> reduce encounters the nullifier, it can return immediately. This can >> provide >> a significant improvement in performance in some cases. >> >> The change is simple. Here, for example, is the "rough equivalent" for >> functools.reduce from the docs: >> >> def reduce(function, iterable, initializer=None): >> it = iter(iterable) >> if initializer is None: >> try: >> initializer = next(it) >> except StopIteration: >> raise TypeError('reduce() of empty sequence with no >> initial value') >> accum_value = initializer >> for x in it: >> accum_value = function(accum_value, x) >> return accum_value >> >> Here's how it looks with the optional nullifier argument; the only >> change is the new argument and an 'if' statement in the 'for' loop. >> >> def reduce(function, iterable, initializer=None, nullifier=None): >> it = iter(iterable) >> if initializer is None: >> try: >> initializer = next(it) >> except StopIteration: >> raise TypeError('reduce() of empty sequence with no >> initial value') >> accum_value = initializer >> for x in it: >> if nullifier is not None and accum_value == nullifier: >> break >> accum_value = function(accum_value, x) >> return accum_value >> >> (It might be better to use a distinct singleton for the default >> value of nullifier, to allow None to be a valid nullifier.) >> >> The actual implementation is in the extension module _functoolsmodule.c. >> It looks like the changes to the C code should be straightforward. >> >> >> Warren >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Aug 23 20:43:42 2014 From: mertz at gnosis.cx (David Mertz) Date: Sat, 23 Aug 2014 11:43:42 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: <20140823174358.GA25957@ando> References: <20140823174358.GA25957@ando> Message-ID: It's true, Steven, that we'd have to use itertools.accumulate() rather than functools.reduce() to do the heavy lifting here (as Bob suggested). But using that, here are three lines: from itertools import * from operator import add def reduce_with_attractor(func, it, start=?, end_if=?): it = iter(it) start = start if start?? else it.__next__() return list(takewhile(? x: x?end_if, accumulate(chain([start],it), func)))[-1] Oh, that's cute... my copy-paste was using https://github.com/ehamberg/vim-cute-python (somewhat customized further by me). In regular ASCII: def reduce_with_attractor(func, it, start=None, end_if=None): it = iter(it) start = start if start!=None else it.__next__() return list(takewhile(lambda x: x!=end_if, accumulate(chain([start],it), func)))[-1] This gives you the accumulation up-to-but-not-including the attractor. I guess the OP wanted to return the attractor itself (although that seems slightly less useful to me). I'd have to play around to get that version in three lines... maybe it would take 4 or 5 lines to do it. On Sat, Aug 23, 2014 at 10:43 AM, Steven D'Aprano wrote: > On Sat, Aug 23, 2014 at 10:25:06AM -0700, David Mertz wrote: > > This "nullifier" is mathematically called an "absorbing element", but > > saying an "attractor" might be a little more general. I.e. think of a > > local optimization problem, where multiple local min/max points might > > occur. If you reach one, further iteration won't budge from that point, > > even if it's not the "global absorbing element." > > Yes. Note that arbitrary systems may have more than one attractors, > including cycles (a -> b -> c -> a) and even "strange attractors" of > infinite complexity. It's probably too much to expect reduce to deal > with cycles, but a really general solution should at least deal with > multiple attractors. > > > Given that one can easily write one's own three line wrapper > > 'reduce_with_attractor()' for this special semantics > > I don't think you can. Although it's 3:30am here and it's past my bed > time, so perhaps I'm wrong. The problem is that the wrapper cannot see > the reduced value until reduce() returns, and we want to short-circuit > the call once the reduced value is the attractor. > > Still, easy or not, I think the semantics are too specialised to > justify in the standard library, especially given that Guido doesn't > like reduce and it almost got removed. A better solution, I think, would > be to stick this reduce_with_attractor() in some third-party functional > tool library. > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Aug 23 20:54:37 2014 From: mertz at gnosis.cx (David Mertz) Date: Sat, 23 Aug 2014 11:54:37 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: <20140823174358.GA25957@ando> Message-ID: Completely off topic, but the symbol substitutions in https://github.com/ehamberg/vim-cute-python/tree/moresymbols used '?' for both 'set()' and 'None' which seemed confusing to me. Using aleph ('?') both somewhat resembles a Roman 'N(one)' and also makes a certain kind of sense if you think of aleph0 as the first "inaccessible cardinal" in set theory. It's too bad I didn't have any loops or 'in' tests in my three liner, since I love the look of 'i ? collection' on screen. On Sat, Aug 23, 2014 at 11:43 AM, David Mertz wrote: > It's true, Steven, that we'd have to use itertools.accumulate() rather > than functools.reduce() to do the heavy lifting here (as Bob suggested). > But using that, here are three lines: > > from itertools import * > from operator import add > > def reduce_with_attractor(func, it, start=?, end_if=?): > it = iter(it) > start = start if start?? else it.__next__() > return list(takewhile(? x: x?end_if, > accumulate(chain([start],it), func)))[-1] > > Oh, that's cute... my copy-paste was using > https://github.com/ehamberg/vim-cute-python (somewhat customized further > by me). In regular ASCII: > > def reduce_with_attractor(func, it, start=None, end_if=None): > it = iter(it) > start = start if start!=None else it.__next__() > return list(takewhile(lambda x: x!=end_if, > accumulate(chain([start],it), func)))[-1] > > This gives you the accumulation up-to-but-not-including the attractor. I > guess the OP wanted to return the attractor itself (although that seems > slightly less useful to me). I'd have to play around to get that version > in three lines... maybe it would take 4 or 5 lines to do it. > > > On Sat, Aug 23, 2014 at 10:43 AM, Steven D'Aprano > wrote: > >> On Sat, Aug 23, 2014 at 10:25:06AM -0700, David Mertz wrote: >> > This "nullifier" is mathematically called an "absorbing element", but >> > saying an "attractor" might be a little more general. I.e. think of a >> > local optimization problem, where multiple local min/max points might >> > occur. If you reach one, further iteration won't budge from that point, >> > even if it's not the "global absorbing element." >> >> Yes. Note that arbitrary systems may have more than one attractors, >> including cycles (a -> b -> c -> a) and even "strange attractors" of >> infinite complexity. It's probably too much to expect reduce to deal >> with cycles, but a really general solution should at least deal with >> multiple attractors. >> >> > Given that one can easily write one's own three line wrapper >> > 'reduce_with_attractor()' for this special semantics >> >> I don't think you can. Although it's 3:30am here and it's past my bed >> time, so perhaps I'm wrong. The problem is that the wrapper cannot see >> the reduced value until reduce() returns, and we want to short-circuit >> the call once the reduced value is the attractor. >> >> Still, easy or not, I think the semantics are too specialised to >> justify in the standard library, especially given that Guido doesn't >> like reduce and it almost got removed. A better solution, I think, would >> be to stick this reduce_with_attractor() in some third-party functional >> tool library. >> >> >> -- >> Steven >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua at landau.ws Sat Aug 23 21:23:16 2014 From: joshua at landau.ws (Joshua Landau) Date: Sat, 23 Aug 2014 20:23:16 +0100 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On 23 August 2014 17:15, Devin Jeanpierre wrote: > On Sat, Aug 23, 2014 at 9:01 AM, Joshua Landau wrote: >> A better answer in my opinion would be something more like >> "reduce(..., stop_on=sentinel)". > > How is this any different? It's not. I'm a fool who doesn't read. Apologies for the noise. From tomirendo at gmail.com Sat Aug 23 21:23:56 2014 From: tomirendo at gmail.com (Yotam Vaknin) Date: Sat, 23 Aug 2014 22:23:56 +0300 Subject: [Python-ideas] Proposal : allowing grouping by relation Message-ID: Hi, I am using groupby (from itertools) to group objects by a key. It would be very useful for me to be able to group objects by the relation of two consecutive objects or by an object relation to the first object in the current group. I think it should be done by adding a "relation" keyword to the function, that accept two argument functions that return true or false. It would also be useful to create a function that enable easily creating relation functions. (Like attrgetter does for keys) ls = "aaabcdddefgjklm" groupby(ls, relation=difference(3,key = ord)) #[['a', 'a', 'a', 'b', 'c', 'd', 'd', 'd', 'e', 'f', 'g'], ['j', 'k', 'l', 'm']] I think in this case the function won't return a key-group tuple, but just a group iterable. This is already very useful for me, to group event objects in a list if they are close enough in time. I wrote most of what I had in mind here: https://github.com/tomirendo/Grouper -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua at landau.ws Sat Aug 23 21:29:32 2014 From: joshua at landau.ws (Joshua Landau) Date: Sat, 23 Aug 2014 20:29:32 +0100 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On 23 August 2014 17:27, Warren Weckesser wrote: > > On Sat, Aug 23, 2014 at 12:01 PM, Joshua Landau wrote: >> >> On 23 August 2014 16:30, Warren Weckesser >> wrote: >>> >>> >> >> > > I suggest you just ignore what I wrote, as I was obviously too out of it to read properly. Let me try again: "This sound good, but it lacks a use-case and has a somewhat poor name." From stephen at xemacs.org Sat Aug 23 22:42:53 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 24 Aug 2014 05:42:53 +0900 Subject: [Python-ideas] Proposal : allowing grouping by relation In-Reply-To: References: Message-ID: <87bnrbc4z6.fsf@uwakimon.sk.tsukuba.ac.jp> Yotam Vaknin writes: > Hi, > > I am using groupby (from itertools) to group objects by a key. It > would be very useful for me to be able to group objects by the > relation of two consecutive objects or by an object relation to > the first object in the current group. I don't think you need to extend groupby. You can just cache the object to compare to. How about a decorator like def relate_to_first(is_related): def wrapped(this, _first=[]): if _first: if is_related(this, _first[0]): pass else: _first[0] = this _first[1] += 1 else: _first[0] = this _first[1] = 0 return _first[1] return wrapped @relate_to_first def some_relation(this, that): pass and similarly for a decorator relate_to_last? There are probably more elegant ways to do this, such as a class whose instances are callable. Such a class could also provide a reset method so you could reuse the relation Warning: that code is untested. From abarnert at yahoo.com Sat Aug 23 22:48:26 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 23 Aug 2014 13:48:26 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140823051357.GX25957@ando> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> Message-ID: On Aug 22, 2014, at 22:13, Steven D'Aprano wrote: > I can't speak for the author of mypy, Jukka Lehtosalo, but for me, I > think the answer is that type declarations make a *mini-language*, not > full-blown Python. It is unclear to me just how powerful the type > language will be, but surely we don't expect it to evaluate arbitrarily > complex Python expressions at compile time? Do we? > > I find it unlikely that we expect a static linter to make sense of this: > > def bizarre(param:int if random.random() > 0.5 else str)->float: > > except to say it returns a float and takes who-knows-what as argument. > An extreme case, I grant, but I expect that there are going to be limits > to what can be checked at compile time based on static analysis. I admit > that I don't fully understand all the implications of static versus > runtime type checking, but it doesn't seem unreasonable to expect static > declarations to be considerably simpler than what can be expressed at > runtime. Hence, a simpler mini-language is appropriate. I agree. Trying to make the declarative type system more powerful is possible, but probably not a good idea. Look at C++98's template system. It's a complete compile-time language for defining types declaratively (in terms of other types, integer literals, and pointer literals), except that it's missing the ability to recursively decompose recursive types (like a compile-time cons). While that sounds nice in theory, in practice it's painful to use, and completely different from programming in C++. That's what happens if you take a declarative type system and try to expand it until it's fully general: you end up with a type system that looks like ML instead of your language. C++14 lets you use almost the entire runtime language for computing at compile time (no goto or try, some limits on inheritance) with constexpr values and functions. You can always use the declarative language, but you can also write complex stuff imperatively, in a way that looks like C++. If we really need something as powerful as ML or Haskell for our compile-time type system, the latter would be the way to do it. And if this obviously takes too much effort to be worth doing, that means we probably don't actually need the former either. So, trying to extend MyPy's declarative system to be fully general probably isn't worth doing. >> Really, for it to be powerful enough, the type description system has to >> use its own set of classes. It can't try to hack away existing builtins >> and ABCs in the hope of expressing different things with them, or it >> *will* hit a wall some day (and, IMO, sooner rather than later). > > How powerful is "powerful enough"? Powerful enough for what? > > You've probably heard of the famous "the compiler found my infinite > loop", where the ML type checker was able to detect that code would > never terminate. I find that remarkable, and even more astonishing that, > according to some, solving the halting problem is "nothing special" for > type systems.[2] But many languages make do with less powerful type systems, > and tools such as IDEs and editors surely don't need something that > powerful. So: > > - how powerful do we expect the type system to be? > > - and how much of that power needs to be expressed using > function annotations? > > The second question is critical, because there are alternatives to > function annotations: decorators, docstring annotations, and external > stub files. > > > > > [1] Although if you do so, it is not clear to me how much the static > checker will be able to use it. > > > [2] http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Sat Aug 23 23:57:46 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 23 Aug 2014 14:57:46 -0700 Subject: [Python-ideas] Proposal : allowing grouping by relation In-Reply-To: References: Message-ID: On Aug 23, 2014, at 12:23, Yotam Vaknin wrote: > Hi, > > I am using groupby (from itertools) to group objects by a key. It would be very useful for me to be able to group objects by the relation of two consecutive objects or by an object relation to the first object in the current group. > > I think it should be done by adding a "relation" keyword to the function, that accept two argument functions that return true or false. This _should be_ easy to write as a wrapper around groupby with a key that checks your relation. But there's one problem: groupby checks the _first_ key in a group against each new key, instead of the most recent one. I wrote a blog post last year about this (http://stupidpythonideas.blogpost.com/2014/01/grouping-into-runs-of-adjacent-values.html). It turns our to be pretty easy if your relation is symmetric, but only one of the obvious ways to do it actually works. Anyway, it might be worth changing groupby so it never compares x==y instead of y==x, and making the C implementation and the Python equivalent in the docs actually equivalent. Beyond that, I think it might make sense to add a relation_to_key function and/or to change cmp_to_key so it's directly usable with groupby. Then, it should be possible to make groupby_relation into a 3-line wrapper around groupby, in which case I think it might be better as a recipe (and submitted to more_itertools on PyPI) than to add it to itertools itself. > It would also be useful to create a function that enable easily creating relation functions. (Like attrgetter does for keys) > ls = "aaabcdddefgjklm" > groupby(ls, relation=difference(3,key = ord)) > #[['a', 'a', 'a', 'b', 'c', 'd', 'd', 'd', 'e', 'f', 'g'], ['j', 'k', 'l', 'm']] > > I think in this case the function won't return a key-group tuple, but just a group iterable. The key actually can be useful here. You can use it as a label for the "column". Especially if you've written your key function so it keeps track of both the first and most recent values, instead of just the most recent, so you can label it "a-_", where that _ is the current value at any given point, and the last value once you've consumed the group iterator. Sure, you _could_ recover that information from the group itself if you need it, but isn't it even easier to discard it if you don't need it? > > This is already very useful for me, to group event objects in a list if they are close enough in time. > > I wrote most of what I had in mind here: > https://github.com/tomirendo/Grouper > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Sun Aug 24 01:00:54 2014 From: antoine at python.org (Antoine Pitrou) Date: Sat, 23 Aug 2014 19:00:54 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140823172536.GZ25957@ando> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> Message-ID: Le 23/08/2014 13:25, Steven D'Aprano a ?crit : >> >> It's not a new use. A type class is a class, and calling it is just >> instantiating that class. There's nothing new here. If you think that's >> a bit "meta", it's no different than e.g. higher-order functions. > > There's no instantiation during *static* analysis, because the code > hasn't run yet. In your idea of "static analysis", it hasn't. Because you think it should involve some kind of separate syntax analysis tool that has nothing to do with regular Python. But Python is powerful enough to let you do that using normal introspection of modules. And it's *exactly* how we are exposing function annotations (and also docstrings, etc.): using runtime-accessible introspection information which is gathered by importing modules and therefore actually *executing* toplevel module code. Not merely compiling it. The rest of your message I'm not motivated to respond to, sorry. Regards Antoine. From cfkaran2 at gmail.com Sun Aug 24 02:37:34 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sat, 23 Aug 2014 20:37:34 -0400 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: <846C4603-A7EF-4394-8CE9-4B8C614E1E13@gmail.com> As others have said on the list, 'go away for a few days, and find a mega-thread in your inbox'. Yay. On Aug 19, 2014, at 1:01 PM, Guido van Rossum wrote: > On Tue, Aug 19, 2014 at 6:27 AM, Nick Coghlan wrote: > [Agreeable musings about team size...] > > Ultimately, my perspective is that Guido's proposal boils down to > having a nice syntax where: > > def myfunc(a : KindX, b: KindY, c: KindZ): > ... > > is the moral equivalent of: > > def myfunc(a, b, c): > assert isinstance(a, KindX) > assert isinstance(b, KindY) > assert isinstance(c, KindZ) > > Please no. The asserts affect runtime. The type declarations are for linting, IDEs and docs. I don't want the difference to be swept under the rug. I want it to be the key feature of the proposal. Going slightly sideways on this -- I think this is why we should use decorators instead of annotations; as has already been mentioned twice on the list, http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a good job pointing out that static and dynamic typing systems are separate but overlapping concepts. It also points out that both have their place. If we define decorators that can be turned on or off easily (command-line option? Environment variable? other?), then the end user can choose if if her or she is going to do static, dynamic, both, or none. This could be useful when trying to track down that annoying bug in long-running production code. Also, Types and Programming Languages by Dr. Pierce has been mentioned at least twice as well; would it be useful to ask him to join in the discussion? Thanks, Cem Karan From greg.ewing at canterbury.ac.nz Sun Aug 24 03:11:34 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 24 Aug 2014 13:11:34 +1200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53F93BC6.6030307@canterbury.ac.nz> Georg Brandl wrote: > Another thought is whether, assuming > > class MyList(list): > ... > > the type "MyList[int]" would do the expected thing. (And if not, > how you would write the correct type generation code.) I would expect that to be a error, unless MyList were declared as accepting a type parameter, using whatever means is decided on for that in the type language. -- Greg From steve at pearwood.info Sun Aug 24 03:53:26 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Aug 2014 11:53:26 +1000 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: <20140823174358.GA25957@ando> Message-ID: <20140824015325.GB25957@ando> On Sat, Aug 23, 2014 at 11:43:42AM -0700, David Mertz wrote: > def reduce_with_attractor(func, it, start=None, end_if=None): > it = iter(it) > start = start if start!=None else it.__next__() > return list(takewhile(lambda x: x!=end_if, > accumulate(chain([start],it), func)))[-1] A couple of points: - Don't use it.__next__(), use the built-in next(it). - To match the behaviour of reduce, you need to catch the StopIteration and raise TypeError. - Your implementation eagerly generates a potentially enormous list of intermediate results, which strikes me as somewhat unfortunate for a functional tool like reduce. In other words, for some inputs, this is going to perform like a dog, generating a HUGE list up front, then throwing it all away except for the final value. > This gives you the accumulation up-to-but-not-including the attractor. I > guess the OP wanted to return the attractor itself (although that seems > slightly less useful to me). Not really. His use-case seems to be to short-cut a lot of unnecessary calculations, e.g. suppose you write product() as reduce(operator.mul, iterable). In the event that the product reaches zero, you can[1] short-circuit the rest of the iterable and just return 0: product([1, 2, 3, 0] + [4]*1000000) ought to reduce 0, not 6, and the intent is for it to do so *quickly*, ignoring the 4s at the end of the list. [1] Actually you can't. 0 is no longer an attractor in the presence of INF or NAN. -- Steven From mertz at gnosis.cx Sun Aug 24 04:19:25 2014 From: mertz at gnosis.cx (David Mertz) Date: Sat, 23 Aug 2014 19:19:25 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: <20140824015325.GB25957@ando> References: <20140823174358.GA25957@ando> <20140824015325.GB25957@ando> Message-ID: On Sat, Aug 23, 2014 at 6:53 PM, Steven D'Aprano wrote: > On Sat, Aug 23, 2014 at 11:43:42AM -0700, David Mertz wrote: > > > def reduce_with_attractor(func, it, start=None, end_if=None): > > it = iter(it) > > start = start if start!=None else it.__next__() > > return list(takewhile(lambda x: x!=end_if, > > accumulate(chain([start],it), func)))[-1] > > A couple of points: > > - Don't use it.__next__(), use the built-in next(it). > Yeah, good point. > - To match the behaviour of reduce, you need to catch the StopIteration > and raise TypeError. > Oh, OK. Hadn't thought of that. > - Your implementation eagerly generates a potentially enormous list of > intermediate results, which strikes me as somewhat unfortunate for a > functional tool like reduce. In other words, for some inputs, this is > going to perform like a dog, generating a HUGE list up front, then > throwing it all away except for the final value. > I know. I realized this flaw right away. I was trying to be cute and fit it in my promised 3 lines. It would be better to put it in a loop to realize the successive values, of course--but would take an extra line or two. Maybe there's a way to squeeze it in one line with itertools rather than a regular loop though. > > This gives you the accumulation up-to-but-not-including the attractor. I > > guess the OP wanted to return the attractor itself (although that seems > > slightly less useful to me). > > Not really. His use-case seems to be to short-cut a lot of unnecessary > calculations, e.g. suppose you write product() as reduce(operator.mul, > iterable). In the event that the product reaches zero, you can[1] > short-circuit the rest of the iterable and just return 0: > > product([1, 2, 3, 0] + [4]*1000000) > > ought to reduce 0, not 6, and the intent is for it to do so *quickly*, > ignoring the 4s at the end of the list. > I guess that's true. Although I can certainly imagine being interested not only in the final attractor, but *that* it reached an attractor and ended early. Not sure how best to signal that. Actually, now that I think of it, it would be kinda nice to make the function 'reduce_with_attractorS()' instead, and allow specification of multiple attractors. I welcome your improved version of the code :-). Feel free to take a whole 10 lines to do it right. > [1] Actually you can't. 0 is no longer an attractor in the presence of > INF or NAN. > I was sort of thinking of a "we're all adults here" attitude. That is, the "attractor" might not really be a genuine attractor, but we still trust the caller to say it is. I.e. my function would accept this call: reduce_with_attractor(operator.mul, range(1,1e6), end_if=6)) I'm making the claim that reaching '6' is a stopping point... which, well it is. No, it's not an actual attractor, but maybe a caller really does want to stop iterating if it gets to that value anyway. Hence 'end_if' is actually an accurate name. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cfkaran2 at gmail.com Sun Aug 24 04:18:42 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sat, 23 Aug 2014 22:18:42 -0400 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: Message-ID: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> On Aug 20, 2014, at 11:08 AM, Paul Moore wrote: > Sigh. I go away for a week and come back to a mega-thread I can never > hope to catch up on :-) > > TL; DR; - Although mypy looks interesting, I think it's too soon to > close the door on all other uses of annotations. Let's find a solution > that allows exploration of alternative uses for a while longer. > > OK, can I just make some points regarding the static typing thread. > First of all, I have no issue with the idea of static typing, and in > fact I look forward to seeing what benefits it might have (if nothing > else, the pointer to mypy, which I'd never heard of before, is > appreciated). It won't be something I use soon (see below) but that's > fine. > > But Guido seems to be saying (on a number of occasions) that nobody is > really using annotations, so he wants to focus on the static typing > use case alone. I think this is a mistake. First of all, I see no > reason why functions using typing annotations could not be introduced > with a decorator[1]. So why insist that this is the *only* use of > annotations, when it's pretty easy to allow others to co-exist? > > Also, the "nobody is using annotations" argument? Personally, I know > of a few other uses: > > 1. Argument parsers - at least 3 have been mentioned in the thread. > 2. Structure unpacking - I think there is a library that uses > annotations for this, although I may be wrong. > 3. FFI bindings. I know I've seen this discussed, although I can't > find a reference just now. > > There are probably other ideas around as well (GUI bindings, > documentation generation, ...) None are particularly mature, and most > are just at the "ideas" stage, but typically the ideas I have seen are > the sort of thing you'd write a library for, and Python 3 only > libraries *really* aren't common yet. > > The problem for people wanting to experiment with annotations, is that > they need to be writing Python 3 only code. While Python 3 adoption is > growing rapidly, I suspect that large applications are typically still > focused on either going through, or tidying up after, a 2-3 migration. > And new projects, while they may be developed from the ground up using > Python 3, will typically be using programmers skilled in Python 2, to > whom Python 3 features are not yet an "instinctive" part of the > toolset. Apart from large standalone applications, there are smaller > scripts (which are typically going to be too small to need > programming-in-the-large features like annotations) and libraries > (which really aren't yet in a position to drop Python 2.x totally, > unless they have a fairly small user base). > > So I don't see it as compelling that usage of annotations in the wild > is not yet extensive. > > Rather than close the door on alternative uses of annotations, can I suggest: > > 1. By all means bless mypy syntax as the standard static typing > notation - this seems like a good thing. > 2. Clarify that static typing annotations should be introduced with a > decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy > declaration in the stdlib, and have a registration protocols for tools > to hook their own implementation into it.[2] > 3. Leave the door open for other uses of decorators, at least until > some of the more major libraries drop Python 2.x support completely > (and hence can afford to have a dependency on a Python 3 only module > that uses annotations). See at that stage if annotations take off. > 4. If we get to a point where even libraries that *could* use > annotations don't, then revisit the idea of restricting usage to just > type information. > > Paul > > [1] Also, a decorator could allow a Python 2 compatible form by using > decorator arguments as an alternative to annotations. > [2] I've yet to see a clear explanation of how "a tool using type > annotations" like an linter, editor, IDE or Python compiler would use > them in such a way that precludes decoration as a means of signalling > the annotation semantics. A long while back I proposed a mechanism for sharing __annotations__ between multiple, non-cooperating projects. The basic idea is that each annotation becomes a dictionary. Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary. The value is up to the project. The advantage to this is manifold: - Annotations can still have multiple uses by different groups without stepping on each other's toes. - If someone wants to make a standard, all they have to do is publish the UUID associated with their standard. For example, we might choose UUID('2cca6238-9fca-4053-aa3d-db9050e6b26b') as the official type information UUID. All projects that want to develop linters, documentation generators, etc., will use that UUID for all annotations, and the PEP will require it. - de facto standards can become de jure standards by blessing a particular UUID. - Guessing if this method is being used is relatively easy; if its a dictionary, and if every key is a UUID, it probably follows this standard. We can tighten it further by requiring some key-value pair be in every dictionary (i.e., {UUID('1ad60d50-8237-4b98-b2b1-69fd08ed575c'):"PEPXXXX"} is always in the dictionary). This makes it fairly simple to add without stomping on what people are already doing. - Finding the standard on the web should also be easy; while you might not find the PEP instantly, you'll probably zoom into it fairly fast. Disadvantages: - Typing out UUIDs is PAINFUL. I highly recommend using decorators instead. - Reading the __annotations__ dictionary will be difficult. pprint() should make this easier. I have working proof-of-concept code at https://github.com/oranguman/annotizer that defines a decorator class that handles the UUID for you. It needs to be extended to parse out information, but it handles the 'other use cases' problem fairly well. Thanks, Cem Karan From steve at pearwood.info Sun Aug 24 05:13:30 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Aug 2014 13:13:30 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53F93BC6.6030307@canterbury.ac.nz> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F93BC6.6030307@canterbury.ac.nz> Message-ID: <20140824031329.GD25957@ando> On Sun, Aug 24, 2014 at 01:11:34PM +1200, Greg Ewing wrote: > Georg Brandl wrote: > >Another thought is whether, assuming > > > > class MyList(list): > > ... > > > >the type "MyList[int]" would do the expected thing. (And if not, > >how you would write the correct type generation code.) > > I would expect that to be a error, unless MyList were > declared as accepting a type parameter, using whatever > means is decided on for that in the type language. I don't really understand what you're trying to say here, so I may be misinterpreting you. I *think* that you're trying to say that for every type in the standard library, and every class created by third parties (including subclasses), the author will have to "declare" (in some unknown sense) that it can be used for type annotations like MyList[T], for some type T. I expect that will be painful and unnecessary. I expect that if we go down this path, somebody will have to add the appropriate code to the metaclass of all types (i.e. type itself) so that AnyArbitraryType[spam] will do the right thing without the author of AnyArbitraryType needing to do any additional work. This is not entirely uncharted territory. Mypy has blazed the trail for us. The way I think Mypy works is that the classes in typing.py return themselves when subscripted. The arguments to __getitem__ are ignored except during static analysis. Here's the metaclass from Mypy: class GenericMeta(type): """Metaclass for generic classes that support indexing by types.""" def __getitem__(self, args): # Just ignore args; they are for compile-time checks only. return self https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py As I understand it, we are considering two possibilities: (1) all types are directly usable, so we can say def func(param:list[int])->list: This will require type to grow a __getitem__ method similar to the metaclass above. (2) Standard library types are not directly usable, you have to go through the typing module, which has wrappers that include the __getitem__ metaclass magic: from typing import List def func(param:List[int])->List: There are pros and cons to both approaches. -- Steven From ncoghlan at gmail.com Sun Aug 24 06:24:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Aug 2014 14:24:53 +1000 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On 24 August 2014 01:30, Warren Weckesser wrote: > I'd like to add an additional optional argument to functools.reduce. > The argument is the "nullifier" of the reducing operation. It is a value > such that function(nullifier, anything) returns nullifier. For example, if > function(x, y) computes x*y, the nullifier is 0. If function(x, y) is > the intersection of the sets x and y, the nullifier is the empty set. When it comes to judging the usefulness of functional programming features these days, my first question is generally going to be "Does PyToolz offer this?" Grumblings about the name aside, it's still the solution I recommend to folks that wish Python had more functional programming tools in the standard library: http://toolz.readthedocs.org/en/latest/api.html There's even a Cython accelerated version available (Cytoolz). "pip install toolz" for the pure Python version, "pip install cytoolz" for the accelerated one. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Sun Aug 24 07:54:22 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 24 Aug 2014 17:54:22 +1200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140824031329.GD25957@ando> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F93BC6.6030307@canterbury.ac.nz> <20140824031329.GD25957@ando> Message-ID: <53F97E0E.6030908@canterbury.ac.nz> Steven D'Aprano wrote: > I don't really understand what you're trying to say here, so I may be > misinterpreting you. I *think* that you're trying to say that for every > type in the standard library, and every class created by third parties > (including subclasses), the author will have to "declare" (in some > unknown sense) that it can be used for type annotations like MyList[T], > for some type T. I suppose the run-time incarnations of the type descriptions could be looser, but if you want to use them for static checking, the static checker is going to have to know what MyList[T] means in some detail (what effect the parameter has on the method types, etc.) The programmer will have to specify all that somehow. The way this is done in other languages with static type checking is to give the class declaration a parameter list. I was envisaging that the mypy type description syntax would have something equivalent. Not sure what form it would take, though. -- Greg From g.brandl at gmx.net Sun Aug 24 10:00:32 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 24 Aug 2014 10:00:32 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53F97E0E.6030908@canterbury.ac.nz> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F93BC6.6030307@canterbury.ac.nz> <20140824031329.GD25957@ando> <53F97E0E.6030908@canterbury.ac.nz> Message-ID: On 08/24/2014 07:54 AM, Greg Ewing wrote: > Steven D'Aprano wrote: >> I don't really understand what you're trying to say here, so I may be >> misinterpreting you. I *think* that you're trying to say that for every >> type in the standard library, and every class created by third parties >> (including subclasses), the author will have to "declare" (in some >> unknown sense) that it can be used for type annotations like MyList[T], >> for some type T. > > I suppose the run-time incarnations of the type descriptions > could be looser, but if you want to use them for static checking, > the static checker is going to have to know what MyList[T] means > in some detail (what effect the parameter has on the method > types, etc.) The programmer will have to specify all that > somehow. > > The way this is done in other languages with static type > checking is to give the class declaration a parameter list. > I was envisaging that the mypy type description syntax would have > something equivalent. Not sure what form it would take, though. Exactly. Does mypy handle that? For example, for a custom mapping type class Mapping(object): def setitem(self, key, value): ... how would one specify a) that you can use Mapping[T1, T2] as a type annotation and b) the type annotations for the "key" and "value" arguments? Georg From steve at pearwood.info Sun Aug 24 10:50:03 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Aug 2014 18:50:03 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F93BC6.6030307@canterbury.ac.nz> <20140824031329.GD25957@ando> <53F97E0E.6030908@canterbury.ac.nz> Message-ID: <20140824085003.GE25957@ando> On Sun, Aug 24, 2014 at 10:00:32AM +0200, Georg Brandl wrote: > On 08/24/2014 07:54 AM, Greg Ewing wrote: > > Steven D'Aprano wrote: > >> I don't really understand what you're trying to say here, so I may be > >> misinterpreting you. I *think* that you're trying to say that for every > >> type in the standard library, and every class created by third parties > >> (including subclasses), the author will have to "declare" (in some > >> unknown sense) that it can be used for type annotations like MyList[T], > >> for some type T. > > > > I suppose the run-time incarnations of the type descriptions > > could be looser, but if you want to use them for static checking, > > the static checker is going to have to know what MyList[T] means > > in some detail (what effect the parameter has on the method > > types, etc.) The programmer will have to specify all that > > somehow. > > > > The way this is done in other languages with static type > > checking is to give the class declaration a parameter list. > > I was envisaging that the mypy type description syntax would have > > something equivalent. Not sure what form it would take, though. > > Exactly. Does mypy handle that? For example, for a custom mapping > type > > class Mapping(object): > def setitem(self, key, value): > ... > > how would one specify > a) that you can use Mapping[T1, T2] as a type annotation and > b) the type annotations for the "key" and "value" arguments? I'm not an expert on Mypy, but I think the answer is, you can't. In order for Mypy to recognise your Mapping as an actual mapping, you have to either inherit from dict or some other mapping which Mypy knows about, or from typing.Generic. http://mypy-lang.org/tutorial.html#genericclasses from typing import typevar, Generic T = typevar('T') class Mapping(Generic[T, T]): ... Now Mypy will know that your Mapping class should be considered a mapping from some type to another type, and you can use it: def func(x:Mapping[int, str], n:int)->str: return x[n+1] which will pass the static type checker. (This is just what I understand from reading some of the docs, I welcome correction from anyone who knows better.) But that's just Mypy. Another type checker might just trust you, regardless of what terrible lies you tell it, so long as they're consistent lies: def func(x:set[int, str], n:int)->str: return x[n+1] Since you've said x is a set which maps ints to strs, the code will pass the static check, but fail at run-time since sets don't actually support __getitem__. A third type checker might do structural typing instead of nominal typing, and be able to recognise that set doesn't have __getitem__ but Mapping does. Remember that this proposal isn't about adding Mypy to the standard library or merging it with CPython. It remains a third-party implementation. People are encouraged to work on Mypy, or fork it, or build their own static tools, which may or may not do a better job of static analysis. Or runtime tools for that matter. -- Steven From abarnert at yahoo.com Sun Aug 24 12:04:09 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 24 Aug 2014 03:04:09 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140824085003.GE25957@ando> References: <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <53F93BC6.6030307@canterbury.ac.nz> <20140824031329.GD25957@ando> <53F97E0E.6030908@canterbury.ac.nz> <20140824085003.GE25957@ando> Message-ID: <389F0F28-31B1-4DBC-BA24-499CDE06FA38@yahoo.com> On Aug 24, 2014, at 1:50, Steven D'Aprano wrote: > On Sun, Aug 24, 2014 at 10:00:32AM +0200, Georg Brandl wrote: >> On 08/24/2014 07:54 AM, Greg Ewing wrote: >>> Steven D'Aprano wrote: >>>> I don't really understand what you're trying to say here, so I may be >>>> misinterpreting you. I *think* that you're trying to say that for every >>>> type in the standard library, and every class created by third parties >>>> (including subclasses), the author will have to "declare" (in some >>>> unknown sense) that it can be used for type annotations like MyList[T], >>>> for some type T. >>> >>> I suppose the run-time incarnations of the type descriptions >>> could be looser, but if you want to use them for static checking, >>> the static checker is going to have to know what MyList[T] means >>> in some detail (what effect the parameter has on the method >>> types, etc.) The programmer will have to specify all that >>> somehow. >>> >>> The way this is done in other languages with static type >>> checking is to give the class declaration a parameter list. >>> I was envisaging that the mypy type description syntax would have >>> something equivalent. Not sure what form it would take, though. >> >> Exactly. Does mypy handle that? For example, for a custom mapping >> type >> >> class Mapping(object): >> def setitem(self, key, value): >> ... >> >> how would one specify >> a) that you can use Mapping[T1, T2] as a type annotation and >> b) the type annotations for the "key" and "value" arguments? > > I'm not an expert on Mypy, but I think the answer is, you can't. In > order for Mypy to recognise your Mapping as an actual mapping, you have > to either inherit from dict or some other mapping which Mypy knows > about, or from typing.Generic. > > http://mypy-lang.org/tutorial.html#genericclasses > > > from typing import typevar, Generic > T = typevar('T') > class Mapping(Generic[T, T]): > ... > > > Now Mypy will know that your Mapping class should be considered a > mapping from some type to another type, and you can use it: No it doesn't, it just knows that it's a generic type, meaning that you can index it with types, and those types will be used to fill in the corresponding typevars on its methods. Consider Tuple[int, str]. That isn't a mapping from int to str, it's a 2-element tuple whose first element is int and whose second is str. And I'm sure you can come up with other uses for generic types of two type parameters that aren't mappings. This is exactly the same as in C++, ML and its descendants, and Java and its descendants. For example: class MyMapping(Generic[T, U]): def __getitem__(self, key: T) -> U: # ... Now a function that takes a MyMapping[int, str] will bind T to int and U to str, so the type checker knows that the argument to __getitem__ is an int and the return value is a str. And that still doesn't make it a Mapping. If collections.abc.Mapping (or typing.Mapping, if they're separate in the final version) is a Protocol, or otherwise implements structural matching, _that_ is what makes MyMapping a Mapping. And if if doesn't, then only inheriting or registering can make it a Mapping. Just like ABCs at runtime. From edk141 at gmail.com Sun Aug 24 13:39:30 2014 From: edk141 at gmail.com (Ed Kellett) Date: Sun, 24 Aug 2014 12:39:30 +0100 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> Message-ID: I have a few questions: - How often do multiple kinds of annotation end up on the same function? - Why UUIDs (rather than for e.g. PyPI packages, or some other namespace)? - What is the point? A decorator could process the annotations and put some information in func._projectname__something instead of doing the UUID dance - What would pydoc print for the function signature? On 24 August 2014 03:18, Cem Karan wrote: > > On Aug 20, 2014, at 11:08 AM, Paul Moore wrote: > >> Sigh. I go away for a week and come back to a mega-thread I can never >> hope to catch up on :-) >> >> TL; DR; - Although mypy looks interesting, I think it's too soon to >> close the door on all other uses of annotations. Let's find a solution >> that allows exploration of alternative uses for a while longer. >> >> OK, can I just make some points regarding the static typing thread. >> First of all, I have no issue with the idea of static typing, and in >> fact I look forward to seeing what benefits it might have (if nothing >> else, the pointer to mypy, which I'd never heard of before, is >> appreciated). It won't be something I use soon (see below) but that's >> fine. >> >> But Guido seems to be saying (on a number of occasions) that nobody is >> really using annotations, so he wants to focus on the static typing >> use case alone. I think this is a mistake. First of all, I see no >> reason why functions using typing annotations could not be introduced >> with a decorator[1]. So why insist that this is the *only* use of >> annotations, when it's pretty easy to allow others to co-exist? >> >> Also, the "nobody is using annotations" argument? Personally, I know >> of a few other uses: >> >> 1. Argument parsers - at least 3 have been mentioned in the thread. >> 2. Structure unpacking - I think there is a library that uses >> annotations for this, although I may be wrong. >> 3. FFI bindings. I know I've seen this discussed, although I can't >> find a reference just now. >> >> There are probably other ideas around as well (GUI bindings, >> documentation generation, ...) None are particularly mature, and most >> are just at the "ideas" stage, but typically the ideas I have seen are >> the sort of thing you'd write a library for, and Python 3 only >> libraries *really* aren't common yet. >> >> The problem for people wanting to experiment with annotations, is that >> they need to be writing Python 3 only code. While Python 3 adoption is >> growing rapidly, I suspect that large applications are typically still >> focused on either going through, or tidying up after, a 2-3 migration. >> And new projects, while they may be developed from the ground up using >> Python 3, will typically be using programmers skilled in Python 2, to >> whom Python 3 features are not yet an "instinctive" part of the >> toolset. Apart from large standalone applications, there are smaller >> scripts (which are typically going to be too small to need >> programming-in-the-large features like annotations) and libraries >> (which really aren't yet in a position to drop Python 2.x totally, >> unless they have a fairly small user base). >> >> So I don't see it as compelling that usage of annotations in the wild >> is not yet extensive. >> >> Rather than close the door on alternative uses of annotations, can I suggest: >> >> 1. By all means bless mypy syntax as the standard static typing >> notation - this seems like a good thing. >> 2. Clarify that static typing annotations should be introduced with a >> decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy >> declaration in the stdlib, and have a registration protocols for tools >> to hook their own implementation into it.[2] >> 3. Leave the door open for other uses of decorators, at least until >> some of the more major libraries drop Python 2.x support completely >> (and hence can afford to have a dependency on a Python 3 only module >> that uses annotations). See at that stage if annotations take off. >> 4. If we get to a point where even libraries that *could* use >> annotations don't, then revisit the idea of restricting usage to just >> type information. >> >> Paul >> >> [1] Also, a decorator could allow a Python 2 compatible form by using >> decorator arguments as an alternative to annotations. >> [2] I've yet to see a clear explanation of how "a tool using type >> annotations" like an linter, editor, IDE or Python compiler would use >> them in such a way that precludes decoration as a means of signalling >> the annotation semantics. > > A long while back I proposed a mechanism for sharing __annotations__ between multiple, non-cooperating projects. The basic idea is that each annotation becomes a dictionary. Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary. The value is up to the project. > > The advantage to this is manifold: > > - Annotations can still have multiple uses by different groups without stepping on each other's toes. > - If someone wants to make a standard, all they have to do is publish the UUID associated with their standard. For example, we might choose UUID('2cca6238-9fca-4053-aa3d-db9050e6b26b') as the official type information UUID. All projects that want to develop linters, documentation generators, etc., will use that UUID for all annotations, and the PEP will require it. > - de facto standards can become de jure standards by blessing a particular UUID. > - Guessing if this method is being used is relatively easy; if its a dictionary, and if every key is a UUID, it probably follows this standard. We can tighten it further by requiring some key-value pair be in every dictionary (i.e., {UUID('1ad60d50-8237-4b98-b2b1-69fd08ed575c'):"PEPXXXX"} is always in the dictionary). This makes it fairly simple to add without stomping on what people are already doing. > - Finding the standard on the web should also be easy; while you might not find the PEP instantly, you'll probably zoom into it fairly fast. > > Disadvantages: > > - Typing out UUIDs is PAINFUL. I highly recommend using decorators instead. > - Reading the __annotations__ dictionary will be difficult. pprint() should make this easier. > > I have working proof-of-concept code at https://github.com/oranguman/annotizer that defines a decorator class that handles the UUID for you. It needs to be extended to parse out information, but it handles the 'other use cases' problem fairly well. > > Thanks, > Cem Karan > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From steve at pearwood.info Sun Aug 24 13:59:55 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Aug 2014 21:59:55 +1000 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: References: <53F25EE8.8000900@stoneleaf.us> Message-ID: <20140824115954.GF25957@ando> On Thu, Aug 21, 2014 at 02:30:03PM +0000, Brett Cannon wrote: > P.S.: Off-topic for this email but something I don't think has been > mentioned more than once, type hinting like we are proposing is exactly > what Dart does and it is nice. Built-in API documentation of interfaces > along with IDE support at the API makes all of this worth it. For anyone not familiar with Dart, I think it is worth reading the Dart justification for static type hinting in a dynamic language: https://www.dartlang.org/articles/optional-types/ https://www.dartlang.org/articles/why-dart-types/ It's not quite the same as Guido's proposal, for example in Dart the type hints have no runtime effect at all, but I think it demonstrates that this is a proven approach and can work well with a dynamic language like Python without compromising the dynamic nature of the language. -- Steven From __peter__ at web.de Sun Aug 24 14:35:57 2014 From: __peter__ at web.de (Peter Otten) Date: Sun, 24 Aug 2014 14:35:57 +0200 Subject: [Python-ideas] Add nullifier argument to functools.reduce? References: <20140823174358.GA25957@ando> Message-ID: David Mertz wrote: > def reduce_with_attractor(func, it, start=None, end_if=None): > it = iter(it) > start = start if start!=None else it.__next__() > return list(takewhile(lambda x: x!=end_if, > accumulate(chain([start],it), func)))[-1] Wouldn't it be better to break this into a function that limits a sequence and to combine that with the original reduce()? >>> from functools import reduce >>> from itertools import takewhile >>> def stop_on(value, items): ... return takewhile(lambda item: item != value, items) ... >>> list(stop_on(0, [1, 2, 3, 0, 4])) [1, 2, 3] >>> from operator import mul >>> reduce(mul, stop_on(0, [1, 2, 3, 0, 4])) 6 My suggestion is to add a value-limited version of itertools.takewhile() rather than making reduce() more more powerful/complex. From steve at pearwood.info Sun Aug 24 14:41:39 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Aug 2014 22:41:39 +1000 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: <846C4603-A7EF-4394-8CE9-4B8C614E1E13@gmail.com> References: <53F25EE8.8000900@stoneleaf.us> <846C4603-A7EF-4394-8CE9-4B8C614E1E13@gmail.com> Message-ID: <20140824124139.GG25957@ando> On Sat, Aug 23, 2014 at 08:37:34PM -0400, Cem Karan wrote: > Going slightly sideways on this -- I think this is why we should use > decorators instead of annotations; as has already been mentioned twice > on the list, > http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a > good job pointing out that static and dynamic typing systems are > separate but overlapping concepts. It also points out that both have > their place. If we define decorators that can be turned on or off > easily (command-line option? Environment variable? other?), then the > end user can choose if if her or she is going to do static, dynamic, > both, or none. This could be useful when trying to track down that > annoying bug in long-running production code. This applies equally to annotations, and in fact that's exactly what Mypy already does. You can run Mypy to do static type checking, or not run it, and the annotations will be ignored. E.g. given a module program.py, you can: # Type check and then run the program: mypy program.py # Just run it, with no extra type checks: python3 program.py Guido's proposal is *not* to add static types to the CPython interpreter, at least not yet. It is just to standardise on annotations for type hinting, agree on a syntax for those type hints, and then allow third-part tools (linters, editors, IDEs, etc.) and alternative interpreters (like mypy) to actually use the type hints. Looking forward to the distant future, if CPython gains its own built-in type checker, it will probably come with a runtime switch to enable or disable such type checking. But that's possible regardless of whether we use decorators, annotations, or both. > Also, Types and Programming Languages by Dr. Pierce has been mentioned > at least twice as well; would it be useful to ask him to join in the > discussion? Does he know anything about Python? Will he care? There are hundreds of programming languages, unless he has a particular interest in Python I can't see why he would care about this discussion. But if you are a colleague or friend of his, by all means invite him to join up, this is a public forum. -- Steven From steve at pearwood.info Sun Aug 24 15:09:08 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Aug 2014 23:09:08 +1000 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: <20140823174358.GA25957@ando> Message-ID: <20140824130908.GH25957@ando> On Sun, Aug 24, 2014 at 02:35:57PM +0200, Peter Otten wrote: > David Mertz wrote: > > > def reduce_with_attractor(func, it, start=None, end_if=None): > > it = iter(it) > > start = start if start!=None else it.__next__() > > return list(takewhile(lambda x: x!=end_if, > > accumulate(chain([start],it), func)))[-1] > > Wouldn't it be better to break this into a function that limits a sequence > and to combine that with the original reduce()? In general, if x is an attractor, then we want to stop and return x if either of these two scenarios occur: - we come across x in the input; - or the sequence of intermediate values reaches x. You can do the first by filtering the input stream, but not the second. Here's a concrete example, sticking to product() where 0 is an attractor: # best viewed with a fixed-width font product([1, 2, 3, 9, 0, 8, 7, 4]) .....................^ stop here But multiplication can underflow to zero too, and once it does, we likewise want to stop: product([1e-70, 2e-71, 6e-69, 4e-68, 1e-48, 2e-71, 1e-69, 3e-70]) .....................................^ stop here Notice that 1e-48 is not only non-zero, but it's considerably bigger than the other numbers in the sequence. (About 100000000000000000000 times bigger, give or take a factor of 10.) Yet it's enough to cause the product to underflow to zero, after which the product will never shift away from zero. (Ignoring NANs and INFs.) I'm still not convinced this belongs in the standard library, but it's a nice functional, er, function to add to your private library or as a third-party module. -- Steven From cfkaran2 at gmail.com Sun Aug 24 15:32:38 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 24 Aug 2014 09:32:38 -0400 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> Message-ID: On Aug 24, 2014, at 7:39 AM, Ed Kellett wrote: > I have a few questions: > > - How often do multiple kinds of annotation end up on the same function? In this thread, we've already talked about type checkers and documentation generators, both of which can use the __annotations__ dictionary legitimately. Now imagine that you are an end user that has installed a documentation generator and a static type analyzer. If both tools were to use the __annotations__ dictionary, then right now you could choose one or the other tool, but not both. However, if the standard I'm proposing was adopted, then each tool would choose its own UUID as its key, which would mean they could share entries in the annotations dictionary. > - Why UUIDs (rather than for e.g. PyPI packages, or some other namespace)? Multiple reasons: - Using PyPI means that every programmer that is trying to follow the standard (including anyone who is just learning python) will create some name while practicing. That name will need to be pushed up to PyPI to ensure uniqueness. Since most of these names are only for learning, PyPI will immediately get flooded with a bunch of project names that are probably going to be abandoned almost immediately (I'm thinking of beginning programming classes especially). This would significantly degrade the utility of PyPI, which I want to avoid. All similar centrally-managed systems will suffer from the same problem. UUIDs don't have this problem; create and abandon them at will. - All other namespace systems will either suffer from the possibility of collisions, require a centrally managed repository of names, or will eventually reinvent UUIDs. We already have UUIDs, lets skip the first couple of headaches and just solve the problem. - Centrally managed systems have a much higher barrier to entry than simple UUIDs. Getting a new UUID to experiment with is trivial; "import uuid; uuid.uuid4()" is our complete program, requires no management on the part of PyPI (or any other third party), doesn't require internet access, etc. - UUIDs have no built-in human significance; it is VERY unlikely that multiple projects will accidentally choose the same UUID. E.g., it is likely that programmers developing different type checkers would choose 'type checker' as a key, and each project will have incompatible meanings/values for the 'type checker' key. This doesn't sound too bad until you start pulling in multiple frameworks from different sources, each of which uses a different, mutually incompatible type checker system. At that point, running any type checker will cause a crash as the type checker tries to read information from the frameworks you've just pulled in. - Google for a UUID. Any UUID. If you've just generated the UUID, you are unlikely to get one come up. Now google for 'UUID('f9bbc165-d904-4452-b858-fc5c9f104c87')'. I've just added it to the README for my annotizer project, and I expect googlebot to pick it up in the next few days. At that point, the only two places you should find mention of that UUID is on github, and in this thread. If you google for 'type checker', etc., how many hits do you get? How many of them relate to mypy, or even this thread? Once people are used to the standard, they'll know that to find out information about what project is associated with a given UUID they just need to google for it. This is a big win. Actually, do this as an experiment: don't look at the URL below, instead, wait a few days and google for the UUID above. See what comes up. - Making a standard for __annotations__ at this point isn't easy; we need a simple way of deciding if someone is complying the the standard. This is pretty easy if we adopt some UUID as a required key as I mentioned earlier. > - What is the point? A decorator could process the annotations and put > some information in func._projectname__something instead of doing the > UUID dance Again, this isn't for consumption within a project, it is for users across projects. What if I want to use Sphinx (http://sphinx-doc.org/) and mypy (http://www.mypy-lang.org/) at the same time in my project? What happens in the following code? """ @sphinx_decorator(a, "Some documentation about a") @mypy_decorator(a, int)) def foo(a): pass """ Is it the same as: """ @mypy_decorator(a, int)) @sphinx_decorator(a, "Some documentation about a") def foo(a): pass """ Right now, as I understand it, the last applied decorator would win, which means 'func._projectname__something' would be set to either sphinx or mypy. That means that order matters for completely orthogonal concepts. This is bad. UUIDs solve this, and all the earlier problems. > - What would pydoc print for the function signature? As I mentioned earlier, certain UUIDs might become de facto or de jure standards. In this case, projects that have common goals could settle on a common standard and publish a common UUID. Pydoc would know about these UUIDs (they would be published), and would know what to do for them. For UUIDs it doesn't understand, it could raise a warning, or simply ignore them. Before you take my comments above as proof the we don't need UUIDs, consider the fact that we are currently discussing type systems, and our thoughts may change in the future. I don't mean that there will be successive standards, I mean that there may be competing standards, at least until we really know what the best one will be. This is a case where creating and abandoning UUIDs will be trivial, but where using 'type checker' is going to lead to confusion. > On 24 August 2014 03:18, Cem Karan wrote: >> >> On Aug 20, 2014, at 11:08 AM, Paul Moore wrote: >> >>> Sigh. I go away for a week and come back to a mega-thread I can never >>> hope to catch up on :-) >>> >>> TL; DR; - Although mypy looks interesting, I think it's too soon to >>> close the door on all other uses of annotations. Let's find a solution >>> that allows exploration of alternative uses for a while longer. >>> >>> OK, can I just make some points regarding the static typing thread. >>> First of all, I have no issue with the idea of static typing, and in >>> fact I look forward to seeing what benefits it might have (if nothing >>> else, the pointer to mypy, which I'd never heard of before, is >>> appreciated). It won't be something I use soon (see below) but that's >>> fine. >>> >>> But Guido seems to be saying (on a number of occasions) that nobody is >>> really using annotations, so he wants to focus on the static typing >>> use case alone. I think this is a mistake. First of all, I see no >>> reason why functions using typing annotations could not be introduced >>> with a decorator[1]. So why insist that this is the *only* use of >>> annotations, when it's pretty easy to allow others to co-exist? >>> >>> Also, the "nobody is using annotations" argument? Personally, I know >>> of a few other uses: >>> >>> 1. Argument parsers - at least 3 have been mentioned in the thread. >>> 2. Structure unpacking - I think there is a library that uses >>> annotations for this, although I may be wrong. >>> 3. FFI bindings. I know I've seen this discussed, although I can't >>> find a reference just now. >>> >>> There are probably other ideas around as well (GUI bindings, >>> documentation generation, ...) None are particularly mature, and most >>> are just at the "ideas" stage, but typically the ideas I have seen are >>> the sort of thing you'd write a library for, and Python 3 only >>> libraries *really* aren't common yet. >>> >>> The problem for people wanting to experiment with annotations, is that >>> they need to be writing Python 3 only code. While Python 3 adoption is >>> growing rapidly, I suspect that large applications are typically still >>> focused on either going through, or tidying up after, a 2-3 migration. >>> And new projects, while they may be developed from the ground up using >>> Python 3, will typically be using programmers skilled in Python 2, to >>> whom Python 3 features are not yet an "instinctive" part of the >>> toolset. Apart from large standalone applications, there are smaller >>> scripts (which are typically going to be too small to need >>> programming-in-the-large features like annotations) and libraries >>> (which really aren't yet in a position to drop Python 2.x totally, >>> unless they have a fairly small user base). >>> >>> So I don't see it as compelling that usage of annotations in the wild >>> is not yet extensive. >>> >>> Rather than close the door on alternative uses of annotations, can I suggest: >>> >>> 1. By all means bless mypy syntax as the standard static typing >>> notation - this seems like a good thing. >>> 2. Clarify that static typing annotations should be introduced with a >>> decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy >>> declaration in the stdlib, and have a registration protocols for tools >>> to hook their own implementation into it.[2] >>> 3. Leave the door open for other uses of decorators, at least until >>> some of the more major libraries drop Python 2.x support completely >>> (and hence can afford to have a dependency on a Python 3 only module >>> that uses annotations). See at that stage if annotations take off. >>> 4. If we get to a point where even libraries that *could* use >>> annotations don't, then revisit the idea of restricting usage to just >>> type information. >>> >>> Paul >>> >>> [1] Also, a decorator could allow a Python 2 compatible form by using >>> decorator arguments as an alternative to annotations. >>> [2] I've yet to see a clear explanation of how "a tool using type >>> annotations" like an linter, editor, IDE or Python compiler would use >>> them in such a way that precludes decoration as a means of signalling >>> the annotation semantics. >> >> A long while back I proposed a mechanism for sharing __annotations__ between multiple, non-cooperating projects. The basic idea is that each annotation becomes a dictionary. Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary. The value is up to the project. >> >> The advantage to this is manifold: >> >> - Annotations can still have multiple uses by different groups without stepping on each other's toes. >> - If someone wants to make a standard, all they have to do is publish the UUID associated with their standard. For example, we might choose UUID('2cca6238-9fca-4053-aa3d-db9050e6b26b') as the official type information UUID. All projects that want to develop linters, documentation generators, etc., will use that UUID for all annotations, and the PEP will require it. >> - de facto standards can become de jure standards by blessing a particular UUID. >> - Guessing if this method is being used is relatively easy; if its a dictionary, and if every key is a UUID, it probably follows this standard. We can tighten it further by requiring some key-value pair be in every dictionary (i.e., {UUID('1ad60d50-8237-4b98-b2b1-69fd08ed575c'):"PEPXXXX"} is always in the dictionary). This makes it fairly simple to add without stomping on what people are already doing. >> - Finding the standard on the web should also be easy; while you might not find the PEP instantly, you'll probably zoom into it fairly fast. >> >> Disadvantages: >> >> - Typing out UUIDs is PAINFUL. I highly recommend using decorators instead. >> - Reading the __annotations__ dictionary will be difficult. pprint() should make this easier. >> >> I have working proof-of-concept code at https://github.com/oranguman/annotizer that defines a decorator class that handles the UUID for you. It needs to be extended to parse out information, but it handles the 'other use cases' problem fairly well. >> >> Thanks, >> Cem Karan Thanks, Cem Karan From ncoghlan at gmail.com Sun Aug 24 15:57:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Aug 2014 23:57:20 +1000 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> Message-ID: On 24 August 2014 23:32, Cem Karan wrote: > > - Using PyPI means that every programmer that is trying to follow the standard (including anyone who is just learning python) will create some name while practicing. That name will need to be pushed up to PyPI to ensure uniqueness. Since most of these names are only for learning, PyPI will immediately get flooded with a bunch of project names that are probably going to be abandoned almost immediately (I'm thinking of beginning programming classes especially). This would significantly degrade the utility of PyPI, which I want to avoid. All similar centrally-managed systems will suffer from the same problem. UUIDs don't have this problem; create and abandon them at will. If someone is just experimenting locally, it doesn't matter if they pick a conflicting name. If they're distributing, they're going to need to register a name on PyPI anyway. The fact that potential conflicts only matter once people start to consider distribution is crystal clear for distribution metadata, which is why PyPI names are the proposed namespacing mechanism for avoiding naming conflicts for extensions to metadata 2.0 (see the PEP 426 draft for details). (Note I don't really like this idea in general - if it's not for humans, then it doesn't need to be in the annotations dict, it can go in a separate file. But the UUIDs for namespacing idea fails badly on the "Readability counts" front) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Sun Aug 24 16:03:14 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 25 Aug 2014 00:03:14 +1000 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> Message-ID: <20140824140313.GI25957@ando> On Sun, Aug 24, 2014 at 09:32:38AM -0400, Cem Karan wrote: > > On Aug 24, 2014, at 7:39 AM, Ed Kellett wrote: > > > I have a few questions: > > > > - How often do multiple kinds of annotation end up on the same function? > > In this thread, we've already talked about type checkers and > documentation generators, both of which can use the __annotations__ > dictionary legitimately. Now imagine that you are an end user that > has installed a documentation generator and a static type analyzer. > If both tools were to use the __annotations__ dictionary, then right > now you could choose one or the other tool, but not both. Not necessarily. It depends on what the tools expect to find in the annotations. If type hints become the standard for annotations, which is Guido's proposal, then both tools will expect to find type hints, and both can use them. As for the cases where the tool or library wants to use annotations for something else, I'm quite happy to say that, in those cases, you can't use both. It's like docstrings. Docstrings are for documentation, including the interactive help system. If a library wants to use __doc__ for something else, it can (within the limits that it can only be a string, or None). But a user of that library cannot expect to *both* use the function __doc__ as a docstring *and* for this other purpose at the same time. They have to pick one. > However, if > the standard I'm proposing was adopted, then each tool would choose > its own UUID as its key, which would mean they could share entries in > the annotations dictionary. I don't think Guido wants to encourage multiple, incompatible uses for function annotations. It is going to be tricky enough to persuade him to merely allow other uses. My feeling is that this is a case of YAGNI. I don't think there are going to be so many competing uses for function annotations in common use that this will be necessary. -- Steven From edk141 at gmail.com Sun Aug 24 16:05:19 2014 From: edk141 at gmail.com (Ed Kellett) Date: Sun, 24 Aug 2014 15:05:19 +0100 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> Message-ID: > In this thread, we've already talked about type checkers and documentation generators, both of which can use the __annotations__ dictionary legitimately. They shouldn't be putting things in it, though, they should be extracting data from it when they're told to by means of a decorator. (I know the original proposal didn't mention using a decorator for standard type information annotations and advocated deprecating other annotations instead, but I think that was wrong and judging by some of the other posts in the thread it seems I'm not the only one). You pick the thing you're going to use annotations for and decorate your function with its "process this function by means of its annotations" decorator; there's no potential for conflict between tools since each function can have only one set of annotations anyway. > Right now, as I understand it, the last applied decorator would win, which means 'func._projectname__something' would be set to either sphinx or mypy. That means that order matters for completely orthogonal concepts. This is bad. UUIDs solve this, and all the earlier problems. You've missed my point (that's not your fault; I could have expressed it better): the key '_projectname__something' won't be used, the keys _mypy__types and _sphinx__doc (or similar) would be. > >> - What would pydoc print for the function signature? > > As I mentioned earlier, certain UUIDs might become de facto or de jure standards. In this case, projects that have common goals could settle on a common standard and publish a common UUID. Pydoc would know about these UUIDs (they would be published), and would know what to do for them. For UUIDs it doesn't understand, it could raise a warning, or simply ignore them. > > Before you take my comments above as proof the we don't need UUIDs, consider the fact that we are currently discussing type systems, and our thoughts may change in the future. I don't mean that there will be successive standards, I mean that there may be competing standards, at least until we really know what the best one will be. This is a case where creating and abandoning UUIDs will be trivial, but where using 'type checker' is going to lead to confusion. I don't think your comments prove that we don't need UUIDs, I just don't think they prove that we do. Decorators solve this problem adequately; the only issue that will arise is if annotations without a decorator are supposed to have some standard semantics. As more or less a tangential point, I think most of the advantages you've listed of UUIDs are counteracted by how unintuitive, unreadable and forgettable they are. From cfkaran2 at gmail.com Sun Aug 24 16:11:30 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 24 Aug 2014 10:11:30 -0400 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: <20140824124139.GG25957@ando> References: <53F25EE8.8000900@stoneleaf.us> <846C4603-A7EF-4394-8CE9-4B8C614E1E13@gmail.com> <20140824124139.GG25957@ando> Message-ID: <7C2692DB-36FE-43E8-B445-A87E7146E739@gmail.com> On Aug 24, 2014, at 8:41 AM, Steven D'Aprano wrote: > On Sat, Aug 23, 2014 at 08:37:34PM -0400, Cem Karan wrote: > >> Going slightly sideways on this -- I think this is why we should use >> decorators instead of annotations; as has already been mentioned twice >> on the list, >> http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a >> good job pointing out that static and dynamic typing systems are >> separate but overlapping concepts. It also points out that both have >> their place. If we define decorators that can be turned on or off >> easily (command-line option? Environment variable? other?), then the >> end user can choose if if her or she is going to do static, dynamic, >> both, or none. This could be useful when trying to track down that >> annoying bug in long-running production code. > > This applies equally to annotations, and in fact that's exactly what > Mypy already does. You can run Mypy to do static type checking, or not > run it, and the annotations will be ignored. > > E.g. given a module program.py, you can: > > # Type check and then run the program: > mypy program.py > > # Just run it, with no extra type checks: > python3 program.py > > > Guido's proposal is *not* to add static types to the CPython > interpreter, at least not yet. It is just to standardise on annotations > for type hinting, agree on a syntax for those type hints, and then allow > third-part tools (linters, editors, IDEs, etc.) and alternative > interpreters (like mypy) to actually use the type hints. OK, I think I see what you're saying. Since the annotations will always be available, the static/dynamic nature doesn't matter, since we just choose the flags/interpreter/whatever, and it will check the annotations, correct? If so, then you're right, and we don't need decorators for this. > Looking forward to the distant future, if CPython gains its own built-in > type checker, it will probably come with a runtime switch to enable or > disable such type checking. But that's possible regardless of whether we > use decorators, annotations, or both. > >> Also, Types and Programming Languages by Dr. Pierce has been mentioned >> at least twice as well; would it be useful to ask him to join in the >> discussion? > > Does he know anything about Python? Will he care? There are hundreds of > programming languages, unless he has a particular interest in Python I > can't see why he would care about this discussion. But if you are a > colleague or friend of his, by all means invite him to join up, this is > a public forum. I just invited him. Thanks, Cem Karan From antoine at python.org Sun Aug 24 16:28:00 2014 From: antoine at python.org (Antoine Pitrou) Date: Sun, 24 Aug 2014 10:28:00 -0400 Subject: [Python-ideas] [OT] Re: Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: Le 24/08/2014 00:24, Nick Coghlan a ?crit : > On 24 August 2014 01:30, Warren Weckesser wrote: >> I'd like to add an additional optional argument to functools.reduce. >> The argument is the "nullifier" of the reducing operation. It is a value >> such that function(nullifier, anything) returns nullifier. For example, if >> function(x, y) computes x*y, the nullifier is 0. If function(x, y) is >> the intersection of the sets x and y, the nullifier is the empty set. > > When it comes to judging the usefulness of functional programming > features these days, my first question is generally going to be "Does > PyToolz offer this?" > > Grumblings about the name aside, I suppose it's for functional whizkidz and cowboyz? Regards Antoine. From warren.weckesser at gmail.com Sun Aug 24 16:45:11 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 24 Aug 2014 10:45:11 -0400 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On Sat, Aug 23, 2014 at 1:25 PM, David Mertz wrote: > This "nullifier" is mathematically called an "absorbing element", but > saying an "attractor" might be a little more general. I.e. think of a > local optimization problem, where multiple local min/max points might > occur. If you reach one, further iteration won't budge from that point, > even if it's not the "global absorbing element." > > I took the name "nullifier" from http://www.mayhematics.com/m/m2_operations.htm but a less "mathy" name would probably be better. For consistency, I'll stick with it, but you can think of it as a placeholder for a better name to be determined later. > However, any such argument--including the much more useful > sentinel/'stop_on' idea--significantly changes the semantics of reduce. In > particular, as is reduce() always consumes its iterator. Under these > changes, it may or may not consume the iterator, depending on what elements > occur. > > I don't agree that this change "significantly changes the semantics of reduce". The nullifier is optional. The short-circuit can only occur when the caller has specified a nullifier, so anyone using it will be aware that the iterable might not be consumed. Indeed, that's the *point* of using it. Assuming the nullifier given is truly a nullifier of the function, the result of the call to reduce should be the same (other than the amount by which the iterable has been consumed) whether or not the nullifier is given. (That was also Hernan's point.) > Given that one can easily write one's own three line wrapper > 'reduce_with_attractor()' for this special semantics which hasn't been > given a use case, > My interest in the enhancement is purely performance. Here's an example (call it a use case, if you like) that is obviously cooked up to maximize the benefit of short-circuiting. In the following ipython session, `reduce` is the builtin function (I'm using python 2.7 here), and `myreduce.reduce` is the python implementation with the nullifier argument: In [40]: a = range(100) In [41]: %timeit reduce(lambda x, y: x*y, a) 100000 loops, best of 3: 8.89 ?s per loop In [42]: %timeit myreduce.reduce(lambda x, y: x*y, a, nullifier=0) 1000000 loops, best of 3: 455 ns per loop > I can't see a point of including the argument in the stdlib. > > -1 on proposal. > > > I think the main objection (here and in other comments) is that the benefit (performance gain in certain cases) does not outweigh the cost (a more complicated API for reduce(), and more code and documentation to maintain in the standard library). That's a compelling argument! I think the enhancement would be useful, but I understand that it might not be useful enough to accept such a change, especially since the cost of *not* having it in the library to a user who wants such a feature is pretty low (i.e. it is easy to "roll your own"). Warren > > On Sat, Aug 23, 2014 at 8:30 AM, Warren Weckesser < > warren.weckesser at gmail.com> wrote: > >> I'd like to add an additional optional argument to functools.reduce. >> The argument is the "nullifier" of the reducing operation. It is a value >> such that function(nullifier, anything) returns nullifier. For example, >> if >> function(x, y) computes x*y, the nullifier is 0. If function(x, y) is >> the intersection of the sets x and y, the nullifier is the empty set. >> >> The argument would allow reduce to "short circuit" its calculation. When >> reduce encounters the nullifier, it can return immediately. This can >> provide >> a significant improvement in performance in some cases. >> >> The change is simple. Here, for example, is the "rough equivalent" for >> functools.reduce from the docs: >> >> def reduce(function, iterable, initializer=None): >> it = iter(iterable) >> if initializer is None: >> try: >> initializer = next(it) >> except StopIteration: >> raise TypeError('reduce() of empty sequence with no >> initial value') >> accum_value = initializer >> for x in it: >> accum_value = function(accum_value, x) >> return accum_value >> >> Here's how it looks with the optional nullifier argument; the only >> change is the new argument and an 'if' statement in the 'for' loop. >> >> def reduce(function, iterable, initializer=None, nullifier=None): >> it = iter(iterable) >> if initializer is None: >> try: >> initializer = next(it) >> except StopIteration: >> raise TypeError('reduce() of empty sequence with no >> initial value') >> accum_value = initializer >> for x in it: >> if nullifier is not None and accum_value == nullifier: >> break >> accum_value = function(accum_value, x) >> return accum_value >> >> (It might be better to use a distinct singleton for the default >> value of nullifier, to allow None to be a valid nullifier.) >> >> The actual implementation is in the extension module _functoolsmodule.c. >> It looks like the changes to the C code should be straightforward. >> >> >> Warren >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Aug 24 16:47:02 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 24 Aug 2014 10:47:02 -0400 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On Sun, Aug 24, 2014 at 12:24 AM, Nick Coghlan wrote: > On 24 August 2014 01:30, Warren Weckesser > wrote: > > I'd like to add an additional optional argument to functools.reduce. > > The argument is the "nullifier" of the reducing operation. It is a value > > such that function(nullifier, anything) returns nullifier. For example, > if > > function(x, y) computes x*y, the nullifier is 0. If function(x, y) is > > the intersection of the sets x and y, the nullifier is the empty set. > > When it comes to judging the usefulness of functional programming > features these days, my first question is generally going to be "Does > PyToolz offer this?" > I took a look, and I couldn't find it there. > > Grumblings about the name aside, it's still the solution I recommend > to folks that wish Python had more functional programming tools in the > standard library: http://toolz.readthedocs.org/en/latest/api.html > > There's even a Cython accelerated version available (Cytoolz). > > "pip install toolz" for the pure Python version, "pip install cytoolz" > for the accelerated one. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Aug 24 16:56:26 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 24 Aug 2014 10:56:26 -0400 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: <20140824130908.GH25957@ando> References: <20140823174358.GA25957@ando> <20140824130908.GH25957@ando> Message-ID: On Sun, Aug 24, 2014 at 9:09 AM, Steven D'Aprano wrote: > On Sun, Aug 24, 2014 at 02:35:57PM +0200, Peter Otten wrote: > > David Mertz wrote: > > > > > def reduce_with_attractor(func, it, start=None, end_if=None): > > > it = iter(it) > > > start = start if start!=None else it.__next__() > > > return list(takewhile(lambda x: x!=end_if, > > > accumulate(chain([start],it), func)))[-1] > > > > Wouldn't it be better to break this into a function that limits a > sequence > > and to combine that with the original reduce()? > > In general, if x is an attractor, then we want to stop and return x if > either of these two scenarios occur: > > - we come across x in the input; > > - or the sequence of intermediate values reaches x. > > You can do the first by filtering the input stream, but not the second. > > Here's a concrete example, sticking to product() where 0 is an > attractor: > > # best viewed with a fixed-width font > product([1, 2, 3, 9, 0, 8, 7, 4]) > .....................^ stop here > > > But multiplication can underflow to zero too, and once it does, > we likewise want to stop: > > product([1e-70, 2e-71, 6e-69, 4e-68, 1e-48, 2e-71, 1e-69, 3e-70]) > .....................................^ stop here > > > Notice that 1e-48 is not only non-zero, but it's considerably bigger > than the other numbers in the sequence. (About 100000000000000000000 > times bigger, give or take a factor of 10.) Yet it's enough to cause the > product to underflow to zero, after which the product will never > shift away from zero. (Ignoring NANs and INFs.) > > An example I gave earlier is the reduction of a collection of sets using set intersection. It is possible that (set1 & set2) gives the empty set, so the nullifier can occur during the reduction without being in the given iterable. E.g.: reduce(lambda x, y: x & y, [{1,2}, {3}, {4, 5}, {6}], nullifier={}) That should stop iterating after the first operation. > I'm still not convinced this belongs in the standard library, but it's a > nice functional, er, function to add to your private library or as a > third-party module. > > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sun Aug 24 17:27:35 2014 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 24 Aug 2014 16:27:35 +0100 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: <20140823174358.GA25957@ando> <20140824130908.GH25957@ando> Message-ID: <53FA0467.2050305@mrabarnett.plus.com> On 2014-08-24 15:56, Warren Weckesser wrote: [snip] > > An example I gave earlier is the reduction of a collection of sets > using set intersection. It is possible that (set1 & set2) gives the > empty set, so the nullifier can occur during the reduction without being > in the given iterable. E.g.: > > reduce(lambda x, y: x & y, [{1,2}, {3}, {4, 5}, {6}], nullifier={}) > > That should stop iterating after the first operation. > I think it would be more flexible if, instead of a 'nullifier' object, there were an 'until' or 'while_' predicate: reduce(lambda x, y: x & y, [{1,2}, {3}, {4, 5}, {6}], while_=bool) From mertz at gnosis.cx Sun Aug 24 19:27:13 2014 From: mertz at gnosis.cx (David Mertz) Date: Sun, 24 Aug 2014 10:27:13 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: <20140823174358.GA25957@ando> <20140824015325.GB25957@ando> Message-ID: So here's a version of my function that addresses Steven's points, and also adds what I think is more useful behavior. A couple points about my short--but more than 3-line function. * The notion of "attractor" here still follows the "we're all adults" philosophy. That is, the 'stop_on' and 'end_if' arguments may very well not actually specify genuine attractors, but simply "satisfaction" values. In the examples below these are completely unlike attractors, but this could be useful also for something like numeric approximation algorithms where some non-final reduction value is considered "good enough" anyway. * To be more like the reduce() builtin, I raise TypeError on an empty iterable with no 'start' value. * Although it's not as cute in using itertools as much, but simply a loop-with-test, I got rid of the "golf" artifact of potentially allocating a large, useless list. * Most interesting, I think, I separated a 'stop_on' collection of values from a 'end_if' predicate. This lets us generalize attractors, I believe. For a simple cyclic attractor, you could list several value in the 'stop_on' collection. But even for a strange attractor (or simply a complex one, or e.g. an attractor to a continuous set of values--think an attractor to a circular, real-valued orbit), the predicate could, in principle, capture the fact we reached the attractor. ################################################# % cat reduce_with_attractor.py #!/usr/bin/env python3 from itertools import * from operator import add def reduce_with_attractors(func, it, start=None, stop_on=(), end_if=None): it = iter(it) try: start = start if start!=None else next(it) except StopIteration: raise TypeError for x in accumulate(chain([start],it), func): if x in stop_on or (end_if and end_if(x)): break return x print(reduce_with_attractors(add, range(100), stop_on=(2, 6, 17))) print(reduce_with_attractors(add, range(100), stop_on=('foo','bar'))) print(reduce_with_attractors(add, range(100), end_if=lambda x: x>100)) ################################################# On Sat, Aug 23, 2014 at 7:19 PM, David Mertz wrote: > > On Sat, Aug 23, 2014 at 6:53 PM, Steven D'Aprano wrote: >> >> On Sat, Aug 23, 2014 at 11:43:42AM -0700, David Mertz wrote: >> >> > def reduce_with_attractor(func, it, start=None, end_if=None): >> > it = iter(it) >> > start = start if start!=None else it.__next__() >> > return list(takewhile(lambda x: x!=end_if, >> > accumulate(chain([start],it), func)))[-1] >> >> A couple of points: >> >> - Don't use it.__next__(), use the built-in next(it). > > > Yeah, good point. > >> >> - To match the behaviour of reduce, you need to catch the StopIteration >> and raise TypeError. > > > Oh, OK. Hadn't thought of that. > >> >> - Your implementation eagerly generates a potentially enormous list of >> intermediate results, which strikes me as somewhat unfortunate for a >> functional tool like reduce. In other words, for some inputs, this is >> going to perform like a dog, generating a HUGE list up front, then >> throwing it all away except for the final value. > > > I know. I realized this flaw right away. I was trying to be cute and fit it in my promised 3 lines. It would be better to put it in a loop to realize the successive values, of course--but would take an extra line or two. Maybe there's a way to squeeze it in one line with itertools rather than a regular loop though. > >> >> > This gives you the accumulation up-to-but-not-including the attractor. I >> > guess the OP wanted to return the attractor itself (although that seems >> > slightly less useful to me). >> >> Not really. His use-case seems to be to short-cut a lot of unnecessary >> calculations, e.g. suppose you write product() as reduce(operator.mul, >> iterable). In the event that the product reaches zero, you can[1] >> short-circuit the rest of the iterable and just return 0: >> >> product([1, 2, 3, 0] + [4]*1000000) >> >> ought to reduce 0, not 6, and the intent is for it to do so *quickly*, >> ignoring the 4s at the end of the list. > > > I guess that's true. Although I can certainly imagine being interested not only in the final attractor, but *that* it reached an attractor and ended early. Not sure how best to signal that. Actually, now that I think of it, it would be kinda nice to make the function 'reduce_with_attractorS()' instead, and allow specification of multiple attractors. > > I welcome your improved version of the code :-). Feel free to take a whole 10 lines to do it right. > >> >> [1] Actually you can't. 0 is no longer an attractor in the presence of >> INF or NAN. > > > I was sort of thinking of a "we're all adults here" attitude. That is, the "attractor" might not really be a genuine attractor, but we still trust the caller to say it is. I.e. my function would accept this call: > > reduce_with_attractor(operator.mul, range(1,1e6), end_if=6)) > > I'm making the claim that reaching '6' is a stopping point... which, well it is. No, it's not an actual attractor, but maybe a caller really does want to stop iterating if it gets to that value anyway. Hence 'end_if' is actually an accurate name. > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbfox at gmail.com Sun Aug 24 19:48:51 2014 From: jsbfox at gmail.com (Thomas Allen) Date: Sun, 24 Aug 2014 21:48:51 +0400 Subject: [Python-ideas] Negative limit for traceback.* Message-ID: I'd like to propose adding the ability to extract last n stack trace entries from a traceback using functions from traceback module. Simply put, passing limit=-n would make functions to return (or print or yield) last abs(n) entries. No-brainer implementation for extract_tb: extracted = list(_extract_tb_iter(tb, limit=(limit, None)[limit < 0])) if limit is not None and limit < 0: return extracted[limit:] return extracted The motivation: http://stackoverflow.com/q/25472412/2301450 From guido at python.org Sun Aug 24 19:52:44 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 24 Aug 2014 10:52:44 -0700 Subject: [Python-ideas] Negative limit for traceback.* In-Reply-To: References: Message-ID: I like this. It's kind of sad that when the limit causes truncation of the traceback it drops the most recent frames, which might give a hint as to what happens, and keeps the oldest frames, which are usually less interesting (I know my program started at main() :-). On Sun, Aug 24, 2014 at 10:48 AM, Thomas Allen wrote: > I'd like to propose adding the ability to extract last n stack trace > entries from a traceback using functions from traceback module. Simply > put, passing limit=-n would make functions to return (or print or > yield) last abs(n) entries. No-brainer implementation for extract_tb: > > extracted = list(_extract_tb_iter(tb, limit=(limit, None)[limit < 0])) > if limit is not None and limit < 0: > return extracted[limit:] > return extracted > > The motivation: http://stackoverflow.com/q/25472412/2301450 > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Aug 24 21:00:01 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 24 Aug 2014 12:00:01 -0700 Subject: [Python-ideas] Optional Static Typing -- the Python Way In-Reply-To: <20140824115954.GF25957@ando> References: <53F25EE8.8000900@stoneleaf.us> <20140824115954.GF25957@ando> Message-ID: <9214C846-2317-4DD1-82BF-195B91154038@yahoo.com> On Aug 24, 2014, at 4:59, Steven D'Aprano wrote: > On Thu, Aug 21, 2014 at 02:30:03PM +0000, Brett Cannon wrote: > >> P.S.: Off-topic for this email but something I don't think has been >> mentioned more than once, type hinting like we are proposing is exactly >> what Dart does and it is nice. Built-in API documentation of interfaces >> along with IDE support at the API makes all of this worth it. > > For anyone not familiar with Dart, I think it is worth reading the Dart > justification for static type hinting in a dynamic language: > > https://www.dartlang.org/articles/optional-types/ > > https://www.dartlang.org/articles/why-dart-types/ > > It's not quite the same as Guido's proposal, for example in Dart the > type hints have no runtime effect at all, but I think it demonstrates > that this is a proven approach and can work well with a dynamic language > like Python without compromising the dynamic nature of the language. Great reference. It's also worth looking at Boo (http://boo.codehaus.org), which was intentionally designed to read like Python, while also having static typing. So, it was designed in the opposite direction: given the CLR compile-time type system, modify Python syntax and semantics as little as possible to implement it. And it was also designed for completely different goals (mainly performance). But it does show how a type system pretty similar to the one Jukka and Guido are envisioning can fit into a language pretty similar to Python, which makes Boo code a useful example for "how readable could this be in real-life code?" questions. From abarnert at yahoo.com Sun Aug 24 22:47:46 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 24 Aug 2014 13:47:46 -0700 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: <20140823174358.GA25957@ando> <20140824015325.GB25957@ando> Message-ID: <1408913266.15289.YahooMailNeo@web181002.mail.ne1.yahoo.com> I think it would be both simpler and more useful to define accumulate_with_attractors, and then define reduce_with_attractors as a wrapper around that. Assuming you're doing a lot of this kind of stuff, you're either going to be using a library like more_itertools or pytoolz or your own custom library, so I'll assume you already have a "takewhileplus" that's like takewhile but also yields the first failing value, and an "ilast" that returns the last value in an iterable and raises a TypeError if empty (maybe with an optional default argument, but I'm not going to use it). _sentinel = object() def accumulate_with_attractors(iterable, func=operator.add, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?stop_on=(), end_if=None): ? ? yield from takewhileplus(lambda x: x not in stop_on and not (end_if and end_if(x)), ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?itertools.accumulate(iterable, func)) def reduce_with_attractors(func, iterable, start=_sentinel, ? ? ? ? ? ? ? ? ? ? ? ? ? ?stop_on=(), end_if=None): ? ? if start is not _sentinel: ? ? ? ? iterable = itertools.chain([start], iterable) ? ? return ilast(accumulate_with_attractors(iterable, func, stop_on, end_if)) You can easily optimize this in a few ways. You can use a takeuntilplus so the lambda doesn't have to negate its condition, you can use stop_on.__contains__ if end_if isn't passed, etc.?With those optimizations, using C implementations of takeuntilplus and ilast, this takes about 40% as long as your version. With no optimizations, using Python takewhileplus and last, it takes about 140% as long. But I think the added simplicity, the fewer edge cases to get wrong, and the fact that you get a usable accumulate_with_attractors out of it makes it a worthwhile tradeoff even at 40% slower. On Sunday, August 24, 2014 10:27 AM, David Mertz wrote: > > >So here's a version of my function that addresses Steven's points, and also adds what I think is more useful behavior. ?A couple points about my short--but more than 3-line function. > >* The notion of "attractor" here still follows the "we're all adults" philosophy. That is, the 'stop_on' and 'end_if' arguments may very well not actually specify genuine attractors, but simply "satisfaction" values. ?In the examples below these are completely unlike attractors, but this could be useful also for something like numeric approximation algorithms where some non-final reduction value is considered "good enough" anyway. > >* To be more like the reduce() builtin, I raise TypeError on an empty iterable with no 'start' value. > >* Although it's not as cute in using itertools as much, but simply a loop-with-test, I got rid of the "golf" artifact of potentially allocating a large, useless list. > >* Most interesting, I think, I separated a 'stop_on' collection of values from a 'end_if' predicate. ?This lets us generalize attractors, I believe. ?For a simple cyclic attractor, you could list several value in the 'stop_on' collection. ?But even for a strange attractor (or simply a complex one, or e.g. an attractor to a continuous set of values--think an attractor to a circular, real-valued orbit), the predicate could, in principle, capture the fact we reached the attractor. > >################################################# >% cat reduce_with_attractor.py >#!/usr/bin/env python3 >from itertools import * >from operator import add > >def reduce_with_attractors(func, it, start=None, stop_on=(), end_if=None): >? ? it = iter(it) >? ? try: >? ? ? ? start = start if start!=None else next(it) >? ? except StopIteration: >? ? ? ? raise TypeError >? ? for x in accumulate(chain([start],it), func): >? ? ? ? if x in stop_on or (end_if and end_if(x)): >? ? ? ? ? ? break >? ? return x > >print(reduce_with_attractors(add, range(100), stop_on=(2, 6, 17))) >print(reduce_with_attractors(add, range(100), stop_on=('foo','bar'))) >print(reduce_with_attractors(add, range(100), end_if=lambda x: x>100)) >################################################# > > >On Sat, Aug 23, 2014 at 7:19 PM, David Mertz wrote: >> >> On Sat, Aug 23, 2014 at 6:53 PM, Steven D'Aprano wrote: >>> >>> On Sat, Aug 23, 2014 at 11:43:42AM -0700, David Mertz wrote: >>> >>> > ? def reduce_with_attractor(func, it, start=None, end_if=None): >>> > ? ? ? it = iter(it) >>> > ? ? ? start = start if start!=None else it.__next__() >>> > ? ? ? return list(takewhile(lambda x: x!=end_if, >>> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? accumulate(chain([start],it), func)))[-1] >>> >>> A couple of points: >>> >>> - Don't use it.__next__(), use the built-in next(it). >> >> >> Yeah, good point. >> ? >>> >>> - To match the behaviour of reduce, you need to catch the StopIteration >>> and raise TypeError. >> >> >> Oh, OK. ?Hadn't thought of that. >> ? >>> >>> - Your implementation eagerly generates a potentially enormous list of >>> intermediate results, which strikes me as somewhat unfortunate for a >>> functional tool like reduce. In other words, for some inputs, this is >>> going to perform like a dog, generating a HUGE list up front, then >>> throwing it all away except for the final value. >> >> >> I know. ?I realized this flaw right away. ?I was trying to be cute and fit it in my promised 3 lines. ?It would be better to put it in a loop to realize the successive values, of course--but would take an extra line or two. ?Maybe there's a way to squeeze it in one line with itertools rather than a regular loop though. >> ? >>> >>> > This gives you the accumulation up-to-but-not-including the attractor. ?I >>> > guess the OP wanted to return the attractor itself (although that seems >>> > slightly less useful to me). >>> >>> Not really. His use-case seems to be to short-cut a lot of unnecessary >>> calculations, e.g. suppose you write product() as reduce(operator.mul, >>> iterable). In the event that the product reaches zero, you can[1] >>> short-circuit the rest of the iterable and just return 0: >>> >>> product([1, 2, 3, 0] + [4]*1000000) >>> >>> ought to reduce 0, not 6, and the intent is for it to do so *quickly*, >>> ignoring the 4s at the end of the list. >> >> >> I guess that's true. ?Although I can certainly imagine being interested not only in the final attractor, but *that* it reached an attractor and ended early. ?Not sure how best to signal that. ?Actually, now that I think of it, it would be kinda nice to make the function 'reduce_with_attractorS()' instead, and allow specification of multiple attractors. >> >> I welcome your improved version of the code :-). ?Feel free to take a whole 10 lines to do it right. >> ? >>> >>> [1] Actually you can't. 0 is no longer an attractor in the presence of >>> INF or NAN. >> >> >> I was sort of thinking of a "we're all adults here" attitude. ?That is, the "attractor" might not really be a genuine attractor, but we still trust the caller to say it is. ?I.e. my function would accept this call: >> >> ? ? reduce_with_attractor(operator.mul, range(1,1e6), end_if=6)) >> >> I'm making the claim that reaching '6' is a stopping point... which, well it is. ?No, it's not an actual attractor, but maybe a caller really does want to stop iterating if it gets to that value anyway. ?Hence 'end_if' is actually an accurate name. >> >> -- >> Keeping medicines from the bloodstreams of the sick; food >> from the bellies of the hungry; books from the hands of the >> uneducated; technology from the underdeveloped; and putting >> advocates of freedom in prisons. ?Intellectual property is >> to the 21st century what the slave trade was to the 16th. > > > > > >-- >Keeping medicines from the bloodstreams of the sick; food >from the bellies of the hungry; books from the hands of the >uneducated; technology from the underdeveloped; and putting >advocates of freedom in prisons. ?Intellectual property is >to the 21st century what the slave trade was to the 16th. > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ > > From ram.rachum at gmail.com Sun Aug 24 23:41:01 2014 From: ram.rachum at gmail.com (Ram Rachum) Date: Sun, 24 Aug 2014 14:41:01 -0700 (PDT) Subject: [Python-ideas] Add an introspection API to Executor Message-ID: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Sometimes I want to take a live executor, like a `ThreadPoolExecutor`, and check up on it. I want to know how many threads there are, how many are handling tasks and which tasks, how many are free, and which tasks are in the queue. I asked on Stack Overflow: http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-futures-threadpoolexecutor There's an answer there, but it uses private variables and it's not part of the API. I suggest it become a part of the API. There should be an API for checking on what the executor is currently doing and answering all the questions I raised above. Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Aug 25 00:17:30 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 25 Aug 2014 10:17:30 +1200 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> Message-ID: <53FA647A.2030108@canterbury.ac.nz> > On 24 August 2014 03:18, Cem Karan wrote: > >> Each project (and 'project' is a very >> loosely defined concept here) chooses a UUID that it uses as key into the >> dictionary. Please, not UUIDs. As something meant to be used by humans, they're horrible. -- Greg From cfkaran2 at gmail.com Mon Aug 25 01:17:46 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 24 Aug 2014 19:17:46 -0400 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: <53FA647A.2030108@canterbury.ac.nz> References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> <53FA647A.2030108@canterbury.ac.nz> Message-ID: On Aug 24, 2014, at 6:17 PM, Greg Ewing wrote: >> On 24 August 2014 03:18, Cem Karan wrote: >>> Each project (and 'project' is a very >>> loosely defined concept here) chooses a UUID that it uses as key into the >>> dictionary. > > Please, not UUIDs. As something meant to be used by > humans, they're horrible. > > -- > Greg I understand your concerns about them, I really, really do. But can you think of a better way? All my prior points still hold. In my mind, UUIDs are the least bad alternative. We have to educate people as to what the annotations dictionary standard is, and then encourage libraries to use the standard and to hide the UUIDs from end users. My code does this already; if someone wants me to extend it in some way, put in a request, and I will. If the PSF wants it in the standard library, it's theirs! Please, download it and play with it, it's short to the point of almost being trivial (https://github.com/oranguman/annotizer). Don't let the fact that the bare __annotations__ dict is going to look ugly; if we try to get a really powerful typing system in there, people will probably not want to write the annotations by hand anyways for the simple reason that it will be too hard to read. I mean, which would you rather read? """ @doc(a, "docs about a") @type(a, set(range(10))) def foo(a): pass def bar(a: {__docs_project_ID: "docs about a", __types_project_ID: set(range(10))}): pass """ Even if __project_ID was something human readable, that line would be PAINFUL for a human to parse. UUIDs won't make it much worse, and they do avoid a lot of the other headaches. Thanks, Cem Karan From rosuav at gmail.com Mon Aug 25 01:22:51 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 25 Aug 2014 09:22:51 +1000 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> <53FA647A.2030108@canterbury.ac.nz> Message-ID: On Mon, Aug 25, 2014 at 9:17 AM, Cem Karan wrote: > @doc(a, "docs about a") > @type(a, set(range(10))) > def foo(a): > pass If this is your proposal, then how do UUIDs even help? You already effectively piggy-back off the standard module import system for uniqueness. Surely that's going to do everything that UUIDs would? ChrisA From cfkaran2 at gmail.com Mon Aug 25 01:28:36 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 24 Aug 2014 19:28:36 -0400 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> <53FA647A.2030108@canterbury.ac.nz> Message-ID: On Aug 24, 2014, at 7:22 PM, Chris Angelico wrote: > On Mon, Aug 25, 2014 at 9:17 AM, Cem Karan wrote: >> @doc(a, "docs about a") >> @type(a, set(range(10))) >> def foo(a): >> pass > > If this is your proposal, then how do UUIDs even help? You already > effectively piggy-back off the standard module import system for > uniqueness. Surely that's going to do everything that UUIDs would? I'm not sure if I follow what you're saying. Do you mean 'if everyone is using my annotizer library, why bother with UUIDs, because each decorator will have a different meaning enforced by the import system'? Thanks, Cem Karan From rosuav at gmail.com Mon Aug 25 01:57:37 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 25 Aug 2014 09:57:37 +1000 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> <53FA647A.2030108@canterbury.ac.nz> Message-ID: On Mon, Aug 25, 2014 at 9:28 AM, Cem Karan wrote: > On Aug 24, 2014, at 7:22 PM, Chris Angelico wrote: > >> On Mon, Aug 25, 2014 at 9:17 AM, Cem Karan wrote: >>> @doc(a, "docs about a") >>> @type(a, set(range(10))) >>> def foo(a): >>> pass >> >> If this is your proposal, then how do UUIDs even help? You already >> effectively piggy-back off the standard module import system for >> uniqueness. Surely that's going to do everything that UUIDs would? > > I'm not sure if I follow what you're saying. Do you mean 'if everyone is using my annotizer library, why bother with UUIDs, because each decorator will have a different meaning enforced by the import system'? doc and type can't conflict, because they're separate names. (Though "type" would need to be renamed to avoid a conflict, unless you're suggesting that the type type should grow this functionality.) If they use their own selves as the keys (nobody ever said the keys have to be strings), there cannot be a conflict. The UUID scheme doesn't add anything that can't be done with existing objects with no hassle. ChrisA From rosuav at gmail.com Mon Aug 25 02:05:27 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 25 Aug 2014 10:05:27 +1000 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On Sun, Aug 24, 2014 at 2:27 AM, Warren Weckesser wrote: > That wouldn't help in an example such as > > reduce(lambda x, y: x & y, [{1,2}, {3, 4}, {5, 6}], nullifier={}) > > The nullifier is the empty set, but the empty set does not occur in the > iterable. Caution: Your nullifier is an empty dict, not an empty set, if you spell it that way. I think this is a cute concept, but not something that is going to be needed all that often. Most Python programs don't even use reduce() in its default form, much less need a specific optimization. ChrisA From ethan at stoneleaf.us Mon Aug 25 01:59:02 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 24 Aug 2014 16:59:02 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> Message-ID: <53FA7C46.1000407@stoneleaf.us> On 08/23/2014 04:00 PM, Antoine Pitrou wrote: > > Le 23/08/2014 13:25, Steven D'Aprano a ?crit : >>> >>> It's not a new use. A type class is a class, and calling it is just >>> instantiating that class. There's nothing new here. If you think that's >>> a bit "meta", it's no different than e.g. higher-order functions. >> >> There's no instantiation during *static* analysis, because the code >> hasn't run yet. > > In your idea of "static analysis", it hasn't. Because you think it should involve some kind of separate syntax analysis > tool that has nothing to do with regular Python. But Python is powerful enough to let you do that using normal > introspection of modules. > > And it's *exactly* how we are exposing function annotations (and also docstrings, etc.): using runtime-accessible > introspection information which is gathered by importing modules and therefore actually *executing* toplevel module > code. Not merely compiling it. Is this workable? If I have a nested class in a function: def spam(): class fribble: def __init__(self, x:int): blahblah I would expect MyPy to be able to retrieve and use the annotations from fribble without actually calling spam. If static checking is only available at the top level I think a lot of power/usefulness is gone. -- ~Ethan~ From abarnert at yahoo.com Mon Aug 25 02:21:30 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 24 Aug 2014 17:21:30 -0700 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> <53FA647A.2030108@canterbury.ac.nz> Message-ID: <1472A8DB-2427-4ABA-857A-41DBB5707900@yahoo.com> On Aug 24, 2014, at 16:17, Cem Karan wrote: > > On Aug 24, 2014, at 6:17 PM, Greg Ewing wrote: > >>> On 24 August 2014 03:18, Cem Karan wrote: >>>> Each project (and 'project' is a very >>>> loosely defined concept here) chooses a UUID that it uses as key into the >>>> dictionary. >> >> Please, not UUIDs. As something meant to be used by >> humans, they're horrible. >> >> -- >> Greg > > I understand your concerns about them, I really, really do. But can you think of a better way? All my prior points still hold. In my mind, UUIDs are the least bad alternative. We have to educate people as to what the annotations dictionary standard is, and then encourage libraries to use the standard and to hide the UUIDs from end users. My code does this already; if someone wants me to extend it in some way, put in a request, and I will. If the PSF wants it in the standard library, it's theirs! Please, download it and play with it, it's short to the point of almost being trivial (https://github.com/oranguman/annotizer). > > Don't let the fact that the bare __annotations__ dict is going to look ugly; if we try to get a really powerful typing system in there, people will probably not want to write the annotations by hand anyways for the simple reason that it will be too hard to read. I mean, which would you rather read? > > """ > @doc(a, "docs about a") > @type(a, set(range(10))) > def foo(a): > pass > > def bar(a: {__docs_project_ID: "docs about a", __types_project_ID: set(range(10))}): > pass > """ Or: @doc(a, "docs about a") def foo(a: set[10]): pass Why does doc have to set an annotation in the first place? Annotations are just syntactic sugar for sticking stuff in a dict attached to the function as an attribute; when that syntactic sugar doesn't make your code more readable, there's no benefit at all to annotations. You could just make the @doc decorator store and find its stuff in a special __docinfo__ attribute instead, and it will look exactly the same to your users. The only question is: which data should be privileges to use the syntactic sugar? That's up to each project to decide; all Guido is suggesting is that we add a default presumption that it's the MyPy-style static types that get to use annotations. As a side note, I'm not sure how your decorator syntax is going to work unless you've predefined a global named "a" with some kind of value that can be used to match parameters named "a". While there might be some way to do that dynamically (create a module subclass with a custom __dict__ that implements __missing__, and an import hook that applies that to your modules?), it seems pretty hacky, and doesn't buy you that much. From antoine at python.org Mon Aug 25 02:24:27 2014 From: antoine at python.org (Antoine Pitrou) Date: Sun, 24 Aug 2014 20:24:27 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53FA7C46.1000407@stoneleaf.us> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <53FA7C46.1000407@stoneleaf.us> Message-ID: Le 24/08/2014 19:59, Ethan Furman a ?crit : > > Is this workable? If I have a nested class in a function: > > def spam(): > class fribble: > def __init__(self, x:int): > blahblah > > I would expect MyPy to be able to retrieve and use the annotations from > fribble without actually calling spam. I don't know. But that's not really the point, because you are still able to do, say: my_type = some_invocation(...) def spam(): class fribble: def __init__(self, x: my_type): blahblah ... as far as the type checker uses module imports, that is. Of course, /after/ importing the module it can also walk the bytecode defined in that module. Regards Antoine. From steve at pearwood.info Mon Aug 25 02:51:31 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 25 Aug 2014 10:51:31 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <53FA7C46.1000407@stoneleaf.us> References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <53FA7C46.1000407@stoneleaf.us> Message-ID: <20140825005131.GJ25957@ando> On Sun, Aug 24, 2014 at 04:59:02PM -0700, Ethan Furman wrote: > Is this workable? If I have a nested class in a function: > > def spam(): > class fribble: > def __init__(self, x:int): > blahblah > > I would expect MyPy to be able to retrieve and use the annotations from > fribble without actually calling spam. If static checking is only > available at the top level I think a lot of power/usefulness is gone. That's entirely up to Mypy or whatever linter, compiler, editor or other tool being used, and has nothing to do with this proposal, which is only to standardize on the syntax. I would expect different static tools would have different levels of ability to deal with dynamically created classes like that. -- Steven From warren.weckesser at gmail.com Mon Aug 25 03:17:22 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 24 Aug 2014 21:17:22 -0400 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: On Sun, Aug 24, 2014 at 8:05 PM, Chris Angelico wrote: > On Sun, Aug 24, 2014 at 2:27 AM, Warren Weckesser > wrote: > > That wouldn't help in an example such as > > > > reduce(lambda x, y: x & y, [{1,2}, {3, 4}, {5, 6}], nullifier={}) > > > > The nullifier is the empty set, but the empty set does not occur in the > > iterable. > > Caution: Your nullifier is an empty dict, not an empty set, if you > spell it that way. > Ah, right--thanks! It should be `nullifier=set()`. > > I think this is a cute concept, but not something that is going to be > needed all that often. Most Python programs don't even use reduce() in > its default form, much less need a specific optimization. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Aug 25 03:30:23 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 25 Aug 2014 11:30:23 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> Message-ID: <20140825013023.GK25957@ando> On Sat, Aug 23, 2014 at 07:00:54PM -0400, Antoine Pitrou wrote: > > Le 23/08/2014 13:25, Steven D'Aprano a ?crit : > >> > >>It's not a new use. A type class is a class, and calling it is just > >>instantiating that class. There's nothing new here. If you think that's > >>a bit "meta", it's no different than e.g. higher-order functions. > > > >There's no instantiation during *static* analysis, because the code > >hasn't run yet. > > In your idea of "static analysis", it hasn't. Because you think it > should involve some kind of separate syntax analysis tool that has > nothing to do with regular Python. That's not "my" idea of static analysis, that is the standard definition of "static" as happening at compile-time. If it happens at runtime, it's not static. > But Python is powerful enough to let > you do that using normal introspection of modules. This proposal isn't just about the Python interpreter, its also about static tools like linters, IDEs and editors. > And it's *exactly* how we are exposing function annotations (and also > docstrings, etc.): using runtime-accessible introspection information > which is gathered by importing modules and therefore actually > *executing* toplevel module code. Not merely compiling it. Correct, the annotations will be available at runtime as well as compile-time. But tools like linters and editors will rely on static analysis, not dynamic run-time checks, and that's Guido's intention. Here's his initial post: https://mail.python.org/pipermail/python-ideas/2014-August/028618.html -- Steven From cfkaran2 at gmail.com Mon Aug 25 04:04:18 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 24 Aug 2014 22:04:18 -0400 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ References: <1C09BD80-90E6-4B61-9CA8-850D9F99EE41@gmail.com> Message-ID: <91DD0A3D-24F1-4169-944B-24A06BA74D57@gmail.com> On Aug 24, 2014, at 9:22 PM, Chris Angelico wrote: > On Mon, Aug 25, 2014 at 11:15 AM, Cem Karan wrote: >> Because of this, using 'annotizer' as a key is meaningless; there could be many instances live at the same time. > > But that's exactly my point! You don't use "annotizer" as the key, you > use annotizer. You use the object itself, not the string. And now I feel dumb... You're right of course, by passing in the object itself, there are no namespace conflicts, inter-project coordination is nil, and on top of it all, you can perform introspection of the annotizer object itself to find out more information about it. UUIDs aren't required at all. Right-o, with your permission Chris, I'd like to amend my proposal so that instead of using UUIDs, the annotizer object itself is used as the key. Depending on what work is like this week, I'll modify my code to do this. Thanks, Cem Karan From cfkaran2 at gmail.com Mon Aug 25 04:11:12 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 24 Aug 2014 22:11:12 -0400 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: <1472A8DB-2427-4ABA-857A-41DBB5707900@yahoo.com> References: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com> <53FA647A.2030108@canterbury.ac.nz> <1472A8DB-2427-4ABA-857A-41DBB5707900@yahoo.com> Message-ID: <05B80116-58B6-4E33-9EDE-F3D5BC7607FB@gmail.com> On Aug 24, 2014, at 8:21 PM, Andrew Barnert wrote: > On Aug 24, 2014, at 16:17, Cem Karan wrote: > >> >> On Aug 24, 2014, at 6:17 PM, Greg Ewing wrote: >> >>>> On 24 August 2014 03:18, Cem Karan wrote: >>>>> Each project (and 'project' is a very >>>>> loosely defined concept here) chooses a UUID that it uses as key into the >>>>> dictionary. >>> >>> Please, not UUIDs. As something meant to be used by >>> humans, they're horrible. >>> >>> -- >>> Greg >> >> I understand your concerns about them, I really, really do. But can you think of a better way? All my prior points still hold. In my mind, UUIDs are the least bad alternative. We have to educate people as to what the annotations dictionary standard is, and then encourage libraries to use the standard and to hide the UUIDs from end users. My code does this already; if someone wants me to extend it in some way, put in a request, and I will. If the PSF wants it in the standard library, it's theirs! Please, download it and play with it, it's short to the point of almost being trivial (https://github.com/oranguman/annotizer). >> >> Don't let the fact that the bare __annotations__ dict is going to look ugly; if we try to get a really powerful typing system in there, people will probably not want to write the annotations by hand anyways for the simple reason that it will be too hard to read. I mean, which would you rather read? >> >> """ >> @doc(a, "docs about a") >> @type(a, set(range(10))) >> def foo(a): >> pass >> >> def bar(a: {__docs_project_ID: "docs about a", __types_project_ID: set(range(10))}): >> pass >> """ > > Or: > > @doc(a, "docs about a") > def foo(a: set[10]): > pass > > Why does doc have to set an annotation in the first place? Annotations are just syntactic sugar for sticking stuff in a dict attached to the function as an attribute; when that syntactic sugar doesn't make your code more readable, there's no benefit at all to annotations. You could just make the @doc decorator store and find its stuff in a special __docinfo__ attribute instead, and it will look exactly the same to your users. True, but I was worried about namespace conflicts. More than one documentation library might decide to use the __docinfo__ attribute, which could lead to problems. The __annotations__ dictionary seemed like a good place to store it since it didn't have any predefined meaning (except that now its being proposed that it should have a meaning). > The only question is: which data should be privileges to use the syntactic sugar? That's up to each project to decide; all Guido is suggesting is that we add a default presumption that it's the MyPy-style static types that get to use annotations. > > As a side note, I'm not sure how your decorator syntax is going to work unless you've predefined a global named "a" with some kind of value that can be used to match parameters named "a". While there might be some way to do that dynamically (create a module subclass with a custom __dict__ that implements __missing__, and an import hook that applies that to your modules?), it seems pretty hacky, and doesn't buy you that much. My code seems to work correctly (see https://github.com/oranguman/annotizer/blob/master/annotizer/annotizer.py) via some hackery involving inspect.signature(), but please feel free to take a look at it and tell me if I should do something different. Thanks, Cem Karan From antoine at python.org Mon Aug 25 04:11:07 2014 From: antoine at python.org (Antoine Pitrou) Date: Sun, 24 Aug 2014 22:11:07 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140825013023.GK25957@ando> References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: Le 24/08/2014 21:30, Steven D'Aprano a ?crit : >> >> In your idea of "static analysis", it hasn't. Because you think it >> should involve some kind of separate syntax analysis tool that has >> nothing to do with regular Python. > > That's not "my" idea of static analysis, that is the standard definition > of "static" as happening at compile-time. If it happens at runtime, it's > not static No, that's the standard definition of "static" in a certain category of languages such as C. Python has "static methods" and they don't happen at compile-time: "staticmethod" is a regular callable which is invoked at runtime. And there are other precedents, such as RPython which is also "statically" typed but... using code loaded from imported modules (in other words, I'm not inventing anything new here). > This proposal isn't just about the Python interpreter, its also about > static tools like linters, IDEs and editors. Those so-called "static" tools are free to invoke the interpreter and use Python's *runtime* introspection facilities. And I'm sure some of them do. Really, it's amusing that I tried several times to explain you that "static" analysis didn't need to happen in an entirely separate evaluation context, and you still don't get it. You speak like some C programmer who has newly discovered Python and doesn't realize the magnitude of the differences between the two languages. Therefore I think the word "static" should be banned from this entire discussion, otherwise folks like you will keep associating irrelevant concepts with it. Regards Antoine. From rosuav at gmail.com Mon Aug 25 04:16:12 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 25 Aug 2014 12:16:12 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On Mon, Aug 25, 2014 at 12:11 PM, Antoine Pitrou wrote: > No, that's the standard definition of "static" in a certain category of > languages such as C. Python has "static methods" and they don't happen at > compile-time: "staticmethod" is a regular callable which is invoked at > runtime. "Static method" is a quite different meaning of static - they always execute at run-time. https://en.wikipedia.org/wiki/Method_(computer_programming)#Static_methods ChrisA From rosuav at gmail.com Mon Aug 25 04:19:44 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 25 Aug 2014 12:19:44 +1000 Subject: [Python-ideas] Was: Annotations (and static typing), Now:Sharing __annotations__ In-Reply-To: <91DD0A3D-24F1-4169-944B-24A06BA74D57@gmail.com> References: <1C09BD80-90E6-4B61-9CA8-850D9F99EE41@gmail.com> <91DD0A3D-24F1-4169-944B-24A06BA74D57@gmail.com> Message-ID: On Mon, Aug 25, 2014 at 12:04 PM, Cem Karan wrote: > On Aug 24, 2014, at 9:22 PM, Chris Angelico wrote: > >> On Mon, Aug 25, 2014 at 11:15 AM, Cem Karan wrote: >>> Because of this, using 'annotizer' as a key is meaningless; there could be many instances live at the same time. >> >> But that's exactly my point! You don't use "annotizer" as the key, you >> use annotizer. You use the object itself, not the string. > > And now I feel dumb... Ehh, don't feel dumb, my explanation wasn't all that clear - it's hard to be clear in email. It's like when Rapunzel says, "I like Eugene Fitzherbert better than Flynn Rider", and the scriptwriters are (presumably deliberately) ambiguous as to whether she's putting quotes around those names ("I prefer to call you Eugene than Flynn") or not ("I prefer the persona of Eugene to that of Flynn"). As long as the objects have useful reprs, you can get most of the benefit of strings anyway - you can print them out and get an idea of what they're all about, which UUIDs wouldn't give you. You might have a collision of representation (if you have multiple annotizer instances), or you might fall back on incorporating the id() in the repr, but either way can't be worse than nothing. ChrisA From guido at python.org Mon Aug 25 04:30:35 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 24 Aug 2014 19:30:35 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: This argument over the word "static" is getting ridiculous. The meaning of "static analysis" is perfectly well understood (just Google it) and using Python run-time introspection for the code being analyzed is exactly what you *don't* want in this case, for a variety of reasons. (And that includes doing the analysis as part of the byte-code compilation that may happen at import/load time. Once any part of the program is executing it's simply too late for the kind of analysis we're interested in.) People who are talking about what should happen if a decorator changes __annotations__ are likewise missing the point. (This doesn't mean that decorators aren't allowed to affect the signature. But the static analyzer has to use its static powers to figure out what the decorator does, or it has to be told, e.g. via a stub file.) Note that I am okay with being somewhat skittish about the term "static typing", given Python's long struggle for legitimacy in a world where professors everywhere used to equate static typing and strong typing and tell their impressionable students that it was the Right Way. We should probably avoid "static typing" and use "type hinting" instead (following TypeScript's lead I believe). But the process of looking for violations against the hinted types before the code run is still called static analysis. On Sun, Aug 24, 2014 at 7:11 PM, Antoine Pitrou wrote: > > Le 24/08/2014 21:30, Steven D'Aprano a ?crit : > > >>> In your idea of "static analysis", it hasn't. Because you think it >>> should involve some kind of separate syntax analysis tool that has >>> nothing to do with regular Python. >>> >> >> That's not "my" idea of static analysis, that is the standard definition >> of "static" as happening at compile-time. If it happens at runtime, it's >> not static >> > > No, that's the standard definition of "static" in a certain category of > languages such as C. Python has "static methods" and they don't happen at > compile-time: "staticmethod" is a regular callable which is invoked at > runtime. > > And there are other precedents, such as RPython which is also "statically" > typed but... using code loaded from imported modules (in other words, I'm > not inventing anything new here). > > > This proposal isn't just about the Python interpreter, its also about >> static tools like linters, IDEs and editors. >> > > Those so-called "static" tools are free to invoke the interpreter and use > Python's *runtime* introspection facilities. And I'm sure some of them do. > > Really, it's amusing that I tried several times to explain you that > "static" analysis didn't need to happen in an entirely separate evaluation > context, and you still don't get it. You speak like some C programmer who > has newly discovered Python and doesn't realize the magnitude of the > differences between the two languages. > > Therefore I think the word "static" should be banned from this entire > discussion, otherwise folks like you will keep associating irrelevant > concepts with it. > > Regards > > Antoine. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Mon Aug 25 04:35:49 2014 From: antoine at python.org (Antoine Pitrou) Date: Sun, 24 Aug 2014 22:35:49 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: Le 24/08/2014 22:16, Chris Angelico a ?crit : > On Mon, Aug 25, 2014 at 12:11 PM, Antoine Pitrou wrote: >> No, that's the standard definition of "static" in a certain category of >> languages such as C. Python has "static methods" and they don't happen at >> compile-time: "staticmethod" is a regular callable which is invoked at >> runtime. > > "Static method" is a quite different meaning of static - they always > execute at run-time. Oh, really, do they? You missed the entire point. Python's static methods not only execute at runtime, they are *defined* at runtime. The compiler doesn't know that something will be a "static method", a regular function, a "class method"... or whatever. The meaning of "static" is entirely different from C's or Java's, where the information about static methods is (and has to be) known at compile time. Of course, this isn't limited to static methods. The same can be said about docstrings, which use separate analysis methods in languages such as C++ or Java (e.g. doxygen, javadoc...), but regular runtime introspection capabilities in Python. Regards Antoine. From antoine at python.org Mon Aug 25 04:51:15 2014 From: antoine at python.org (Antoine Pitrou) Date: Sun, 24 Aug 2014 22:51:15 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: Le 24/08/2014 22:30, Guido van Rossum a ?crit : > > People who are talking about what should happen if a decorator changes > __annotations__ are likewise missing the point. But you shouldn't use annotations for something that refuses to use runtime introspection abilities. There's no reason to waste annotations if the need can be fullfilled by separate description files in a DSL that doesn't even need to be Python code (but cutely looks like so). And it's not surprising that some people may be missing the point when a feature pretends to use a runtime facility but interprets it from a separate channel which eschews any code execution. Your proposal seems determined by the fact that mypy has much grander ambitions (and therefore requires its insertion into regular Python), but it doesn't make use of that power at all, worse, it forbids others to use it. Regards Antoine. From antoine at python.org Mon Aug 25 05:03:46 2014 From: antoine at python.org (Antoine Pitrou) Date: Sun, 24 Aug 2014 23:03:46 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: Le 24/08/2014 22:51, Antoine Pitrou a ?crit : > > Le 24/08/2014 22:30, Guido van Rossum a ?crit : >> >> People who are talking about what should happen if a decorator changes >> __annotations__ are likewise missing the point. > > But you shouldn't use annotations for something that refuses to use > runtime introspection abilities. There's no reason to waste annotations > if the need can be fullfilled by separate description files in a DSL > that doesn't even need to be Python code (but cutely looks like so). (*) or a docstring-embedded DSL, as others acutely remarked. Regards Antoine. From guido at python.org Mon Aug 25 05:12:07 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 24 Aug 2014 20:12:07 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On Sun, Aug 24, 2014 at 7:51 PM, Antoine Pitrou wrote: > > Le 24/08/2014 22:30, Guido van Rossum a ?crit : > > People who are talking about what should happen if a decorator changes >> __annotations__ are likewise missing the point. >> > > But you shouldn't use annotations for something that refuses to use > runtime introspection abilities. There's no reason to waste annotations if > the need can be fullfilled by separate description files in a DSL that > doesn't even need to be Python code (but cutely looks like so). > It doesn't feel like a waste to me -- having it syntactically be part of the source code to me is a strong advantage over stubs or special comments or even special syntax in docstrings. As I have said several times now, it fulfills exactly the original (pre-PEP 3107) goal I had in mind for them. > And it's not surprising that some people may be missing the point when a > feature pretends to use a runtime facility but interprets it from a > separate channel which eschews any code execution. > There is no pretense here. It is simply useful to make the type annotations *also* available at run time. > Your proposal seems determined by the fact that mypy has much grander > ambitions (and therefore requires its insertion into regular Python), but > it doesn't make use of that power at all, worse, it forbids others to use > it. > This remark about mypy's ambitions sounds delusional and paranoid. I wouldn't care about mypy at all if it wasn't right in line with my intentions and desires for Python. And while its author *originally* intended to create a whole new language (since that is what academics do :-), he gracefully changed his design (over a year ago) when I suggested that it might be more useful if it fit neatly into Python 3 annotations. Finally, I would actually be okay if we found a way to let type hints and other annotations coexist -- I just prefer type hints to be the default use, since I see them as much more generally useful than all other uses combined. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Aug 25 05:29:08 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 24 Aug 2014 20:29:08 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: Message-ID: On Fri, Aug 22, 2014 at 7:30 AM, Bob Ippolito wrote: > > > On Friday, August 22, 2014, Skip Montanaro wrote: > >> There's been a lot to read in this and related threads over the past nine >> days. (According to Gmail. It qualitatively seems much longer to me.) I >> think I've followed along reasonably well. Forgive me if I missed the >> answers to these questions. The proposal on the table is to adopt MyPy's >> type annotations for Python 3.mumble. I presume MyPy already supports them. >> >> 1. Can MyPy be used today as a standalone static type checker for Python >> 3.x code without actually compiling anything? That is, can I just sprinkle >> type annotations into my code and run a front-end pass of MyPy much the >> same way I'd run pylint, vulture, flake8, or other lint-ish program? >> > > Yes. mypy -S performs checking without execution. > > 2. Assuming the answer to #1 is "yes," if you start sprinkling type >> annotations into your code and running "mypy *.py", will it tell you when >> it needs a missing type annotation to more fully check things, or will it >> silently process code without annotations and not let you know that it's >> not really checking much? >> > > No. It happily and silently accepts dynamic code with no annotations. It > can't do much for you with that code, but it doesn't complain. > However, mypy has a flag --html-report which generates a marked-up source code listing that shows which regions of the code have or haven't been checked, using similar color-coding as coverage.py. > 3. Will we get into a phase like the early days of "const" in ANSI C where >> addition of "const" in one location in your existing code base forced you >> into a never-ending iterations of adding const all over the place? I forget >> what that was called ("const propagation"?), but I recall that it generally >> wasn't a fun activity. >> > > Nope. > The Python interpreter won't force you. Of course you can still play this game (e.g. by insisting that no red zones should exist in the above-mentioned source listing) but that's no different from insisting on zero lint warnings -- it is a choice a developer (usually a team) makes, not something the software enforces. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Aug 25 08:18:12 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 25 Aug 2014 02:18:12 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On 8/24/2014 11:12 PM, Guido van Rossum wrote: > Finally, I would actually be okay if we found a way to let type hints > and other annotations coexist -- I just prefer type hints to be the > default use, since I see them as much more generally useful than all > other uses combined. I believe co-existence is possible, but the details will depend on the form type hints take. First, other annotations should be easily distinguished from type hints. Second, other annotations that would interfere with the runtime use of __annotations__ should be pulled out by a decorator and possibly moved to another attribute specific to the decorator (such as '_deco_name'). -- Terry Jan Reedy From edk141 at gmail.com Mon Aug 25 12:20:03 2014 From: edk141 at gmail.com (Ed Kellett) Date: Mon, 25 Aug 2014 11:20:03 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On 25 August 2014 07:18, Terry Reedy wrote: > On 8/24/2014 11:12 PM, Guido van Rossum wrote: > >> Finally, I would actually be okay if we found a way to let type hints >> and other annotations coexist -- I just prefer type hints to be the >> default use, since I see them as much more generally useful than all >> other uses combined. > > > I believe co-existence is possible, but the details will depend on the form > type hints take. First, other annotations should be easily distinguished > from type hints. Second, other annotations that would interfere with the > runtime use of __annotations__ should be pulled out by a decorator and > possibly moved to another attribute specific to the decorator (such as > '_deco_name'). ? or we could have a decorator for type hints, and ascribe no new meaning at all to __annotations__. Or assume __annotations__ contains type hints iff all the annotations present are instances of typing.*. That might be better anyway, since a decorator could then add (or augment) type information without assigning to __annotations__ (which would be weird). Even if conveying type information is the most useful use of annotations, there's no reason it can't be explicit (and consistent with other uses of annotations). Ed Kellett From cfkaran2 at gmail.com Mon Aug 25 12:57:23 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Mon, 25 Aug 2014 06:57:23 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com> On Aug 25, 2014, at 6:20 AM, Ed Kellett wrote: > On 25 August 2014 07:18, Terry Reedy wrote: >> On 8/24/2014 11:12 PM, Guido van Rossum wrote: >> >>> Finally, I would actually be okay if we found a way to let type hints >>> and other annotations coexist -- I just prefer type hints to be the >>> default use, since I see them as much more generally useful than all >>> other uses combined. >> >> >> I believe co-existence is possible, but the details will depend on the form >> type hints take. First, other annotations should be easily distinguished >> from type hints. Second, other annotations that would interfere with the >> runtime use of __annotations__ should be pulled out by a decorator and >> possibly moved to another attribute specific to the decorator (such as >> '_deco_name'). > > ? or we could have a decorator for type hints, and ascribe no new > meaning at all to __annotations__. Or assume __annotations__ contains > type hints iff all the annotations present are instances of typing.*. > That might be better anyway, since a decorator could then add (or > augment) type information without assigning to __annotations__ (which > would be weird). > > Even if conveying type information is the most useful use of > annotations, there's no reason it can't be explicit (and consistent > with other uses of annotations). I'm going to beat on my drum some more, but if annotations are dictionaries, where the keys are objects known to be associated with the type checker, then we don't have to guess, we'll know. E.g.: """ # type_checker.py class type_checker(object): # decorator magic TYPE_CHECKER = type_checker() """ """ # Your file from type_checker import TYPE_CHECKER @TYPE_CHECKER(a, int) def foo(a): pass """ which is morally equivalent to: """ from type_checker import TYPE_CHECKER def foo(a: {TYPE_CHECKER: int}): pass """ Chris Angelico pointed this trick out to me, and I think it's a good one. Assuming the type checker has enough brains to check and ensure that TYPE_CHECKER is never reassigned, it is guaranteed by the runtime system that TYPE_CHECKER is unique, which means that it can do the moral equivalent of 'TYPE_CHECKER in foo.__annotations__['a']'. That gives a VERY simple method of knowing what the annotations are being used for. Thanks, Cem Karan From rosuav at gmail.com Mon Aug 25 13:03:48 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 25 Aug 2014 21:03:48 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com> References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com> Message-ID: On Mon, Aug 25, 2014 at 8:57 PM, Cem Karan wrote: > from type_checker import TYPE_CHECKER > > def foo(a: {TYPE_CHECKER: int}): > pass I still don't think this offers much benefit over def foo(a: int): pass It's a lot wordier and the flexibility will almost never be needed. So is that multiplexing really worth it? ChrisA From oreilldf at gmail.com Mon Aug 25 15:44:12 2014 From: oreilldf at gmail.com (Dan O'Reilly) Date: Mon, 25 Aug 2014 09:44:12 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: Adding active/idle/total worker counts for both ThreadPoolExecutor and ProcessPoolExecutor is pretty straightforward; I just threw a patch together for both in 30 minutes or so. However, I don't think its possible to inspect the contents of a ProcessPoolExecutor's queue without actually consuming items from it. While it *is* possible with ThreadPoolExecutor, I don't think we should expose it - the queue.Queue() implementation ThreadPoolExecutor relies on doesn't have a public API for inspecting its contents, so ThreadPoolExecutor probably shouldn't expose one, either. Identifying which task each worker is processing is possible, but would perhaps require more work than its worth, at least for ProcessPoolExecutor. I do think adding worker count APIs is reasonable, and in-line with a TODO item in the ThreadPoolExecutor source: # TODO(bquinlan): Should avoid creating new threads if there are more # idle threads than items in the work queue. So, at the very least there have been plans to internally keep track active/idle thread counts. If others agree it's a good idea, I'll open an issue on the tracker for this and include my patch (which also addresses that TODO item). On Sun, Aug 24, 2014 at 5:41 PM, Ram Rachum wrote: > Sometimes I want to take a live executor, like a `ThreadPoolExecutor`, and > check up on it. I want to know how many threads there are, how many are > handling tasks and which tasks, how many are free, and which tasks are in > the queue. > > I asked on Stack Overflow: > http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-futures-threadpoolexecutor > > There's an answer there, but it uses private variables and it's not part > of the API. > > I suggest it become a part of the API. There should be an API for checking > on what the executor is currently doing and answering all the questions I > raised above. > > > Thanks, > Ram. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Mon Aug 25 15:57:17 2014 From: antoine at python.org (Antoine Pitrou) Date: Mon, 25 Aug 2014 09:57:17 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: Le 24/08/2014 23:12, Guido van Rossum a ?crit : > > As I have said several times now, it fulfills exactly the original > (pre-PEP 3107) goal I had in mind for them. But were you envisioning back then that said annotations would be looked up without the regular execution environment? PEP 3107 doesn't say anything about that (or rather, it says that annotations can be looked up on the function, but function objects only exist at run-time). Actually, the bytecode isn't very practical to work with to extract annotations, it seems: >>> def g(): ... def f(a: "foo") -> "bar": pass ... >>> dis.dis(g) 2 0 LOAD_CONST 1 ('foo') 3 LOAD_CONST 2 ('bar') 6 LOAD_CONST 3 (('a', 'return')) 9 LOAD_CONST 4 (", line 2>) 12 LOAD_CONST 5 ('g..f') 15 EXTENDED_ARG 3 18 MAKE_FUNCTION 196608 21 STORE_FAST 0 (f) 24 LOAD_CONST 0 (None) 27 RETURN_VALUE ... so I suppose people would want to run an AST pass instead? (and then evaluate the "annotations" parts of the AST by hand... uh) > Your proposal seems determined by the fact that mypy has much > grander ambitions (and therefore requires its insertion into regular > Python), but it doesn't make use of that power at all, worse, it > forbids others to use it. > > This remark about mypy's ambitions sounds delusional and paranoid. It is entirely uncritical about mypy. It's fine to have other Python (or Python-like) implementations, even with deliberately different semantics. Such experimentations actually make the community much livelier than, say, PHP's or Ruby's. I don't know at which point mypy changed goals (if it has), but there are still signs in the website of the goal of building a separate runtime (not necessarily a separate syntax), e.g. """Also some language features that are evaluated at runtime in Python may happen during compilation in mypy when using the native semantics. For example, mypy base classes may be bound during compilation (or program loading, before evaluation), unlike Python.""" Also the fact that mypy supports constructs such as "List[int]()". This is a set of design constraints that you're not bound to. Regards Antoine. From apalala at gmail.com Mon Aug 25 16:57:45 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Mon, 25 Aug 2014 10:27:45 -0430 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On Sun, Aug 24, 2014 at 10:00 PM, Guido van Rossum wrote: > We should probably avoid "static typing" and use "type hinting" instead > (following TypeScript's lead I believe). But the process of looking for > violations against the hinted types before the code run is still called > static analysis. +1 -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Aug 25 17:12:04 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 25 Aug 2014 08:12:04 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On Sun, Aug 24, 2014 at 11:18 PM, Terry Reedy wrote: > On 8/24/2014 11:12 PM, Guido van Rossum wrote: > > Finally, I would actually be okay if we found a way to let type hints >> and other annotations coexist -- I just prefer type hints to be the >> default use, since I see them as much more generally useful than all >> other uses combined. >> > > I believe co-existence is possible, but the details will depend on the > form type hints take. First, other annotations should be easily > distinguished from type hints. Second, other annotations that would > interfere with the runtime use of __annotations__ should be pulled out by a > decorator and possibly moved to another attribute specific to the decorator > (such as '_deco_name'). What exactly do you mean by "other annotations that would interfere with the runtime use of __annotations__"? I can only assume that you are thinking of a situation where you are introspecting some function/method of unknown origin and you are trying to see if it has any annotations, in which case you are going to use them for a locally-defined use (e.g. generate an HTML form -- contrived example). It sounds as if you are worried about being passed a function that in the past would not have any annotations (so you would just generate a default form based on the argument names) but which now has been annotated by a zealous programmer with type hints. And you are worried that those type hints will confuse (perhaps crash) the form-generation code. I think initially we will just tell the user "don't use type annotations for form-handling functions", and also "don't run the type checker on that code". The next stage would probably be to create a decorator (say, @form_handler) and request that all form handlers are thus decorated. The decorator wouldn't need to do anything -- we could just teach the type checker that @form_handler means that argument annotations mean something that's unrelated to type checking, and we would still have to tell the user "don't use type annotations for @form_handler functions", but at least they could run the type checker on files that contain form handlers (the handlers themselves just wouldn't be type-checked). It should be easy enough to teach the type checker about such decorators; all you need is some kind of marker on the decorator function (perhaps itself a decorator :-). Eventually users might encourage the form library's maintainers to move the form info elsewhere (e.g. in the decorator arguments) and then the marker on the decorator can be taken off. But the initial stage seems a totally fine solution for 3.5, and it doesn't require anyone to do anything (it just requires some people to refrain from doing some new things -- which is much easier, because people are naturally aware that new things can cause new problems :-). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Aug 25 17:29:55 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 25 Aug 2014 08:29:55 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On Mon, Aug 25, 2014 at 3:20 AM, Ed Kellett wrote: > > On 25 August 2014 07:18, Terry Reedy wrote: > > On 8/24/2014 11:12 PM, Guido van Rossum wrote: > > > >> Finally, I would actually be okay if we found a way to let type hints > >> and other annotations coexist -- I just prefer type hints to be the > >> default use, since I see them as much more generally useful than all > >> other uses combined. > > > > > > I believe co-existence is possible, but the details will depend on the form > > type hints take. First, other annotations should be easily distinguished > > from type hints. Second, other annotations that would interfere with the > > runtime use of __annotations__ should be pulled out by a decorator and > > possibly moved to another attribute specific to the decorator (such as > > '_deco_name'). > > ? or we could have a decorator for type hints, and ascribe no new > meaning at all to __annotations__. Or assume __annotations__ contains > type hints iff all the annotations present are instances of typing.*. > That might be better anyway, since a decorator could then add (or > augment) type information without assigning to __annotations__ (which > would be weird). > > Even if conveying type information is the most useful use of > annotations, there's no reason it can't be explicit (and consistent > with other uses of annotations). All that sounds fine, but you still have to have a way to convey all that information to the type checker. Remember, the type checker cannot (or doesn't want to) execute the code and it can only see annotations in their original syntactic form. (But it can indeed be told about certain decorators.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Mon Aug 25 18:07:37 2014 From: edk141 at gmail.com (Ed Kellett) Date: Mon, 25 Aug 2014 17:07:37 +0100 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On 25 Aug 2014 16:29, "Guido van Rossum" wrote: > > ? or we could have a decorator for type hints, and ascribe no new > > meaning at all to __annotations__. Or assume __annotations__ contains > > type hints iff all the annotations present are instances of typing.*. > > That might be better anyway, since a decorator could then add (or > > augment) type information without assigning to __annotations__ (which > > would be weird). > > > > Even if conveying type information is the most useful use of > > annotations, there's no reason it can't be explicit (and consistent > > with other uses of annotations). > > All that sounds fine, but you still have to have a way to convey all that information to the type checker. Remember, the type checker cannot (or doesn't want to) execute the code and it can only see annotations in their original syntactic form. (But it can indeed be told about certain decorators.) That's reasonable - I was imagining the decorator for type hints wouldn't do anything (apart from possibly marking the function as having type hints), it would just be there to tell the static checker "this function should be type-checked ". Ed Kellett -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Aug 25 19:26:01 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 25 Aug 2014 10:26:01 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On Mon, Aug 25, 2014 at 6:57 AM, Antoine Pitrou wrote: > > Le 24/08/2014 23:12, Guido van Rossum a ?crit : > > > As I have said several times now, it fulfills exactly the original > > (pre-PEP 3107) goal I had in mind for them. > > But were you envisioning back then that said annotations would be > looked up without the regular execution environment? > > PEP 3107 doesn't say anything about that (or rather, it says that > annotations can be looked up on the function, but function objects only > exist at run-time). Actually, the bytecode isn't very practical to work > with to extract annotations, it seems: > > >>> def g(): > ... def f(a: "foo") -> "bar": pass > ... > >>> dis.dis(g) > 2 0 LOAD_CONST 1 ('foo') > 3 LOAD_CONST 2 ('bar') > 6 LOAD_CONST 3 (('a', 'return')) > 9 LOAD_CONST 4 ( 0x7f17339e2580, file "", line 2>) > 12 LOAD_CONST 5 ('g..f') > 15 EXTENDED_ARG 3 > 18 MAKE_FUNCTION 196608 > 21 STORE_FAST 0 (f) > 24 LOAD_CONST 0 (None) > 27 RETURN_VALUE > > ... so I suppose people would want to run an AST pass instead? > (and then evaluate the "annotations" parts of the AST by hand... uh) > Actually mypy uses an entirely different parser. This is mostly for historical reasons: it started out as a compiler/checker/runtime for a different, somewhat-Python-like statically-typed language. But I assume that static checkers often have no choice in the matter and must write their own parser -- e.g. PyCharm is itself written in Java and its parser must work even if the code is not syntactically correct. Another reason to use a different parser is that mypy needs to see the comments so it can look for magic comments starting with "#type:". This is typical for linters. In general a type checker should not attempt to cope with clever things (decorators or otherwise) that modify __annotations__; it should take the expression tree of the annotation at face value. (It should of course know about certain decorators like @property and other Python ideosyncracies like self-passing for methods, and I think there should be a way to teach it about new decorators. But executing Python code should not be part of that.) > > Your proposal seems determined by the fact that mypy has much > > grander ambitions (and therefore requires its insertion into regular > > Python), but it doesn't make use of that power at all, worse, it > > forbids others to use it. > > > > This remark about mypy's ambitions sounds delusional and paranoid. > > It is entirely uncritical about mypy. It's fine to have other Python > (or Python-like) implementations, even with deliberately different > semantics. Such experimentations actually make the community much livelier > than, say, PHP's or Ruby's. > OK, thanks for clarifying that. I took "grand ambitions" as a (sarcastically) pejorative term, but you're right that it doesn't need to be read like that. > I don't know at which point mypy changed goals (if it has), but there are > still signs in the website of the goal of building a separate runtime (not > necessarily a separate syntax), e.g. > > """Also some language features that are evaluated at runtime in Python > may happen during compilation in mypy when using the native semantics. > For example, mypy base classes may be bound during compilation (or > program loading, before evaluation), unlike Python.""" > I have talked to Jukka about this, and he is definitely on board with reducing mypy's functionality to that of a linter; I think there's even an issue in mypy's tracker about removing the ability to execute the code (currently there's a flag you must pass to prevent it from trying to run the code). Updating the website is an ongoing low-priority project (you can probably send him pull requests for it). > Also the fact that mypy supports constructs such as "List[int]()". This > is a set of design constraints that you're not bound to. > Yeah, I'm not crazy about that syntax (I think nobody is). It mostly exists for the common use case where you have a function that is declared to return e.g. a list of ints and you want to start off with an empty list; the current mypy implementation complains about such code. Example: def squares(xs: Sequence[float]) -> List[float]: sqs = [] for x in xs: sqs.append(x**2) return sqs mypy complains about the unannotated initialization of sqs. The two ways to currently address this are sqs = [] # type: List[float] or sqs = List[float]() Neither is very attractive; mypy should just infer the type of sqs. Nevertheless, I don't want to use call syntax to set parameters for generic types, since a generic type still "feels" sufficiently like a class that calling it is easily confused with instantiation -- even though ABCs are typically not instantiable. (There's no hard rule for that though -- it is merely the result of typical ABCs having at least one abstract method.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Aug 25 19:30:59 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 25 Aug 2014 10:30:59 -0700 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: Doesn't queue.Queue also have methods qsize(), empty() and full()? We could easily wrap those. There's always the caveat that the numbers may be out of date as soon as you print them. On Mon, Aug 25, 2014 at 6:44 AM, Dan O'Reilly wrote: > Adding active/idle/total worker counts for both ThreadPoolExecutor and > ProcessPoolExecutor is pretty straightforward; I just threw a patch > together for both in 30 minutes or so. However, I don't think its possible > to inspect the contents of a ProcessPoolExecutor's queue without actually > consuming items from it. While it *is* possible with ThreadPoolExecutor, I > don't think we should expose it - the queue.Queue() implementation > ThreadPoolExecutor relies on doesn't have a public API for inspecting its > contents, so ThreadPoolExecutor probably shouldn't expose one, either. > Identifying which task each worker is processing is possible, but would > perhaps require more work than its worth, at least for ProcessPoolExecutor. > > I do think adding worker count APIs is reasonable, and in-line with a TODO > item in the ThreadPoolExecutor source: > > # TODO(bquinlan): Should avoid creating new threads if there are more > # idle threads than items in the work queue. > > So, at the very least there have been plans to internally keep track > active/idle thread counts. If others agree it's a good idea, I'll open an > issue on the tracker for this and include my patch (which also addresses > that TODO item). > > > > On Sun, Aug 24, 2014 at 5:41 PM, Ram Rachum wrote: > >> Sometimes I want to take a live executor, like a `ThreadPoolExecutor`, >> and check up on it. I want to know how many threads there are, how many are >> handling tasks and which tasks, how many are free, and which tasks are in >> the queue. >> >> I asked on Stack Overflow: >> http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-futures-threadpoolexecutor >> >> There's an answer there, but it uses private variables and it's not part >> of the API. >> >> I suggest it become a part of the API. There should be an API for >> checking on what the executor is currently doing and answering all the >> questions I raised above. >> >> >> Thanks, >> Ram. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Mon Aug 25 19:34:43 2014 From: antoine at python.org (Antoine Pitrou) Date: Mon, 25 Aug 2014 13:34:43 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: Le 25/08/2014 09:44, Dan O'Reilly a ?crit : > > So, at the very least there have been plans to internally keep track > active/idle thread counts. If others agree it's a good idea, I'll open > an issue on the tracker for this and include my patch (which also > addresses that TODO item). I agree that basic executor parameters could be reflected, and I also agree that some other pieces of runtime state cannot be reliably computed and therefore shouldn't be exposed. Don't hesitate to open an issue with your patch. Regards Antoine. From ram at rachum.com Mon Aug 25 19:37:25 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 25 Aug 2014 20:37:25 +0300 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: "some other pieces of runtime state cannot be reliably computed" Can you please specify which ones you mean, and why not reliable? On Mon, Aug 25, 2014 at 8:34 PM, Antoine Pitrou wrote: > Le 25/08/2014 09:44, Dan O'Reilly a ?crit : > > >> So, at the very least there have been plans to internally keep track >> active/idle thread counts. If others agree it's a good idea, I'll open >> an issue on the tracker for this and include my patch (which also >> addresses that TODO item). >> > > I agree that basic executor parameters could be reflected, and I also > agree that some other pieces of runtime state cannot be reliably computed > and therefore shouldn't be exposed. > > Don't hesitate to open an issue with your patch. > > Regards > > Antoine. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/python-ideas/pl3r5SsbLLU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Mon Aug 25 19:57:08 2014 From: antoine at python.org (Antoine Pitrou) Date: Mon, 25 Aug 2014 13:57:08 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: Le 25/08/2014 13:37, Ram Rachum a ?crit : > "some other pieces of runtime state cannot be reliably computed" > > Can you please specify which ones you mean, and why not reliable? I cannot say for sure without taking a more detailed look at concurrent.futures :-) However, any runtime information such as "the tasks current being processes" (as opposed to, say, waiting) may not be available to the calling thread or process, or may be unreliable once it returns to the function's caller (since the actual state may have changed in-between). In the former case (information not available to the main process), we can't expose the information at all; in the latter case, we may still choose to expose it with the usual caveats in the documentation (exactly like Queue.qsize()). Regards Antoine. From ram at rachum.com Mon Aug 25 20:16:56 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 25 Aug 2014 21:16:56 +0300 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: Maybe I'm missing something, but I don't think that's something that should block implementation. Information not available? Change the executor code to make that information available. Information could have been changed? So what? That is to be expected. When I read a file in Python, by the time the line finished someone could have written something to that file so the result of the read may not be current. Even if I read just a simple variable, by the next line it might have been changed by another thread. I really don't see why any of that deserves special consideration. On Mon, Aug 25, 2014 at 8:57 PM, Antoine Pitrou wrote: > Le 25/08/2014 13:37, Ram Rachum a ?crit : > > "some other pieces of runtime state cannot be reliably computed" >> >> Can you please specify which ones you mean, and why not reliable? >> > > I cannot say for sure without taking a more detailed look at > concurrent.futures :-) However, any runtime information such as "the tasks > current being processes" (as opposed to, say, waiting) may not be available > to the calling thread or process, or may be unreliable once it returns to > the function's caller (since the actual state may have changed in-between). > > In the former case (information not available to the main process), we > can't expose the information at all; in the latter case, we may still > choose to expose it with the usual caveats in the documentation (exactly > like Queue.qsize()). > > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/python-ideas/pl3r5SsbLLU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Mon Aug 25 20:43:50 2014 From: antoine at python.org (Antoine Pitrou) Date: Mon, 25 Aug 2014 14:43:50 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: Le 25/08/2014 14:16, Ram Rachum a ?crit : > Maybe I'm missing something, but I don't think that's something that > should block implementation. > > Information not available? Change the executor code to make that > information available. Not if that would make the implementation much more complicated, or significantly slower. Regards Antoine. From guido at python.org Mon Aug 25 20:54:03 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 25 Aug 2014 11:54:03 -0700 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: It might be worth it to make the implementation somewhat more complicated if it serves a good purpose, for example giving the user of the program insights into how well the executor is performing. Without such insight you may be attempting to tune parameters (like the pool size) without being able to evaluate their effect. On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou wrote: > Le 25/08/2014 14:16, Ram Rachum a ?crit : > > Maybe I'm missing something, but I don't think that's something that >> should block implementation. >> >> Information not available? Change the executor code to make that >> information available. >> > > Not if that would make the implementation much more complicated, or > significantly slower. > > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From oreilldf at gmail.com Mon Aug 25 22:03:47 2014 From: oreilldf at gmail.com (Dan O'Reilly) Date: Mon, 25 Aug 2014 16:03:47 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: I'll take a look at this again tonight and see if more detailed information (e.g. which tasks are actually being processed) can be determined without too much added complexity and/or performance penalties. If I can come up with something reasonable for both ProcessPool/ThreadPool, I'll add it to the changes I've already made. Either way, I'll create an issue to track this. On Mon, Aug 25, 2014 at 2:54 PM, Guido van Rossum wrote: > It might be worth it to make the implementation somewhat more complicated > if it serves a good purpose, for example giving the user of the program > insights into how well the executor is performing. Without such insight you > may be attempting to tune parameters (like the pool size) without being > able to evaluate their effect. > > > On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou > wrote: > >> Le 25/08/2014 14:16, Ram Rachum a ?crit : >> >> Maybe I'm missing something, but I don't think that's something that >>> should block implementation. >>> >>> Information not available? Change the executor code to make that >>> information available. >>> >> >> Not if that would make the implementation much more complicated, or >> significantly slower. >> >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Aug 25 22:05:18 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Aug 2014 13:05:18 -0700 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> Message-ID: <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> I don't think there's any issue with letting people introspect the executor. The problem is that the main thing you get is a queue, and there's a limit to how introspectable a queue can be. In particular, if you want to iterate the waiting tasks, you have to iterate the queue, and there's no safe way to do that. Since CPython's queue.Queue happens to be just a deque and a mutex, you could make it iterable at the cost of blocking all producers and consumers (which might be fine for many uses, like debugging or exploratory programming), or provide a snapshot API to return a copy of the deque. But do you want to make that a requirement on all subclasses of Queue, and all other implementations' queue modules? Does ProirityQueue have to nondestructively iterate a heap in order? Does Jython have to use a mutex and a deque instead of a more efficient (and possibly lock-free) queue from the Java stdlib? What does multiprocessing.Queue do on each implementation? I don't think the costs are worth the benefit. And I assume that's why the existing queue API doesn't provide an iteration or snapshot mechanism. But there's an option that might be worth doing: Provide a queue.IntrospectableQueue type that _is_ defined to have such a mechanism, but to otherwise work like a Queue (except maybe less efficiently). Then provide an optional parameter for the Executor that lets you specify an alternate queue constructor in place of the default. So, when exploring or debugging, you could pass queue.IntrospectableQueue (or multiprocessing.IntrospectableQueue for ProcessPoolExecutor). Whether the interface is "lock_and_return_iterator" or "snapshot", this would be trivial to implement in CPython, and other Pythons could just copy the CPython version instead of extending their native queue types. Sent from a random iPhone On Aug 25, 2014, at 11:54, Guido van Rossum wrote: > It might be worth it to make the implementation somewhat more complicated if it serves a good purpose, for example giving the user of the program insights into how well the executor is performing. Without such insight you may be attempting to tune parameters (like the pool size) without being able to evaluate their effect. > > > On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou wrote: >> Le 25/08/2014 14:16, Ram Rachum a ?crit : >> >>> Maybe I'm missing something, but I don't think that's something that >>> should block implementation. >>> >>> Information not available? Change the executor code to make that >>> information available. >> >> Not if that would make the implementation much more complicated, or significantly slower. >> >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Mon Aug 25 23:07:18 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 26 Aug 2014 00:07:18 +0300 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> Message-ID: Sounds good to me. Having to specify `IntrospectableQueue` to the executor is a bit of a chore, but not too bad to get this functionality. I also bet that the performance difference wouldn't be an issue for most uses. On Mon, Aug 25, 2014 at 11:05 PM, 'Andrew Barnert' via python-ideas < python-ideas at googlegroups.com> wrote: > I don't think there's any issue with letting people introspect the > executor. The problem is that the main thing you get is a queue, and > there's a limit to how introspectable a queue can be. > > In particular, if you want to iterate the waiting tasks, you have to > iterate the queue, and there's no safe way to do that. > > Since CPython's queue.Queue happens to be just a deque and a mutex, you > could make it iterable at the cost of blocking all producers and consumers > (which might be fine for many uses, like debugging or exploratory > programming), or provide a snapshot API to return a copy of the deque. > > But do you want to make that a requirement on all subclasses of Queue, and > all other implementations' queue modules? Does ProirityQueue have to > nondestructively iterate a heap in order? Does Jython have to use a mutex > and a deque instead of a more efficient (and possibly lock-free) queue from > the Java stdlib? What does multiprocessing.Queue do on each implementation? > > I don't think the costs are worth the benefit. And I assume that's why the > existing queue API doesn't provide an iteration or snapshot mechanism. > > But there's an option that might be worth doing: > > Provide a queue.IntrospectableQueue type that _is_ defined to have such a > mechanism, but to otherwise work like a Queue (except maybe less > efficiently). Then provide an optional parameter for the Executor that lets > you specify an alternate queue constructor in place of the default. So, > when exploring or debugging, you could pass queue.IntrospectableQueue (or > multiprocessing.IntrospectableQueue for ProcessPoolExecutor). > > Whether the interface is "lock_and_return_iterator" or "snapshot", this > would be trivial to implement in CPython, and other Pythons could just copy > the CPython version instead of extending their native queue types. > > Sent from a random iPhone > > On Aug 25, 2014, at 11:54, Guido van Rossum wrote: > > It might be worth it to make the implementation somewhat more complicated > if it serves a good purpose, for example giving the user of the program > insights into how well the executor is performing. Without such insight you > may be attempting to tune parameters (like the pool size) without being > able to evaluate their effect. > > > On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou > wrote: > >> Le 25/08/2014 14:16, Ram Rachum a ?crit : >> >> Maybe I'm missing something, but I don't think that's something that >>> should block implementation. >>> >>> Information not available? Change the executor code to make that >>> information available. >>> >> >> Not if that would make the implementation much more complicated, or >> significantly slower. >> >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/pl3r5SsbLLU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Aug 26 01:36:53 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 25 Aug 2014 19:36:53 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On 8/25/2014 11:12 AM, Guido van Rossum wrote: > On Sun, Aug 24, 2014 at 11:18 PM, Terry Reedy > I believe co-existence is possible, but the details will depend on > the form type hints take. First, other annotations should be easily > distinguished from type hints. Second, other annotations that would > interfere with the runtime use of __annotations__ should be pulled > out by a decorator and possibly moved to another attribute specific > to the decorator (such as '_deco_name'). > > > What exactly do you mean by "other annotations that would interfere with > the runtime use of __annotations__"? I am referring to the current stdlib runtime use of .__annotations__ directly by inspect.getfullargspec and now inspect.signature and indirectly by Idle calltips. def f(n:[int, 'random non-type info'])->(int, 'more non-type info'): pass import inspect as ip print(ip.formatargspec(*ip.getfullargspec(f))) print(str(ip.signature(f))) (n: [, 'random non-type info']) -> (, 'more non-type info') (n:[, 'random non-type info']) -> (, 'more non-type info') (Idle currently displays the first, will change to the second. The display difference is more pronounced (now, at least) for C functions with .__text_signature__ from Argument Clinic.) To me, 'argument specification' and especially 'signature' imply arguments names and types, but not other stuff. Certainly, name and type is all that is wanted for a tip on how to write a call. Extra stuff would typically be noise in this context -- especially for beginners. > I can only assume that you are thinking of a situation where you are > introspecting some function/method of unknown origin and you are trying > to see if it has any annotations, in which case you are going to use > them for a locally-defined use (e.g. generate an HTML form -- contrived > example). > > It sounds as if you are worried about being passed a function that in > the past would not have any annotations (so you would just generate a > default form based on the argument names) but which now has been > annotated by a zealous programmer with type hints. And you are worried > that those type hints will confuse (perhaps crash) the form-generation code. I am worried about annotations other than type hints. Compact type hints, especially for type-opaque parameter names, should improve the usefulness of call tips. Extra noise detracts. If type hints were standardized so as to be easily recognizable, .signature could be given an option to ignore anything else. -- Terry Jan Reedy From guido at python.org Tue Aug 26 01:58:31 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 25 Aug 2014 16:58:31 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On Mon, Aug 25, 2014 at 4:36 PM, Terry Reedy wrote: > On 8/25/2014 11:12 AM, Guido van Rossum wrote: > >> On Sun, Aug 24, 2014 at 11:18 PM, Terry Reedy >> > > I believe co-existence is possible, but the details will depend on >> the form type hints take. First, other annotations should be easily >> distinguished from type hints. Second, other annotations that would >> interfere with the runtime use of __annotations__ should be pulled >> out by a decorator and possibly moved to another attribute specific >> to the decorator (such as '_deco_name'). >> >> >> What exactly do you mean by "other annotations that would interfere with >> the runtime use of __annotations__"? >> > > I am referring to the current stdlib runtime use of .__annotations__ > directly by inspect.getfullargspec and now inspect.signature and indirectly > by Idle calltips. > > def f(n:[int, 'random non-type info'])->(int, 'more non-type info'): pass > import inspect as ip > print(ip.formatargspec(*ip.getfullargspec(f))) > print(str(ip.signature(f))) > > (n: [, 'random non-type info']) -> (, 'more > non-type info') > (n:[, 'random non-type info']) -> (, 'more > non-type info') > > (Idle currently displays the first, will change to the second. The display > difference is more pronounced (now, at least) for C functions with > .__text_signature__ from Argument Clinic.) > > To me, 'argument specification' and especially 'signature' imply arguments > names and types, but not other stuff. Certainly, name and type is all that > is wanted for a tip on how to write a call. Extra stuff would typically be > noise in this context -- especially for beginners. > I see two possible concerns here -- can you clarify? (1) If a function is annotated with multiple *categories* of annotations (e.g. type hints and html form specifiers) you want to change inspect to only return the type hints. (2) If a function is annotated for a non-type-hinting category (e.g. only html form specifiers) you want to change inspect to return nothing. I hadn't thought of either of these; I was only concerned about making sure that a static type checker wouldn't be unduly confused by non-type-hinting annotations. I assume __annotations__ should always return the full expression found in the syntactic position of the annotation (though decorators can change this). > I can only assume that you are thinking of a situation where you are >> introspecting some function/method of unknown origin and you are trying >> to see if it has any annotations, in which case you are going to use >> them for a locally-defined use (e.g. generate an HTML form -- contrived >> example). >> >> It sounds as if you are worried about being passed a function that in >> the past would not have any annotations (so you would just generate a >> default form based on the argument names) but which now has been >> annotated by a zealous programmer with type hints. And you are worried >> that those type hints will confuse (perhaps crash) the form-generation >> code. >> > > I am worried about annotations other than type hints. Compact type hints, > especially for type-opaque parameter names, should improve the usefulness > of call tips. Extra noise detracts. > > If type hints were standardized so as to be easily recognizable, > .signature could be given an option to ignore anything else. > So I think you are actually not excited about any mechanism to keep other uses of annotations alive, and you would just as well only have them used for type hints? (That's fine with me, but there is the backward compatibility issue, and perhaps legitimate cool alternate uses of annotations. We still have the choice to allow only one category of annotations per function.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Aug 26 03:48:40 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 26 Aug 2014 11:48:40 +1000 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: <20140826014839.GO25957@ando> On Mon, Aug 25, 2014 at 07:36:53PM -0400, Terry Reedy wrote: > I am referring to the current stdlib runtime use of .__annotations__ > directly by inspect.getfullargspec and now inspect.signature and > indirectly by Idle calltips. > > def f(n:[int, 'random non-type info'])->(int, 'more non-type info'): pass > import inspect as ip > print(ip.formatargspec(*ip.getfullargspec(f))) > print(str(ip.signature(f))) > > (n: [, 'random non-type info']) -> (, 'more > non-type info') > (n:[, 'random non-type info']) -> (, 'more > non-type info') > > (Idle currently displays the first, will change to the second. The > display difference is more pronounced (now, at least) for C functions > with .__text_signature__ from Argument Clinic.) > > To me, 'argument specification' and especially 'signature' imply > arguments names and types, but not other stuff. I don't think that's a useful defintion for Python. There is no hard definition of "function signature" in computer science, since different languages provide different information in the function declaration. But here's a definition from the C++ world which seems to capture the gist of it to me: A function signature consists of the function prototype. What it tells you is the general information about a function, its name, parameters, what scope it is in, and other miscellaneous information. http://www.cs.unm.edu/~storm/C++/ProgrammingTerms/FunctionSignatures.html To put it another way, it's anything in the function declaration apart from the actual body of the function. In the case of Python, that includes annotations, which currently have no official semantics, but may include type hints, or documentation, or anything else. > Certainly, name and type > is all that is wanted for a tip on how to write a call. Extra stuff > would typically be noise in this context -- especially for beginners. I think that it is particularly for beginners that "extra stuff" in the form of documentation could be useful. Don't think of "random non-type info", as you state above, think of NON-random information about the parameter. The most obvious example is documentation about what the parameter represents. Here's a hypothetical example of what I might have done had annotations not been prohibited in the standard library: print(inspect.signature(statistics.pvariance)) => (data, mu:"pre-calculated mean of the data, if already known"=None) So I think that there's no good reason to decide, ahead of time, that the information in an annotation is "noise" in a tool-tip. I think the tool-tip should show whatevr information the author of the function thinks is important enough to go in the annotation. (However, the tool-tip might choose to format it a bit more nicely than it currently does. But that's a separate discussion.) > >I can only assume that you are thinking of a situation where you are > >introspecting some function/method of unknown origin and you are trying > >to see if it has any annotations, in which case you are going to use > >them for a locally-defined use (e.g. generate an HTML form -- contrived > >example). > > > >It sounds as if you are worried about being passed a function that in > >the past would not have any annotations (so you would just generate a > >default form based on the argument names) but which now has been > >annotated by a zealous programmer with type hints. And you are worried > >that those type hints will confuse (perhaps crash) the form-generation > >code. > > I am worried about annotations other than type hints. Compact type > hints, especially for type-opaque parameter names, should improve the > usefulness of call tips. Extra noise detracts. Certainly "noise" detracts, but you have no reason to assume that annotations can only be two things: type hints, or noise. In fact, I would argue that for a beginner, type hints will often be noise. Do you expect beginners to understand ABCs? If not, what are they to make of something like this? flatten(it:Iterable[Any])->Sequence[Any] -- Steven From stephen at xemacs.org Tue Aug 26 03:43:42 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 26 Aug 2014 10:43:42 +0900 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: References: Message-ID: <87a96sc9f5.fsf@uwakimon.sk.tsukuba.ac.jp> Warren Weckesser writes: > I don't agree that this change "significantly changes the semantics of > reduce". The nullifier is optional. Many programs deal with domains that seem to have nullifiers but don't. See Steven's example of floating point multiplication where the "tail" might contain Inf or NaN. You can argue that this is a "consenting adults" issue, but I consider this possibility an attractive nuisance, and I'd rather it not be in the stdlib. IMHO YMMV From cfkaran2 at gmail.com Tue Aug 26 04:12:05 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Mon, 25 Aug 2014 22:12:05 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com> Message-ID: <752DAB4F-C271-4B37-8F6D-7F76D0626086@gmail.com> On Aug 25, 2014, at 7:03 AM, Chris Angelico wrote: > On Mon, Aug 25, 2014 at 8:57 PM, Cem Karan wrote: >> from type_checker import TYPE_CHECKER >> >> def foo(a: {TYPE_CHECKER: int}): >> pass > > I still don't think this offers much benefit over > > def foo(a: int): > pass > > It's a lot wordier and the flexibility will almost never be needed. So > is that multiplexing really worth it? I'm just trying to preserve annotations as they currently are; that is, without a clearly defined meaning. Right now, I see annotations as a great place to store both documentation and type information, but without some kind of multiplexing standard you can't have both at the same time. I also suspect that if some kind of standard is in place that allows multiplexing, then other uses will be invented to take advantage of it. The other alternative is to define more attributes that functions are supposed to have, but that feels like a hack. It feels cleaner to me to store information about the arguments and return values with the arguments and return values directly, rather than somewhere else. I know that with decorators that isn't a problem, but it doesn't feel as clean to me. Thanks, Cem Karan From oreilldf at gmail.com Tue Aug 26 04:51:18 2014 From: oreilldf at gmail.com (Dan O'Reilly) Date: Mon, 25 Aug 2014 22:51:18 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> Message-ID: The IntrospectableQueue idea seems reasonable to me. I think I would prefer passing an introspectable (or similar) keyword to the Executor rather than a queue class, though. Adding support for identifying which tasks are active introduces some extra overhead, which I think can reasonably be made optional. If we're going to use a different Queue class to enable introspection, we might as well disable the other stuff that we're doing to make introspection work. It also makes it easier to raise an exception if an API is called that won't work without IntrospectableQueue being used. >> Does Jython have to use a mutex and a deque instead of a more efficient (and possibly lock-free) queue from the Java stdlib? For what it's worth, Jython just uses CPython's queue.Queue implementation, as far as I can tell. >> What does multiprocessing.Queue do on each implementation? In addition to a multiprocessing.Queue, the ProcessPoolExecutor maintains a dict of all submitted work items, so that can be used instead of trying to inspect the queue itself. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Aug 26 05:02:24 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Aug 2014 20:02:24 -0700 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> Message-ID: <53FBF8C0.3070300@stoneleaf.us> On 08/25/2014 07:51 PM, Dan O'Reilly wrote: > > The IntrospectableQueue idea seems reasonable to me. I think I would prefer passing an introspectable (or similar) > keyword to the Executor rather than a queue class, though. Passing the class is the better choice -- it means that future needs can be more easily met by designing the queue variant needed and passing it in -- having a keyword to select only one option is unnecessarily limiting. -- ~Ethan~ From oreilldf at gmail.com Tue Aug 26 05:31:21 2014 From: oreilldf at gmail.com (Dan O'Reilly) Date: Mon, 25 Aug 2014 23:31:21 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: <53FBF8C0.3070300@stoneleaf.us> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> Message-ID: In that case, what's the best way to disallow use of APIs that require an introspectable queue implementation? Using isinstance(self._work_queue, IntrospectableQueue) would work, but seems to be nearly as limiting as using an introspectable keyword. Perhaps IntrospectableQueue could support __iter__ as a way of iterating over a snapshot of enqueued items - The Executor could try iterating over the queue when it needs to inspect its contents, raising an appropriate exception (something like "Provided queue class must be introspectable") if that fails. If people prefer __iter__ isn't used for that purpose, we could just do the same thing with whatever public method ends up being used to get the snapshot instead. On Mon, Aug 25, 2014 at 11:02 PM, Ethan Furman wrote: > On 08/25/2014 07:51 PM, Dan O'Reilly wrote: > >> >> The IntrospectableQueue idea seems reasonable to me. I think I would >> prefer passing an introspectable (or similar) >> keyword to the Executor rather than a queue class, though. >> > > Passing the class is the better choice -- it means that future needs can > be more easily met by designing the queue variant needed and passing it in > -- having a keyword to select only one option is unnecessarily limiting. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Aug 26 05:38:20 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 25 Aug 2014 20:38:20 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <752DAB4F-C271-4B37-8F6D-7F76D0626086@gmail.com> References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com> <752DAB4F-C271-4B37-8F6D-7F76D0626086@gmail.com> Message-ID: On Mon, Aug 25, 2014 at 7:12 PM, Cem Karan wrote: > > On Aug 25, 2014, at 7:03 AM, Chris Angelico wrote: > > > On Mon, Aug 25, 2014 at 8:57 PM, Cem Karan wrote: > >> from type_checker import TYPE_CHECKER > >> > >> def foo(a: {TYPE_CHECKER: int}): > >> pass > > > > I still don't think this offers much benefit over > > > > def foo(a: int): > > pass > > > > It's a lot wordier and the flexibility will almost never be needed. So > > is that multiplexing really worth it? > > I'm just trying to preserve annotations as they currently are; that is, > without a clearly defined meaning. Right now, I see annotations as a great > place to store both documentation and type information, but without some > kind of multiplexing standard you can't have both at the same time. I also > suspect that if some kind of standard is in place that allows multiplexing, > then other uses will be invented to take advantage of it. > > The other alternative is to define more attributes that functions are > supposed to have, but that feels like a hack. It feels cleaner to me to > store information about the arguments and return values with the arguments > and return values directly, rather than somewhere else. I know that with > decorators that isn't a problem, but it doesn't feel as clean to me. > I appreciate your efforts, but in my mind this is a lost cause -- whatever notation you use to put multiple annotations on a single argument is just too noisy to bear, and I don't think the need is strong enough. Let's focus instead on coming up with a way to indicate to which category the annotations for a given function belong. As I've said before, I think a simple decorator should suffice. In fact, let me propose a straw-man so we have something to shoot down. The only thing we need is a way to tell the static type checker (which for the rest of this message I'm just going to abbreviate as "mypy") "these are not the annotations you are looking for." The simplest thing that could possibly work would be a single designated decorator. Let's say we import it from typing.py: from typing import no_type_checks @no_type_checks def speeder(short: 'r2d2', tall: 'c3po') -> 'luke': print('These are not the droids you are looking for') (Note that mypy uses string literals as forward class references, so without the decorator mypy would probably issue some confused complaints.) However, I think we need to be slightly more flexible. Most likely a function that uses annotations in some pre-mypy way is already decorated by a decorator that interprets the annotations. It would be annoying to have to double-decorate such functions, so I think there should be a way to decorate such decorators. Assuming such a decorator is implemented as a function, we can just decorate the decorator, as follows: from typing import no_type_checks @no_type_checks def clone_trooper(func): print('Looking for', func.__annotations__, 'in', func.__name__) func.searched = True return func @clone_trooper def speeder(short: 'r2d2', tall: 'c3po') -> 'luke': print('These are not the droids you are looking for') This is probably going to put off some people for its lack of purity -- how does mypy know that the @no_type_checks flag rubs off on anything decorated with it? In fact, how would it know that in the first example speeder() isn't a decorator? I personally think decorators are rare enough that no confusion will happen in practice, but I'll concede that perhaps it would be better to use two different decorators. Then @no_type_checks could be a simple no-op decorator which is itself decorated with e.g. @no_type_checks_decorator; both defined by typing.py. PS. In case someone asks, mypy is smart enough to follow imports, so @clone_trooper doesn't need to be defined in the same file where it is used -- I just did that to simplify the example. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Tue Aug 26 05:43:04 2014 From: antoine at python.org (Antoine Pitrou) Date: Mon, 25 Aug 2014 23:43:04 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: <53FBF8C0.3070300@stoneleaf.us> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> Message-ID: Le 25/08/2014 23:02, Ethan Furman a ?crit : > On 08/25/2014 07:51 PM, Dan O'Reilly wrote: >> >> The IntrospectableQueue idea seems reasonable to me. I think I would >> prefer passing an introspectable (or similar) >> keyword to the Executor rather than a queue class, though. > > Passing the class is the better choice -- it means that future needs can > be more easily met by designing the queue variant needed and passing it > in -- having a keyword to select only one option is unnecessarily limiting. What if an implementation wants to use something other than a queue? It seems you're breaking the abstraction here. Regards Antoine. From abarnert at yahoo.com Tue Aug 26 05:48:41 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Aug 2014 20:48:41 -0700 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> Message-ID: <1409024921.35699.YahooMailNeo@web181001.mail.ne1.yahoo.com> On Monday, August 25, 2014 7:52 PM, Dan O'Reilly wrote: >The IntrospectableQueue idea seems reasonable to me. I think I would prefer passing an introspectable (or similar) keyword to the Executor rather than a queue class, though. Adding support for identifying which tasks are active introduces some extra overhead, which I think can reasonably be made optional. If we're going to use a different Queue class to enable introspection, we might as well disable the other stuff that we're doing to make introspection work. It also makes it easier to raise an exception if an API is called that won't work without IntrospectableQueue being used. Even though this was my suggestion, let me play devil's advocate for a second? The main reason to use this is for debugging or exploratory programming.? In the debugger, of course, it's not necessary, because you can just break and suspend all the threads while you do what you want. Would it be reasonable to do the same thing outside the debugger, by providing a threading.Thread.suspend API (and of course the pool and executor APIs have a suspend method that suspends all their threads) so you can safely access the queue's internals? Obviously suspending threads in general is a bad thing to do unless you're a big fan of deadlocks, but for debugging and exploration it seems reasonable; if a program occasionally deadlocks or crashes while you're screwing with its threads to see what happens, well, you were screwing with its threads to see what happens? That might be a horrible attractive nuisance, but if you required an extra flag to be passed in at construction time to make these methods available, and documented that it was unsafe and potentially inefficient, it might be acceptable. On the other hand, it's hard to think of a case where this is a good answer but "just run it in the debugger" isn't a better answer? >>>?Does Jython have to use a mutex and a deque instead of a more efficient (and possibly lock-free) queue from the Java stdlib? > >For what it's worth, Jython just uses CPython's queue.Queue implementation, as far as I can tell. Now that I think about it, that makes sense; if I really need a lock-free thread pool and queue in Jython, I'm probably going to use the native Java executors, not the Python ones, right? >>>?What does multiprocessing.Queue do on each implementation? > >In addition to a multiprocessing.Queue, the ProcessPoolExecutor maintains a dict of all submitted work items, so that can be used instead of trying to inspect the queue itself. Interesting. This implies that supplying an inspectable queue class may not be the best answer here; instead, we could have an option for an inspectable work dict, which would just expose the existing one for ProcessPoolExecutor, while it would make ThreadPoolExecutor maintain an equivalent dict as a thread-local in the launching thread.?(I'm assuming you only need to inspect the jobs from the launching process/thread here? I'm not sure if that's sufficient for the OP's intended use or not.) From abarnert at yahoo.com Tue Aug 26 05:56:30 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Aug 2014 20:56:30 -0700 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> Message-ID: <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> On Monday, August 25, 2014 8:44 PM, Antoine Pitrou wrote: > > Le 25/08/2014 23:02, Ethan Furman a ?crit : >> On 08/25/2014 07:51 PM, Dan O'Reilly wrote: >>> >>> The IntrospectableQueue idea seems reasonable to me. I think I would >>> prefer passing an introspectable (or similar) >>> keyword to the Executor rather than a queue class, though. >> >> Passing the class is the better choice -- it means that future needs can >> be more easily met by designing the queue variant needed and passing it >> in -- having a keyword to select only one option is unnecessarily limiting. > > What if an implementation wants to use something other than a queue? > It seems you're breaking the abstraction here. A collection of threads and a shared queue is almost the definition of a thread pool. What else would you use? Also, this could make it a lot easier to create variations on ThreadPoolExecutor without subclassing or forking it. For example, if you want your tasks to run in priority order, just give it a priority queue keyed on task.priority. If you want a scheduled executor, just give it a priority queue whose get method blocks until the first task's task.timestamp or a new task is added ahead of the first. And so on. I'm not sure if that's a good idea or not, but it's an interesting possibility at least? From guido at python.org Tue Aug 26 06:08:17 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 25 Aug 2014 21:08:17 -0700 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: Hm. Maybe we should not complicate the API after all. This seems a lot of theorizing without enough of a use case. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Tue Aug 26 06:09:24 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Tue, 26 Aug 2014 00:09:24 -0400 Subject: [Python-ideas] Add nullifier argument to functools.reduce? In-Reply-To: <87a96sc9f5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87a96sc9f5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Aug 25, 2014 at 9:43 PM, Stephen J. Turnbull wrote: > Warren Weckesser writes: > > > I don't agree that this change "significantly changes the semantics of > > reduce". The nullifier is optional. > > Many programs deal with domains that seem to have nullifiers but don't. > See Steven's example of floating point multiplication where the "tail" > might contain Inf or NaN. You can argue that this is a "consenting > adults" issue, but I consider this possibility an attractive nuisance, > and I'd rather it not be in the stdlib. > > Gotcha. To spell it out: one might naively think 0 is a nullifier for floating point multiplication, but 0*inf is nan, not 0. That means reduce(lambda x,y: x*y, [1e-50]*10 + [float('inf')], nullifier=0), which underflows to 0 during the intermediate steps, would incorrectly return 0. So yeah, consenting adults, know your data, etc. :) (nan, on the other hand, appears to be a true nullifier for the product in IEEE 754 floating point arithmetic.) Warren IMHO YMMV > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Tue Aug 26 06:13:53 2014 From: antoine at python.org (Antoine Pitrou) Date: Tue, 26 Aug 2014 00:13:53 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: Le 25/08/2014 23:56, Andrew Barnert a ?crit : >> >> What if an implementation wants to use something other than a queue? >> It seems you're breaking the abstraction here. > > A collection of threads and a shared queue is almost the definition of a thread pool. What else would you use? Definitions don't necessarily have any relationship with the way a feature is implemented. Perhaps some version of concurrent.futures would like to use some advanced dispatch mechanism provided by the OS (or shared memory, or whatever). (I'll note that such "flexibility" has been chosen for the API of threading.Condition and it is making it difficult to write an optimized implementation that would you use OS-native facilities, such as pthread condition variables) We have come from a simple proposal to introspect some runtime properties of an executor to the idea of swapping out a building block with another. That doesn't sound reasonable. Regards Antoine. From ethan at stoneleaf.us Tue Aug 26 06:22:38 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Aug 2014 21:22:38 -0700 Subject: [Python-ideas] Executor Docs [was: Re: Add an introspection API to Executor] In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <53FC0B8E.1010309@stoneleaf.us> On 08/25/2014 09:08 PM, Guido van Rossum wrote: > > Hm. Maybe we should not complicate the API after all. This seems a lot of theorizing without enough of a use case. I went in search of docs to see what the API actually was, and while I know the source code is a great place to go look for education and finer points, should we have to go looking there just to see what the __init__ parameters are? I'm going to go out on a limb and say that ThreadPoolExecutor takes a max_workers param, but I only have that because it's in the example. On the up side, having a link to the source is really cool. Having clicked on that I now know that max_workers is the only param taken. ;) -- ~Ethan~ From ethan at stoneleaf.us Tue Aug 26 06:27:37 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Aug 2014 21:27:37 -0700 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> Message-ID: <53FC0CB9.1070202@stoneleaf.us> On 08/25/2014 09:08 PM, Guido van Rossum wrote: > > Hm. Maybe we should not complicate the API after all. This seems a lot of theorizing without enough of a use case. Introspection (aka debugging) is an important use case. Having looked at the code, and with Antoine's comments in mind, I'd be happy with whatever Dan can get in there without changing the queuing implementation -- if anyone needs that much flexibility, they can take the Python code and massage it to their own desires. -- ~Ethan~ From antoine at python.org Tue Aug 26 06:37:52 2014 From: antoine at python.org (Antoine Pitrou) Date: Tue, 26 Aug 2014 00:37:52 -0400 Subject: [Python-ideas] Executor Docs [was: Re: Add an introspection API to Executor] In-Reply-To: <53FC0B8E.1010309@stoneleaf.us> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> <53FC0B8E.1010309@stoneleaf.us> Message-ID: Le 26/08/2014 00:22, Ethan Furman a ?crit : > > I went in search of docs to see what the API actually was, and while I > know the source code is a great place to go look for education and finer > points, should we have to go looking there just to see what the __init__ > parameters are? So, you didn't find the docs? https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor """ class concurrent.futures.ThreadPoolExecutor(max_workers) An Executor subclass that uses a pool of at most max_workers threads to execute calls asynchronously. """ https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor """ class concurrent.futures.ProcessPoolExecutor(max_workers=None) An Executor subclass that executes calls asynchronously using a pool of at most max_workers processes. If max_workers is None or not given, it will default to the number of processors on the machine. """ Regards Antoine. From tjreedy at udel.edu Tue Aug 26 07:42:31 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Aug 2014 01:42:31 -0400 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations Message-ID: On 8/25/2014 7:58 PM, Guido van Rossum wrote: > On Mon, Aug 25, 2014 at 4:36 PM, Terry Reedy > wrote: > I am referring to the current stdlib runtime use of .__annotations__ > directly by inspect.getfullargspec and now inspect.signature and > indirectly by Idle calltips. > > def f(n:[int, 'random non-type info'])->(int, 'more non-type info'): > pass > import inspect as ip > print(ip.formatargspec(*ip.__getfullargspec(f))) > print(str(ip.signature(f))) > > (n: [, 'random non-type info']) -> (, > 'more non-type info') > (n:[, 'random non-type info']) -> (, 'more > non-type info') > > (Idle currently displays the first, will change to the second. The > display difference is more pronounced (now, at least) for C > functions with .__text_signature__ from Argument Clinic.) > > To me, 'argument specification' and especially 'signature' imply > arguments names and types, but not other stuff. Certainly, name and > type is all that is wanted for a tip on how to write a call. Extra > stuff would typically be noise in this context -- especially for > beginners. > > > I see two possible concerns here -- can you clarify? > > (1) If a function is annotated with multiple *categories* of annotations > (e.g. type hints and html form specifiers) you want to change inspect to > only return the type hints. > > (2) If a function is annotated for a non-type-hinting category (e.g. > only html form specifiers) you want to change inspect to return nothing. Both are covered by "I would either like non-type-hinting annotations moved elsewhere by decorators (my older idea) *or* I would like inspect.signature to at least optionally ignore them (today's idea). The latter is more dependable since we control the inspect functions. I an not too worried that anyone would use the blessed type-hint syntax for something completely different. On the other hand, I can imagine that there might someday be more that one annotation system in general use. > I hadn't thought of either of these; I was only concerned about making > sure that a static type checker wouldn't be unduly confused by > non-type-hinting annotations. If, for instance, everything in typing or produced by the function is a subclass of BaseTyping or an instance of MetaTyping, then post-compile typing objects would be easy to distinguish. Source annotations are a harder problem. >I assume __annotations__ should always > return the full expression found in the syntactic position of the > annotation I think that should continue. > (though decorators can change this). Because .__annotations__ is a regular dict. annotation syntax in code --compiler>> initial .__annotation__ dict --decorators>> exposed .__annotation__ dict + any other attributes added. > I am worried about annotations other than type hints. Compact type > hints, especially for type-opaque parameter names, should improve > the usefulness of call tips. Extra noise detracts. > > If type hints were standardized so as to be easily recognizable, > .signature could be given an option to ignore anything else. > > So I think you are actually not excited about any mechanism to keep > other uses of annotations alive, I want there to be such a mechanism even though it is does not 'excite' me . Just as you are focused on type hints for static hints, I, *as an Idle maintainer* who just happens to have spent considerable time on the call tip feature, am focused on type hints for dynamic documentation. > and you would just as well only have them used for type hints? Too strong. > (That's fine with me, but there is the backward compatibility issue, I said in my first response that I don't think we should break code. This is me speaking a as core developer and community member. I do want multiple ramifications of annotation use considered, including run-time use. > and perhaps legitimate cool alternate uses of annotations. Most experiments do not amount to much, but a few, such as the metaclass hack, certainly have. Python should continue to allow experiments. > We still have the choice to allow only one category of > annotations per function.) This seems dubious to me. If strings, tuples, or dicts are allowed as annotations, it seems to me that they can be used to combine annotations. -- Terry Jan Reedy From jlehtosalo at gmail.com Tue Aug 26 07:47:14 2014 From: jlehtosalo at gmail.com (Jukka Lehtosalo) Date: Mon, 25 Aug 2014 22:47:14 -0700 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: On Mon, Aug 25, 2014 at 10:26 AM, Guido van Rossum wrote: > On Mon, Aug 25, 2014 at 6:57 AM, Antoine Pitrou > wrote: > >> PEP 3107 doesn't say anything about that (or rather, it says that >> annotations can be looked up on the function, but function objects only >> exist at run-time). Actually, the bytecode isn't very practical to work >> with to extract annotations, >> > I don't know at which point mypy changed goals (if it has), but there are >> still signs in the website of the goal of building a separate runtime (not >> necessarily a separate syntax), e.g. >> >> """Also some language features that are evaluated at runtime in Python >> may happen during compilation in mypy when using the native semantics. >> For example, mypy base classes may be bound during compilation (or >> program loading, before evaluation), unlike Python.""" >> > > I have talked to Jukka about this, and he is definitely on board with > reducing mypy's functionality to that of a linter; I think there's even an > issue in mypy's tracker about removing the ability to execute the code > (currently there's a flag you must pass to prevent it from trying to run > the code). Updating the website is an ongoing low-priority project (you can > probably send him pull requests for it). > Yeah, the website wasn't very up-to-date. The project goals have shifted a lot in the last 18 months. I just updated the tutorial and the FAQ to not talk about the native semantics. > mypy complains about the unannotated initialization of sqs. The two ways > to currently address this are > > sqs = [] # type: List[float] > > or > > sqs = List[float]() > > Neither is very attractive; mypy should just infer the type of sqs. > Agreed, but this is very difficult to do in all possible contexts. For example, __init__ methods often use an empty list as an attribute initializer, and here it's more tricky to infer the item type, as we'd have to look into other methods to figure out the item type. > > Nevertheless, I don't want to use call syntax to set parameters for > generic types, since a generic type still "feels" sufficiently like a class > that calling it is easily confused with instantiation -- even though ABCs > are typically not instantiable. (There's no hard rule for that though -- it > is merely the result of typical ABCs having at least one abstract method.) > Generic types don't need to be abstract, and for those the call syntax would be ambiguous (and confusing). Jukka -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Aug 26 08:12:08 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Aug 2014 23:12:08 -0700 Subject: [Python-ideas] Executor Docs [was: Re: Add an introspection API to Executor] In-Reply-To: References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> <53FC0B8E.1010309@stoneleaf.us> Message-ID: <53FC2538.4010109@stoneleaf.us> On 08/25/2014 09:37 PM, Antoine Pitrou wrote: > Le 26/08/2014 00:22, Ethan Furman a ?crit : >> >> I went in search of docs to see what the API actually was, and while I >> know the source code is a great place to go look for education and finer >> points, should we have to go looking there just to see what the __init__ >> parameters are? > > So, you didn't find the docs? > > https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor > > """ > class concurrent.futures.ThreadPoolExecutor(max_workers) > > An Executor subclass that uses a pool of at most max_workers threads to execute calls asynchronously. > """ > > https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor > > """ > class concurrent.futures.ProcessPoolExecutor(max_workers=None) > > An Executor subclass that executes calls asynchronously using a pool of at most max_workers processes. If > max_workers is None or not given, it will default to the number of processors on the machine. > """ I did find the docs, and even with your plain text guide I almost didn't see them when I looked just now. Too much fancy going on there, and all the green examples -- yeah, it's hard to read. For comparison, here's what help(ThreadPoolExecutor) shows: class ThreadPoolExecutor(concurrent.futures._base.Executor) | Method resolution order: | ThreadPoolExecutor | concurrent.futures._base.Executor | builtins.object | | Methods defined here: | | __init__(self, max_workers) | Initializes a new ThreadPoolExecutor instance. | | Args: | max_workers: The maximum number of threads that can be used to | execute the given calls. | | shutdown(self, wait=True) | Clean-up the resources associated with the Executor. | | It is safe to call this method several times. Otherwise, no other | methods can be called after this one. | | Args: | wait: If True then shutdown will not return until all running | futures have finished executing and the resources used by the | executor have been reclaimed. | | submit(self, fn, *args, **kwargs) | Submits a callable to be executed with the given arguments. | | Schedules the callable to be executed as fn(*args, **kwargs) and returns | a Future instance representing the execution of the callable. | | Returns: | A Future representing the given call. Much easier to understand. Looking at the docs again, I think the biggest hurdle to finding that line and recognizing it for what it is is the fact that it comes /after/ all the examples. That's backwards. Why would you need examples for something you haven't read yet? -- ~Ethan~ From ncoghlan at gmail.com Tue Aug 26 11:49:48 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Aug 2014 19:49:48 +1000 Subject: [Python-ideas] Executor Docs [was: Re: Add an introspection API to Executor] In-Reply-To: <53FC2538.4010109@stoneleaf.us> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> <53FC0B8E.1010309@stoneleaf.us> <53FC2538.4010109@stoneleaf.us> Message-ID: On 26 Aug 2014 16:12, "Ethan Furman" wrote: > Looking at the docs again, I think the biggest hurdle to finding that line and recognizing it for what it is is the fact that it comes /after/ all the examples. That's backwards. Why would you need examples for something you haven't read yet? Many of our module docs serve a dual purpose as a tutorial *and* as an API reference. That's actually a problem, and often a sign of a separate "HOWTO" guide trying to get out. Actually doing the work to split them is rather tedious though, so it tends not to happen very often. Cheers, Nick. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaiser.yann at gmail.com Tue Aug 26 12:49:02 2014 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Tue, 26 Aug 2014 12:49:02 +0200 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> Message-ID: Given / assuming that most annotated functions will only concern themselves with one use of annotations (either typing, web forms, cli parameters), maybe inspect could be extended to have an annotation-filtering function. Given a namespace(could be any python object), its protocol regarding annotations for each parameter could be: * If the annotation is a mapping(c.abc.Mapping): assume the namespace object is a key of that mapping. If the key is not present, return None for that annotation. * If the annotation is not a mapping, it is always returned whole. The implications would be: * A stdlib-approved way of namespacing annotations would exist. Annotations for different introspectors can coexist. * inspect.signature's purpose remains unchanged and still tells the whole truth * Most annotated functions, typing-related or not are unchanged. (except those that consist of a mapping ?) In Python: import inspect import collections.abc def filter_annotation(annotation, namespace): if isinstance(annotation, collections.abc.Mapping): return annotation.get(namespace, inspect.Signature.empty) return annotation def filter_annotations(signature, namespace): params = [ p.replace(annotation=filter_annotation(p.annotation, namespace)) for p in signature.parameters.values() ] ret = filter_annotation(signature.return_annotation, namespace) return signature.replace(parameters=params, return_annotation=ret) >>> def classic(param:'annot'): pass ... >>> cliparser = object() >>> typing = object() >>> def namespaced(*, param:{typing:int, cliparser:'p'}): pass ... >>> s_classic = inspect.signature(classic) >>> s_namespaced = inspect.signature(namespaced) >>> print(filter_annotations(s_classic, cliparser)) (param:'annot') >>> print(filter_annotations(s_classic, typing)) (param:'annot') >>> print(filter_annotations(s_namespaced, cliparser)) (*, param:'p') >>> print(filter_annotations(s_namespaced, typing)) (*, param:int) On 26 August 2014 07:47, Jukka Lehtosalo wrote: > On Mon, Aug 25, 2014 at 10:26 AM, Guido van Rossum > wrote: > >> On Mon, Aug 25, 2014 at 6:57 AM, Antoine Pitrou >> wrote: >> >>> PEP 3107 doesn't say anything about that (or rather, it says that >>> annotations can be looked up on the function, but function objects only >>> exist at run-time). Actually, the bytecode isn't very practical to work >>> with to extract annotations, >>> >> I don't know at which point mypy changed goals (if it has), but there are >>> still signs in the website of the goal of building a separate runtime (not >>> necessarily a separate syntax), e.g. >>> >>> >>> """Also some language features that are evaluated at runtime in Python >>> may happen during compilation in mypy when using the native semantics. >>> For example, mypy base classes may be bound during compilation (or >>> program loading, before evaluation), unlike Python.""" >>> >> >> I have talked to Jukka about this, and he is definitely on board with >> reducing mypy's functionality to that of a linter; I think there's even an >> issue in mypy's tracker about removing the ability to execute the code >> (currently there's a flag you must pass to prevent it from trying to run >> the code). Updating the website is an ongoing low-priority project (you can >> probably send him pull requests for it). >> > > Yeah, the website wasn't very up-to-date. The project goals have shifted a > lot in the last 18 months. I just updated the tutorial and the FAQ to not > talk about the native semantics. > > >> mypy complains about the unannotated initialization of sqs. The two ways >> to currently address this are >> >> sqs = [] # type: List[float] >> >> or >> >> sqs = List[float]() >> >> Neither is very attractive; mypy should just infer the type of sqs. >> > > Agreed, but this is very difficult to do in all possible contexts. For > example, __init__ methods often use an empty list as an attribute > initializer, and here it's more tricky to infer the item type, as we'd have > to look into other methods to figure out the item type. > > >> >> Nevertheless, I don't want to use call syntax to set parameters for >> generic types, since a generic type still "feels" sufficiently like a class >> that calling it is easily confused with instantiation -- even though ABCs >> are typically not instantiable. (There's no hard rule for that though -- it >> is merely the result of typical ABCs having at least one abstract method.) >> > > Generic types don't need to be abstract, and for those the call syntax > would be ambiguous (and confusing). > > Jukka > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oreilldf at gmail.com Wed Aug 27 03:59:24 2014 From: oreilldf at gmail.com (Dan O'Reilly) Date: Tue, 26 Aug 2014 21:59:24 -0400 Subject: [Python-ideas] Add an introspection API to Executor In-Reply-To: <53FC0CB9.1070202@stoneleaf.us> References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com> <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com> <53FBF8C0.3070300@stoneleaf.us> <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com> <53FC0CB9.1070202@stoneleaf.us> Message-ID: As promised, I've opened issue22281 (http://bugs.python.org/issue22281), and attached a patch that makes an attempt at implementing this. Let's continue any further discussion on this topic there. On Tue, Aug 26, 2014 at 12:27 AM, Ethan Furman wrote: > On 08/25/2014 09:08 PM, Guido van Rossum wrote: > >> >> Hm. Maybe we should not complicate the API after all. This seems a lot of >> theorizing without enough of a use case. >> > > Introspection (aka debugging) is an important use case. Having looked at > the code, and with Antoine's comments in mind, I'd be happy with whatever > Dan can get in there without changing the queuing implementation -- if > anyone needs that much flexibility, they can take the Python code and > massage it to their own desires. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony at lownds.com Wed Aug 27 19:20:18 2014 From: tony at lownds.com (Tony Lownds) Date: Wed, 27 Aug 2014 17:20:18 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Proposal=3A_Use_mypy_syntax_for_function?= =?utf-8?q?=09annotations?= References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> <20140823051357.GX25957@ando> <20140823172536.GZ25957@ando> <20140825013023.GK25957@ando> <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com> <752DAB4F-C271-4B37-8F6D-7F76D0626086@gmail.com> Message-ID: Guido van Rossum writes: > from typing import no_type_checks > > @no_type_checks > def speeder(short: 'r2d2', tall: 'c3po') -> 'luke': > ??? print('These are not the droids you are looking for') > > @no_type_checks > def clone_trooper(func): > ??? print('Looking for', func.__annotations__, 'in', func.__name__) > ??? func.searched = True > ??? return func > > @clone_trooper > def speeder(short: 'r2d2', tall: 'c3po') -> 'luke': >??? print('These are not the droids you are looking for') > Can I propose a slight modification -- the designated decorator takes another decorator, in the decoration line: @non_typing_annotation(None) def speeder(short: 'r2d2', tall: 'c3po') -> 'luke': ??? print('These are not the droids you are looking for') @non_typing_annotation(clone_trooper) def speeder(short: 'r2d2', tall: 'c3po') -> 'luke': ?? print('These are not the droids you are looking for') This way the library defining `clone_trooper` doesn't have to change to support the typing improvements, and the reader always knows whether the annotations will be treated as type hints without needing to consult another libraries' source. Also, I'd like to suggest that mypy should use the import of 'typing' as an indicator that the module's annotations are in fact type hints. -Tony From burak.arslan at arskom.com.tr Thu Aug 28 08:36:28 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Thu, 28 Aug 2014 09:36:28 +0300 Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations In-Reply-To: <20140823013646.GV25957@ando> References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io> <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp> <20140823013646.GV25957@ando> Message-ID: <53FECDEC.4030709@arskom.com.tr> On 08/23/14 04:36, Steven D'Aprano wrote: > But another reason for disliking call syntax for this is that it gets > confusing: > > def function(param:Sequence(Spam, Eggs))->Mapping(Cheese, Toast): > > suffers compared to this: > > def function(param:Sequence[Spam, Eggs])->Mapping[Cheese, Toast]: > > In the second case, the type annotations stand out more easily because > they use square brackets instead of round. > You're only giving examples with builtins. What about: def function(param:SomeClass(Spam, Eggs))->SomeOtherClass(Cheese, Toast): If there are template instantiations (in the C++ sense) there that's another meaning for parens which already have two meanings. And now they're also context-dependent!.. At least they were all the same everywhere before. So SomeClass[Spam, Eggs] is a fabulous idea -- in my mind it's just a special way of looking up a certain type. But as I said before[1] I'd hate to see annotations only be used for type checking. I'd love to be able to do: Integer[ge=5] but this comes with the question about what should actually passed to __getitem__ I'd avoid changing its signature to accept **kwargs. So I'd say that Unicode[15, pattern='[0-9a-z]+'] should translate to: {0: 15, 'pattern': '[0-9a-z]+'} which is a frozen ordered dict. As that's actually a better tuple that also happens to accept string keys, I doubt that'd interfere with the existing usage. We need this for duck typing to work as well. To my understanding, the type marker "File" means "only accept what's returned by open()". If however, we want to accept any file-like object, we could use Object[write=Callable]. Or maybe more specifically: Object[write=Callable[Bytes]]. > -1 on re-using call syntax: it is more confusing to read inside a > parameter list, too flexible, and clashes with existing use calling > types. > > +1 on subscripting syntax: it is more easily distinguishable from > calling, just flexible enough, and doesn't clash with existing > functionality. > Well, we need that flexibility at the very least for making duck typing work with type-annotated functions. best, burak [1]: https://mail.python.org/pipermail/python-ideas/2014-August/028988.html [2]: https://github.com/arskom/spyne/pull/313#issuecomment-29029067 From steve at pearwood.info Fri Aug 29 07:45:27 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 29 Aug 2014 15:45:27 +1000 Subject: [Python-ideas] Query available codecs and error handlers Message-ID: <20140829054527.GC23931@ando> Unless I am badly misinformed, there is no way to programmatically check what codecs and error handlers are available. I propose two functions in the codecs module: get_codecs() get_error_handlers() which each return a set of the available codec, or error handler, names. Use-cases --------- (1) At the interactive interpreter, it is often useful to experiment with different codecs. Being able to get a list of available codecs avoids the frustration of having to look them up elsewhere, or guess. (2) Applications which offer to read or write text files may wish to allow the user to select an encoding. With no obvious way of finding out what encodings are available, the application is limited to hard-coding some, and hoping that they don't get renamed or removed. Why sets, rather than lists? ---------------------------- Sets emphasis that the names are in no particular order. Since the names should always be strings and therefore hashable, and the order is irrelevant, there is no particular need for a list. Should they be frozensets rather than sets? ------------------------------------------- I don't care :-) Using sets may make it easier for the application to filter the results in some way, but I really don't think it matters. Thoughts or comments? -- Steven From rosuav at gmail.com Fri Aug 29 09:27:19 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 29 Aug 2014 17:27:19 +1000 Subject: [Python-ideas] Query available codecs and error handlers In-Reply-To: <20140829054527.GC23931@ando> References: <20140829054527.GC23931@ando> Message-ID: On Fri, Aug 29, 2014 at 3:45 PM, Steven D'Aprano wrote: > Unless I am badly misinformed, there is no way to programmatically check > what codecs and error handlers are available. I propose two functions in > the codecs module: > > get_codecs() > get_error_handlers() > > which each return a set of the available codec, or error handler, names. A quick look at the codecs module suggests that this may not be as simple as returning a list/set; when _codecs.lookup() is called, it does a search. So this is actually like asking how to list importable modules. It may not be possible, but if it is, I would suggest calling it "list_codecs" or something, as it's basically going to be tracing the search path and enumerating every codec it finds. But it sounds like a quite useful facility. +1. ChrisA From mal at egenix.com Fri Aug 29 09:33:47 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 29 Aug 2014 09:33:47 +0200 Subject: [Python-ideas] Query available codecs and error handlers In-Reply-To: <20140829054527.GC23931@ando> References: <20140829054527.GC23931@ando> Message-ID: <54002CDB.9060704@egenix.com> On 29.08.2014 07:45, Steven D'Aprano wrote: > Unless I am badly misinformed, there is no way to programmatically check > what codecs and error handlers are available. I propose two functions in > the codecs module: > > get_codecs() > get_error_handlers() > > which each return a set of the available codec, or error handler, names. Question is: how would you implement these ? The codec registry uses lookup functions to find codecs, so we'd have to extend those lookup functions to also support reporting known installed codecs. For the stdlib encodings package we could simply put a list into the package, e.g. encodings.available_codecs returning a dictionary of mappings from codec name to CodecInfo tuples and then extend the CodecInfo with some extra information such as supported error handlers, alternative names and information about the supported input/output types. At the moment, the available codecs are documented here: https://docs.python.org/3.5/library/codecs.html?highlight=codecs#standard-encodings It's probably a good idea to add information about supported error handlers to that list. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 29 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ... 21 days to go 2014-09-27: PyDDF Sprint 2014 ... 29 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Fri Aug 29 09:49:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 29 Aug 2014 17:49:07 +1000 Subject: [Python-ideas] Query available codecs and error handlers In-Reply-To: <54002CDB.9060704@egenix.com> References: <20140829054527.GC23931@ando> <54002CDB.9060704@egenix.com> Message-ID: On 29 August 2014 17:33, M.-A. Lemburg wrote: > On 29.08.2014 07:45, Steven D'Aprano wrote: >> Unless I am badly misinformed, there is no way to programmatically check >> what codecs and error handlers are available. I propose two functions in >> the codecs module: >> >> get_codecs() >> get_error_handlers() >> >> which each return a set of the available codec, or error handler, names. > > Question is: how would you implement these ? > > The codec registry uses lookup functions to find codecs, so we'd > have to extend those lookup functions to also support reporting > known installed codecs. I'd actually be a fan of a PEP to add such an introspection API that also made it easier to register new codecs just by adding them to a suitable namespace package. I believe MvL's original idea was to use the existing encodings package for that, but that doesn't seem feasible due to the non-empty __init__ > For the stdlib encodings package we could simply put a list into > the package, e.g. encodings.available_codecs returning a dictionary > of mappings from codec name to CodecInfo tuples and then extend > the CodecInfo with some extra information such as supported > error handlers, alternative names and information about the > supported input/output types. > > At the moment, the available codecs are documented here: > > https://docs.python.org/3.5/library/codecs.html?highlight=codecs#standard-encodings > > It's probably a good idea to add information about supported > error handlers to that list. Those tables are already pretty busy though - I'm not sure how we could add supported error handler details without making them hard to read. Agreed it would be good to make the info more readily available, though (I had actually hoped to get some proposed revisions together for the codecs module docs before 3.4 went out the door, but alas, it was not to be). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Fri Aug 29 09:50:51 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 29 Aug 2014 17:50:51 +1000 Subject: [Python-ideas] Query available codecs and error handlers In-Reply-To: References: <20140829054527.GC23931@ando> Message-ID: <20140829075050.GE23931@ando> On Fri, Aug 29, 2014 at 05:27:19PM +1000, Chris Angelico wrote: > On Fri, Aug 29, 2014 at 3:45 PM, Steven D'Aprano wrote: > > Unless I am badly misinformed, there is no way to programmatically check > > what codecs and error handlers are available. I propose two functions in > > the codecs module: > > > > get_codecs() > > get_error_handlers() > > > > which each return a set of the available codec, or error handler, names. > > A quick look at the codecs module suggests that this may not be as > simple as returning a list/set; when _codecs.lookup() is called, it > does a search. Sure, but codecs have to be registered before they can be used. The register function can cache the names. Perhaps any builtin codecs might need to be explicitly added to the cache in order to support this, I don't know the details of _codecs. If the codec registry were a dict, we might even return a view of the dict.keys(). I'm sure there is some solution which is not quite as difficult as enumerating all importable modules. There's a lot fewer codecs, they don't have platform-dependent suffixes, and unlike modules in the PYTHONPATH, once Python starts up the available codecs won't change unless register() is called. -- Steven From p.f.moore at gmail.com Fri Aug 29 09:51:00 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 29 Aug 2014 08:51:00 +0100 Subject: [Python-ideas] Subprocess: Add an encoding argument Message-ID: At the moment, subprocess offers two options for handlingthe standard IO streams of the child. By default, the streams are binary, or you can set universal_newlines to get text-mode streams with universal newline handling enabled. With universal_newlines, the encoding of the streams is the default value for the environment (whatever locale.getpreferredencoding() returns). However, there can be cases where you want finer control over the encoding to use (for example, if you run Python in a subprocess and set PYTHONIOENCODING). I propose adding an "encoding" parameter to subprocess.Popen (and the various wrapper routines) to allow specifying the actual encoding to use. Obviously, you can simply wrap the binary streams yourself - the main use for this facility would be in the higher level functions like check_output and communicate. Does this seem like a reasonable suggestion? Paul From steve at pearwood.info Fri Aug 29 10:16:15 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 29 Aug 2014 18:16:15 +1000 Subject: [Python-ideas] Query available codecs and error handlers In-Reply-To: <54002CDB.9060704@egenix.com> References: <20140829054527.GC23931@ando> <54002CDB.9060704@egenix.com> Message-ID: <20140829081615.GF23931@ando> On Fri, Aug 29, 2014 at 09:33:47AM +0200, M.-A. Lemburg wrote: > On 29.08.2014 07:45, Steven D'Aprano wrote: > > Unless I am badly misinformed, there is no way to programmatically check > > what codecs and error handlers are available. I propose two functions in > > the codecs module: > > > > get_codecs() > > get_error_handlers() > > > > which each return a set of the available codec, or error handler, names. > > Question is: how would you implement these ? > > The codec registry uses lookup functions to find codecs, so we'd > have to extend those lookup functions to also support reporting > known installed codecs. Arrgggh, you're right -- I've been working on a wrong assumption. I had forgotten that the codecs.register function takes a function, not a name. I always forgot that, because it's such a strange and unhelpful API compared to, say, a mapping of name:codec. It's probably water under the bridge now, but is there any documentation for why this API was used in the first place? -- Steven From mal at egenix.com Fri Aug 29 10:30:55 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 29 Aug 2014 10:30:55 +0200 Subject: [Python-ideas] Query available codecs and error handlers In-Reply-To: <20140829081615.GF23931@ando> References: <20140829054527.GC23931@ando> <54002CDB.9060704@egenix.com> <20140829081615.GF23931@ando> Message-ID: <54003A3F.2050408@egenix.com> On 29.08.2014 10:16, Steven D'Aprano wrote: > On Fri, Aug 29, 2014 at 09:33:47AM +0200, M.-A. Lemburg wrote: >> On 29.08.2014 07:45, Steven D'Aprano wrote: >>> Unless I am badly misinformed, there is no way to programmatically check >>> what codecs and error handlers are available. I propose two functions in >>> the codecs module: >>> >>> get_codecs() >>> get_error_handlers() >>> >>> which each return a set of the available codec, or error handler, names. >> >> Question is: how would you implement these ? >> >> The codec registry uses lookup functions to find codecs, so we'd >> have to extend those lookup functions to also support reporting >> known installed codecs. > > Arrgggh, you're right -- I've been working on a wrong assumption. I had > forgotten that the codecs.register function takes a function, not a > name. > > I always forgot that, because it's such a strange and unhelpful API > compared to, say, a mapping of name:codec. It's probably water under the > bridge now, but is there any documentation for why this API was used in > the first place? Because at the time I designed the API in 1999/2000 it wasn't clear how people would start writing codecs. Note that codecs do not use a simple name to codec mapping to figure out the implementation module name. The name typically goes through a few layers of normalization and then a alias dictionary to find the name of the implementation. The lookup functions were meant to implement these more complex n-1 mappings. I also thought that codec implementation might want to tap into system registries of codecs, use file based tables as basis for encodings, or even implement load on demand. Today, it's rather obvious that apparently no one has considered doing any of this, so it would have been better to design a system where you explicitly register individual codecs (together with a set of attributes). It should be possible to phase out the lookup API and expose the encodings package lookup mechanism directly in the codecs module. I can help guide people, if they are willing to do the work, but don't have time to work on this myself. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 29 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ... 21 days to go 2014-09-27: PyDDF Sprint 2014 ... 29 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Fri Aug 29 10:39:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 29 Aug 2014 18:39:35 +1000 Subject: [Python-ideas] Subprocess: Add an encoding argument In-Reply-To: References: Message-ID: On 29 Aug 2014 17:52, "Paul Moore" wrote: > > At the moment, subprocess offers two options for handlingthe standard > IO streams of the child. By default, the streams are binary, or you > can set universal_newlines to get text-mode streams with universal > newline handling enabled. > > With universal_newlines, the encoding of the streams is the default > value for the environment (whatever locale.getpreferredencoding() > returns). However, there can be cases where you want finer control > over the encoding to use (for example, if you run Python in a > subprocess and set PYTHONIOENCODING). > > I propose adding an "encoding" parameter to subprocess.Popen (and the > various wrapper routines) to allow specifying the actual encoding to > use. > > Obviously, you can simply wrap the binary streams yourself - the main > use for this facility would be in the higher level functions like > check_output and communicate. > > Does this seem like a reasonable suggestion? This actually gets a little messy once you start digging into it, as you actually have up to 3 streams to deal with (stdin, stdout, stderr), and may want to set the error handler in addition to the encoding. http://bugs.python.org/issue6135 has the many gory details. It's a problem that definitely needs solving, but it may be better approached by making it easier to create the pipes *separately*, and then pass relevant details into the subprocess call. As with win_unicode_console (and even contextlib2), it's probably worth experimenting in a PyPI module, as that will make it easier for people to try out with existing Python 3 releases. Cheers, Nick. > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Aug 29 10:44:57 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 29 Aug 2014 10:44:57 +0200 Subject: [Python-ideas] Query available codecs and error handlers In-Reply-To: References: <20140829054527.GC23931@ando> <54002CDB.9060704@egenix.com> Message-ID: <54003D89.10606@egenix.com> On 29.08.2014 09:49, Nick Coghlan wrote: > On 29 August 2014 17:33, M.-A. Lemburg wrote: >> On 29.08.2014 07:45, Steven D'Aprano wrote: >>> Unless I am badly misinformed, there is no way to programmatically check >>> what codecs and error handlers are available. I propose two functions in >>> the codecs module: >>> >>> get_codecs() >>> get_error_handlers() >>> >>> which each return a set of the available codec, or error handler, names. >> >> Question is: how would you implement these ? >> >> The codec registry uses lookup functions to find codecs, so we'd >> have to extend those lookup functions to also support reporting >> known installed codecs. > > I'd actually be a fan of a PEP to add such an introspection API that > also made it easier to register new codecs just by adding them to a > suitable namespace package. I believe MvL's original idea was to use > the existing encodings package for that, but that doesn't seem > feasible due to the non-empty __init__ It's fairly easy to have the lookup function in the encodings package to also look in say a "siteencodings" package for codecs it cannot find in the stdlib encodings package. This new siteencodings package could then be setup as namespace package to make installation easier. That doesn't answer the original question, though, since introspection of available codecs would still not be possible. >> For the stdlib encodings package we could simply put a list into >> the package, e.g. encodings.available_codecs returning a dictionary >> of mappings from codec name to CodecInfo tuples and then extend >> the CodecInfo with some extra information such as supported >> error handlers, alternative names and information about the >> supported input/output types. >> >> At the moment, the available codecs are documented here: >> >> https://docs.python.org/3.5/library/codecs.html?highlight=codecs#standard-encodings >> >> It's probably a good idea to add information about supported >> error handlers to that list. > > Those tables are already pretty busy though - I'm not sure how we > could add supported error handler details without making them hard to > read. Here's one idea: don't use a table, but instead have one subsection per codec. There are already a few subsections for specific codecs on the page. > Agreed it would be good to make the info more readily available, > though (I had actually hoped to get some proposed revisions together > for the codecs module docs before 3.4 went out the door, but alas, it > was not to be). Since it's not a feature, the doc change could potentially be backported. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 29 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ... 21 days to go 2014-09-27: PyDDF Sprint 2014 ... 29 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Fri Aug 29 11:23:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 29 Aug 2014 19:23:07 +1000 Subject: [Python-ideas] Query available codecs and error handlers In-Reply-To: <54003D89.10606@egenix.com> References: <20140829054527.GC23931@ando> <54002CDB.9060704@egenix.com> <54003D89.10606@egenix.com> Message-ID: On 29 August 2014 18:44, M.-A. Lemburg wrote: > On 29.08.2014 09:49, Nick Coghlan wrote: >> Agreed it would be good to make the info more readily available, >> though (I had actually hoped to get some proposed revisions together >> for the codecs module docs before 3.4 went out the door, but alas, it >> was not to be). > > Since it's not a feature, the doc change could potentially > be backported. Right, I think that's probably worth doing. The tricky part is figuring out *how* to change them. At the moment, there isn't an especially clear distinction between the text encoding specific parts (e.g. several of the error handlers, most of the encodings) and the underlying general purpose codecs machinery. However, beyond a vague notion of "that distinction should be made clearer in the docs now that it affects which codecs can be used with str.encode, bytes.decode and bytearray.decode, and which can *only* be used with codecs.encode and codecs.decode", I don't have any specific suggestions to make. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From zedobject at gmail.com Fri Aug 29 21:12:56 2014 From: zedobject at gmail.com (Milind Khadilkar) Date: Sat, 30 Aug 2014 00:42:56 +0530 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. Message-ID: Hi! I do appreciate that indentation in Python code shows intention, and I am all for it, but problems do arise if code is copied from one place to another, even from specialized Python editors.... and there is little you can do to recover intentions/indentations lost in the process. Besides, spaces become difficult to count and distinguish from tabs, adding to the problem. (Why allow tabs at all?) Why can't Python be made to accept the following indentation of code (in *addition* to the current schemes)? There could be some flexibility of choosing the indentation character. (Pipe was mentioned as an alternative in another thread) for i in range(10): .if i>5: ..print 10-i .else: ..print i print "!" If not in the interpreter for some reason, could it be "advised" as a mode in a python oriented editor, or a "dictated" mechanism for copy/paste/transfer of code? Sorry for posing an elementary question. Thanks and regards MK-zedobject -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Fri Aug 29 21:24:59 2014 From: skip at pobox.com (Skip Montanaro) Date: Fri, 29 Aug 2014 14:24:59 -0500 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. In-Reply-To: References: Message-ID: On Fri, Aug 29, 2014 at 2:12 PM, Milind Khadilkar wrote: > I do appreciate that indentation in Python code shows intention, and I am > all for it, but problems do arise if code is copied from one place to > another, even from specialized Python editors.... and there is little you > can do to recover intentions/indentations lost in the process. It's a known issue. I wouldn't necessarily call it a problem. I don't know if there are good strategies for tackling that other than "don't do that." If you find yourself copying around blocks of text which don't correspond to whole functions or classes, it tells me you might have some refactoring to do. Tools like pylint can help identify common chunks of code between two or more files. I don't know what support there is in existing editors or IDEs for code refactoring. There used to be Bicycle Repair Man, but I think he's long gone: http://bicyclerepair.sourceforge.net/ There is also a library called ROPE: http://rope.sourceforge.net/ Never heard of it before. Don't know if it's currently supported (it at least has Py3K support). Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Fri Aug 29 21:40:02 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Fri, 29 Aug 2014 20:40:02 +0100 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. In-Reply-To: References: Message-ID: On 29/08/2014 20:12, Milind Khadilkar wrote: > Hi! > I do appreciate that indentation in Python code shows intention, and I > am all for it, but problems do arise if code is copied from one place to > another, even from specialized Python editors.... and there is little > you can do to recover intentions/indentations lost in the process. > Besides, spaces become difficult to count and distinguish from tabs, > adding to the problem. (Why allow tabs at all?) > > Why can't Python be made to accept the following indentation of code (in > *addition* to the current schemes)? There could be some flexibility of > choosing the indentation character. (Pipe was mentioned as an > alternative in another thread) > > for i in range(10): > .if i>5: > ..print 10-i > .else: > ..print i > print "!" > > If not in the interpreter for some reason, could it be "advised" as a > mode in a python oriented editor, or a "dictated" mechanism for > copy/paste/transfer of code? > > Sorry for posing an elementary question. > > Thanks and regards > MK-zedobject > It strikes me that these "problems" has existed for 23 years and somehow people have survived. Could it be a classic example of a bad workman always blames his tools? See Skip Montanaro's reply for an explanation as to why. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From abarnert at yahoo.com Fri Aug 29 23:04:24 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 29 Aug 2014 14:04:24 -0700 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. In-Reply-To: References: Message-ID: <539EE1FB-7384-451F-8BE1-247FB892D28F@yahoo.com> On Aug 29, 2014, at 12:12, Milind Khadilkar wrote: > Hi! > I do appreciate that indentation in Python code shows intention, and I am all for it, but problems do arise if code is copied from one place to another, even from specialized Python editors.... and there is little you can do to recover intentions/indentations lost in the process. I copy and paste code between editors all the time--e.g., from emacs to the graphical editors on sites like Github or StackOverflow--and I can't remember the last time I've had this problem, except when copying someone else's code that mixed spaces and tabs (and even in that case, it more often helps me find existing invisible errors than it causes errors). > Besides, spaces become difficult to count and distinguish from tabs, adding to the problem. (Why allow tabs at all?) That one I could get behind, but it's been suggested and rejected enough times to have a PEP assigned to it. > Why can't Python be made to accept the following indentation of code (in addition to the current schemes)? There could be some flexibility of choosing the indentation character. (Pipe was mentioned as an alternative in another thread) > > for i in range(10): > .if i>5: > ..print 10-i > .else: > ..print i > print "!" The first problem is trying to come up with a syntax that isn't ambiguous to the parser or to a human. You haven't succeeded there: .2 Is that the int literal `2` indented, or the float literal `.2` in the left column? Even the pipe character had this problem: |2 That pipe could mean the or operator if the previous line ended with a continuation character. But even discounting that, the tokenizer and any human experienced with Python will read it as the or operator and then have to follow some rule saying "an or operator at the start of a line that isn't part of a valid expression counts as a space". And that means guessing that it wasn't intended as an or operator, possibly turning what should have been identified as a simple SyntaxError into something baffling. I suppose you could suggest bringing back ` for this purpose, but I doubt anyone will like that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat Aug 30 07:32:07 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 30 Aug 2014 14:32:07 +0900 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. In-Reply-To: References: Message-ID: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp> Milind Khadilkar writes: > Besides, spaces become difficult to count and distinguish from > tabs, adding to the problem. (Why allow tabs at all?) Working code should not be broken gratuitously. Sure, it was an unfortunate decision in the first place, but we can't change that original decision now. > Why can't Python be made to accept the following indentation of > code (in *addition* to the current schemes)? Because we *already have it*: the character is ' '. If *you* follow that rule, you won't have problems copy/pasting your own code into well-behaved code. If you're working with somebody else's code which mixes tabs and spaces, *using an alternative character doesn't help*, whether the "bad" code is source or target. From zedobject at gmail.com Sat Aug 30 08:19:41 2014 From: zedobject at gmail.com (Milind Khadilkar) Date: Sat, 30 Aug 2014 11:49:41 +0530 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. In-Reply-To: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Thanks. The problem is mostly when code from different people using different tools needs to be combined, or sample code from the internet (not from the well-behaved ones like github and stackoverflow) has to be tested and understood. Of course it is fun to discover the lost intentions of the original programmer, but it is not practical in a day job. QUOTE Because we *already have it*: the character is ' '. UNQUOTE The problem with ' ' is that a sequence of ' 's can't easily be counted, especially on feature rich editors. A sequence of '.' or some visible character can be counted. Of course, with the previous line being continued on the current line, '.' could present problems, as Andrew B. points out, and they can be messy. Could the solution lie in providing a "transfer" or "import-export" mode in editors where the indentation is given in terms of a chosen special character which is converted to spaces/tabs? Thanks again MK-Zedobject On Sat, Aug 30, 2014 at 11:02 AM, Stephen J. Turnbull wrote: > Milind Khadilkar writes: > > > Besides, spaces become difficult to count and distinguish from > > tabs, adding to the problem. (Why allow tabs at all?) > > Working code should not be broken gratuitously. Sure, it was an > unfortunate decision in the first place, but we can't change that > original decision now. > > > Why can't Python be made to accept the following indentation of > > code (in *addition* to the current schemes)? > > Because we *already have it*: the character is ' '. If *you* follow > that rule, you won't have problems copy/pasting your own code into > well-behaved code. If you're working with somebody else's code which > mixes tabs and spaces, *using an alternative character doesn't help*, > whether the "bad" code is source or target. > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sat Aug 30 10:02:40 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 30 Aug 2014 18:02:40 +1000 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. References: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <85ha0uxv4v.fsf@benfinney.id.au> Milind Khadilkar writes: > The problem with ' ' is that a sequence of ' 's can't easily be > counted, especially on feature rich editors. You have a ?feature rich? editor which is unable to count a sequence of characters? I don't think that qualifies as ?feature rich? for purposes of programming. Also, your proposal then only seems to be valid if you have an editor which: * is capable of the highly specific task of ?convert Python code between ? ? for indentation and ?.? for indentation?; but * is so feature-poor that it *is not* capable of the generally-useful task ?count a sequence of characters?. I think that's a text editor so bizarre we can discount it as irrelevant until someone is demonstrated to be using it seriously and can't be convinced to use a better editor. -- \ ?If you ever catch on fire, try to avoid seeing yourself in the | `\ mirror, because I bet that's what REALLY throws you into a | _o__) panic.? ?Jack Handey | Ben Finney From steve at pearwood.info Sat Aug 30 11:55:07 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 30 Aug 2014 19:55:07 +1000 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. In-Reply-To: References: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140830095507.GN23931@ando> On Sat, Aug 30, 2014 at 11:49:41AM +0530, Milind Khadilkar wrote: > Thanks. > The problem is mostly when code from different people using different tools > needs to be combined, or sample code from the internet (not from the > well-behaved ones like github and stackoverflow) has to be tested and > understood. > Of course it is fun to discover the lost intentions of the original > programmer, but it is not practical in a day job. Are you suggesting that indentation is the *only* reason to test and understand arbitrary code you copy and paste from the Internet? I don't that's a reasonable argument to make. Whether it is your day job, or just a hobby, you should have some understanding of what the code is supposed to do before pasting it into your own work. And there is so much more than just indentation that you have to be concerned about: the imports, the variables it uses, which functions it calls, can the code be trusted. Indentation is the least of these. I accept that there are occasional situations where we might want to copy code from a website or email, and the indentation has been lost. But that ought to be rare. In my personal experience, I've needed to reconstruct the indentation from scratch perhaps three or four times in the last ten years. Even if I'm off by a factor of ten, let's call it three or four times a year. That's still not very important. Worrying that this is "not practical" in a day job seems to be worrying over nothing. > QUOTE > Because we *already have it*: the character is ' '. > UNQUOTE > The problem with ' ' is that a sequence of ' 's can't easily be counted, > especially on feature rich editors. > A sequence of '.' or some visible character can be counted. I don't think that's a reasonable argument. I trust you're not using Notepad, are you? In a bare-bones editor, it's hard to count *any* long sequence of characters. Quick, how many dots before the X? ..................X If you are a professional programmer, or even just a serious amateur, you should be using professional tools. With professional tools, you rarely need to *count the spaces*. You just need to notice changes to the indent level: one level another level different level You don't need to care precisely how many spaces (or tabs) are there, you just need to ensure things line up. And any decent programming editor will give you the tools to increase or decrease indentation over a block of lines, without caring about the specific number of spaces or counting exactly how many spaces are needed. That includes a wide range of IDEs and editors of all sorts of power, on many different platforms: Notepad++ or geany on Windows, kate or kwrite on KDE, vim and emacs for pretty much any Unix or Linux system, to mention only a few. Which brings us to another problem with your suggestion. Professional editors already know about indentation with tabs, and spaces, but I don't know any editor which gives you the ability to indent with arbitrary characters. (Although I daresay somebody would soon write a macro for Emacs to do that.) Which means you are swapping from a system where professional-quality programming tools can do the counting for you, to a situation where you actually do need to count the dots yourself. And that's a big step backwards. I don't think this change is practical, or useful. And it's ugly. -- Steven From solipsis at pitrou.net Sat Aug 30 13:07:29 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 30 Aug 2014 13:07:29 +0200 Subject: [Python-ideas] Subprocess: Add an encoding argument References: Message-ID: <20140830130729.6e8b67ec@fsol> On Fri, 29 Aug 2014 18:39:35 +1000 Nick Coghlan wrote: > > This actually gets a little messy once you start digging into it, as you > actually have up to 3 streams to deal with (stdin, stdout, stderr), and may > want to set the error handler in addition to the encoding. At this point, I'd suggest creating a clean, separate TextPopen class (or subclass) without any legacy argument baggage. As for per-stream settings, we could allow passing each of *encoding* and *errors* in two forms: - as a string, in which case it applies to all three pipes - as a dict, in which case it is looked up for each of the "stdin", "stdout", "stderr" pipes (if configured as pipes) We could then deprecate the bogus-named "universal_newlines" in the main Popen class. Regards Antoine. From p.f.moore at gmail.com Sat Aug 30 14:05:02 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 30 Aug 2014 13:05:02 +0100 Subject: [Python-ideas] Subprocess: Add an encoding argument In-Reply-To: <20140830130729.6e8b67ec@fsol> References: <20140830130729.6e8b67ec@fsol> Message-ID: On 30 August 2014 12:07, Antoine Pitrou wrote: > On Fri, 29 Aug 2014 18:39:35 +1000 > Nick Coghlan wrote: >> >> This actually gets a little messy once you start digging into it, as you >> actually have up to 3 streams to deal with (stdin, stdout, stderr), and may >> want to set the error handler in addition to the encoding. > > At this point, I'd suggest creating a clean, separate TextPopen class > (or subclass) without any legacy argument baggage. > > As for per-stream settings, we could allow passing each of *encoding* > and *errors* in two forms: > - as a string, in which case it applies to all three pipes > - as a dict, in which case it is looked up for each of the "stdin", > "stdout", "stderr" pipes (if configured as pipes) > > We could then deprecate the bogus-named "universal_newlines" in the > main Popen class. Sounds reasonable. I'll look into that (no promises on timescales :-)) In practice, I doubt we'd need per-stram encodings particularly often, so I like the idea of *not* clutteringthe API to cope with them. I'm curious, by the way - what arguments do you consider as "legacy baggage" (a lot of them seem to me to be OS-specific and/or specialised rather than legacy). In practice, we'd probably need to do something about the utility functions like check_output and communicate as well. Paul From solipsis at pitrou.net Sat Aug 30 14:10:52 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 30 Aug 2014 14:10:52 +0200 Subject: [Python-ideas] Subprocess: Add an encoding argument References: <20140830130729.6e8b67ec@fsol> Message-ID: <20140830141052.35226537@fsol> On Sat, 30 Aug 2014 13:05:02 +0100 Paul Moore wrote: > > Sounds reasonable. I'll look into that (no promises on timescales :-)) > In practice, I doubt we'd need per-stram encodings particularly often, > so I like the idea of *not* clutteringthe API to cope with them. I'm > curious, by the way - what arguments do you consider as "legacy > baggage" (a lot of them seem to me to be OS-specific and/or > specialised rather than legacy). I was thinking mostly about universal_newlines. Perhaps preexec_fn applies too, since it's dangerous (read: unstable). Regards Antoine. From moritz.beber at gmail.com Sat Aug 30 14:37:10 2014 From: moritz.beber at gmail.com (Moritz Beber) Date: Sat, 30 Aug 2014 14:37:10 +0200 Subject: [Python-ideas] Subprocess: Add an encoding argument In-Reply-To: <20140830141052.35226537@fsol> References: <20140830130729.6e8b67ec@fsol> <20140830141052.35226537@fsol> Message-ID: Hi, On Sat, Aug 30, 2014 at 2:10 PM, Antoine Pitrou wrote: > On Sat, 30 Aug 2014 13:05:02 +0100 > Paul Moore wrote: > > > > Sounds reasonable. I'll look into that (no promises on timescales :-)) > > In practice, I doubt we'd need per-stram encodings particularly often, > > so I like the idea of *not* clutteringthe API to cope with them. I'm > > curious, by the way - what arguments do you consider as "legacy > > baggage" (a lot of them seem to me to be OS-specific and/or > > specialised rather than legacy). > > I was thinking mostly about universal_newlines. Perhaps preexec_fn > applies too, since it's dangerous (read: unstable). > > > preexec_fn is important though if you want to run something with different uid and gid from a sudo script, for example. Cheers, Moritz -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sat Aug 30 17:36:56 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 30 Aug 2014 10:36:56 -0500 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. In-Reply-To: References: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 08/30/2014 01:19 AM, Milind Khadilkar wrote: > Thanks. > The problem is mostly when code from different people using different tools > needs to be combined, or sample code from the internet (not from the > well-behaved ones like github and stackoverflow) has to be tested and > understood. > Of course it is fun to discover the lost intentions of the original > programmer, but it is not practical in a day job. > QUOTE > Because we *already have it*: the character is ' '. > UNQUOTE > The problem with ' ' is that a sequence of ' 's can't easily be counted, > especially on feature rich editors. > A sequence of '.' or some visible character can be counted. > > Of course, with the previous line being continued on the current line, '.' > could present problems, as Andrew B. points out, and they can be messy. > Could the solution lie in providing a "transfer" or "import-export" mode in > editors where the indentation is given in terms of a chosen special > character which is converted to spaces/tabs? > Thanks again Unfortunately there probably isn't much you can do about uses of tabs or spaces outside your immediate team. If they are all are part of your team, probably the best approach is to agree to set your editors to hi-light tabs in an obvious way, and also set them to use spaces in place of tabs. Over time you should see less uses of tabs and it won't be as much of an issue. As for cutting and pasting lines, you will still probably need to reindent anyway. For that I use the indent, dedent capability of the editor if it has one, or a different editor that has that, if it doesn't. It's one of my must have requirements. I don't think an explicit indent character would help much, but possibly a relative indent marker might be useful for putting multi-line code on a single line. Currently we have the ";" which only keeps the current indentation. It probably wouldn't be that hard to add ";+", and ";-", symbols (or something equivalent), but I think it would need it's own pep and discussion if anyone cares enough about it. Probably the biggest issue is "is it needed?", so it would need some convincing examples of how it would be useful. -Ron From tjreedy at udel.edu Sat Aug 30 21:58:00 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 30 Aug 2014 15:58:00 -0400 Subject: [Python-ideas] Indentation, again: Problems with moving and combining code across editors. In-Reply-To: References: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 8/30/2014 11:36 AM, Ron Adam wrote: > Unfortunately there probably isn't much you can do about uses of tabs or > spaces outside your immediate team. Except to convert to local standard. > If they are all are part of your > team, probably the best approach is to agree to set your editors to > hi-light tabs in an obvious way, Hmmm. Idle does not have an option to do that, but it would be useful. -- Terry Jan Reedy From ncoghlan at gmail.com Sun Aug 31 01:30:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 31 Aug 2014 09:30:32 +1000 Subject: [Python-ideas] Subprocess: Add an encoding argument In-Reply-To: <20140830130729.6e8b67ec@fsol> References: <20140830130729.6e8b67ec@fsol> Message-ID: On 30 Aug 2014 21:08, "Antoine Pitrou" wrote: > > On Fri, 29 Aug 2014 18:39:35 +1000 > Nick Coghlan wrote: > > > > This actually gets a little messy once you start digging into it, as you > > actually have up to 3 streams to deal with (stdin, stdout, stderr), and may > > want to set the error handler in addition to the encoding. > > At this point, I'd suggest creating a clean, separate TextPopen class > (or subclass) without any legacy argument baggage. > > As for per-stream settings, we could allow passing each of *encoding* > and *errors* in two forms: > - as a string, in which case it applies to all three pipes > - as a dict, in which case it is looked up for each of the "stdin", > "stdout", "stderr" pipes (if configured as pipes) > > We could then deprecate the bogus-named "universal_newlines" in the > main Popen class. That sounds like a plausible plan to me - splitting the API like that also reflects what was needed to get IO working sensibly in the first place. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbfox at gmail.com Sun Aug 31 18:04:30 2014 From: jsbfox at gmail.com (Thomas Allen) Date: Sun, 31 Aug 2014 20:04:30 +0400 Subject: [Python-ideas] List available tests without running them Message-ID: I'd like to propose the ability to list all available unit tests without executing them by providing a specific command line option (-l/--list for instance). python3 -m unittest discover -ls . would list all tests in current directory. Motivation & possible implementation: http://stackoverflow.com/q/24478727/2301450