From guido at python.org Thu Sep 1 00:05:59 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 31 Aug 2016 21:05:59 -0700 Subject: [Python-ideas] real numbers with SI scale factors: next steps In-Reply-To: <22471.42809.452598.705168@turnbull.sk.tsukuba.ac.jp> References: <20160830203427.GE2363@kundert.designers-guide.com> <20160831020551.GN26300@ando.pearwood.info> <20160831040801.GF2363@kundert.designers-guide.com> <1472665380.41376.711754849.48190D2E@webmail.messagingengine.com> <1472665844.45040.711818297.6A0BCBBB@webmail.messagingengine.com> <22471.42809.452598.705168@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Aug 31, 2016 at 8:57 PM, Stephen J. Turnbull wrote: > Random832 writes: > > > Also, interesting quirk - it always rounds up. 1025 bytes is "1.1K", and > > in SI mode, 1001 bytes is "1.1k" > > That seems to be right approach: in system administration, these > numbers are used mostly to understand resource usage, and > underestimates are almost never what you want, while quite large > overestimates are tolerable, and are typically limited because the > actual precision of calculations is much higher than that of the > "human-readable" output. That would seem to apply to "space used" but not to "space available". -- --Guido van Rossum (python.org/~guido) From python-ideas at shalmirane.com Thu Sep 1 01:08:48 2016 From: python-ideas at shalmirane.com (Ken Kundert) Date: Wed, 31 Aug 2016 22:08:48 -0700 Subject: [Python-ideas] real numbers with SI scale factors: next steps In-Reply-To: References: <20160830203427.GE2363@kundert.designers-guide.com> <20160831020551.GN26300@ando.pearwood.info> <20160831040801.GF2363@kundert.designers-guide.com> Message-ID: <20160901050848.GA3380@kundert.designers-guide.com> All, Armed with all of your requirements, suggestions and good ideas, I believe I am ready to try to put something together. Thank you all, and once again let me apologize for 'all the drama'. I'll let you know when I have something. -Ken From shane at hathawaymix.org Thu Sep 1 01:24:33 2016 From: shane at hathawaymix.org (Shane Hathaway) Date: Wed, 31 Aug 2016 23:24:33 -0600 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> Message-ID: On 08/31/2016 07:25 PM, Guido van Rossum wrote: > On Wed, Aug 31, 2016 at 2:46 PM, Shane Hathaway wrote: > [...] >> I'd like to suggest a small change to the Python parser that would make long >> expressions read better: >> >> rows = dbsession.query(Table1) ... >> .join( >> Table2, Table2.y = Table1.y) >> .filter(Table1.x = xx) >> .all() >> >> [...] > (And no, this isn't equivalent to using '\'.) Exactly. > Would this be enforced in the grammar or by the lexer? Since you say > you expect the indentation to be enforced, that suggests it would be > done by the grammar, but then the question is how you would modify the > grammar? You could take the rule that says an expression can be > followed by ".NAME" and extended it to also allow "... INDENT xxxxx > DEDENT" where the xxxxx is whatever's allowed at ".NAME" (i.e. ".NAME" > followed by other tails like "(......)" or "[.......]". > > But then you could only use this new idea for chaining method calls, > and not for spreading other large expressions across multiple lines. Yes, I was hoping the enhancement might be useful for more than just chaining method calls; if possible, the ellipsis should be allowed between any expression tokens. I'll study the grammar and see if my idea fits cleanly somehow. Thanks! Shane From greg.ewing at canterbury.ac.nz Thu Sep 1 01:50:07 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Sep 2016 17:50:07 +1200 Subject: [Python-ideas] real numbers with SI scale factors: next steps In-Reply-To: <20160831121406.GR26300@ando.pearwood.info> References: <20160830203427.GE2363@kundert.designers-guide.com> <20160831020551.GN26300@ando.pearwood.info> <20160831040801.GF2363@kundert.designers-guide.com> <20160831121406.GR26300@ando.pearwood.info> Message-ID: <57C7C18F.7070202@canterbury.ac.nz> Steven D'Aprano wrote: > On Tue, Aug 30, 2016 at 09:08:01PM -0700, Ken Kundert wrote: > >> My thinking was that r stands for real like f stands for float. The next available letter in the e, f, g sequence would be 'h'. If you want it to stand for something, it could be "human-readable" or "human-oriented". (There's a precedent for this in the "df" unix utility which has a -H option producing SI prefixes.) > I'm talking about chosing between "M" or "mega". The actual unit > itself is up to the caller to supply. Maybe 'h' for abbreviations and 'H' for full prefixes? -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 1 02:17:50 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Sep 2016 18:17:50 +1200 Subject: [Python-ideas] real numbers with SI scale factors: next steps In-Reply-To: <1472665380.41376.711754849.48190D2E@webmail.messagingengine.com> References: <20160830203427.GE2363@kundert.designers-guide.com> <20160831020551.GN26300@ando.pearwood.info> <20160831040801.GF2363@kundert.designers-guide.com> <1472665380.41376.711754849.48190D2E@webmail.messagingengine.com> Message-ID: <57C7C80E.8060708@canterbury.ac.nz> Random832 wrote: > One thing to consider is that this is very likely to be used with a unit > (e.g. "%hA" intending to display in amperes), so maybe it should put a > space after it? Though really people are probably going to want "1 A" vs > "1 kA" in that case, rather than "1 A" vs "1kA". I don't think a space should be automatic. The typographical recommendation is to put a thin non-breaking space between the value and the unit, but this is not possible with a monospaced font, so some people might decide that it's better without a space, or they might want to use a character other than 0x20. Better to let the user put the space in the format string if wanted. > Engineering or SI-scale-factor format suggests a third > possibility: number of decimal places to be shown after the displayed > decimal point, e.g. "%.1h" % 1.2345 * 10 ** x for x in range(10): "1.2", > "12.3", "123.5", "1.2k", "12.3k", "123.5k", "1.2M", "12.3M", "123.5M". I'm inclined to think it should be the number of significant digits, not decimal places, to give a more consistent precision as the magnitude of the number changes. For example, if you're displaying some resistor values that are accurate to 2 digits, you would want to see 2.7k, 27k, 270k, but not 27.0k or 270.0k as those would suggest spurious precision. This would also help with fitting the value into a fixed width, since you would know that a precision of n would use at most n+1 characters for the numeric part. -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 1 02:30:25 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Sep 2016 18:30:25 +1200 Subject: [Python-ideas] real numbers with SI scale factors: next steps In-Reply-To: <87h9a0olrn.fsf@thinkpad.rath.org> References: <20160830203427.GE2363@kundert.designers-guide.com> <20160831020551.GN26300@ando.pearwood.info> <20160831040801.GF2363@kundert.designers-guide.com> <87h9a0olrn.fsf@thinkpad.rath.org> Message-ID: <57C7CB01.9090803@canterbury.ac.nz> Nikolaus Rath wrote: > There's also the important nitpick if 32e7 is best rendered as 320 M or > 0.32 G. There's valid applications for both. If you want 0.32 G it's probably because you're showing it alongside other values >= 1 G, so you're really getting into the business of letting the user choose the prefix. The default should be 320 M, I think. (Unless it's a capacitor value, where there's a long-standing convention in some circles to use uF or pF, but never nF. :-) -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 1 02:40:19 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Sep 2016 18:40:19 +1200 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> Message-ID: <57C7CD53.9070805@canterbury.ac.nz> Guido van Rossum wrote: > Would this be enforced in the grammar or by the lexer? Since you say > you expect the indentation to be enforced, that suggests it would be > done by the grammar, I think it could be done by having the lexer enter a mode where it swallows a newline that is followed by an indentation to a level greater than the starting level of the construct. Then no change would be needed to the grammar. -- Greg From liik.joonas at gmail.com Thu Sep 1 03:43:17 2016 From: liik.joonas at gmail.com (Joonas Liik) Date: Thu, 1 Sep 2016 10:43:17 +0300 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <57C7CD53.9070805@canterbury.ac.nz> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <57C7CD53.9070805@canterbury.ac.nz> Message-ID: On 1 September 2016 at 09:40, Greg Ewing wrote: > Guido van Rossum wrote: >> >> Would this be enforced in the grammar or by the lexer? Since you say >> you expect the indentation to be enforced, that suggests it would be >> done by the grammar, > > > I think it could be done by having the lexer enter a mode > where it swallows a newline that is followed by an indentation > to a level greater than the starting level of the construct. > Then no change would be needed to the grammar. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ Not sure if this is a good idea but it might also make some sense to in stead have an operator at the beginning of the line for example some languages have a chainging operator for method calls: my_object.m1() ..m2() being equivalent to my_object.m1() my_object.m2() It could also be possible to have a special swallow-the-preceding-newline operator or something of that effect. side note: the chainging operator does not work on the return value of the method thus the method no longer has to choose between returning useful info or `this` for chaining convenience. From sjoerdjob at sjoerdjob.com Thu Sep 1 04:10:32 2016 From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus) Date: Thu, 1 Sep 2016 10:10:32 +0200 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <57C7CD53.9070805@canterbury.ac.nz> Message-ID: <20160901081032.GM6830@sjoerdjob.com> On Thu, Sep 01, 2016 at 10:43:17AM +0300, Joonas Liik wrote: > Not sure if this is a good idea but it might also make some sense to > in stead have an operator at the beginning of the line > > for example some languages have a chainging operator for method calls: > > my_object.m1() > ..m2() > being equivalent to > my_object.m1() > my_object.m2() > > It could also be possible to have a special > swallow-the-preceding-newline operator or something of that effect. > > side note: the chainging operator does not work on the return value of > the method thus the method no longer has to choose between returning > useful info or `this` for chaining convenience. Not sure if I would appreciate this. Most of the cases where I do chaining method calls, it's the return value that's being chained on. Of course that is logical, because that's what's currently possible. Also, would it be called 'call forking' instead of 'call chaining'...? But also, what does this mean: my_object.value.m1() ..m2() is that my_object.value.m1() my_object.value.m2() or my_object.value.m1() my_object.m2() I do think the syntax you suggest is readable, I just think the semantics is confusing and ambiguous in non-trivial examples. And the extra '.' to signify it is chaining is not really a syntactical necessity I think. What I think might be a neat idea is to do the following: if: - we have an 'unexpected indent', and - the line starts with a '.' then: - interpret the physical line as continuation of the previous line. In any current Python version this is a syntax error. In a new Python version it could be *the* obvious way to do method chaining. inactive_admins = User.objects .filter(is_staff=True) .exclude(last_login__gt=three_weeks_ago) In current versions of Python, you have to add a `\` to the end, in a possible future version of Python, you could leave out the `\`. To be fair, I think this only makes sense for when the next line starts with a `.`. In PEP8 we could add a rule about 'aligning the . with the last dot on the previous line' (or an indent of 4 if the previous line has no dot). Even though I think it would make for a good enhancement to the Python language, I can not currently estimate how much of a change to the Python parser this would need to be. Is it a three-hour straight-forward job, or a three-month bug-prone assignment? Regards, Sjoerd Job From greg.ewing at canterbury.ac.nz Thu Sep 1 02:22:53 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Sep 2016 18:22:53 +1200 Subject: [Python-ideas] real numbers with SI scale factors: next steps In-Reply-To: References: <20160830203427.GE2363@kundert.designers-guide.com> <20160831020551.GN26300@ando.pearwood.info> <20160831040801.GF2363@kundert.designers-guide.com> <233afe30-c9b1-5f89-8839-fd73fb48023a@mrabarnett.plus.com> Message-ID: <57C7C93D.6050607@canterbury.ac.nz> >>On 2016-08-31 17:19, Guido van Rossum wrote: >> >>>I guess we need to debate what it should do if the value is >>>way out of range of the SI scale system -- what's it going to do when >>>I pass it 1e50? I propose that it should fall back to 'g' style then, >>>but use "engineering" style where exponents are always a multiple of >>>3.) An alternative would be to use the largest or smallest scale factor available, and use e format to make up the difference. Not sure whether that would be better or worse. -- Greg From liik.joonas at gmail.com Thu Sep 1 06:38:19 2016 From: liik.joonas at gmail.com (Joonas Liik) Date: Thu, 1 Sep 2016 13:38:19 +0300 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <20160901081032.GM6830@sjoerdjob.com> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <57C7CD53.9070805@canterbury.ac.nz> <20160901081032.GM6830@sjoerdjob.com> Message-ID: On 1 September 2016 at 11:10, Sjoerd Job Postmus wrote: > On Thu, Sep 01, 2016 at 10:43:17AM +0300, Joonas Liik wrote: >> Not sure if this is a good idea but it might also make some sense to >> in stead have an operator at the beginning of the line >> >> for example some languages have a chainging operator for method calls: >> >> my_object.m1() >> ..m2() >> being equivalent to >> my_object.m1() >> my_object.m2() >> >> It could also be possible to have a special >> swallow-the-preceding-newline operator or something of that effect. >> >> side note: the chainging operator does not work on the return value of >> the method thus the method no longer has to choose between returning >> useful info or `this` for chaining convenience. > > Not sure if I would appreciate this. Most of the cases where I do > chaining method calls, it's the return value that's being chained on. Of > course that is logical, because that's what's currently possible. Also, > would it be called 'call forking' instead of 'call chaining'...? > > But also, what does this mean: > > my_object.value.m1() > ..m2() > > is that > > my_object.value.m1() > my_object.value.m2() > > or > > my_object.value.m1() > my_object.m2() > > > I do think the syntax you suggest is readable, I just think the > semantics is confusing and ambiguous in non-trivial examples. And the > extra '.' to signify it is chaining is not really a syntactical > necessity I think. > > What I think might be a neat idea is to do the following: > > if: > - we have an 'unexpected indent', and > - the line starts with a '.' > > then: > - interpret the physical line as continuation of the previous line. > > > In any current Python version this is a syntax error. In a new Python > version it could be *the* obvious way to do method chaining. > > inactive_admins = User.objects > .filter(is_staff=True) > .exclude(last_login__gt=three_weeks_ago) > > In current versions of Python, you have to add a `\` to the end, in a > possible future version of Python, you could leave out the `\`. > > To be fair, I think this only makes sense for when the next line starts > with a `.`. > > In PEP8 we could add a rule about 'aligning the . with the last dot on > the previous line' (or an indent of 4 if the previous line has no dot). > > Even though I think it would make for a good enhancement to the Python > language, I can not currently estimate how much of a change to the > Python parser this would need to be. Is it a three-hour straight-forward > job, or a three-month bug-prone assignment? > > Regards, > Sjoerd Job its just some syntax i've come across in some other language (can't recall which one unfortunately), the semantics being .. #you have this r = a.b()..c()..d() #is equivalent to t=a a.b() a.c() r=a.d() of yourse if you want to maintain readability you want most calls to be on separate lines so r = a.b() ..c() ..d() # the return value of the last function is the value of the entire expression with some some names that are actually descriptive this could quite nicely represent a sequence of transformations to something for example (is this why jQuery has so much method chaining?) anyway with method chaing you sometimes (often? rarely?) have an issue where you have a meaningful return value so you have to chooze if you want to return meaningful output or return self/this to enable chaining. if you need the return value then you obviously need to capture it but sometimes you just want the side effect (mutating state of `self`) and don't care about what the method returns. if you have this you probably want method chaining to be usable in expressions tho, having subsets of language usable in different contexts rly gets annoying mmighty fast. From sjoerdjob at sjoerdjob.com Thu Sep 1 07:26:58 2016 From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus) Date: Thu, 1 Sep 2016 13:26:58 +0200 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <57C7CD53.9070805@canterbury.ac.nz> <20160901081032.GM6830@sjoerdjob.com> Message-ID: <20160901112658.GN6830@sjoerdjob.com> On Thu, Sep 01, 2016 at 01:38:19PM +0300, Joonas Liik wrote: > On 1 September 2016 at 11:10, Sjoerd Job Postmus > wrote: > > On Thu, Sep 01, 2016 at 10:43:17AM +0300, Joonas Liik wrote: > >> Not sure if this is a good idea but it might also make some sense to > >> in stead have an operator at the beginning of the line > >> > >> for example some languages have a chainging operator for method calls: > >> > >> my_object.m1() > >> ..m2() > >> being equivalent to > >> my_object.m1() > >> my_object.m2() > >> > >> It could also be possible to have a special > >> swallow-the-preceding-newline operator or something of that effect. > >> > >> side note: the chainging operator does not work on the return value of > >> the method thus the method no longer has to choose between returning > >> useful info or `this` for chaining convenience. > > > > Not sure if I would appreciate this. Most of the cases where I do > > chaining method calls, it's the return value that's being chained on. Of > > course that is logical, because that's what's currently possible. Also, > > would it be called 'call forking' instead of 'call chaining'...? > > > > But also, what does this mean: > > > > my_object.value.m1() > > ..m2() > > > > is that > > > > my_object.value.m1() > > my_object.value.m2() > > > > or > > > > my_object.value.m1() > > my_object.m2() > > > > > > I do think the syntax you suggest is readable, I just think the > > semantics is confusing and ambiguous in non-trivial examples. And the > > extra '.' to signify it is chaining is not really a syntactical > > necessity I think. > > > > What I think might be a neat idea is to do the following: > > > > if: > > - we have an 'unexpected indent', and > > - the line starts with a '.' > > > > then: > > - interpret the physical line as continuation of the previous line. > > > > > > In any current Python version this is a syntax error. In a new Python > > version it could be *the* obvious way to do method chaining. > > > > inactive_admins = User.objects > > .filter(is_staff=True) > > .exclude(last_login__gt=three_weeks_ago) > > > > In current versions of Python, you have to add a `\` to the end, in a > > possible future version of Python, you could leave out the `\`. > > > > To be fair, I think this only makes sense for when the next line starts > > with a `.`. > > > > In PEP8 we could add a rule about 'aligning the . with the last dot on > > the previous line' (or an indent of 4 if the previous line has no dot). > > > > Even though I think it would make for a good enhancement to the Python > > language, I can not currently estimate how much of a change to the > > Python parser this would need to be. Is it a three-hour straight-forward > > job, or a three-month bug-prone assignment? > > > > Regards, > > Sjoerd Job > > its just some syntax i've come across in some other language (can't > recall which one unfortunately), the semantics being .. After doing some research, turns out this is called 'method cascading' (not to be confused with 'method chaining'). It is found in Dart, which has exactly this syntax. > > #you have this > r = a.b()..c()..d() > > #is equivalent to > t=a > a.b() > a.c() > r=a.d() After reading up on this, it seems your example is a bit mistaken. It would be t = a.b() a.c() r = a.d() (for the other behaviour, you should do `a..b()..c()..d()`) It turns out that Dart also has assignment cascading: r = Address() ..street = "East Avenue" ..housenumber = 12 > > of yourse if you want to maintain readability you want most calls to > be on separate lines so > r = a.b() > ..c() > ..d() # the return value of the last function is the value of the > entire expression > > with some some names that are actually descriptive this could quite > nicely represent a sequence of transformations to something for > example (is this why jQuery has so much method chaining?) Seeing "transformations" makes me think of method chaining more than the method cascading as described above. In jQuery (and also Django querysets), you don't get the same object back, but a new object. So Django again: qs = User.objects.filter(is_staff=True).filter(is_active=True) is not equal to qs = User.objects qs.filter(is_staff=True) qs.filter(is_active=True) but qs = User.objects qs = qs.filter(is_staff=True) qs = qs.filter(is_active=True) > anyway with method chaing you sometimes (often? rarely?) have an issue > where you have a meaningful return value so you have to chooze if you > want to return meaningful output or return self/this to enable > chaining. > if you need the return value then you obviously need to capture it but > sometimes you just want the side effect (mutating state of `self`) and > don't care about what the method returns. > > if you have this you probably want method chaining to be usable in > expressions tho, having subsets of language usable in different > contexts rly gets annoying mmighty fast. Method chaining is already possible in Python, just sometimes you have to add a bit of extra characters to make it prettily span multiple lines. Inside an expression it already works. Method cascading is not yet possible in Python. To be honest, I think it is off-topic in this thread, but also that it is something that might be a good idea. It is however already covered in the following thread... https://mail.python.org/pipermail//python-ideas/2013-November/024124.html Not to say that the opinions might not have changed since then... From random832 at fastmail.com Thu Sep 1 10:03:19 2016 From: random832 at fastmail.com (Random832) Date: Thu, 01 Sep 2016 10:03:19 -0400 Subject: [Python-ideas] real numbers with SI scale factors: next steps In-Reply-To: <57C7C80E.8060708@canterbury.ac.nz> References: <20160830203427.GE2363@kundert.designers-guide.com> <20160831020551.GN26300@ando.pearwood.info> <20160831040801.GF2363@kundert.designers-guide.com> <1472665380.41376.711754849.48190D2E@webmail.messagingengine.com> <57C7C80E.8060708@canterbury.ac.nz> Message-ID: <1472738599.1237985.712749665.6F5F6127@webmail.messagingengine.com> On Thu, Sep 1, 2016, at 02:17, Greg Ewing wrote: > I don't think a space should be automatic. The typographical > recommendation is to put a thin non-breaking space between > the value and the unit, but this is not possible with a > monospaced font, so some people might decide that it's > better without a space, or they might want to use a > character other than 0x20. Better to let the user put the > space in the format string if wanted. If the space needs to be between the number and the unit there's no good way to do this. I think this is an argument for a separate function that returns a tuple of (formatted number, prefix). Incidentally, do we have a good primitive to return (string of digits, exponent or position of decimal point) a la C's ecvt/fcvt? This would be something that might be useful in allowing users to build their own formatting code. It can be worked around though ("guess" the exponent, scale with multiplication or division, round to an integer to get the string of digits) so I guess it's not that important. > I'm inclined to think it should be the number of significant > digits, not decimal places, to give a more consistent > precision as the magnitude of the number changes. > > For example, if you're displaying some resistor values that > are accurate to 2 digits, you would want to see 2.7k, > 27k, 270k, but not 27.0k or 270.0k as those would suggest > spurious precision. What I was getting at is that there are two different use cases possible here. > This would also help with fitting the value into a fixed > width, since you would know that a precision of n would > use at most n+1 characters for the numeric part. Exactly n+1, surely? And on the other hand a fixed number of decimal places allows easy alignment by right-justifying the text within a field (and will use at most n+4 characters for the numeric part). From ryan at ryanhiebert.com Thu Sep 1 11:04:12 2016 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Thu, 1 Sep 2016 10:04:12 -0500 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <57C7CD53.9070805@canterbury.ac.nz> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <57C7CD53.9070805@canterbury.ac.nz> Message-ID: <7BAAAE43-BD07-4CF8-8979-C21E071EF990@ryanhiebert.com> > On Sep 1, 2016, at 1:40 AM, Greg Ewing wrote: > > Guido van Rossum wrote: >> Would this be enforced in the grammar or by the lexer? Since you say >> you expect the indentation to be enforced, that suggests it would be >> done by the grammar, > > I think it could be done by having the lexer enter a mode > where it swallows a newline that is followed by an indentation > to a level greater than the starting level of the construct. > Then no change would be needed to the grammar. +1 This is what I would like. It wouldn't require the ellipsis marker, AFAICT, and would eliminate so many parenthesis and backslashes from my code. From steve at pearwood.info Thu Sep 1 11:35:19 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Sep 2016 01:35:19 +1000 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> Message-ID: <20160901153519.GW26300@ando.pearwood.info> On Wed, Aug 31, 2016 at 03:46:13PM -0600, Shane Hathaway wrote: > Cons: > > - Extra parentheses are required. You have to have extra *something* no matter how you do it. An extra semi-colon at the end of the statement, in semi-colon language. An extra backslash at the end of the line. An extra pair of parens. I don't see this as a meaningful negative. > - The indentation is not enforced by the parser, so I have unnecessary > freedom that could let various mistakes slip through. I don't see how. Its precisely because the indentation is not enforced that you *can't* get errors related to the indentation. Of course you can still get other errors, like dropping a comma between function arguments, or failing to close a pair of brackets, but enforcing indentation doesn't protect you against those. Can you demonstrate an error that can be prevented by enforcing indentation inside an expression? In fact, can you explain what indentation rules you want to enforce? Its not clear to me exactly what you want. > - The closing parenthesis has to move every time I append to or reorder > the expression, leading to diff noise in version control. > (Alternatively, I could put the closing parenthesis on its own line, but > that consumes precious vertical reading space.) I don't get this argument either. There are two situations: you have the closing paren just after the final token, or you have it on a line of its own. If its on a line of its own, you don't need to change it if you modify the expression (as you state yourself). But if its on the same line as the final token, and you modify that line by appending or re-ordering, there's going to be a diff regardless: result = ( blah blah blah blah blah blah blah + x - y or spam.eggs()) becomes: result = ( blah blah blah blah blah x - y blah blah or spam.eggs() or default) The presence or absence of that outermost pair of brackets doesn't affect the diff in any way. You either change the line, and get a diff, or you don't, and don't. At least that's how I see it. Can you show the sort of diff noise that you're talking about? > I'd like to suggest a small change to the Python parser that would make > long expressions read better: > > rows = dbsession.query(Table1) ... > .join( > Table2, Table2.y = Table1.y) > .filter(Table1.x = xx) > .all() > > The idea is to use an ellipsis at the end of a line to spread an > expression over multiple indented lines, terminated by a return to an > earlier indentation level. You can still indent more deeply as needed, > as shown above by the join() method call. Three dots is already syntax for Ellipsis, so this is going to be ambiguous. There will be expressions where ... is valid, and the parser cannot tell if it is intended as a line continuation or not. result = spam + ... (eggs or cheese)(arg) Is that a line continuation, or an accidentally indented line which ought to raise a SyntaxError? > This syntax has all the pros of the existing syntax and resolves all the > cons: I'm not convinced that any of your "cons" actually are cons. See my questions above. > - No extra parentheses are required. So instead of two extra characters, you need three extra characters. I'm not seeing how this is a benefit. > - The indentation is enforced, so my mistakes are more likely to be > caught early. > - Without a closing parenthesis, there is no diff noise when I append to > or reorder an expression. As above, I'm not seeing how this is supposed to work. -- Steve From steve at pearwood.info Thu Sep 1 12:26:02 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Sep 2016 02:26:02 +1000 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <7BAAAE43-BD07-4CF8-8979-C21E071EF990@ryanhiebert.com> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <57C7CD53.9070805@canterbury.ac.nz> <7BAAAE43-BD07-4CF8-8979-C21E071EF990@ryanhiebert.com> Message-ID: <20160901162601.GY26300@ando.pearwood.info> On Thu, Sep 01, 2016 at 10:04:12AM -0500, Ryan Hiebert wrote: > > > On Sep 1, 2016, at 1:40 AM, Greg Ewing wrote: > > > > Guido van Rossum wrote: > >> Would this be enforced in the grammar or by the lexer? Since you say > >> you expect the indentation to be enforced, that suggests it would be > >> done by the grammar, > > > > I think it could be done by having the lexer enter a mode > > where it swallows a newline that is followed by an indentation > > to a level greater than the starting level of the construct. > > Then no change would be needed to the grammar. > > +1 > > This is what I would like. It wouldn't require the ellipsis marker, > AFAICT, and would eliminate so many parenthesis and backslashes from > my code. -1 on implicit line continuations. Having explicit parens (or even the much-maligned backslash line continuation) is a good thing, making it clear that, yes, the extra indentation is intentional. -- Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Sep 1 13:00:43 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Fri, 2 Sep 2016 02:00:43 +0900 Subject: [Python-ideas] real numbers with SI scale factors: next steps In-Reply-To: References: <20160830203427.GE2363@kundert.designers-guide.com> <20160831020551.GN26300@ando.pearwood.info> <20160831040801.GF2363@kundert.designers-guide.com> <1472665380.41376.711754849.48190D2E@webmail.messagingengine.com> <1472665844.45040.711818297.6A0BCBBB@webmail.messagingengine.com> <22471.42809.452598.705168@turnbull.sk.tsukuba.ac.jp> Message-ID: <22472.24251.214953.677331@turnbull.sk.tsukuba.ac.jp> Guido van Rossum writes: > On Wed, Aug 31, 2016 at 8:57 PM, Stephen J. Turnbull > wrote: > > That seems to be right approach: in system administration, these > > numbers are used mostly to understand resource usage, and > > underestimates are almost never what you want, > > That would seem to apply to "space used" but not to "space available". True, but I don't think the implications are symmetric. I buy storage to handle space (expected to be) used, not space available. But when I find myself caring about the "slop" in space available, the fact that I care about that is already very bad news. Time to head for Fry's Electronics! As I wrote before, I don't think the same argument applies to scientific computing. From mistersheik at gmail.com Thu Sep 1 13:36:53 2016 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 1 Sep 2016 10:36:53 -0700 (PDT) Subject: [Python-ideas] Improve the error message for impossible multiple inheritance Message-ID: <5db3d11b-272d-4698-afc3-b93f9e588fc3@googlegroups.com> I sometimes get the error message: TypeError: Cannot create a consistent method resolution order (MRO) for bases ? I suggest that Python report * cycles (if there are any), or * a list of orderings of direct base classes (if there are no cycles). A cycle for the base classes A, B, C, D can be reported as follows: TypeError: Cannot create a consistent method resolution order (MRO) because A inherits from B (through X, Y), but B inherits from A (through Z). The topological sort could be reported as follows for base classes A, B, C, D: TypeError: Cannot create a consistent method resolution order (MRO) because B inherit from C (through X, Y), but B follows C in inheritance order. It might be nice to have an exposed tool somewhere in the standard library that spits out for A, B, C, D: A, B inherit from C (through X, Y) C inherit from D (through Z) Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From shane at hathawaymix.org Thu Sep 1 14:44:58 2016 From: shane at hathawaymix.org (Shane Hathaway) Date: Thu, 1 Sep 2016 12:44:58 -0600 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <20160901153519.GW26300@ando.pearwood.info> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <20160901153519.GW26300@ando.pearwood.info> Message-ID: <8eb05df2-4bcd-eac1-961e-d5727731c13c@hathawaymix.org> On 09/01/2016 09:35 AM, Steven D'Aprano wrote: > On Wed, Aug 31, 2016 at 03:46:13PM -0600, Shane Hathaway wrote: > >> Cons: >> >> - Extra parentheses are required. > > You have to have extra *something* no matter how you do it. An extra > semi-colon at the end of the statement, in semi-colon language. An extra > backslash at the end of the line. An extra pair of parens. I don't see > this as a meaningful negative. I agree that multi-line expressions should always be explicit. Neither of us would want implicit multi-line expressions. I admire the style Python uses to format classes, functions, loops, conditions, and so on. There's always a block start marker, an indented block, and an implicit block end (implied by a reduction in the indentation level). That's a very tidy pattern and I try to use it as much as possible. I currently can't use that pattern for multi-line expressions. Because my SQLAlchemy code has a lot of multi-line expressions, some of my code doesn't look very much like Python. I'd like to keep my style more consistent so it's easier to read. >> - The indentation is not enforced by the parser, so I have unnecessary >> freedom that could let various mistakes slip through. > > I don't see how. Its precisely because the indentation is not enforced > that you *can't* get errors related to the indentation. Of course you > can still get other errors, like dropping a comma between function > arguments, or failing to close a pair of brackets, but enforcing > indentation doesn't protect you against those. > > Can you demonstrate an error that can be prevented by enforcing > indentation inside an expression? Sometimes I fix unbalanced parentheses incorrectly. Here's something I might type. There should be another closing parenthesis in the middle: def update(names, value): (dbsession.query(Table1) .filter(Table1.name.in_(names)) .update({'value': value}) (dbsession.query(Table2) .filter(Table2.name.in_(names)) .update({'value': value})) Either Python or flake8 will tell me there's some kind of syntax error, so I might fix it by adding a closing parenthesis at the end: def update(names, value): (dbsession.query(Table1) .filter(Table1.name.in_(names)) .update({'value': value}) (dbsession.query(Table2) .filter(Table2.name.in_(names)) .update({'value': value}))) This will fix the syntax error but fail at runtime. With my proposed syntax, I would probably never create the error in the first place because I would only need to scan for balanced parentheses on each line, not over multiple lines: def update(names, value): dbsession.query(Table1) ... .filter(Table1.name.in_(names)) .update({'value': value}) dbsession.query(Table2) ... .filter(Table2.name.in_(names)) .update({'value': value}) > In fact, can you explain what indentation rules you want to enforce? Its > not clear to me exactly what you want. The indentation must be consistent, and the expression ends when the indentation level drops to the same level as the first line or an earlier line. >> - The closing parenthesis has to move every time I append to or reorder >> the expression, leading to diff noise in version control. >> (Alternatively, I could put the closing parenthesis on its own line, but >> that consumes precious vertical reading space.) > > I don't get this argument either. There are two situations: you have the > closing paren just after the final token, or you have it on a line of > its own. If its on a line of its own, you don't need to change it if you > modify the expression (as you state yourself). But if its on the same > line as the final token, and you modify that line by appending or > re-ordering, there's going to be a diff regardless: > > result = ( blah blah blah > blah blah blah > blah + x - y or spam.eggs()) > > becomes: > > result = ( blah blah blah > blah blah x - y blah > blah or spam.eggs() or default) > > > The presence or absence of that outermost pair of brackets doesn't > affect the diff in any way. You either change the line, and get a diff, > or you don't, and don't. At least that's how I see it. > > Can you show the sort of diff noise that you're talking about? Let's say I write this code: rows = ( dbsession.query(Table1) .filter(Table1.name.in_(names))) That code will work, but it will be slow (because it will fetch rows one at a time). I later realize I need to call the all() method to make it fast: rows = ( dbsession.query(Table1) .filter(Table1.name.in_(names)) .all()) One line of code had to change and another had to be added. With my proposed syntax, let's say I write the same code again: rows = dbsession.query(Table1) ... .filter(Table1.name.in_(names)) To make it fast, I add a line: rows = dbsession.query(Table1) ... .filter(Table1.name.in_(names)) .all() In this case, I only added one line of code and didn't change any existing lines, so the signal to noise ratio is better. > Three dots is already syntax for Ellipsis, so this is going to be > ambiguous. There will be expressions where ... is valid, and the parser > cannot tell if it is intended as a line continuation or not. > > result = spam + ... > (eggs or cheese)(arg) > > Is that a line continuation, or an accidentally indented line which > ought to raise a SyntaxError? Ah, that's very interesting. I had forgotten until now that "..." in Python 3 is a literal value. For anyone else reading along, this now works in Python 3: >>> x = ... >>> print(x, ...) Ellipsis Ellipsis I wasn't aware of that complication. Maybe I should limit the idea to method chaining as Guido hinted. Shane From brenbarn at brenbarn.net Thu Sep 1 15:24:53 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Thu, 01 Sep 2016 12:24:53 -0700 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> Message-ID: <57C88085.3020706@brenbarn.net> On 2016-08-31 14:46, Shane Hathaway wrote: > Hi, > > I write a lot of SQLAlchemy code that looks more or less like this: > > rows = ( > dbsession.query(Table1) > .join( > Table2, Table2.y = Table1.y) > .filter(Table1.x = xx) > .all()) > > The expressions get very long and nearly always need to be spread to > multiple lines. I've tried various styles and have chosen the style > above as the most tasteful available. > > Pros of the existing syntax: > > - It's possible to indent clearly and consistently. > - Nested indentation works out OK. > - It's flexible; I can combine lines or separate them for emphasis. > > Cons: > > - Extra parentheses are required. > - The indentation is not enforced by the parser, so I have unnecessary > freedom that could let various mistakes slip through. > - The closing parenthesis has to move every time I append to or reorder > the expression, leading to diff noise in version control. > (Alternatively, I could put the closing parenthesis on its own line, but > that consumes precious vertical reading space.) I sometimes write similar code with a sequence of pandas operations. I think you can get a lot of mileage out of accepting the loss in precious vertical space and just putting the closing parenthesis on its own line, indented to the same level as the beginning of the line where the corresponding open parenthesis was. Your example then becomes: rows = ( dbsession.query(Table1) .join( Table2, Table2.y = Table1.y ) .filter(Table1.x = xx) .all() ) This is essentially the way I write such code. This gives you most of the important advantages you seek. You do need one extra set of parentheses around the whole thing, but you never have to move or alter those parentheses. You can freely add new lines (i.e., add the ".all()" line if it wasn't there, as you suggested in a later post) without interfering with existing lines. Moreover, some of the reasons you cite in your later post for wanting indentation enforcement become less important if you do things this way, because you are less likely to add a parenthesis in the wrong place when your parentheses are clearly set off like this. (It's true, though, that it might help even more if Python would enforce the requirement that, when a line ends with an open parenthesis, its counterpart must appear at the same indentation level.) In my experience, the tradeoff between compactness and clarity is rarely a major issue, because most expressions are either simple enough that clarity is not at risk, or complex enough that compactness is not a real benefit. If your expression is two or three lines long with no long nested parentheses, it will probably be clear enough even if you fudge a bit on the formatting. If your expression is ten lines long, having to add an extra line or two (or even three) for closing parentheses is not such a big deal. Basically, adding extra lines for parentheses is only a pain if the expression was previously "short" and now becomes "long", but not many expressions are very close to that boundary. All that said, I do think it is worth considering some syntax to make this sort of thing easier. I'm not sure the Ellipsis approach is the right one, but I'm interested to see where this thread goes. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From p.f.moore at gmail.com Thu Sep 1 16:04:49 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Sep 2016 21:04:49 +0100 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <8eb05df2-4bcd-eac1-961e-d5727731c13c@hathawaymix.org> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <20160901153519.GW26300@ando.pearwood.info> <8eb05df2-4bcd-eac1-961e-d5727731c13c@hathawaymix.org> Message-ID: On 1 September 2016 at 19:44, Shane Hathaway wrote: > Sometimes I fix unbalanced parentheses incorrectly. Here's something I > might type. There should be another closing parenthesis in the middle: > > def update(names, value): > (dbsession.query(Table1) > .filter(Table1.name.in_(names)) > .update({'value': value}) > (dbsession.query(Table2) > .filter(Table2.name.in_(names)) > .update({'value': value})) > > Either Python or flake8 will tell me there's some kind of syntax error, so I > might fix it by adding a closing parenthesis at the end: > > def update(names, value): > (dbsession.query(Table1) > .filter(Table1.name.in_(names)) > .update({'value': value}) > (dbsession.query(Table2) > .filter(Table2.name.in_(names)) > .update({'value': value}))) > > This will fix the syntax error but fail at runtime. With my proposed > syntax, I would probably never create the error in the first place because I > would only need to scan for balanced parentheses on each line, not over > multiple lines: > > def update(names, value): > dbsession.query(Table1) ... > .filter(Table1.name.in_(names)) > .update({'value': value}) > dbsession.query(Table2) ... > .filter(Table2.name.in_(names)) > .update({'value': value}) Thanks. That's a nice example of how the proposal might help. But you could of course have written your original code as def update(names, value): (dbsession.query(Table1) .filter(Table1.name.in_(names)) .update({'value': value}) (dbsession.query(Table2) .filter(Table2.name.in_(names)) .update({'value': value})) That's not going to completely alleviate the problem, but it does make the intent clearer. And it's possible that you could propose a style rule that a dedent in a bracketed expression is not allowed - that might well be something that flake8 could add, and then you'd get a much clearer error message (but only if you ran flake8 - if you just saw a syntax error from Python, you'd probably be just as likely to "fix" it as you said above, without even trying to run flake8). Also, of course, most text editors would highlight matching parentheses - which makes it much easier to spot the "correct" place for the missing parenthesis. One other thing, I'm not at all keen on using "..." for the syntax - it's almost completely invisible when I read this in gmail, and as others have pointed out, it already has a meaning, as Ellipsis. But I don't have a better suggestion to offer, I'm afraid. Overall, though, I'm cautiously in favour of the proposal. I'm not convinced the benefit is huge, and I'm a little concerned that it may be supporting a style of code that isn't ideal (the method chaining from SQLAlchemy you use to illustrate the examples is common enough in languages like JavaScript, but outside of SQLAlchemy I haven't seen it used much in Python). Also, it's very definitely "yet another way to write expressions across multiple lines". But the indented expression format is pretty readable for cases when you *do* have a long expression and no convenient way to bracket it. Paul From shane at hathawaymix.org Thu Sep 1 17:05:16 2016 From: shane at hathawaymix.org (Shane Hathaway) Date: Thu, 1 Sep 2016 15:05:16 -0600 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: <57C88085.3020706@brenbarn.net> References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <57C88085.3020706@brenbarn.net> Message-ID: <083eae61-d4e5-addb-4370-1091df6d9026@hathawaymix.org> On 09/01/2016 01:24 PM, Brendan Barnwell wrote: > I sometimes write similar code with a sequence of pandas > operations. I think you can get a lot of mileage out of accepting the > loss in precious vertical space and just putting the closing parenthesis > on its own line, indented to the same level as the beginning of the line > where the corresponding open parenthesis was. Your example then becomes: > > rows = ( > dbsession.query(Table1) > .join( > Table2, Table2.y = Table1.y > ) > .filter(Table1.x = xx) > .all() > ) > > This is essentially the way I write such code. This gives you most > of the important advantages you seek. You do need one extra set of > parentheses around the whole thing, but you never have to move or alter > those parentheses. You can freely add new lines (i.e., add the ".all()" > line if it wasn't there, as you suggested in a later post) without > interfering with existing lines. Moreover, some of the reasons you cite > in your later post for wanting indentation enforcement become less > important if you do things this way, because you are less likely to add > a parenthesis in the wrong place when your parentheses are clearly set > off like this. (It's true, though, that it might help even more if > Python would enforce the requirement that, when a line ends with an open > parenthesis, its counterpart must appear at the same indentation level.) That's a good point. I've been waffling about where to put the closing parenthesis. If flake8 had an option to force the closing parenthesis to be on its own line and at the same indentation level as the line with the opening parenthesis, I would certainly use it. > In my experience, the tradeoff between compactness and clarity is > rarely a major issue, because most expressions are either simple enough > that clarity is not at risk, or complex enough that compactness is not a > real benefit. If your expression is two or three lines long with no > long nested parentheses, it will probably be clear enough even if you > fudge a bit on the formatting. If your expression is ten lines long, > having to add an extra line or two (or even three) for closing > parentheses is not such a big deal. Basically, adding extra lines for > parentheses is only a pain if the expression was previously "short" and > now becomes "long", but not many expressions are very close to that > boundary. You may be right, but I'm not sure your experience matches mine. I seem to recall changing single-line expressions into multi-line expressions fairly often. I'll pay more attention in the future. > All that said, I do think it is worth considering some syntax to > make this sort of thing easier. I'm not sure the Ellipsis approach is > the right one, but I'm interested to see where this thread goes. Thanks for the well considered thoughts. Shane From shane at hathawaymix.org Thu Sep 1 17:35:23 2016 From: shane at hathawaymix.org (Shane Hathaway) Date: Thu, 1 Sep 2016 15:35:23 -0600 Subject: [Python-ideas] Extending expressions using ellipsis In-Reply-To: References: <49797ff7-5066-df52-c962-1148ddd4f2e4@hathawaymix.org> <20160901153519.GW26300@ando.pearwood.info> <8eb05df2-4bcd-eac1-961e-d5727731c13c@hathawaymix.org> Message-ID: On 09/01/2016 02:04 PM, Paul Moore wrote: > Thanks. That's a nice example of how the proposal might help. But you > could of course have written your original code as > > def update(names, value): > (dbsession.query(Table1) > .filter(Table1.name.in_(names)) > .update({'value': value}) > (dbsession.query(Table2) > .filter(Table2.name.in_(names)) > .update({'value': value})) > > That's not going to completely alleviate the problem, but it does make > the intent clearer. And it's possible that you could propose a style > rule that a dedent in a bracketed expression is not allowed - that > might well be something that flake8 could add, and then you'd get a > much clearer error message (but only if you ran flake8 - if you just > saw a syntax error from Python, you'd probably be just as likely to > "fix" it as you said above, without even trying to run flake8). Also, > of course, most text editors would highlight matching parentheses - > which makes it much easier to spot the "correct" place for the missing > parenthesis. Good points. That style does look clearer than my example. > One other thing, I'm not at all keen on using "..." for the syntax - > it's almost completely invisible when I read this in gmail, and as > others have pointed out, it already has a meaning, as Ellipsis. But I > don't have a better suggestion to offer, I'm afraid. Now that I remember that "..." is a literal value in Python 3 (surprise!), I've been thinking about alternatives. I wonder if a double backslash would work. Double backslash at the end of a line is currently a syntax error. Let's see how it looks: def update(names, value): dbsession.query(Table1) \\ .filter(Table1.name.in_(names)) .update({'value': value}) dbsession.query(Table2) \\ .filter(Table2.name.in_(names)) .update({'value': value}) That looks OK to me right now. Double backslash seems to convey the idea that it's not only a line continuation, but a line continuation with special properties. I would never write it this way, BTW: def update(names, value): dbsession.query(Table1) \ .filter(Table1.name.in_(names)) \ .update({'value': value}) dbsession.query(Table2) \ .filter(Table2.name.in_(names)) \ .update({'value': value}) I never seem to be able to sprinkle those single backslashes in the right places consistently. I forget one, or I leave one in after rearranging lines, leading to mayhem. > Also, it's very definitely "yet another way to > write expressions across multiple lines". But the indented expression > format is pretty readable for cases when you *do* have a long > expression and no convenient way to bracket it. Well, you could say the same about triple quoted strings. Sure there are other ways to write strings longer than a single line, but triple quoted strings are clearly worth the extra parser complexity. Shane From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Sep 3 11:17:32 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 4 Sep 2016 00:17:32 +0900 Subject: [Python-ideas] [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526) In-Reply-To: <57CA24EA.1080002@canterbury.ac.nz> References: <57CA24EA.1080002@canterbury.ac.nz> Message-ID: <22474.59788.248766.537464@turnbull.sk.tsukuba.ac.jp> Please respect Reply-To, set to python-ideas. Greg Ewing writes: > Chris Angelico wrote: > > Forcing people to write 1.0 just to be compatible with 1.5 will cause > > a lot of annoyance. > > Indeed, this would be unacceptable IMO. But "forcing" won't happen. Just ignore the warning. *All* such Python programs will continue to run (or crash) exactly as if the type declarations weren't there. If you don't like the warning, either don't run the typechecker, or change your code to placate it. But allowing escapes from a typechecker means allowing escapes. All of them, not just the ones you or I have preapproved. I want my typechecker to be paranoid, and loud about it. That doesn't mean I would never use a type like "Floatable" (ie, any type subject to implicit conversion to float). But in the original example, I would probably placate the typechecker. YMMV, of course. From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Sep 3 11:28:08 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 4 Sep 2016 00:28:08 +0900 Subject: [Python-ideas] What should a good typechecker do? In-Reply-To: References: <57C982D4.1060405@hotpy.org> <20160902164035.GB26300@ando.pearwood.info> Message-ID: <22474.60424.61709.728801@turnbull.sk.tsukuba.ac.jp> Guido van Rossum writes: > But that's not what type comments mean! They don't annotate the > expression. They annotate the variable. In PEP 484. But syntactically, AFAICS in an initialization that's a distinction without a difference. It would be perfectly possible to write a checker that allows if cond: x: str = "a string, what else?" else: x: int = 1 and infers the union, and even infers the types from the expressions in if cond: x = "a string, what else?" else: x = 1 Once you have that, then the only things that are really being typed are the initializer expressions. I don't understand why some people seem to think that is a *good* typechecker (if that's it's normal mode of operation). But as a tool to identify the types that untyped (or ambiguously-typed) code actually uses, review them, and help produce a precisely typed version, it seems perfectly plausible to me. From k7hoven at gmail.com Sat Sep 3 11:44:29 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 3 Sep 2016 18:44:29 +0300 Subject: [Python-ideas] [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526) In-Reply-To: <22474.59788.248766.537464@turnbull.sk.tsukuba.ac.jp> References: <57CA24EA.1080002@canterbury.ac.nz> <22474.59788.248766.537464@turnbull.sk.tsukuba.ac.jp> Message-ID: What's up with the weird subthreads, Stephen?! On Guido's suggestion, I'm working on posting those type-checking thoughts here. -- Koos On Sat, Sep 3, 2016 at 6:17 PM, Stephen J. Turnbull wrote: > Please respect Reply-To, set to python-ideas. > > Greg Ewing writes: > > Chris Angelico wrote: > > > Forcing people to write 1.0 just to be compatible with 1.5 will cause > > > a lot of annoyance. > > > > Indeed, this would be unacceptable IMO. > > But "forcing" won't happen. Just ignore the warning. *All* such > Python programs will continue to run (or crash) exactly as if the type > declarations weren't there. If you don't like the warning, either > don't run the typechecker, or change your code to placate it. > > But allowing escapes from a typechecker means allowing escapes. All > of them, not just the ones you or I have preapproved. I want my > typechecker to be paranoid, and loud about it. > > That doesn't mean I would never use a type like "Floatable" (ie, any > type subject to implicit conversion to float). But in the original > example, I would probably placate the typechecker. YMMV, of course. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com -- + Koos Zevenhoven + http://twitter.com/k7hoven + From guido at python.org Sat Sep 3 11:51:41 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 3 Sep 2016 08:51:41 -0700 Subject: [Python-ideas] What should a good typechecker do? In-Reply-To: <22474.60424.61709.728801@turnbull.sk.tsukuba.ac.jp> References: <57C982D4.1060405@hotpy.org> <20160902164035.GB26300@ando.pearwood.info> <22474.60424.61709.728801@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sat, Sep 3, 2016 at 8:28 AM, Stephen J. Turnbull wrote: > Guido van Rossum writes: > > > But that's not what type comments mean! They don't annotate the > > expression. They annotate the variable. > > In PEP 484. But syntactically, AFAICS in an initialization that's a > distinction without a difference. It would be perfectly possible to > write a checker that allows > > if cond: > x: str = "a string, what else?" > else: > x: int = 1 > > and infers the union, and even infers the types from the expressions > in > > if cond: > x = "a string, what else?" > else: > x = 1 > > Once you have that, then the only things that are really being typed > are the initializer expressions. > > I don't understand why some people seem to think that is a *good* > typechecker (if that's it's normal mode of operation). But as a tool > to identify the types that untyped (or ambiguously-typed) code > actually uses, review them, and help produce a precisely typed > version, it seems perfectly plausible to me. That's is why we're not restricting this syntactically. I've seen a fair amount of untyped code that reuses the same variable name (in the same scope, often in unit tests), but usually it was serially, e.g. target = 42 assert fooi(target) == '42' target = 3.14 assert foof(target) == '3.14' target = 'x' assert foos(target) == "'x'" -- --Guido van Rossum (python.org/~guido) From k7hoven at gmail.com Sat Sep 3 12:21:21 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 3 Sep 2016 19:21:21 +0300 Subject: [Python-ideas] [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526) In-Reply-To: References: <57CA24EA.1080002@canterbury.ac.nz> <22474.59788.248766.537464@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sat, Sep 3, 2016 at 6:44 PM, Koos Zevenhoven wrote: > On Guido's suggestion, I'm working on posting those type-checking thoughts here. > Except that I just realized I can still make it to the store before it closes, so a little later. But anyone can read my posts on python-dev from yesterday in the meantime if they want to. -- Koos PS. I think I just got reminded about what I disklike most about these lists. From k7hoven at gmail.com Sat Sep 3 17:26:58 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 4 Sep 2016 00:26:58 +0300 Subject: [Python-ideas] Semantics for type checking (was: What should a good type checker do?) Message-ID: On Friday, while replying to a post on python-dev about PEP 526 (variable annotations), I ended up mentioning things that I think a good type checker should do, which seem to differ from the general understanding. The discussion should continue here, though, because the aim of PEP 526 is not to define type-checking semantics. I'll recap the thoughts in this post, but here are links to the original posts, for reference: [1] https://mail.python.org/pipermail/python-dev/2016-September/146069.html [2] https://mail.python.org/pipermail/python-dev/2016-September/146090.html Quoting [1], with added explanations in between: On Fri, Sep 2, 2016 at 1:10 PM, Koos Zevenhoven wrote: [...] > > The more you infer types, the less you check them. It's up to the > programmers to choose the amount of annotation. > [...] > > I'm also a little worried about not being able to reannotate a name. > IOW, this should be allowed: x : int = 1 foo(x) x : float = 0.5 bar(x) It turns out that disallowing this was not the PEP's intention, because type-checking semantics is up to the type checker. This has now been clarified in the PEP. Good. [...] >> def eggs(cond:bool): >> x: Optional[List[int]] >> if cond: >> x = None # Now legal >> else: >> x: = [] >> spam(x) >> The above is an example from Mark Shannon's email, and below is my response: > > A good checker should be able to infer that x is a union type at the > point that it's passed to spam, even without the type annotation. For > example: > > def eggs(cond:bool): > if cond: > x = 1 > else: > x = 1.5 > spam(x) # a good type checker infers that x is of type Union[int, float] > Here, there are two inferred types for x, but in different branches of the if statement. So x, which is dynamically typed at runtime, becomes a Union at "static-check time". > Or with annotations: > > def eggs(cond:bool): > if cond: > x : int = foo() # foo may not have a return type hint > else: > x : float = bar() # bar may not have a return type hint > spam(x) # a good type checker infers that x is of type Union[int, float] > > Two explicit annotations, and again neither of them "wins". The inferred type after the if statement is therefore a Union. Below, I'm quoting [2], my response to an email that disagreed with me about the above being something "a good type checker" would do: On Fri, Sep 2, 2016 at 6:49 PM, Koos Zevenhoven wrote: >>> A good checker should be able to infer that x is a union type at the >>> point that it's passed to spam, even without the type annotation. For >>> example: >>> >>> def eggs(cond:bool): >>> if cond: >>> x = 1 >>> else: >>> x = 1.5 >>> spam(x) # a good type checker infers that x is of type Union[int, float] >> >> Oh I really hope not. I wouldn't call that a *good* type checker. I >> would call that a type checker that is overly permissive. > > I guess it's perfectly fine if we disagree about type checking ideals, > and I can imagine the justification for you thinking that way. There > can also be different type checkers, and which can have different > modes. > > But assume (a) that the above function is perfectly working code, and > spam(...) accepts Union[int, float]. Why would I want the type checker > to complain? > > Then again, (b) instead of that being working code, it might be an > error and spam only takes float. No problem, the type checker will > catch that. > > In case of (b), to get the behavior you want (but in my hypothetical > semantics), this could be annotated as > > def eggs(cond:bool): > x : float > if cond: > x = 1 # type checker says error > else: > x = 1.5 > spam(x) > > So here the programmer thinks the type of x should be more constrained > than what spam(...) accepts. The reasoning here is that x is explicitly annotated without an assignment, so the inferred types subsequently assigned to it (int and float) must be compatible with the annotation. Therefore "the good type checker" would say assigning 1 is an error. > > Or you might have something like this > > def eggs(cond:bool): > if cond: > x = 1 > else: > x = 1.5 > # type checker has inferred x to be Union[int, float] > x : float # type checker finds an error > spam(x) > > Here, the same error is found, but at a different location. > Here, x is explicitly annotated as float while its value remains a Union[int, float], as inferred by "the good type checker". Therefore this is an error. One could consider this a type assertation (at static-check time). "The good type checker" should also be able to tell the programmer that it's "x = 1" that made the subsequent "type assertion" fail. In addition to being type *hints*, the annotations can also be considered as "check points". -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + From k7hoven at gmail.com Sat Sep 3 17:44:54 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 4 Sep 2016 00:44:54 +0300 Subject: [Python-ideas] Semantics for type checking (was: What should a good type checker do?) In-Reply-To: References: Message-ID: Below I respond to Chris Angelico's post in the python-dev thread. On Sat, Sep 3, 2016 at 2:01 AM, Chris Angelico wrote: >>>> >>>> def eggs(cond:bool): >>>> if cond: >>>> x = 1 >>>> else: >>>> x = 1.5 >>>> spam(x) # a good type checker infers that x is of type Union[int, float] >>> > > I wonder if it would be different if you wrote that as a single expression: > > x = 1 if cond else 1.5 > > x = sum([1] + [0.5] * cond) > > What should type inference decide x is in these cases? Assume an > arbitrarily smart type checker that can implement your ideal; it's > equally plausible to pretend that the type checker can recognize an > if/else block (or even if/elif/else tree of arbitrary length) as a > single "assignment" operation. IMO both of these examples - and by > extension, the if/else of the original - should be assigning a Union > type. In the first case it would indeed again be Union[int, float] (the code is basically equivalent after all). In the second case, the checker would infer List[Union[int, float]]. No magic involved, assuming that list.__add__ is annotated precisely enough. > Lots of Python code assumes that smallish integers [1] are > entirely compatible with floats. Is Python 4 going to have to deal > with the int/float distinction the way Python 3 did for bytes/text, or > are they fundamentally compatible concepts? Is the "small integer" > like the "ASCII byte/character" as a kind of hybrid beast that people > treat as simultaneously two types? (Personally, I don't think it's > anything like bytes/text. But I'm open to argument.) > > Forcing people to write 1.0 just to be compatible with 1.5 will cause > a lot of annoyance. I leave it to you to decide whether there's a > fundamental difference that needs to be acknowledged, or just > subtleties of representational limitations to be ignored until they > become a problem. Agreed. However, I think it completely depends on the context, which types are "compatible" in the way that 1 and 1.5 often are. Numbers are just a very typical example. But it could as well be Animal and Person. It's just a matter of whether the code base is written in a way that it's ok to pass on either type. > ChrisA > > [1] And by "smallish" I mean less than 2**53. Big enough for a lot of > purposes. Bigger (by definition) than JavaScript's integers, which cap > out at 2**32. > -- + Koos Zevenhoven + http://twitter.com/k7hoven + From rosuav at gmail.com Sat Sep 3 19:22:06 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Sep 2016 09:22:06 +1000 Subject: [Python-ideas] Semantics for type checking (was: What should a good type checker do?) In-Reply-To: References: Message-ID: On Sun, Sep 4, 2016 at 7:44 AM, Koos Zevenhoven wrote: >> I wonder if it would be different if you wrote that as a single expression: >> >> x = 1 if cond else 1.5 >> >> x = sum([1] + [0.5] * cond) >> >> What should type inference decide x is in these cases? Assume an >> arbitrarily smart type checker that can implement your ideal; it's >> equally plausible to pretend that the type checker can recognize an >> if/else block (or even if/elif/else tree of arbitrary length) as a >> single "assignment" operation. IMO both of these examples - and by >> extension, the if/else of the original - should be assigning a Union >> type. > > In the first case it would indeed again be Union[int, float] (the code > is basically equivalent after all). > > In the second case, the checker would infer List[Union[int, float]]. > No magic involved, assuming that list.__add__ is annotated precisely > enough. I think you missed the sum() call there. Most likely, the sum of List[Union[anything]] would be Union[anything], so these would all be the same (which was my original point). ChrisA From yselivanov.ml at gmail.com Sat Sep 3 19:31:51 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 3 Sep 2016 16:31:51 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions Message-ID: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Hi, Below is a proposal to add support for asynchronous comprehensions and asynchronous generator expressions in Python 3.6. I have a half-working implementation of the proposal which fully implements all required grammar and AST changes. What's left is to update the compiler to emit correct opcodes for async comprehensions. I'm confident that we can have a fully working patch before the feature freeze. Thank you, Yury PEP: 530 Title: Asynchronous Comprehensions Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov Discussions-To: Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2016 Python-Version: 3.6 Post-History: 03-Sep-2016 Abstract ======== PEP 492 and PEP 525 introduce support for native coroutines and asynchronous generators using ``async`` / ``await`` syntax. This PEP proposes to add asynchronous versions of list, set, dict comprehensions and generator expressions. Rationale and Goals =================== Python has extensive support for synchronous comprehensions, allowing to produce lists, dicts, and sets with a simple and concise syntax. We propose implementing similar syntactic constructions for the asynchronous code. To illustrate the readability improvement, consider the following example:: result = [] async for i in aiter(): if i % 2: result.append(i) With the proposed asynchronous comprehensions syntax, the above code becomes as short as:: result = [i async for i in aiter() if i % 2] The PEP also makes it possible to use the ``await`` expressions in all kinds of comprehensions:: result = [await fun() for fun in funcs] Specification ============= Asynchronous Comprehensions --------------------------- We propose to allow using ``async for`` inside list, set and dict comprehensions. Pending PEP 525 approval, we can also allow creation of asynchronous generator expressions. Examples: * set comprehension: ``{i async for i in agen()}``; * list comprehension: ``[i async for i in agen()]``; * dict comprehension: ``{i: i ** 2 async for i in agen()}``; * generator expression: ``(i ** 2 async for i in agen())``. It is allowed to use ``async for`` along with ``if`` and ``for`` clauses in asynchronous comprehensions and generator expressions:: dataset = {data for line in aiter() async for data in line if check(data)} Asynchronous comprehensions are only allowed inside an ``async def`` function. In principle, asynchronous generator expressions are allowed in any context. However, in Python 3.6, due to ``async`` and ``await`` soft-keyword status, asynchronous generator expressions are only allowed in an ``async def`` function. Once ``async`` and ``await`` become reserved keywords in Python 3.7 this restriction will be removed. ``await`` in Comprehensions --------------------------- We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions:: result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs} result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs} This is only valid in ``async def`` function body. Grammar Updates --------------- The proposal requires one change on the grammar level: adding the optional "async" keyword to ``comp_for``:: comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter] The ``comprehension`` AST node will have the new ``is_async`` argument. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From k7hoven at gmail.com Sat Sep 3 19:33:40 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 4 Sep 2016 02:33:40 +0300 Subject: [Python-ideas] Semantics for type checking (was: What should a good type checker do?) In-Reply-To: References: Message-ID: On Sun, Sep 4, 2016 at 2:22 AM, Chris Angelico wrote: > On Sun, Sep 4, 2016 at 7:44 AM, Koos Zevenhoven wrote: >>> I wonder if it would be different if you wrote that as a single expression: >>> >>> x = 1 if cond else 1.5 >>> >>> x = sum([1] + [0.5] * cond) >>> >>> What should type inference decide x is in these cases? Assume an >>> arbitrarily smart type checker that can implement your ideal; it's >>> equally plausible to pretend that the type checker can recognize an >>> if/else block (or even if/elif/else tree of arbitrary length) as a >>> single "assignment" operation. IMO both of these examples - and by >>> extension, the if/else of the original - should be assigning a Union >>> type. >> >> In the first case it would indeed again be Union[int, float] (the code >> is basically equivalent after all). >> >> In the second case, the checker would infer List[Union[int, float]]. >> No magic involved, assuming that list.__add__ is annotated precisely >> enough. > > I think you missed the sum() call there. Most likely, the sum of > List[Union[anything]] would be Union[anything], so these would all be > the same (which was my original point). > Oh, you're right, I missed the sum. Suitable @overloads in the stubs for sum should lead to that being inferred as Union[int, float]. But if the annotations are not that precise, that might become just List (or List[Any]), and the programmer may want to annotate x or improve the stubs. -- Koos > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- + Koos Zevenhoven + http://twitter.com/k7hoven + From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Sep 3 20:28:47 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 4 Sep 2016 09:28:47 +0900 Subject: [Python-ideas] [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526) In-Reply-To: References: <57CA24EA.1080002@canterbury.ac.nz> <22474.59788.248766.537464@turnbull.sk.tsukuba.ac.jp> Message-ID: <22475.27327.519468.498859@turnbull.sk.tsukuba.ac.jp> Koos Zevenhoven writes: > What's up with the weird subthreads, Stephen?! Moving to Python-ideas? Doing what should have been done earlier, and which Guido explicitly requested. If you're referring to the cross-post to python-dev, I was hoping to encourage motion of the thread by setting reply-to and saying so. Evidently that traditional practice is a mistake on Python channels, as it won't be respected or even noticed. From rosuav at gmail.com Sat Sep 3 20:34:03 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Sep 2016 10:34:03 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On Sun, Sep 4, 2016 at 9:31 AM, Yury Selivanov wrote: > Below is a proposal to add support for asynchronous comprehensions and > basynchronous generator expressions in Python 3.6. Looks good to me! No content comments, and +1 on the proposal. One copyedit suggestion: > In principle, asynchronous generator expressions are allowed in > any context. However, in Python 3.6, due to ``async`` and ``await`` > soft-keyword status, asynchronous generator expressions are only > allowed in an ``async def`` function. Once ``async`` and ``await`` > become reserved keywords in Python 3.7 this restriction will be > removed. Does this want a comma after "3.7"? Otherwise, LGTM. Bring on the asynciness! ChrisA From yselivanov.ml at gmail.com Sat Sep 3 20:37:57 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 3 Sep 2016 17:37:57 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: <5da21dc2-210a-759b-3a9d-9ae8591ff4d4@gmail.com> Hi Chris, On 2016-09-03 5:34 PM, Chris Angelico wrote: > On Sun, Sep 4, 2016 at 9:31 AM, Yury Selivanov wrote: >> Below is a proposal to add support for asynchronous comprehensions and >> basynchronous generator expressions in Python 3.6. > Looks good to me! No content comments, and +1 on the proposal. Thanks! > One > copyedit suggestion: > >> In principle, asynchronous generator expressions are allowed in >> any context. However, in Python 3.6, due to ``async`` and ``await`` >> soft-keyword status, asynchronous generator expressions are only >> allowed in an ``async def`` function. Once ``async`` and ``await`` >> become reserved keywords in Python 3.7 this restriction will be >> removed. > Does this want a comma after "3.7"? Fixed! Yury From greg.ewing at canterbury.ac.nz Sat Sep 3 20:40:47 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Sep 2016 12:40:47 +1200 Subject: [Python-ideas] [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526) In-Reply-To: <22474.59788.248766.537464@turnbull.sk.tsukuba.ac.jp> References: <57CA24EA.1080002@canterbury.ac.nz> <22474.59788.248766.537464@turnbull.sk.tsukuba.ac.jp> Message-ID: <57CB6D8F.90504@canterbury.ac.nz> Stephen J. Turnbull wrote: > But "forcing" won't happen. Just ignore the warning. If I'm using a static type checker at all, I'm going to be treating its warnings as errors and doing something to make them go away. So effectively it would force me to deal explicitly with every instance of int-float mixing. I wouldn't enjoy doing that. -- Greg From boekewurm at gmail.com Sat Sep 3 20:55:46 2016 From: boekewurm at gmail.com (Matthias welp) Date: Sun, 4 Sep 2016 02:55:46 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: > In principle, asynchronous generator expressions are allowed in > any context. However, in Python 3.6, due to ``async`` and ``await`` > soft-keyword status, asynchronous generator expressions are only > allowed in an ``async def`` function. Once ``async`` and ``await`` > become reserved keywords in Python 3.7 this restriction will be > removed. Would this mean that with this PEP I can write a for loop like this? for (await?) item in async_generator(): ... code Or am I misunderstanding something? footnote: I am not really up to date with the async PEPs, so please correct me if this has already been discussed somewhere else. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Sat Sep 3 21:03:39 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 3 Sep 2016 18:03:39 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: <4bcff08a-e9a0-3b9f-be2a-742563ace4f0@gmail.com> Hi Matthias, On 2016-09-03 5:55 PM, Matthias welp wrote: > > In principle, asynchronous generator expressions are allowed in > > any context. However, in Python 3.6, due to ``async`` and ``await`` > > soft-keyword status, asynchronous generator expressions are only > > allowed in an ``async def`` function. Once ``async`` and ``await`` > > become reserved keywords in Python 3.7 this restriction will be > > removed. > > Would this mean that with this PEP I can write a for loop like this? > > for (await?) item in async_generator(): > ... code > > Or am I misunderstanding something? No, this is an illegal syntax and will stay that way. The PEP allows to do this: [await item for item in generator()] and this: [await item async for item in async_generator()] The idea is to remove any kind of restrictions that we currently have on async/await. A lot of people just assume that PEP 530 is already implemented in 3.5. Yury From steve at pearwood.info Sat Sep 3 21:20:58 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Sep 2016 11:20:58 +1000 Subject: [Python-ideas] Semantics for type checking (was: What should a good type checker do?) In-Reply-To: References: Message-ID: <20160904012058.GJ26300@ando.pearwood.info> On Sun, Sep 04, 2016 at 12:26:58AM +0300, Koos Zevenhoven wrote: > On Friday, while replying to a post on python-dev about PEP 526 > (variable annotations), I ended up mentioning things that I think a > good type checker should do, which seem to differ from the general > understanding. The discussion should continue here, though, because > the aim of PEP 526 is not to define type-checking semantics. Why should it continue here? The semantics of a type-checker is not part of Python the language. This is "python-ideas", not "third-party- application-ideas". If this thread belongs anywhere, it is probably the code quality mailing list. It is very likely that different linters and type-checkers will take different approaches. Some may infer more, some may infer less, and that's a good thing. Before arguing what a type-checker should or shouldn't do, perhaps we ought to look at what MyPy, PyLint, PyChecker, PyFlakes and Jedi already do. (Apologies if I have missed anyone's favourite linter/checker.) We should not try to specify the behaviour of the type-checker. Let the type-checkers compete in the wider Python eco-system and converge to best practice as found by experience, not because we declare by fiat that Thou Shalt Do It This Way. -- Steve From drekin at gmail.com Sun Sep 4 06:10:11 2016 From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=) Date: Sun, 4 Sep 2016 12:10:11 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions Message-ID: Hello, will await be allowed also in the "if" part of comprehensions? And what about the "in" part? (e.g. if I'm not mistaken, we may have an asynchronous function returning an asynchronous iterator.) Regards, Adam Barto? -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Sun Sep 4 07:06:38 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 4 Sep 2016 14:06:38 +0300 Subject: [Python-ideas] Semantics for type checking (was: What should a good type checker do?) In-Reply-To: <20160904012058.GJ26300@ando.pearwood.info> References: <20160904012058.GJ26300@ando.pearwood.info> Message-ID: On Sun, Sep 4, 2016 at 4:20 AM, Steven D'Aprano wrote: > On Sun, Sep 04, 2016 at 12:26:58AM +0300, Koos Zevenhoven wrote: >> On Friday, while replying to a post on python-dev about PEP 526 >> (variable annotations), I ended up mentioning things that I think a >> good type checker should do, which seem to differ from the general >> understanding. The discussion should continue here, though, because >> the aim of PEP 526 is not to define type-checking semantics. > > Why should it continue here? The semantics of a type-checker is not part > of Python the language. This is "python-ideas", not "third-party- > application-ideas". If this thread belongs anywhere, it is probably the > code quality mailing list. > I don't think you can 100% separate type checking syntax and semantics. We should try to imagine desired future semantics as well as possible before making decisions about syntax. > It is very likely that different linters and type-checkers will take > different approaches. Some may infer more, some may infer less, and > that's a good thing. I agree that's likely, but I'm not sure that is ultimately a good thing. Type-checking semantics affect the way you write code, so at least you cannot fully separate writing code from the kind of type checking you use. What should large code bases do? What if you use different libraries that have been annotated for different semantics? > Before arguing what a type-checker should or shouldn't do, perhaps we > ought to look at what MyPy, PyLint, PyChecker, PyFlakes and Jedi already > do. (Apologies if I have missed anyone's favourite linter/checker.) Sure, arguments in any direction would be interesting. > We > should not try to specify the behaviour of the type-checker. Let the > type-checkers compete in the wider Python eco-system and converge to > best practice as found by experience, not because we declare by fiat > that Thou Shalt Do It This Way. > I'm not looking to fix type-checker behavior now. I do hope that convergence is not going to take forever. -- Koos > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- + Koos Zevenhoven + http://twitter.com/k7hoven + From k7hoven at gmail.com Sun Sep 4 07:28:15 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 4 Sep 2016 14:28:15 +0300 Subject: [Python-ideas] Semantics for type checking (was: What should a good type checker do?) In-Reply-To: References: Message-ID: On Sun, Sep 4, 2016 at 2:33 AM, Koos Zevenhoven wrote: > On Sun, Sep 4, 2016 at 2:22 AM, Chris Angelico wrote: >> On Sun, Sep 4, 2016 at 7:44 AM, Koos Zevenhoven wrote: >>>> I wonder if it would be different if you wrote that as a single expression: >>>> >>>> x = 1 if cond else 1.5 >>>> >>>> x = sum([1] + [0.5] * cond) >>>> >>>> What should type inference decide x is in these cases? Assume an >>>> arbitrarily smart type checker that can implement your ideal; it's >>>> equally plausible to pretend that the type checker can recognize an >>>> if/else block (or even if/elif/else tree of arbitrary length) as a >>>> single "assignment" operation. IMO both of these examples - and by >>>> extension, the if/else of the original - should be assigning a Union >>>> type. >>> >>> In the first case it would indeed again be Union[int, float] (the code >>> is basically equivalent after all). >>> >>> In the second case, the checker would infer List[Union[int, float]]. >>> No magic involved, assuming that list.__add__ is annotated precisely >>> enough. >> >> I think you missed the sum() call there. Most likely, the sum of >> List[Union[anything]] would be Union[anything], so these would all be >> the same (which was my original point). >> > > Oh, you're right, I missed the sum. > > Suitable @overloads in the stubs for sum should lead to that being > inferred as Union[int, float]. But if the annotations are not that > precise, that might become just List (or List[Any]), and the > programmer may want to annotate x or improve the stubs. > Let me try explaining this once more. If list.__add__ is imprecisely annotated, one may end up with just List or list. But the annotation is not that complicated. In the stubs you might have T1 = TypeVar('T1') T2 = TypeVar('T2') def __add__(self : List[T1], other : List[T2]) -> List[Union[T1,T2]]: ... And sum is also implemented in c, so no inference there (unless stubs can provide a reference Python implementation in the future, but I'm not sure I want to propose that). For sum, there could be an overload to handle this case: @overload def sum(iterable : Iterable[Union[int, float]], start : Union[int, float] = 0) -> Union[int, float]: ... In general the following is not strictly true: ?ef sum(iterable : Iterable[T], start: T) -> T: ... because sum uses __add__ which can return anything. However, the counterexamples are probably very rare. -- Koos > -- Koos > >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > + Koos Zevenhoven + http://twitter.com/k7hoven + -- + Koos Zevenhoven + http://twitter.com/k7hoven + From arek.bulski at gmail.com Sun Sep 4 16:06:44 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Sun, 4 Sep 2016 22:06:44 +0200 Subject: [Python-ideas] kw to be ordered dict Message-ID: I have a piece of code that essentially reimplements OrderedDict. I have been wondering why the key ordered is not preserved correctly (tests fail). c = Container(a=1,b=2,c=3,d=4) self.assertEqual(c.keys(), ["a","b","c","d"]) self.assertEqual(c.values(), [1,2,3,4]) self.assertEqual(c.items(), [("a",1),("b",2),("c",3),("d",4)]) Then I looked at collections.OrderedDict ctor docstring and it says "Initialize an ordered dictionary. The signature is the same as regular dictionaries, but keyword arguments are not recommended because their insertion order is arbitrary." I expected the **kw to reproduce the same order as actual keyword arguments. The only right way to solve it, as it seems to me, would be to make **kw an OrderedDict. Arkadiusz Bulski -------------- next part -------------- An HTML attachment was scrubbed... URL: From waultah at gmail.com Sun Sep 4 16:32:45 2016 From: waultah at gmail.com (Riley Banks) Date: Sun, 4 Sep 2016 21:32:45 +0100 Subject: [Python-ideas] kw to be ordered dict In-Reply-To: References: Message-ID: This has been discussed before. See for example https://mail.python.or g/pipermail/python-ideas/2010-October/008445.html There's even PEP about this: https://www.python.org/dev/peps/pep-0468 -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Mon Sep 5 06:28:33 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 5 Sep 2016 19:28:33 +0900 Subject: [Python-ideas] kw to be ordered dict In-Reply-To: References: Message-ID: And there is a patch already. http://bugs.python.org/issue27350 On Mon, Sep 5, 2016 at 5:32 AM, Riley Banks wrote: > This has been discussed before. See for example > https://mail.python.org/pipermail/python-ideas/2010-October/008445.html > > There's even PEP about this: https://www.python.org/dev/peps/pep-0468 > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- INADA Naoki From levkivskyi at gmail.com Mon Sep 5 18:57:58 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 6 Sep 2016 00:57:58 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On 4 September 2016 at 01:31, Yury Selivanov wrote: > Below is a proposal to add support for asynchronous comprehensions and > asynchronous generator expressions in Python 3.6. > Interesting proposal. Would be nice to have this! I have one question: > ``await`` in Comprehensions > --------------------------- > > We propose to allow the use of ``await`` expressions in both > asynchronous and synchronous comprehensions:: > > result = [await fun() for fun in funcs] > result = {await fun() for fun in funcs} > result = {fun: await fun() for fun in funcs} > > result = [await fun() async for fun in funcs] > result = {await fun() async for fun in funcs} > result = {fun: await fun() async for fun in funcs} > > This is only valid in ``async def`` function body. > Do I understand correctly that the limitation that they are allowed only in async def is because await binds to the enclosing coroutine? There is an old "bug" (some people call this a feature) http://bugs.python.org/issue10544 If one uses yield in a comprehension, then it leads to unexpected results: >>> def f(): ... yield ... res = [(yield) for i in range(3)] ... return res ... >>> fg = f() >>> next(fg) >>> next(fg) Traceback (most recent call last): File "", line 1, in StopIteration: . at 0x7fe05b37a498> This function yields only once and then returns another generator. This is because the yield in comprehension "lives" in an auxiliary function scope used to make the comprehension. So that such comprehension are even allowed outside function body: >>> [(yield) for i in range(3)] at 0x7fe05b3d41f0> >>> [(yield from range(3)) for i in range(3)] at 0x7fe05ade11f0> Do I understand correctly that this is not the case with asynchronous comprehensions? If this is not the case, then I like this, but this will be inconsistent with normal comprehensions. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Mon Sep 5 19:27:39 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 01:27:39 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: Hi Yury, just for my understanding: On 04.09.2016 01:31, Yury Selivanov wrote: > We propose to allow the use of ``await`` expressions in both > asynchronous and synchronous comprehensions:: > > result = [await fun() for fun in funcs] > result = {await fun() for fun in funcs} > result = {fun: await fun() for fun in funcs} > This will produce normal lists, sets and dicts, right? Whereas the following will produce some sort of async lists, sets, and dicts? > result = [await fun() async for fun in funcs] > result = {await fun() async for fun in funcs} > result = {fun: await fun() async for fun in funcs} If so, how do I read values from an async list/set/dict? Sven From arek.bulski at gmail.com Mon Sep 5 20:50:02 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Tue, 6 Sep 2016 02:50:02 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: We should add shuffled() analog to sorted. Also shuffle() should return self so mutating methods could be chained. -- Arkadiusz Bulski -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Sep 5 21:03:10 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 6 Sep 2016 11:03:10 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: <20160906010308.GU26300@ando.pearwood.info> On Tue, Sep 06, 2016 at 02:50:02AM +0200, Arek Bulski wrote: > We should add shuffled() analog to sorted. Also shuffle() should return > self so mutating methods could be chained. from random import shuffle shuffle(mylist) Why is that so important that it needs to be a built-in? In 15+ years of using Python, I've probably used shuffle() three or four times. -- Steve From mafagafogigante at gmail.com Mon Sep 5 21:09:27 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Mon, 5 Sep 2016 22:09:27 -0300 Subject: [Python-ideas] Shuffled In-Reply-To: <20160906010308.GU26300@ando.pearwood.info> References: <20160906010308.GU26300@ando.pearwood.info> Message-ID: On 09/05/2016 10:03 PM, Steven D'Aprano wrote: > > from random import shuffle > shuffle(mylist) > > Why is that so important that it needs to be a built-in? In 15+ years of > using Python, I've probably used shuffle() three or four times. > Just adding to this. Sorting is a problem that comes up in almost all applications, while shuffling simply does not. I think that it is relevant enough not to require a third-party library, but it doesn't already. From yselivanov.ml at gmail.com Mon Sep 5 21:16:08 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 5 Sep 2016 18:16:08 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: Hi Sven, On 2016-09-05 4:27 PM, Sven R. Kunze wrote: > Hi Yury, > > just for my understanding: > > On 04.09.2016 01:31, Yury Selivanov wrote: >> We propose to allow the use of ``await`` expressions in both >> asynchronous and synchronous comprehensions:: >> >> result = [await fun() for fun in funcs] >> result = {await fun() for fun in funcs} >> result = {fun: await fun() for fun in funcs} >> > > This will produce normal lists, sets and dicts, right? Right. > > Whereas the following will produce some sort of async lists, sets, and > dicts? > >> result = [await fun() async for fun in funcs] >> result = {await fun() async for fun in funcs} >> result = {fun: await fun() async for fun in funcs} > > If so, how do I read values from an async list/set/dict? Consider "funcs" to be an asynchronous generator/iterable that produces a sequence of awaitables. The above comprehensions will await on each awaitable in funcs, producing regular list, set, and dict. I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable. There is no concept of asynchronous datastructures in Python. Thanks, Yury > > Sven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Mon Sep 5 21:17:22 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Sep 2016 11:17:22 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On Tue, Sep 6, 2016 at 9:27 AM, Sven R. Kunze wrote: > Hi Yury, > > just for my understanding: > > On 04.09.2016 01:31, Yury Selivanov wrote: >> >> We propose to allow the use of ``await`` expressions in both >> asynchronous and synchronous comprehensions:: >> >> result = [await fun() for fun in funcs] >> result = {await fun() for fun in funcs} >> result = {fun: await fun() for fun in funcs} >> > > This will produce normal lists, sets and dicts, right? > > Whereas the following will produce some sort of async lists, sets, and > dicts? > >> result = [await fun() async for fun in funcs] >> result = {await fun() async for fun in funcs} >> result = {fun: await fun() async for fun in funcs} > > > If so, how do I read values from an async list/set/dict? AIUI they won't return "async lists" etc; what they'll do is asynchronously return list/set/dict. Imagine an async database query that returns Customer objects. You could get their names thus: names = [cust.name async for cust in find_customers()] And you could enumerate their invoices (another database query) with a double-async comprehension: invoices = {cust.name: await cust.list_invoices() async for cust in find_customers()} As always, you can unroll a comprehension to the equivalent statement form. _tmp = {} async for cust in find_customers(): _tmp[cust.name] = await cust.list_invoices() invoices = _tmp or with the original example: _tmp = [] async for fun in funcs: _tmp.append(await fun()) result = _tmp Hope that helps. ChrisA From yselivanov.ml at gmail.com Mon Sep 5 21:22:36 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 5 Sep 2016 18:22:36 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: <84f7b987-3583-f5fe-3fb0-093305fc954b@gmail.com> Hi Ivan, On 2016-09-05 3:57 PM, Ivan Levkivskyi wrote: > On 4 September 2016 at 01:31, Yury Selivanov > wrote: > > Below is a proposal to add support for asynchronous comprehensions and > asynchronous generator expressions in Python 3.6. > > > Interesting proposal. Would be nice to have this! > > I have one question: > > ``await`` in Comprehensions > --------------------------- > > We propose to allow the use of ``await`` expressions in both > asynchronous and synchronous comprehensions:: > > result = [await fun() for fun in funcs] > result = {await fun() for fun in funcs} > result = {fun: await fun() for fun in funcs} > > result = [await fun() async for fun in funcs] > result = {await fun() async for fun in funcs} > result = {fun: await fun() async for fun in funcs} > > This is only valid in ``async def`` function body. > > > Do I understand correctly that the limitation that they are allowed > only in async def is because await binds to the enclosing coroutine? Correct. > > There is an old "bug" (some people call this a feature) > http://bugs.python.org/issue10544 > If one uses yield in a comprehension, then it leads to unexpected results: > > >>> def f(): > ... yield > ... res = [(yield) for i in range(3)] > ... return res > ... > >>> fg = f() > >>> next(fg) > >>> next(fg) > Traceback (most recent call last): > File "", line 1, in > StopIteration: . at 0x7fe05b37a498> > > This function yields only once and then returns another generator. > This is because the yield in comprehension "lives" in an auxiliary > function > scope used to make the comprehension. So that such comprehension > are even allowed outside function body: > > >>> [(yield) for i in range(3)] > at 0x7fe05b3d41f0> > >>> [(yield from range(3)) for i in range(3)] > at 0x7fe05ade11f0> > > Do I understand correctly that this is not the case with > asynchronous comprehensions? > > If this is not the case, then I like this, but this will be > inconsistent with > normal comprehensions. I'm not sure what will happen here. Will have an update once a reference implementation is ready. Seems that the bug you're referring to is something that should be just fixed. Yury From rosuav at gmail.com Mon Sep 5 21:22:45 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Sep 2016 11:22:45 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: <20160906010308.GU26300@ando.pearwood.info> References: <20160906010308.GU26300@ando.pearwood.info> Message-ID: On Tue, Sep 6, 2016 at 11:03 AM, Steven D'Aprano wrote: > On Tue, Sep 06, 2016 at 02:50:02AM +0200, Arek Bulski wrote: > >> We should add shuffled() analog to sorted. Also shuffle() should return >> self so mutating methods could be chained. > > from random import shuffle > shuffle(mylist) > > Why is that so important that it needs to be a built-in? In 15+ years of > using Python, I've probably used shuffle() three or four times. What you have isn't quite parallel to shuffled(). It'd need to be more like: mylist = mylist[:] shuffle(mylist) to leave the original list untouched. Might make a nice utility function for someone's personal tool collection, if they're doing this a lot: def shuffled(stuff): lst = list(stuff) random.shuffle(lst) return lst Doesn't need - or want - to be a built-in, partly because shuffling depends on an RNG, which you might want to swap out. At best, it would be a method on the random.Random class... but that's not needed either. ChrisA From arek.bulski at gmail.com Mon Sep 5 21:59:10 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Tue, 6 Sep 2016 03:59:10 +0200 Subject: [Python-ideas] Shuffled Message-ID: shuffled() should be in the random module, of course. I dont suggest a builtin. Although now that you mentioned it, I could go for that too. There are usage cases where its heavily used, in randomized testing for example. I am sure that there are also other domains where randomization of lists is used. Another reason to put it there is that using shuffle is inconvenient. The fact that I CAN write it myself doesnt mean that it doesnt belong in the standard library. Implementing this in pure python wont take a lot of work. pozdrawiam, Arkadiusz Bulski -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Sep 5 22:39:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 6 Sep 2016 12:39:24 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: <20160906023924.GV26300@ando.pearwood.info> On Tue, Sep 06, 2016 at 03:59:10AM +0200, Arek Bulski wrote: > Implementing this in pure python wont take a lot of work. Great. Well, here's a tracker issue for it. http://bugs.python.org/issue27964 Can you provide a patch? Don't forget the tests and documentation. Providing a patch doesn't mean the patch will be accepted. But no patch means that, even if accepted, you are unlikely to see anything before Python 3.7, as the 3.6 feature freeze is less than a week away. -- Steve From mr.eightnoteight at gmail.com Mon Sep 5 23:46:49 2016 From: mr.eightnoteight at gmail.com (srinivas devaki) Date: Tue, 6 Sep 2016 09:16:49 +0530 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On Tue, Sep 6, 2016 at 6:46 AM, Yury Selivanov wrote: > Hi Sven, > > I doubt that anybody ever would write something like that; this is just > examples of what the PEP will enable. using the same object as iterator and async generator, may be not. but isn't it good to write code for sync iteration and async iteration in the same class? so that if I'm writing a library, I can say that the library supports both. Junior (3rd yr) student at Indian School of Mines,(IIT Dhanbad) Computer Science and Engineering Department ph: +91 9491 383 249 telegram_id: @eightnoteight From mahmoud at hatnote.com Tue Sep 6 00:34:05 2016 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Mon, 5 Sep 2016 21:34:05 -0700 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: I tend to agree with Arek. I've been bitten multiple times, including once yesterday, because shuffle works in place, when I really expect a sorted()-like behavior for a standalone function like that. Mahmoud https://github.com/mahmoud http://sedimental.org On Mon, Sep 5, 2016 at 6:59 PM, Arek Bulski wrote: > shuffled() should be in the random module, of course. I dont suggest a > builtin. Although now that you mentioned it, I could go for that too. > > There are usage cases where its heavily used, in randomized testing for > example. I am sure that there are also other domains where randomization of > lists is used. > > Another reason to put it there is that using shuffle is inconvenient. The > fact that I CAN write it myself doesnt mean that it doesnt belong in the > standard library. > > Implementing this in pure python wont take a lot of work. > > pozdrawiam, > Arkadiusz Bulski > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Sep 6 02:51:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 6 Sep 2016 16:51:24 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: <20160906065123.GA10468@ando.pearwood.info> On Mon, Sep 05, 2016 at 09:34:05PM -0700, Mahmoud Hashemi wrote: > I tend to agree with Arek. I've been bitten multiple times, including once > yesterday, because shuffle works in place, when I really expect a > sorted()-like behavior for a standalone function like that. *shrug* And if random.shuffle() returned a new list, other people would be bitten because they expected it to be in-place. You can't please everyone. In any case, the tracker item I opened has already been closed by the module maintainer Raymond Hettinger. I don't intend to pursue this, but if anyone wishes to change his mind, you will need: - good use-cases for the new function; - evidence that this is common enough to justify; - and (optional, but recommended) an actual patch. If you (generic "you", not Mahmoud or Arek specifically) aren't volunteering to do the work yourself, the barrier to convince somebody else to do it is much higher. One moderately stong piece of evidence would be if this function is widely available in third-party libraries and other languages. That is evidence that this is common enough that people are reinventing the wheel, and therefore we should consider adding a standard wheel. But I don't have the time (or that much interest) to do this, but I encourage others to do their homework if they want to make a strong case for this function. -- Steve From arek.bulski at gmail.com Tue Sep 6 05:43:54 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Tue, 6 Sep 2016 11:43:54 +0200 Subject: [Python-ideas] shuffled In-Reply-To: References: Message-ID: They closed it. Will they accept a patch if I send it? Where is the repo for the code? Never submitted anything to py code base. -- Arkadiusz Bulski -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue Sep 6 08:46:17 2016 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 6 Sep 2016 08:46:17 -0400 Subject: [Python-ideas] shuffled In-Reply-To: References: Message-ID: On 09/06/2016 05:43 AM, Arek Bulski wrote: > They closed it. Will they accept a patch if I send it? In all likelihood, no. However, if you want to advocate for this change, and try to change their minds, I suggest you read and follow the suggestions that Steven D'Aprano gave you in an earlier email (https://mail.python.org/pipermail/python-ideas/2016-September/042107.html). > Where is the repo for the code? Never submitted anything to py code base. Everything you need to know is at https://docs.python.org/devguide/ Eric. From steve at pearwood.info Tue Sep 6 10:02:43 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Sep 2016 00:02:43 +1000 Subject: [Python-ideas] shuffled In-Reply-To: References: Message-ID: <20160906140242.GW26300@ando.pearwood.info> On Tue, Sep 06, 2016 at 11:43:54AM +0200, Arek Bulski wrote: > They closed it. Will they accept a patch if I send it? Truth is, probably not. You would need to convince Raymond (and to a lesser extent, Tim) that this is a good idea. See my last email for the *minimum* of what you would need to show. Without a patch, there is no chance this will happen: I'm not going to do it, and Raymond definitely won't since he doesn't think it is necessary. With a patch, you might have a tiny chance. But frankly, I think you would be better off just adding it to your own personal toolbox of functions. Not everything has to be in the standard library. > Where is the repo for the code? Never submitted anything to py code base. https://docs.python.org/devguide/ -- Steve From srkunze at mail.de Tue Sep 6 11:06:26 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 17:06:26 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: <454f59ac-8668-5ecc-7ff1-990131a9d453@mail.de> If so, then I would suggest than each list provides a .shuffle method as well (just as sorted/sort does). On 06.09.2016 06:34, Mahmoud Hashemi wrote: > I tend to agree with Arek. I've been bitten multiple times, including > once yesterday, because shuffle works in place, when I really expect a > sorted()-like behavior for a standalone function like that. > > Mahmoud > https://github.com/mahmoud > http://sedimental.org > > On Mon, Sep 5, 2016 at 6:59 PM, Arek Bulski > wrote: > > shuffled() should be in the random module, of course. I dont > suggest a builtin. Although now that you mentioned it, I could go > for that too. > > There are usage cases where its heavily used, in randomized > testing for example. I am sure that there are also other domains > where randomization of lists is used. > > Another reason to put it there is that using shuffle is > inconvenient. The fact that I CAN write it myself doesnt mean that > it doesnt belong in the standard library. > > Implementing this in pure python wont take a lot of work. > > pozdrawiam, > Arkadiusz Bulski > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Sep 6 11:20:41 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 17:20:41 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: <9ac16998-a479-5a7a-53ec-241aad769047@mail.de> On 06.09.2016 03:16, Yury Selivanov wrote: > >> Whereas the following will produce some sort of async lists, sets, >> and dicts? >> >>> result = [await fun() async for fun in funcs] >>> result = {await fun() async for fun in funcs} >>> result = {fun: await fun() async for fun in funcs} >> >> If so, how do I read values from an async list/set/dict? > > Consider "funcs" to be an asynchronous generator/iterable that > produces a sequence of awaitables. The above comprehensions will > await on each awaitable in funcs, producing regular list, set, and dict. So, what's the "async" good for then? > I doubt that anybody ever would write something like that; this is > just examples of what the PEP will enable. Why do you implement it then? :D Put it differently, why are you sceptic about it? > There is no concept of asynchronous datastructures in Python. I thought so, that's why I asked. ;) "async def" gives me something async, so I assumed it to be the case here as well. Cheers, Sven From arek.bulski at gmail.com Tue Sep 6 11:29:18 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Tue, 6 Sep 2016 17:29:18 +0200 Subject: [Python-ideas] Shuffled Message-ID: > And if random.shuffle() returned a new list, other people would be bitten because they expected it to be in-place. No one proposes that. shuffled() should be a new function. > One moderately stong piece of evidence would be if this function is widely available in third-party libraries and other languages. Wrong. Python is NOT known for doing everything by how it was done before. My father used statically typed variables and his father used statically typed variables so I will use... > If so, then I would suggest than each list provides a .shuffle method as well (just as sorted/sort does). Well, that could be problematic because std lib would have to use a random generator. We should leave things that require a source of randomness somewhere in the random module. Would you agree on that? Randomized unit testing is obvious example. Besides, any argument that can be used for having shuffle() could be also made for adding shuffled(). Randomizing lists has been used for a long time. We only need an non mutating analog. I am willing to make a patch. pozdrawiam, Arkadiusz Bulski -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Sep 6 12:32:57 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Sep 2016 02:32:57 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: <20160906163257.GX26300@ando.pearwood.info> On Tue, Sep 06, 2016 at 05:29:18PM +0200, Arek Bulski wrote: [I think Arek is quoting me here] > > One moderately stong piece of evidence would be if this function is widely > > available in third-party libraries and other languages. > > Wrong. Python is NOT known for doing everything by how it was done before. That's not what I said. I'm telling you what you need to do to have the best chance of your proposal being accepted. It isn't enough to just write a patch. Python is not *your* language where you get to decide what goes in and out, it is Guido's language, and he trusts the community to come to a consensus for these things. Part of that process is to trust respected core developers like Raymond Hettinger and Tim Peters to make decisions. I have been following the Python-Ideas mailing list for, oh, probably ten years. Maybe fifteen. I've seen hundreds, maybe thousands of suggestions and ideas rejected, and dozens accepted, including some of my own. If your only argument is to continue to insist that Python should have a shuffled() function and you'll write a patch, it will go no where. First you have to convince people that the patch is needed. > My father used statically typed variables and his father used statically > typed variables so I will use... And my grandfather and my father used dynamically typed variables. Lisp had dynamically typed variables back in the 1950s, and Smalltalk had them in the 1980s. So what? -- Steve From anthony at xtfx.me Tue Sep 6 12:40:27 2016 From: anthony at xtfx.me (C Anthony Risinger) Date: Tue, 6 Sep 2016 11:40:27 -0500 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <9ac16998-a479-5a7a-53ec-241aad769047@mail.de> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <9ac16998-a479-5a7a-53ec-241aad769047@mail.de> Message-ID: On Tue, Sep 6, 2016 at 10:20 AM, Sven R. Kunze wrote: > On 06.09.2016 03:16, Yury Selivanov wrote: > >> >> Whereas the following will produce some sort of async lists, sets, and >>> dicts? >>> >>> result = [await fun() async for fun in funcs] >>>> result = {await fun() async for fun in funcs} >>>> result = {fun: await fun() async for fun in funcs} >>>> >>> >>> If so, how do I read values from an async list/set/dict? >>> >> >> Consider "funcs" to be an asynchronous generator/iterable that produces a >> sequence of awaitables. The above comprehensions will await on each >> awaitable in funcs, producing regular list, set, and dict. >> > > So, what's the "async" good for then? Maybe I'm off base here, but my understanding is the `async for` version would allow for suspension during the actual iteration, ie. using the __a*__ protocol methods, and not just by awaiting on the produced item itself. IOW, `[await ... async for ...]` will suspend at least twice, once during iteration using the __a*__ protocols and then again awaiting on the produced item, whereas `[await ... for ...]` will synchronously produce items and then suspend on them. So to use the async + await version, your (async) iterator must return awaitables to satisfy the `async for` part with then produce another awaitable we explicitly `await` on. Can someone confirm this understanding? And also that all 4 combinations are possible, each with different meaning: # Normal comprehension, 100% synchronous and blocking [... for ...] # Blocking/sync iterator producing awaitables which suspend before producing a "normal" value [await ... for ...] # Non-blocking/async iterator that suspends before producing "normal" values [... async for ...] # Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value [await ... async for ...] Is this correct? -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 6 13:38:17 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2016 03:38:17 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <84f7b987-3583-f5fe-3fb0-093305fc954b@gmail.com> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <84f7b987-3583-f5fe-3fb0-093305fc954b@gmail.com> Message-ID: On 6 September 2016 at 11:22, Yury Selivanov wrote: > On 2016-09-05 3:57 PM, Ivan Levkivskyi wrote: >> There is an old "bug" (some people call this a feature) >> http://bugs.python.org/issue10544 >> If one uses yield in a comprehension, then it leads to unexpected results: > > I'm not sure what will happen here. Will have an update once a reference > implementation is ready. Seems that the bug you're referring to is > something that should be just fixed. I was going to say that the problem with that last sentence is the "just" part, as for a long time, even though the more desirable behaviour was relatively clear ("make it work the way it did in Python 2"), we didn't have a readily available means of doing that. However, the last time I thought seriously about this problem was before we added "yield from", and that may actually be enough to change the situation. Specifically, the problem is that comprehensions and generator expressions in 3.x create a full closure, so code like: def example(): L = [(yield) for i in range(2)] print(L) is implicitly doing: def example(): def _bad_gen(_arg): result = [] for i in _arg: result.append((yield)) return result L = _bad_gen(range(2)) print(L) Prior to yield from, the code generator had no easy way to fix that, since it couldn't delegate yield operations to the underlying generator. However, given PEP 380, the code generator could potentially detect these situations, and implicitly use "yield from" to hoist the generator behaviour up to the level of the containing function, exactly as happened in Python 2: def example(): def _nested_gen(_arg): result = [] for i in _arg: result.append((yield)) return result L = yield from _nested_gen(range(2)) print(L) >>> gen = example() >>> next(gen) >>> gen.send(1) >>> gen.send(2) [1, 2] Traceback (most recent call last): File "", line 1, in StopIteration I don't think there's anything we can do about generator expressions short of PEP 530 itself though - they already misbehave in Python 2, and misbehave in exactly the same way in Python 3, since there's no way for the interpreter to tell the difference at runtime between the implicit yields from the generator expression itself, and any explicit yields used in generator subexpressions. By contrast, PEP 530 can work, since it doesn't *want* to tell the difference between the implicit awaits and explicit awaits, and the await and yield channels are already distinguished at the language level. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From yselivanov.ml at gmail.com Tue Sep 6 13:38:38 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 6 Sep 2016 10:38:38 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <9ac16998-a479-5a7a-53ec-241aad769047@mail.de> Message-ID: <018ec4f8-b881-9bf4-574a-a833e85afa2b@gmail.com> On 2016-09-06 9:40 AM, C Anthony Risinger wrote: > On Tue, Sep 6, 2016 at 10:20 AM, Sven R. Kunze > wrote: > > On 06.09.2016 03:16, Yury Selivanov wrote: > > > Whereas the following will produce some sort of async > lists, sets, and dicts? > > result = [await fun() async for fun in funcs] > result = {await fun() async for fun in funcs} > result = {fun: await fun() async for fun in funcs} > > > If so, how do I read values from an async list/set/dict? > > > Consider "funcs" to be an asynchronous generator/iterable that > produces a sequence of awaitables. The above comprehensions > will await on each awaitable in funcs, producing regular list, > set, and dict. > > > So, what's the "async" good for then? > > > Maybe I'm off base here, but my understanding is the `async for` > version would allow for suspension during the actual iteration, ie. > using the __a*__ protocol methods, and not just by awaiting on the > produced item itself. > > IOW, `[await ... async for ...]` will suspend at least twice, once > during iteration using the __a*__ protocols and then again awaiting on > the produced item, whereas `[await ... for ...]` will synchronously > produce items and then suspend on them. So to use the async + await > version, your (async) iterator must return awaitables to satisfy the > `async for` part with then produce another awaitable we explicitly > `await` on. > > Can someone confirm this understanding? And also that all 4 > combinations are possible, each with different meaning: > > # Normal comprehension, 100% synchronous and blocking > [... for ...] > > # Blocking/sync iterator producing awaitables which suspend before > producing a "normal" value > [await ... for ...] > > # Non-blocking/async iterator that suspends before producing "normal" > values > [... async for ...] > > # Non-blocking/async iterator that suspends before producing > awaitables which suspend again before producing a "normal" value > [await ... async for ...] > > Is this correct? All correct. I'll update the PEP to better clarify the semantics. Yury From yselivanov.ml at gmail.com Tue Sep 6 13:42:03 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 6 Sep 2016 10:42:03 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: Message-ID: <38d25b0e-6a64-d27b-1718-c68a11a8b779@gmail.com> On 2016-09-04 3:10 AM, Adam Barto? wrote: > Hello, > > will await be allowed also in the "if" part of comprehensions? And > what about the "in" part? (e.g. if I'm not mistaken, we may have an > asynchronous function returning an asynchronous iterator.) Yes, awaits will be allowed. I'll update the PEP. Yury From srkunze at mail.de Tue Sep 6 13:57:23 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 19:57:23 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <018ec4f8-b881-9bf4-574a-a833e85afa2b@gmail.com> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <9ac16998-a479-5a7a-53ec-241aad769047@mail.de> <018ec4f8-b881-9bf4-574a-a833e85afa2b@gmail.com> Message-ID: <1335f97b-2edb-9983-20f0-5d08e1b80391@mail.de> On 06.09.2016 19:38, Yury Selivanov wrote: > > > On 2016-09-06 9:40 AM, C Anthony Risinger wrote: >> On Tue, Sep 6, 2016 at 10:20 AM, Sven R. Kunze > > wrote: >> >> So, what's the "async" good for then? >> >> >> Maybe I'm off base here, but my understanding is the `async for` >> version would allow for suspension during the actual iteration, ie. >> using the __a*__ protocol methods, and not just by awaiting on the >> produced item itself. >> >> IOW, `[await ... async for ...]` will suspend at least twice, once >> during iteration using the __a*__ protocols and then again awaiting >> on the produced item, whereas `[await ... for ...]` will >> synchronously produce items and then suspend on them. So to use the >> async + await version, your (async) iterator must return awaitables >> to satisfy the `async for` part with then produce another awaitable >> we explicitly `await` on. >> >> Can someone confirm this understanding? And also that all 4 >> combinations are possible, each with different meaning: >> >> # Normal comprehension, 100% synchronous and blocking >> [... for ...] >> >> # Blocking/sync iterator producing awaitables which suspend before >> producing a "normal" value >> [await ... for ...] >> >> # Non-blocking/async iterator that suspends before producing "normal" >> values >> [... async for ...] >> >> # Non-blocking/async iterator that suspends before producing >> awaitables which suspend again before producing a "normal" value >> [await ... async for ...] >> >> Is this correct? > > All correct. I'll update the PEP to better clarify the semantics. Still don't understand what the "async" is good for. Seems redundant to me. Sven From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Sep 6 14:12:51 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 7 Sep 2016 03:12:51 +0900 Subject: [Python-ideas] Shuffled In-Reply-To: <20160906163257.GX26300@ando.pearwood.info> References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: <22479.1827.364734.282768@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > If your only argument is to continue to insist that Python should > have a shuffled() function and you'll write a patch, it will go no > where. First you have to convince people that the patch is needed. It's not worth the effort. The previous attempt (issue26393) to get a shuffled function had a patch (which reduces to turning the expression "random.Random.sample(x, len(x))" into a function), and it was summarily rejected then, too. By different people, not quite evidence of a consensus, but put it this way: you're spitting into a gale. That patch gets 95%, btw ... it had docs, tests, and a NEWS entry first try. I'd want a few stylistic changes, but altogether a copacetic patch. Even so, no mercy was shown. From srkunze at mail.de Tue Sep 6 14:15:23 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 20:15:23 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: <20160906163257.GX26300@ando.pearwood.info> References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: Hi Steven, On 06.09.2016 18:32, Steven D'Aprano wrote: > On Tue, Sep 06, 2016 at 05:29:18PM +0200, Arek Bulski wrote: > > [I think Arek is quoting me here] >>> One moderately stong piece of evidence would be if this function is widely >>> available in third-party libraries and other languages. >> Wrong. Python is NOT known for doing everything by how it was done before. > That's not what I said. > > I'm telling you what you need to do to have the best chance of your > proposal being accepted. It isn't enough to just write a patch. > > Python is not *your* language where you get to decide what goes in and > out, it is Guido's language, and he trusts the community to come to a > consensus for these things. Part of that process is to trust respected > core developers like Raymond Hettinger and Tim Peters to make decisions. > > I have been following the Python-Ideas mailing list for, oh, probably > ten years. Maybe fifteen. I've seen hundreds, maybe thousands of > suggestions and ideas rejected, and dozens accepted, including some > of my own. > > If your only argument is to continue to insist that Python should have a > shuffled() function and you'll write a patch, it will go no where. First > you have to convince people that the patch is needed. He already convinced some people. Just not some venerable Python devs, which doesn't necessarily mean something at all. Their response: "Oh, I don't need it, let's close it." Arek: "But I need it." So, who's right now? I would suggest not being so protective. It's community project after all. Maybe, the patch will help him understand that it's not so easy, or other people will see the benefit after reading it. If the patch isn't worth including, he learned something in the process of creating it. >> My father used statically typed variables and his father used statically >> typed variables so I will use... > And my grandfather and my father used dynamically typed variables. Lisp > had dynamically typed variables back in the 1950s, and Smalltalk had > them in the 1980s. So what? > > Best, Sven From rosuav at gmail.com Tue Sep 6 14:17:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Sep 2016 04:17:56 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: On Wed, Sep 7, 2016 at 4:15 AM, Sven R. Kunze wrote: > It's community project after all. What's that mean, exactly? That the community gets to vote on what goes into Python? Because we don't. ChrisA From srkunze at mail.de Tue Sep 6 14:24:41 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 20:24:41 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> Hi Srinivas, On 06.09.2016 05:46, srinivas devaki wrote: > On Tue, Sep 6, 2016 at 6:46 AM, Yury Selivanov wrote: >> Hi Sven, >> >> I doubt that anybody ever would write something like that; this is just >> examples of what the PEP will enable. > using the same object as iterator and async generator, may be not. > but isn't it good to write code for sync iteration and async iteration > in the same class? > so that if I'm writing a library, I can say that the library supports both. oh, wrong territory here! Python async community wants you to write everything twice: for the sync and async case. And don't dare to mentioned code sharing here. They will rip you apart. ;) Just kidding. Of course would it be great to write code only once but Yury want to preserve well-paid Python dev jobs in the industry because everything here needs to be maintained twice then. ;) No really, I have absolutely no idea why you need to put that "async" in all places where Python can detect automatically if it needs to perform an async iteration or not. Maybe, Yury can explain. Cheers, Sven From elazarg at gmail.com Tue Sep 6 14:25:45 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 06 Sep 2016 18:25:45 +0000 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: Naive shuffled() can be emulated using a single expression: sorted(lst, key=lambda _: random()) So there's even less incentive for standardization. ~Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Sep 6 14:37:13 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 20:37:13 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: <53d8e42e-08f3-0d30-1467-24e87bec5786@mail.de> Are you serious? The reason for "we don't need standardization" == "there is a solution, long, and with a lot of special characters". Holy c***. I remember Python being the least verbose language on the planet. I suspect this argument didn't lead to status quo. It's like saying, we don't need variable annotations because we have comments. On 06.09.2016 20:25, ????? wrote: > Naive shuffled() can be emulated using a single expression: > > sorted(lst, key=lambda _: random()) > > So there's even less incentive for standardization. Besides being a silly argument, it's an interesting solution. Does it really work? I remember Microsoft utilizing a similar approach for their browser selection tool which led to a skewed probability distribution. Maybe, I wrong here though. Cheer, Sven From ncoghlan at gmail.com Tue Sep 6 14:37:19 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2016 04:37:19 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> Message-ID: On 7 September 2016 at 04:24, Sven R. Kunze wrote: > Python async community wants you to write everything twice: for the sync and > async case. And don't dare to mentioned code sharing here. They will rip you > apart. ;) > > Just kidding. Of course would it be great to write code only once but Yury > want to preserve well-paid Python dev jobs in the industry because > everything here needs to be maintained twice then. ;) Sven, this is not productive, not funny, and not welcome. Vent your frustrations with the fundamental split between synchronous and explicitly asynchronous software design elsewhere. > No really, I have absolutely no idea why you need to put that "async" in all > places where Python can detect automatically if it needs to perform an async > iteration or not. Maybe, Yury can explain. As Anthony already noted, the "async" keyword switches to the asynchronous version of the iterator protocol - you use this when your *iterator* needs to interact with the event loop, just as you do when deciding whether or not to mark a for loop as asynchronous. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From k7hoven at gmail.com Tue Sep 6 14:45:00 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 6 Sep 2016 21:45:00 +0300 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> Message-ID: On Tue, Sep 6, 2016 at 9:24 PM, Sven R. Kunze wrote: [...] > No really, I have absolutely no idea why you need to put that "async" in all > places where Python can detect automatically if it needs to perform an async > iteration or not. Maybe, Yury can explain. I'm sure he would explain, but it seems I was first ;) [last-minute edit: no, Nick was first, but this is a slightly different angle]. First, the "async" gets inherited from PEP 492, so this has actually already been decided on. While not strictly necessary for a syntax for "async for", it makes it more explicit what happens under the hood -- that __a*__ methods are called and awaited, instead of simply calling __iter__/__next__ etc. as in regular loops/comprehensions. Not a lot to debate, I guess. No surprises here, just implementation work. -- Koos > Cheers, > Sven > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- + Koos Zevenhoven + http://twitter.com/k7hoven + From tim.peters at gmail.com Tue Sep 6 14:46:14 2016 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 6 Sep 2016 13:46:14 -0500 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: [Sven R. Kunze ] > ... > He already convinced some people. Just not some venerable Python devs, which > doesn't necessarily mean something at all. > > Their response: "Oh, I don't need it, let's close it." > Arek: "But I need it." > > So, who's right now? By default, the venerable Python devs ;-) As I said in my comments on the issue report, I searched through all my code for instances of `shuffle()`, and didn't find any that could be improved by `shuffled()`. Nor could I dream up a compelling use case. So it wasn't just a knee-jerk "oh, I don't need it", the "I don't need it" was the outcome of some research and thought. In contrast, the original request was a bare "But I need it", without so much as a single concrete use case to justify it ("testing" is an application area, not a concrete use case - I've made extensive use of shuffling in testing too, but - as I said - found no existing code where `shuffled()` would have helped). In the absence of compelling concrete use cases, new ideas have approximately no chance of being adopted. Otherwise the language & libraries bloat in counterproductive ways. Just ask any venerable Python dev ;-) From mafagafogigante at gmail.com Tue Sep 6 14:46:49 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Tue, 6 Sep 2016 15:46:49 -0300 Subject: [Python-ideas] Shuffled In-Reply-To: <53d8e42e-08f3-0d30-1467-24e87bec5786@mail.de> References: <20160906163257.GX26300@ando.pearwood.info> <53d8e42e-08f3-0d30-1467-24e87bec5786@mail.de> Message-ID: <2b626260-dd26-43d5-89b7-607f6d5fe194@gmail.com> On 09/06/2016 03:37 PM, Sven R. Kunze wrote: > > Besides being a silly argument, it's an interesting solution. > > Does it really work? I remember Microsoft utilizing a similar approach > for their browser selection tool which led to a skewed probability > distribution. Maybe, I wrong here though. > Yes. The key is evaluated only once, so each element gets a pseudo-random number. Sorting this list leads to a shuffle. However, a super-linear shuffle, whilst Fisher-Yates is a linear solution and also requires less additional memory. Lastly, although it is obvious, from a software engineering standpoint, this is a mere weird hack. From srkunze at mail.de Tue Sep 6 14:54:50 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 20:54:50 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> Message-ID: <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> On 06.09.2016 20:37, Nick Coghlan wrote: > On 7 September 2016 at 04:24, Sven R. Kunze wrote: >> Python async community wants you to write everything twice: for the sync and >> async case. And don't dare to mentioned code sharing here. They will rip you >> apart. ;) >> >> Just kidding. Of course would it be great to write code only once but Yury >> want to preserve well-paid Python dev jobs in the industry because >> everything here needs to be maintained twice then. ;) > Sven, this is not productive, not funny, and not welcome. Vent your > frustrations with the fundamental split between synchronous and > explicitly asynchronous software design elsewhere. Don't make a mistake here, Nick. I take this with some humor as it does not concern me in production. It's interesting to see though that people new to the discussion detect this obvious issue very fast. > >> No really, I have absolutely no idea why you need to put that "async" in all >> places where Python can detect automatically if it needs to perform an async >> iteration or not. Maybe, Yury can explain. > As Anthony already noted, the "async" keyword switches to the > asynchronous version of the iterator protocol - you use this when your > *iterator* needs to interact with the event loop, just as you do when > deciding whether or not to mark a for loop as asynchronous. Of course "async" switches to async mode. But that was not the question. I asked WHY that's necessary not what it does. I already noted that Python can detect when to make the switch without a marker. And you fail to explain where the issue with this point of view is. Sven PS: Nick, I noted that while replying, my mail client made me responding to you and the list as cc. Is there something wrong with my config or is this deliberate on your part? From elazarg at gmail.com Tue Sep 6 14:57:23 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 06 Sep 2016 18:57:23 +0000 Subject: [Python-ideas] Shuffled In-Reply-To: <2b626260-dd26-43d5-89b7-607f6d5fe194@gmail.com> References: <20160906163257.GX26300@ando.pearwood.info> <53d8e42e-08f3-0d30-1467-24e87bec5786@mail.de> <2b626260-dd26-43d5-89b7-607f6d5fe194@gmail.com> Message-ID: (Just to be clear, I wasn't trying to suggest this as more than an ad-hoc solution for a throwaway script. But to me, "sorted by random key" is almost as obvious as "shuffled", perhaps more so for non english speakers with little background in CS terms; the words "sorted" and "random" jumps to the eye, and sometimes you don't need more than that) ~Elazar ?????? ??? ??, 6 ????' 2016, 21:48, ??? Bernardo Sulzbach ?< mafagafogigante at gmail.com>: > On 09/06/2016 03:37 PM, Sven R. Kunze wrote: > > > > Besides being a silly argument, it's an interesting solution. > > > > Does it really work? I remember Microsoft utilizing a similar approach > > for their browser selection tool which led to a skewed probability > > distribution. Maybe, I wrong here though. > > > > Yes. The key is evaluated only once, so each element gets a > pseudo-random number. Sorting this list leads to a shuffle. > > However, a super-linear shuffle, whilst Fisher-Yates is a linear > solution and also requires less additional memory. > > Lastly, although it is obvious, from a software engineering standpoint, > this is a mere weird hack. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Sep 6 15:09:47 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 21:09:47 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: On 06.09.2016 20:46, Tim Peters wrote: > [Sven R. Kunze ] >> ... >> He already convinced some people. Just not some venerable Python devs, which >> doesn't necessarily mean something at all. >> >> Their response: "Oh, I don't need it, let's close it." >> Arek: "But I need it." >> >> So, who's right now? > By default, the venerable Python devs ;-) Well, little bit arrogant? I hope technical arguments weigh more than status here. ;) > [I don't need it in my source code which is under version control.] It's okay. You don't need to reiterate that you don't need it. I also don't need it when looking at my checked-in source code (but it seems there are people who do). But I remember using it in interactive sessions (damn, why didn't I check that into git?). And there I remember finding it quite irritating not having a return value at all. >> random.shuffle([1,2,3,4]) Just gives me: nothing. Of course, the period of irritation lasted short and reading __doc__ helped a lot, but, well, I could have saved me some time. You get the idea. Cheers, Sven From srkunze at mail.de Tue Sep 6 15:20:55 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 21:20:55 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: <2b626260-dd26-43d5-89b7-607f6d5fe194@gmail.com> References: <20160906163257.GX26300@ando.pearwood.info> <53d8e42e-08f3-0d30-1467-24e87bec5786@mail.de> <2b626260-dd26-43d5-89b7-607f6d5fe194@gmail.com> Message-ID: <6d195dfe-533f-50a9-1cfb-d8d58639b9e2@mail.de> On 06.09.2016 20:46, Bernardo Sulzbach wrote: > On 09/06/2016 03:37 PM, Sven R. Kunze wrote: >> >> Besides being a silly argument, it's an interesting solution. >> >> Does it really work? I remember Microsoft utilizing a similar approach >> for their browser selection tool which led to a skewed probability >> distribution. Maybe, I wrong here though. >> > > Yes. The key is evaluated only once, so each element gets a > pseudo-random number. Sorting this list leads to a shuffle. Ah yes, that might make it work. > However, a super-linear shuffle, whilst Fisher-Yates is a linear > solution and also requires less additional memory. > > Lastly, although it is obvious, from a software engineering > standpoint, this is a mere weird hack. Sure, but it works. ;) Though I agree that's not really intuitive if you are looking for "just shuffle that list, please". Cheers, Sven PS: here's the statistical analysis of the browser ballot issue with some explanations on the different algorithms on shuffling: http://www.robweir.com/blog/2010/02/microsoft-random-browser-ballot.html From srkunze at mail.de Tue Sep 6 15:27:22 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 6 Sep 2016 21:27:22 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> Message-ID: <9aa2bf24-76de-363c-bd5f-82586bb3766c@mail.de> On 06.09.2016 20:45, Koos Zevenhoven wrote: > On Tue, Sep 6, 2016 at 9:24 PM, Sven R. Kunze wrote: > [...] >> No really, I have absolutely no idea why you need to put that "async" in all >> places where Python can detect automatically if it needs to perform an async >> iteration or not. Maybe, Yury can explain. > I'm sure he would explain, but it seems I was first ;) [last-minute > edit: no, Nick was first, but this is a slightly different angle]. > > First, the "async" gets inherited from PEP 492, so this has actually > already been decided on. While not strictly necessary for a syntax for > "async for", it makes it more explicit what happens under the hood -- > that __a*__ methods are called and awaited, instead of simply calling > __iter__/__next__ etc. as in regular loops/comprehensions. > > Not a lot to debate, I guess. No surprises here, just implementation work. Of course, I would do the same. I value consistency a lot (but the issue here remains). :) Cheers, Sven From tim.peters at gmail.com Tue Sep 6 15:29:56 2016 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 6 Sep 2016 14:29:56 -0500 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: [Sven R. Kunze ] >>> >>> ... >>> He already convinced some people. Just not some venerable Python devs, >>> which>> doesn't necessarily mean something at all. >>> >>> Their response: "Oh, I don't need it, let's close it." >>> Arek: "But I need it." >>> >>> So, who's right now? [Tim] >> By default, the venerable Python devs ;-) [Sven] > Well, little bit arrogant? I hope technical arguments weigh more than status > here. ;) I haven't seen any technical arguments here. You snipped the parts of my reply where I emphasized the need for concrete use cases. In the absence of anything approaching that, it's a matter of taste, and then - yes - decades of Python development experience do - and should - outweigh a newcomer's wish list. Especially when, as appears to be the case here, those with truly extensive experience agree. > ... > But I remember using it in interactive sessions (damn, why didn't I check > that into git?). And there I remember finding it quite irritating not having > a return value at all. > > >>> random.shuffle([1,2,3,4]) > > Just gives me: nothing. > > Of course, the period of irritation lasted short and reading __doc__ helped > a lot, but, well, I could have saved me some time. You get the idea. No, I don't. It's an obvious feature of Python's list object design that mutating methods always return None. Which means "obvious" after someone has learned the language, not obvious the first time they try it. StackOverflow is equally full of newbie complaints that, e.g., some_list.sort().append(100) gives an "incomprehensible" AttributeError: 'NoneType' object has no attribute 'append' error. Once "ah, mutating methods generally return None" sinks in, they never post any complaint like that again. I would be far more annoyed if, e.g., >>> random.shuffle(some_million_element_list) swamped my terminal with mountains of output. You can't both behaviors simultaneously, so the status quo wins. Indeed, the venerable status quo ;-) From tjreedy at udel.edu Tue Sep 6 15:45:29 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 6 Sep 2016 15:45:29 -0400 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> Message-ID: On 9/6/2016 2:54 PM, Sven R. Kunze wrote: > PS: Nick, I noted that while replying, my mail client made me responding > to you and the list as cc. For Thunderbird, this is normal behavior. I suspect you would find the same if your tried 'replay all' to multiple messages. > Is there something wrong with my config No > or is this deliberate on your part? No -- Terry Jan Reedy From mertz at gnosis.cx Tue Sep 6 15:48:39 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 6 Sep 2016 14:48:39 -0500 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: On Tue, Sep 6, 2016 at 1:15 PM, Sven R. Kunze wrote: > > Their response: "Oh, I don't need it, let's close it." > Arek: "But I need it." > This definitely feels like a case of "put it on PyPI." Actually, maybe contribute to `boltons`, it feels like it might fit as a utility function there. While I wouldn't mind being able to type `from random import shuffled`, I don't have a bit problem instead typing `from boltons import shuffled`, nor even `from arek_utils import shuffled`. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue Sep 6 17:35:56 2016 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 6 Sep 2016 16:35:56 -0500 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: On Tue, Sep 6, 2016 at 2:48 PM, David Mertz wrote: > On Tue, Sep 6, 2016 at 1:15 PM, Sven R. Kunze wrote: >> >> Their response: "Oh, I don't need it, let's close it." >> Arek: "But I need it." > > > This definitely feels like a case of "put it on PyPI." Actually, maybe > contribute to `boltons`, it feels like it might fit as a utility function > there. > > While I wouldn't mind being able to type `from random import shuffled`, I > don't have a bit problem instead typing `from boltons import shuffled`, nor > even `from arek_utils import shuffled`. > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From tim.peters at gmail.com Tue Sep 6 18:04:19 2016 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 6 Sep 2016 17:04:19 -0500 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: [David Mertz ] > This definitely feels like a case of "put it on PyPI." Actually, maybe > contribute to `boltons`, it feels like it might fit as a utility function > there. It's trivial to write such a function if it's truly needed - it would be easier to write it from scratch than to remember which module it's hiding in. There's a "clever" 1-liner, but with no imagination at all it's still dead obvious: def shuffled(xs): from random import shuffle xs = xs[:] shuffle(xs) return xs `boltons` doesn't typically bother with screamingly obvious things. > While I wouldn't mind being able to type `from random import shuffled`, I > don't have a bit problem instead typing `from boltons import shuffled`, nor > even `from arek_utils import shuffled`. But would you _use_ it? I'm still asking for use cases. When, e.g., I'm randomizing permutations for testing, the last thing I want is: while whatever: result = function_of_xs(shuffled(xs)) check result and complain if it's wrong Why not? Because no trace remains of _which_ permutation provoked the failure when a failure occurs. Instead code looks like this: while whatever: shuffle(xs) result = function_of_xs(xs) # or xs[:] if the function mutates its arg check result and complain that `xs` specifically provoked a failure Indeed, the only clear "use case" that makes sense I've been able to think of is: xs = shuffled(xs) But that's written more easily and efficiently today as shuffle(xs) This is in stark contrast to sorted(), where clear use cases abound. That didn't get in because it's hard to mimic (it's basically as easy as shuffled()), but because it's exactly what's wanted in all kinds of contexts in all kinds of code. From ckaynor at zindagigames.com Tue Sep 6 20:49:38 2016 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Tue, 6 Sep 2016 17:49:38 -0700 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: On Tue, Sep 6, 2016 at 3:04 PM, Tim Peters wrote: > But would you _use_ it? I'm still asking for use cases. > > When, e.g., I'm randomizing permutations for testing, the last thing I > want is: > > while whatever: > result = function_of_xs(shuffled(xs)) > check result and complain if it's wrong > > Why not? Because no trace remains of _which_ permutation provoked the > failure when a failure occurs. Instead code looks like this: > > while whatever: > shuffle(xs) > result = function_of_xs(xs) # or xs[:] if the function mutates > its arg > check result and complain that `xs` specifically provoked a failure > > Indeed, the only clear "use case" that makes sense I've been able to > think of is: > > xs = shuffled(xs) > > But that's written more easily and efficiently today as > > shuffle(xs) > > This is in stark contrast to sorted(), where clear use cases abound. > That didn't get in because it's hard to mimic (it's basically as easy > as shuffled()), but because it's exactly what's wanted in all kinds of > contexts in all kinds of code. > I'll weigh in and say that I've had a few cases where I've wanted a shuffled function, but not many. The vast majority are interactive uses, where I want to get a sampling of data, and in those cases I'm normally just printing the output (often, by letting the REPL handle it). I'm fairly sure I've never wanted a shuffled in the actual code, and shuffle is almost always what I want (or just pulling items at random). Probably the most common case is to produce a list of random numbers in a (small) range. The current code looks roughly like: import random items = list(range(10)) random.shuffle(items) items # this is interactive, so this prints it for me As this does not come up often, I almost invariably write the following first: import random random.shuffle(range(10)) Then get no output and write the first form. It is not a major difference, and only comes up maybe a few times a year at most for me. Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 6 22:19:42 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Sep 2016 19:19:42 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <38d25b0e-6a64-d27b-1718-c68a11a8b779@gmail.com> References: <38d25b0e-6a64-d27b-1718-c68a11a8b779@gmail.com> Message-ID: On Tue, Sep 6, 2016 at 10:42 AM, Yury Selivanov wrote: > On 2016-09-04 3:10 AM, Adam Barto? wrote: >> will await be allowed also in the "if" part of comprehensions? And what >> about the "in" part? (e.g. if I'm not mistaken, we may have an asynchronous >> function returning an asynchronous iterator.) > Yes, awaits will be allowed. I'll update the PEP. Hasn't happened yet... I see this PEP as written in a bit of haste and very close to the 3.6b1 feature freeze (coming weekend). Usually I wouldn't accept such a hasty PEP, but the ideas in it seem pretty uncontroversial, and in line with the existing expectations for async/await. Yury, if you manage to get a working implementation signed off by one other core dev (not me) I can accept the PEP provisionally, under the same conditions as PEP 525. -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Wed Sep 7 00:23:11 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 06 Sep 2016 21:23:11 -0700 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: <57CF962F.8030202@stoneleaf.us> On 09/06/2016 11:15 AM, Sven R. Kunze wrote: > On 06.09.2016 18:32, Steven D'Aprano wrote: >> If your only argument is to continue to insist that Python should have a >> shuffled() function and you'll write a patch, it will go no where. First >> you have to convince people that the patch is needed. > > He already convinced some people. Just not some venerable Python devs, > which doesn't necessarily mean something at all. It means it's not going to happen. > Their response: "Oh, I don't need it, let's close it." > Arek: "But I need it." > > So, who's right now? Everyone: - if the python devs say it doesn't need to be in the stdlib, it's not going in. - if Arek (or anyone) needs it, then Arek (and those who need it) can write it and put it in their own personal utility library. > I would suggest not being so protective. It's community project after all. No, it isn't. It is Guido's project. > Maybe, the patch will help him understand that it's not so easy, or other > people will see the benefit after reading it. If the patch isn't worth > including, he learned something in the process of creating it. Sure. The point is simply that writing and submitting a patch does not grant automatic acceptance. -- ~Ethan~ From greg.ewing at canterbury.ac.nz Wed Sep 7 03:04:39 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Sep 2016 19:04:39 +1200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> Message-ID: <57CFBC07.9060700@canterbury.ac.nz> Sven R. Kunze wrote: > I already noted that > Python can detect when to make the switch without a marker. And you fail > to explain where the issue with this point of view is. The issue is that Guido wants to see where all the potential suspension points are. -- Greg From storchaka at gmail.com Wed Sep 7 05:20:10 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 7 Sep 2016 12:20:10 +0300 Subject: [Python-ideas] Three ways of paths canonization Message-ID: Before removing provisional state from the pathlib module, we should resolve the issue with Path.resolve(). It corresponds to os.path.realpath(), but behaves differently in case of non-existent path. Actually we can't say that any of these functions is wrong. Both behaviors make sense in different situations. The readlink utility from GNU coreutils has three mode for resolving file path: -f, --canonicalize canonicalize by following every symlink in every component of the given name recursively; all but the last component must exist -e, --canonicalize-existing canonicalize by following every symlink in every component of the given name recursively, all components must exist -m, --canonicalize-missing canonicalize by following every symlink in every component of the given name recursively, without requirements on components existence Current behavior of posixpath.realpath() is matches (besides one minor detail) to `readlink -m`. The behavior of Path.resolve() matches `readlink -e`. I have proposed a patch that adds three-state optional parameter to posixpath.realpath() and I'm going to provide similar patch for Path.resolve(). But I'm not sure this is good API. Are there better variants? [1] http://bugs.python.org/issue19717 [2] http://bugs.python.org/issue27002 From hugo.fisher at gmail.com Wed Sep 7 05:28:30 2016 From: hugo.fisher at gmail.com (Hugh Fisher) Date: Wed, 7 Sep 2016 19:28:30 +1000 Subject: [Python-ideas] Typecheckers: there can be only one Message-ID: There's been discussion here and on python-dev regarding PEP 526 that assumes there will be multiple type checkers for Python. I really can't see this happening. If typing annotations become common or maybe considered best practice, the Zen of Python "there should be one-- and preferably only one --obvious way to do it" will take effect. Firstly, the interpreter will need to have type checking built in. Just about every intro book and tutorial for Python says how great it is that you don't have an edit-save-compile cycle, just fire up the Python interpreter and start typing. Having to run a separate type checker will be considered as ridiculous as a C compiler that didn't run the preprocessor itself. Secondly, PyPI will collapse if there isn't just one. How can we express dependencies between packages that use different type checkers? When type checkers themselves have versions? When a dev team uses one type checker for 1.x and then switches to another for 2.x? That's a special circle of hell. -- cheers, Hugh Fisher From p.f.moore at gmail.com Wed Sep 7 05:41:12 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Sep 2016 10:41:12 +0100 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On 7 September 2016 at 10:28, Hugh Fisher wrote: > Firstly, the interpreter will need to have type checking built in. > Just about every intro book and tutorial for Python says how great it > is that you don't have an edit-save-compile cycle, just fire up the > Python interpreter and start typing. Having to run a separate type > checker will be considered as ridiculous as a C compiler that didn't > run the preprocessor itself. The intention (and current reality) is that type checkers are *static* checkers, not runtime checks, in the same way as linters like pyflakes. So yo run a checker over your code as a QA step as part of your testing cycle, it has no runtime effect. So your users and dependencies are unaffected by your choice of typechecker. Frankly, this is the only model that makes sense, precisely because of the issues you raise. And it's an existing model used by linters like pyflakes, so there's no reason to assume it will be any less acceptable for type checkers. Hope this clarifies the situation. Paul From k7hoven at gmail.com Wed Sep 7 06:59:28 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 7 Sep 2016 13:59:28 +0300 Subject: [Python-ideas] Atomic counter / atomic increment In-Reply-To: References: Message-ID: > import threading > > class AtomicCounter: > def __init__(self, initial=0): > self.value = initial > self._lock = threading.Lock() > > def increment(self, num=1): > with self._lock: > self.value += num > return self.value Some late nitpicking, sorry: This is not an atomic counter, it's a thread-safe counter. And since being thread-safe is rarely a bad idea, especially assuming someone will manage to Kill GIL, I would just call it a "counter" or Counter. (Of course, atomic/lock-free implementations are nice when possible.) -- Koos From ncoghlan at gmail.com Wed Sep 7 07:11:16 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2016 21:11:16 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> Message-ID: On 7 September 2016 at 04:54, Sven R. Kunze wrote: > On 06.09.2016 20:37, Nick Coghlan wrote: >> As Anthony already noted, the "async" keyword switches to the >> asynchronous version of the iterator protocol - you use this when your >> *iterator* needs to interact with the event loop, just as you do when >> deciding whether or not to mark a for loop as asynchronous. > > Of course "async" switches to async mode. But that was not the question. I > asked WHY that's necessary not what it does. I already noted that Python can > detect when to make the switch without a marker. And you fail to explain > where the issue with this point of view is. The Python *runtime* can tell whether the result of an expression is a normal iterator or an asynchronous iterator, but the Python *compiler* can't. So at compile time, it has to decide what code to emit for the four different scenarios Anthony listed: # Normal comprehension, 100% synchronous and blocking squares = [i**i for i in range(10)] # Blocking/sync iterator producing awaitables which suspend before producing a "normal" value results = [await result for result in submit_requests()] # Non-blocking/async iterator that suspends before producing "normal" values results = [result async for submit_requests_and_wait_for_results()] # Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value results = [await result async for submit_requests_asynchronously()] And hence needs the "async" keyword to detect when it has the latter two cases rather than the first two. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Sep 7 07:31:03 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2016 21:31:03 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On 4 September 2016 at 09:31, Yury Selivanov wrote: > With the proposed asynchronous comprehensions syntax, the above code > becomes as short as:: > > result = [i async for i in aiter() if i % 2] After using it a few times in examples, while I'm prepared to accept the agrammatical nature of "async for" in the statement form (where the adjective-noun phrase can be read as a kind of compound noun introducing the whole statement), I think for the comprehension form, we should aim to put the words in the right grammatical order if we can: result = [i for i in async aiter() if i % 2] I think the readability gain from that approach becomes even clearer with nested loops: result = [x for aiterable in async outer() for x in async aiterable] vs the current: result = [x async for aiterable in outer() async for x in async aiterable] In the first form, "async" is clearly a pre-qualifier on "outer()" and "aiterable", indicating they need to be asynchronous iterators rather than synchronous ones. By contrast, in the current form, the first "async" reads like a post-qualifer on "x" (it isn't, it modifies how outer() is handled in the outer loop), while the second looks like a post-qualifier on "outer()" (it isn't, it modified how aiterable is handled in the inner loop) If that means having to postpone full async comprehensions until "async" becomes a proper keyword in 3.7 and only adding "await in comprehensions and generator expressions" support to 3.6, that seems reasonable to me Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From k7hoven at gmail.com Wed Sep 7 07:35:17 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 7 Sep 2016 14:35:17 +0300 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> Message-ID: On Wed, Sep 7, 2016 at 2:11 PM, Nick Coghlan wrote: > On 7 September 2016 at 04:54, Sven R. Kunze wrote: >> On 06.09.2016 20:37, Nick Coghlan wrote: >>> As Anthony already noted, the "async" keyword switches to the >>> asynchronous version of the iterator protocol - you use this when your >>> *iterator* needs to interact with the event loop, just as you do when >>> deciding whether or not to mark a for loop as asynchronous. >> >> Of course "async" switches to async mode. But that was not the question. I >> asked WHY that's necessary not what it does. I already noted that Python can >> detect when to make the switch without a marker. And you fail to explain >> where the issue with this point of view is. > > The Python *runtime* can tell whether the result of an expression is a > normal iterator or an asynchronous iterator, but the Python *compiler* > can't. Yes, so to remove "async" from the syntax, this would be handled at runtime. But there's no way this PEP is going to do that, so I agree there's no point in discussing this here. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + From andrew.svetlov at gmail.com Wed Sep 7 07:37:41 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Wed, 07 Sep 2016 11:37:41 +0000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On Wed, Sep 7, 2016 at 2:31 PM Nick Coghlan wrote: > On 4 September 2016 at 09:31, Yury Selivanov > wrote: > > With the proposed asynchronous comprehensions syntax, the above code > > becomes as short as:: > > > > result = [i async for i in aiter() if i % 2] > > After using it a few times in examples, while I'm prepared to accept > the agrammatical nature of "async for" in the statement form (where > the adjective-noun phrase can be read as a kind of compound noun > introducing the whole statement), I think for the comprehension form, > we should aim to put the words in the right grammatical order if we > can: > > result = [i for i in async aiter() if i % 2] > > Please, no. It may be totally correct from English grammar POV but brings different syntax from regular async for statement, e.g. async for row in db.execute(...): pass Currently regular comprehensions are pretty similar to `for` loop. Why async comprehensions should look different from `async for` counterpart? > I think the readability gain from that approach becomes even clearer > with nested loops: > > result = [x for aiterable in async outer() for x in async aiterable] > > vs the current: > > result = [x async for aiterable in outer() async for x in async > aiterable] > > In the first form, "async" is clearly a pre-qualifier on "outer()" and > "aiterable", indicating they need to be asynchronous iterators rather > than synchronous ones. > > By contrast, in the current form, the first "async" reads like a > post-qualifer on "x" (it isn't, it modifies how outer() is handled in > the outer loop), while the second looks like a post-qualifier on > "outer()" (it isn't, it modified how aiterable is handled in the inner > loop) > > If that means having to postpone full async comprehensions until > "async" becomes a proper keyword in 3.7 and only adding "await in > comprehensions and generator expressions" support to 3.6, that seems > reasonable to me > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Sep 7 07:41:10 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 7 Sep 2016 14:41:10 +0300 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On Wed, Sep 7, 2016 at 2:31 PM, Nick Coghlan wrote: > On 4 September 2016 at 09:31, Yury Selivanov wrote: >> With the proposed asynchronous comprehensions syntax, the above code >> becomes as short as:: >> >> result = [i async for i in aiter() if i % 2] > > After using it a few times in examples, while I'm prepared to accept > the agrammatical nature of "async for" in the statement form (where > the adjective-noun phrase can be read as a kind of compound noun > introducing the whole statement), I think for the comprehension form, > we should aim to put the words in the right grammatical order if we > can: > > result = [i for i in async aiter() if i % 2] > I agree this would be better, but the difference compared to PEP-492 async for loops (and even async with statements) would be awful :S. -- Koos > I think the readability gain from that approach becomes even clearer > with nested loops: > > result = [x for aiterable in async outer() for x in async aiterable] > > vs the current: > > result = [x async for aiterable in outer() async for x in async aiterable] > > In the first form, "async" is clearly a pre-qualifier on "outer()" and > "aiterable", indicating they need to be asynchronous iterators rather > than synchronous ones. > > By contrast, in the current form, the first "async" reads like a > post-qualifer on "x" (it isn't, it modifies how outer() is handled in > the outer loop), while the second looks like a post-qualifier on > "outer()" (it isn't, it modified how aiterable is handled in the inner > loop) > > If that means having to postpone full async comprehensions until > "async" becomes a proper keyword in 3.7 and only adding "await in > comprehensions and generator expressions" support to 3.6, that seems > reasonable to me > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- + Koos Zevenhoven + http://twitter.com/k7hoven + From ncoghlan at gmail.com Wed Sep 7 07:56:12 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2016 21:56:12 +1000 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On 7 September 2016 at 19:41, Paul Moore wrote: > Frankly, this is the only model that makes sense, precisely because of > the issues you raise. And it's an existing model used by linters like > pyflakes, so there's no reason to assume it will be any less > acceptable for type checkers. Exactly - in a dynamic language like Python, running a typechecker is a form of testing, rather than a necessary part of getting the code to work in the first place. For interactive-use-only code like the Jupyter Notebook I use to download background images for my laptop [1], neither testing nor typechecking are needed - I don't care if it works for all possible cases and for all possible users, I only care that it works when I'm the one running it. Code written for learning purposes, personal automation, ad hoc data analysis, etc, largely falls into this category. For most other applications, an automated regression test suite will be a better tool than a typechecker for preventing errors you actually care about preventing. ("print('!dlrow elloH')" will typecheck just fine, but probably isn't what you meant to do) Similarly, a structural linter like pylint is smart enough to pick up basic things like attempts to read non-existent attributes. Where a typechecker starts to become attractive though is when you start to hit the limits of automated testing regimes that involve actually running the software and you've gone beyond the kinds of errors a structural linter can find: - combinatorial explosions make it impossible to test all code path combinations - some error paths are simply difficult to trigger without extensive investment in API mocking - your code coverage has reached the point of diminishing returns (you need a lot more test code for each new line of coverage) - you're integrating multiple projects together and tracing the cause of runtime test failures can be difficult - you're engaging in large scale automated refactoring and want to be confident the result is at least somewhat sensible Like automated tests and structural linters, typecheckers should be adopted only when the pain of *not* having them exceeds (or is expected to exceed) the inconvenience of using them (and the precise location of that threshold will vary by individual and by context). Cheers, Nick. [1] http://nbviewer.jupyter.org/urls/bitbucket.org/ncoghlan/misc/raw/default/notebooks/Digital%20Blasphemy.ipynb -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Sep 7 08:27:32 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2016 22:27:32 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On 7 September 2016 at 21:37, Andrew Svetlov wrote: > On Wed, Sep 7, 2016 at 2:31 PM Nick Coghlan wrote: >> After using it a few times in examples, while I'm prepared to accept >> the agrammatical nature of "async for" in the statement form (where >> the adjective-noun phrase can be read as a kind of compound noun >> introducing the whole statement), I think for the comprehension form, >> we should aim to put the words in the right grammatical order if we >> can: >> >> result = [i for i in async aiter() if i % 2] >> > Please, no. > It may be totally correct from English grammar POV but brings different > syntax from regular async for statement, e.g. > async for row in db.execute(...): > pass The new issue that's specific to comprehensions is that the statement form doesn't have the confounding factor of having an expression to the left of it. Thus, the prefix is unambiguous and can be read as modifying the entire statement rather than just the immediately following keyword: async for row in db.execute(...): process(row) It's also pragmatically necessary right now due to the sleight of hand that Yury used in the code generation pipeline to bring in "async def", "async with", and "async for" without a __future__ statement, but without making them full keywords either. However, when we convert it to the comprehension form, a parsing ambiguity (for humans) arises that creates an inherent readability problem: [process(row) async for row in db.execute(...)] When reading that, is "async" a postfix operator being used in a normal comprehension (wrong, but somewhat plausible)? Or is it part of a compound keyword with "for" that modifies the iteration behaviour of that part of the comprehension (the correct interpretation)? [(process(row) async) for row in db.execute(...)] [process(row) (async for) row in db.execute(...)] The postfix operator interpretation is just plain wrong, but even the correct interpretation as a compound keyword sits between two expressions *neither* of which is the one being modified (that would be "db.execute()") By contrast, if the addition of full async comprehensions is deferred to 3.7 (when async becomes a true keyword), then the infix spelling can be permitted in both the statement and comprehension forms: for row in async db.execute(...): process(row) [process(row) for row in async db.execute(...)] with the prefix spelling of the statement form retained solely for backwards compatibility purposes (just as we retain "from __future__ import feature" flags even after the feature has become the default behaviour). The beauty of the infix form is that it *doesn't matter* whether someone reads it as a compound keyword with "in" or as a prefix modifying the following expression: [process(row) for row (in async) db.execute(...)] [process(row) for row in (async db.execute(...))] In both cases, it clearly suggests something special about the way "db.execute()" is going to be handled, which is the correct interpretation. > Currently regular comprehensions are pretty similar to `for` loop. > Why async comprehensions should look different from `async for` counterpart? Regular "for" loops don't have the problem of their introductory keyword being written as two words, as that's the culprit that creates the ambiguity when you add an expression to the left of it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rymg19 at gmail.com Wed Sep 7 09:50:10 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 7 Sep 2016 08:50:10 -0500 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On Sep 7, 2016 4:28 AM, "Hugh Fisher" wrote: > > There's been discussion here and on python-dev regarding PEP 526 that > assumes there will be multiple type checkers for Python. I really > can't see this happening. It already did: - mypy - pytype - PyCharm has an integrated one - pylint will add support in the future > If typing annotations become common or maybe > considered best practice, the Zen of Python "there should be one-- and > preferably only one --obvious way to do it" will take effect. > > Firstly, the interpreter will need to have type checking built in. A lot of people would disagree. > Just about every intro book and tutorial for Python says how great it > is that you don't have an edit-save-compile cycle, just fire up the > Python interpreter and start typing. Having to run a separate type > checker will be considered as ridiculous as a C compiler that didn't > run the preprocessor itself. > Remember: you don't have to run it. You can prototype to your hearts content and then have it run as a commit hook. So you still have the nice, tight cycle that you mentioned. > Secondly, PyPI will collapse if there isn't just one. Too late! > How can we > express dependencies between packages that use different type > checkers? Add the type checker to the dev dependencies; as long as it's PEP 484-compatible, the user couldn't care less. > When type checkers themselves have versions? CPython has versions. > When a dev team > uses one type checker for 1.x and then switches to another for 2.x? That's the dev team's problem. > That's a special circle of hell. > > > -- > > cheers, > Hugh Fisher > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Sep 7 10:47:36 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 7 Sep 2016 23:47:36 +0900 Subject: [Python-ideas] Three ways of paths canonization In-Reply-To: References: Message-ID: <22480.10376.844689.468617@turnbull.sk.tsukuba.ac.jp> Serhiy Storchaka writes: > The readlink utility from GNU coreutils has three mode for resolving > file path: > > -f, --canonicalize > canonicalize by following every symlink in every > component of the given name recursively; all but the last component must > exist > > -e, --canonicalize-existing > canonicalize by following every symlink in every > component of the given name recursively, all components must exist > > -m, --canonicalize-missing > canonicalize by following every symlink in every > component of the given name recursively, without requirements on > components existence In Mac OS X (and I suppose other BSDs), realpath(3) implements -e. glibc does none of these, instead: GNU extensions If the call fails with either EACCES or ENOENT and resolved_path is not NULL, then the prefix of path that is not readable or does not exist is returned in resolved_path. I suppose this nonstandard behavior is controlled by a #define, but the Linux manpage doesn't specify it. > Current behavior of posixpath.realpath() is matches (besides one minor > detail) to `readlink -m`. The behavior of Path.resolve() matches > `readlink -e`. This looks like a bug in posixpath, while Path.resolve follows POSIX. http://pubs.opengroup.org/onlinepubs/009695399/functions/realpath.html sez: RETURN VALUE Upon successful completion, realpath() shall return a pointer to the resolved name. Otherwise, realpath() shall return a null pointer and set errno to indicate the error, and the contents of the buffer pointed to by resolved_name are undefined. ERRORS The realpath() function shall fail if: [...] [ENOENT] A component of file_name does not name an existing file or file_name points to an empty string. [ENOTDIR] A component of the path prefix is not a directory. which corresponds to -e. > I have proposed a patch that adds three-state optional parameter to > posixpath.realpath() and I'm going to provide similar patch for > Path.resolve(). But I'm not sure this is good API. Are there better > variants? Said parameter will almost always be a constant. Usually in those cases Python prefers to use different functions. Eg, posixpath.realpath -e posixpath.realpath_require_prefix -f posixpath.realpath_allow_missing -m posixpath.realpath_gnuext GNU extension From k7hoven at gmail.com Wed Sep 7 10:57:37 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 7 Sep 2016 17:57:37 +0300 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On Wed, Sep 7, 2016 at 3:27 PM, Nick Coghlan wrote: [...] > > The new issue that's specific to comprehensions is that the statement > form doesn't have the confounding factor of having an expression to > the left of it. Thus, the prefix is unambiguous and can be read as > modifying the entire statement rather than just the immediately > following keyword: > > async for row in db.execute(...): > process(row) > > It's also pragmatically necessary right now due to the sleight of hand > that Yury used in the code generation pipeline to bring in "async > def", "async with", and "async for" without a __future__ statement, > but without making them full keywords either. I don't think this issue strictly has anything to do with where that "async" is put in the syntax, as long as it's used within the definition of an async function: async def function(): # in here, is effectively "async" a keyword. Of course, in French, this would be: def function async(): # dedans, "async" est effectivement un mot cl? I'm sure someone will be able to correct my French, though. [and Nick writes:] > [process(row) async for row in db.execute(...)] > > When reading that, is "async" a postfix operator being used in a > normal comprehension (wrong, but somewhat plausible)? Or is it part of > a compound keyword with "for" that modifies the iteration behaviour of > that part of the comprehension (the correct interpretation)? > > [(process(row) async) for row in db.execute(...)] > [process(row) (async for) row in db.execute(...)] > > The postfix operator interpretation is just plain wrong, but even the > correct interpretation as a compound keyword sits between two > expressions *neither* of which is the one being modified (that would > be "db.execute()") > > By contrast, if the addition of full async comprehensions is deferred > to 3.7 (when async becomes a true keyword), then the infix spelling > can be permitted in both the statement and comprehension forms: That's an interesting suggestion. What exactly is the relation between deferring this PEP and permitting the infix spelling? This would make it more obvious at a first glance, whether something is a with statement or for loop. The word "async" there is still not very easy to miss, especially with highlighted syntax. I didn't realize (or had forgotten) that PEP 492 is provisional. > for row in async db.execute(...): > process(row) > > [process(row) for row in async db.execute(...)] > > with the prefix spelling of the statement form retained solely for > backwards compatibility purposes (just as we retain "from __future__ > import feature" flags even after the feature has become the default > behaviour). > > The beauty of the infix form is that it *doesn't matter* whether > someone reads it as a compound keyword with "in" or as a prefix > modifying the following expression: > > [process(row) for row (in async) db.execute(...)] > [process(row) for row in (async db.execute(...))] > > In both cases, it clearly suggests something special about the way > "db.execute()" is going to be handled, which is the correct > interpretation. And db.execute is an async iterarable after all, so "async" is a suitable adjective for db.execute(...). -- Koos [...] > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- + Koos Zevenhoven + http://twitter.com/k7hoven + From ncoghlan at gmail.com Wed Sep 7 11:53:11 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 01:53:11 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> Message-ID: On 8 September 2016 at 00:57, Koos Zevenhoven wrote: > On Wed, Sep 7, 2016 at 3:27 PM, Nick Coghlan wrote: > [...] >> >> The new issue that's specific to comprehensions is that the statement >> form doesn't have the confounding factor of having an expression to >> the left of it. Thus, the prefix is unambiguous and can be read as >> modifying the entire statement rather than just the immediately >> following keyword: >> >> async for row in db.execute(...): >> process(row) >> >> It's also pragmatically necessary right now due to the sleight of hand >> that Yury used in the code generation pipeline to bring in "async >> def", "async with", and "async for" without a __future__ statement, >> but without making them full keywords either. > > I don't think this issue strictly has anything to do with where that > "async" is put in the syntax, as long as it's used within the > definition of an async function: > > async def function(): > # in here, is effectively "async" a keyword. Good point - I was thinking there was additional cleverness around "async for" and "async with" as well, but you're right that it's specifically "async def" that enables the pseudo-keyword behaviour. > [and Nick writes:] >> [process(row) async for row in db.execute(...)] >> >> When reading that, is "async" a postfix operator being used in a >> normal comprehension (wrong, but somewhat plausible)? Or is it part of >> a compound keyword with "for" that modifies the iteration behaviour of >> that part of the comprehension (the correct interpretation)? >> >> [(process(row) async) for row in db.execute(...)] >> [process(row) (async for) row in db.execute(...)] >> >> The postfix operator interpretation is just plain wrong, but even the >> correct interpretation as a compound keyword sits between two >> expressions *neither* of which is the one being modified (that would >> be "db.execute()") >> >> By contrast, if the addition of full async comprehensions is deferred >> to 3.7 (when async becomes a true keyword), then the infix spelling >> can be permitted in both the statement and comprehension forms: > > That's an interesting suggestion. What exactly is the relation between > deferring this PEP and permitting the infix spelling? Just a mistake on my part regarding how we were currently handling "async" within "async def" statements. With that mistake corrected, there may not be any need to defer the suggestion, since it already behaves as a keyword in the context where it matters. That said, we *are* doing some not-normal things in the code generation pipeline to enable the pseudo-keyword behaviour, so I also wouldn't be surprised if there was a practical limitation on allowing the "async" to appear after the "in" rather than before the "for" prior to 3.7. It's also worth reviewing the minimalist grammar changes in PEP 492 and the associated discussion about "async def" vs "def async": * https://www.python.org/dev/peps/pep-0492/#grammar-updates * https://www.python.org/dev/peps/pep-0492/#why-async-def-and-not-def-async Changing "for_stmt" to allow the "for TARGET in [ASYNC] expr" spelling isn't as tidy a modification as just allowing ASYNC in front of any of def_stmt, for_stmt and with_stmt. > This would make it more obvious at a first glance, whether something > is a with statement or for loop. The word "async" there is still not > very easy to miss, especially with highlighted syntax. > > I didn't realize (or had forgotten) that PEP 492 is provisional. Right, and one of the reasons for that was because we hadn't fully worked through the implications for comprehensions and generator expressions at the time. Now that I see the consequences of attempting to transfer the "async keyword is a statement qualifier " notion to the expression form, I think we may need to tweak things a bit :) >> The beauty of the infix form is that it *doesn't matter* whether >> someone reads it as a compound keyword with "in" or as a prefix >> modifying the following expression: >> >> [process(row) for row (in async) db.execute(...)] >> [process(row) for row in (async db.execute(...))] >> >> In both cases, it clearly suggests something special about the way >> "db.execute()" is going to be handled, which is the correct >> interpretation. > > And db.execute is an async iterarable after all, so "async" is a > suitable adjective for db.execute(...). Exactly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From yselivanov.ml at gmail.com Wed Sep 7 12:34:52 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 7 Sep 2016 09:34:52 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <38d25b0e-6a64-d27b-1718-c68a11a8b779@gmail.com> Message-ID: <53c1e3d6-d8f9-67e5-31bd-994d4fae77f1@gmail.com> On 2016-09-06 7:19 PM, Guido van Rossum wrote: > On Tue, Sep 6, 2016 at 10:42 AM, Yury Selivanov wrote: >> On 2016-09-04 3:10 AM, Adam Barto? wrote: >>> will await be allowed also in the "if" part of comprehensions? And what >>> about the "in" part? (e.g. if I'm not mistaken, we may have an asynchronous >>> function returning an asynchronous iterator.) >> Yes, awaits will be allowed. I'll update the PEP. > Hasn't happened yet... I see this PEP as written in a bit of haste and > very close to the 3.6b1 feature freeze (coming weekend). Usually I > wouldn't accept such a hasty PEP, but the ideas in it seem pretty > uncontroversial, and in line with the existing expectations for > async/await. > > Yury, if you manage to get a working implementation signed off by one > other core dev (not me) I can accept the PEP provisionally, under the > same conditions as PEP 525. > Thank you for accepting the PEP! Will focus on the implementation now. Yury From guido at python.org Wed Sep 7 12:48:35 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Sep 2016 09:48:35 -0700 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: <53c1e3d6-d8f9-67e5-31bd-994d4fae77f1@gmail.com> References: <38d25b0e-6a64-d27b-1718-c68a11a8b779@gmail.com> <53c1e3d6-d8f9-67e5-31bd-994d4fae77f1@gmail.com> Message-ID: On Wed, Sep 7, 2016 at 9:34 AM, Yury Selivanov wrote: > Thank you for accepting the PEP! Will focus on the implementation now. You're welcome! Good luck with the implementation. @Nick, regarding the suggestion to use [ for in async ] instead of [ async for in ] I think the lack of grammaticality is actually a feature -- we don't want people to confuse `async ` with `await `. Python has no other postfix syntax like this (unless you count f(), d[k] or x.a as postfix) so I don't think people are likely to mistake this for an 'async' postfix on rather than a prefix on 'for'. Hopefully they've seen an 'async for' statement before they encounter an async comprehension. -- --Guido van Rossum (python.org/~guido) From srkunze at mail.de Wed Sep 7 15:02:01 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 7 Sep 2016 21:02:01 +0200 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <50d6df33-e2c9-57d8-55ca-2134b444d953@gmail.com> <0c16e1e2-888b-7284-8aeb-5431125de9d3@mail.de> <87132d57-7c8a-7f54-4a5a-94cf997de7bc@mail.de> Message-ID: <7338e78b-dae2-f1d2-9ef9-d6db1d6278f0@mail.de> On 06.09.2016 21:45, Terry Reedy wrote: > On 9/6/2016 2:54 PM, Sven R. Kunze wrote: > >> PS: Nick, I noted that while replying, my mail client made me responding >> to you and the list as cc. > > For Thunderbird, this is normal behavior. I suspect you would find > the same if your tried 'replay all' to multiple messages. Actually no. Replying to your message for example leads to "Reply to List". That's odd. Sven From ncoghlan at gmail.com Wed Sep 7 15:06:15 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 05:06:15 +1000 Subject: [Python-ideas] PEP 530: Asynchronous Comprehensions In-Reply-To: References: <38d25b0e-6a64-d27b-1718-c68a11a8b779@gmail.com> <53c1e3d6-d8f9-67e5-31bd-994d4fae77f1@gmail.com> Message-ID: On 8 Sep 2016 02:48, "Guido van Rossum" wrote: > > On Wed, Sep 7, 2016 at 9:34 AM, Yury Selivanov wrote: > > Thank you for accepting the PEP! Will focus on the implementation now. > > You're welcome! Good luck with the implementation. > > @Nick, regarding the suggestion to use > > [ for in async ] > > instead of > > [ async for in ] > > I think the lack of grammaticality is actually a feature -- we don't > want people to confuse `async ` with `await `. Aye, that problem occurred to me after my last post and is definitely a concern. > Python has no other postfix syntax like this (unless you count f(), > d[k] or x.a as postfix) so I don't think people are likely to mistake > this for an 'async' postfix on rather than a prefix on 'for'. > Hopefully they've seen an 'async for' statement before they encounter > an async comprehension. I also realised that the worst case outcome I can see for the PEP 492 based syntax is some folks avoiding them in favour of the statement version because they don't like how the expression version reads (ala lambda), and as worst case scenarios go, that's not a particularly worrying one :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.fisher at gmail.com Wed Sep 7 17:31:39 2016 From: hugo.fisher at gmail.com (Hugh Fisher) Date: Thu, 8 Sep 2016 07:31:39 +1000 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On Wed, Sep 7, 2016 at 9:56 PM, Nick Coghlan wrote: > > Exactly - in a dynamic language like Python, running a typechecker is > a form of testing, rather than a necessary part of getting the code to > work in the first place. > The average programmer such as myself will expect that if I write code specifying the type of a variable or whatever it should *do something*. It's code, I wrote it, it should be interpreted. Assuming PEP 526 is implemented, suppose as a very new programmer I write foo: dict = 0 (If anyone thinks even new programmers wouldn't write that, you've never taught an introductory programming course.) It takes very little programming experience to know that is flat out wrong. I cannot think of any other programming language with type notation where this would not be immediately flagged as an error. But Python will silently accept it? That's the kind of one liner people put on presentation slides when they want to make fun of how bad programming languages like JavaScript are. A more subtle example that came up recently on one of the lists (which I cannot attribute it properly right now): c : complex = 1.0 My C compiler is smart enough to figure out it should add an imaginary zero to the RHS. As an average programmer, I'd expect that Python would be at least as smart and likewise convert 1.0 into a complex value. Or it could raise an error saying that the RHS is not a complex number. Given the code I've written, either makes sense. What I would not expect is for the interpreter to silently assign a scalar 1.0 to c and continue. That's just ... WTF? Type annotations are code, not tests. -- cheers, Hugh Fisher From srkunze at mail.de Wed Sep 7 17:43:59 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 7 Sep 2016 23:43:59 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> On 06.09.2016 21:29, Tim Peters wrote: > In the absence of anything approaching [use-cases], it's a matter of taste, and then > - yes - decades of Python development experience do - and should - > outweigh a newcomer's wish list. Especially when, as appears to be > the case here, those with truly extensive experience agree. I am not questioning experience which everyone in a team can benefit from. BUT experienced devs also need to recognize and respect the fact that younger/unexperienced developers are just better in detecting inconsistencies and bloody work-arounds. They simply haven't had to live with them for so long. Experienced devs just are stuck in a rut/are routine-blinded: "we've done that for years", "there's no better way". That's the way we do it in our teams. Employing the new guys as some sort of inconsistency detectors. This way, they learn to find their way around the code base and they can improve it by doing so. And I would never allow it in my team, to dismiss this kind of observation from new colleagues. It's invaluable as they will become routine-blinded as well. > [...] I would be far more annoyed if, e.g., > >>>> random.shuffle(some_million_element_list) > swamped my terminal with mountains of output. But you readily accept this behavior for "sorted"? That makes no sense at all. > You can't both behaviors simultaneously, so the status quo wins. > Indeed, the venerable status quo ;-) Nobody said to change "shuffle". Sven From p.f.moore at gmail.com Wed Sep 7 18:00:49 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Sep 2016 23:00:49 +0100 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On 7 September 2016 at 22:31, Hugh Fisher wrote: > The average programmer such as myself will expect that if I write code > specifying the type of a variable or whatever it should *do > something*. It's code, I wrote it, it should be interpreted. Reading the documentation should correct that mistaken expectation. > Type annotations are code, not tests. Not in Python they aren't. Paul From tim.peters at gmail.com Wed Sep 7 18:02:19 2016 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 7 Sep 2016 17:02:19 -0500 Subject: [Python-ideas] Shuffled In-Reply-To: <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> Message-ID: [Sven R. Kunze ] > I am not questioning experience which everyone in a team can benefit from. > > > BUT experienced devs also need to recognize and respect the fact that > younger/unexperienced developers are just better in detecting > inconsistencies and bloody work-arounds. They simply haven't had to live > with them for so long. Experienced devs just are stuck in a rut/are > routine-blinded: "we've done that for years", "there's no better way". > > > That's the way we do it in our teams. Employing the new guys as some sort of > inconsistency detectors. This way, they learn to find their way around the > code base and they can improve it by doing so. > > And I would never allow it in my team, to dismiss this kind of observation > from new colleagues. It's invaluable as they will become routine-blinded as > well. I have been more than willing to discuss it, and I did not close the issue report. I did say I was opposed to it, but that's simply because I am, and I explained there too _why_ I was opposed. Do you have anything to say about the specific proposal? I doubt either of us has found this meta-discussion useful. I'm still looking for a compelling use case. The only concrete thing anyone has noted in `shuffled()`'s favor so far is that sometimes they're surprised by the behavior of random.shuffle(list) returning None in an interactive shell (noted by you, and by another, and I cheerfully own up to being a bit surprised by that too long ago). But that's an observation about `random.shuffle()`, not about the proposed `shuffled()`. >> [...] I would be far more annoyed if, e.g., >> >> >>> random.shuffle(some_million_element_list) >> >> swamped my terminal with mountains of output. > But you readily accept this behavior for "sorted"? That makes no sense at > all. Of course it does. The only analogy to random.shuffle(big_list) returning None that makes a lick of sense here is that big_list.sort() also returns None. IF a `shuffled()` function is introduced, then of course it should return its result - just like `sorted()` returns its result. >> You can't both behaviors simultaneously, so the status quo wins. >> Indeed, the venerable status quo ;-) > Nobody said to change "shuffle". A verbatim quote from the first message in this thread: "Also shuffle() should return self so mutating methods could be chained." From srkunze at mail.de Wed Sep 7 18:10:43 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 8 Sep 2016 00:10:43 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> Message-ID: <0e641553-6d0f-5799-12ea-f9e0ca516ed3@mail.de> On 07.09.2016 02:49, Chris Kaynor wrote: > I'll weigh in and say that I've had a few cases where I've wanted a > shuffled function, but not many. The vast majority are interactive > uses, where I want to get a sampling of data, and in those cases I'm > normally just printing the output (often, by letting the REPL handle it). > > I'm fairly sure I've never wanted a shuffled in the actual code, and > shuffle is almost always what I want (or just pulling items at random). > > Probably the most common case is to produce a list of random numbers > in a (small) range. The current code looks roughly like: > > import random > items = list(range(10)) > random.shuffle(items) > items # this is interactive, so this prints it for me > > As this does not come up often, I almost invariably write the > following first: > import random > random.shuffle(range(10)) > > Then get no output and write the first form. It is not a major > difference, and only comes up maybe a few times a year at most for me. > That sounds extremely familiar. I would say the interactive session is the clear use-case here. But don't ask me why I need a shuffled list of somethings (probably for some highly vicious and immoral reasons ;-) ). @David Your idea of a PyPI package could almost work. However, in an interactive python console, I expect as much batteries included as possible to make it as quick as possible. And considering how simple such wrapper would be, it almost does not warrant the download of a third-party package. @Tim Of course it's easy to write a wrapper function. But from what I gather here, this is not the point. In interactive sessions, I find it highly annoying when I need to define my own functions. Especially because I need to do it again in a new python session. Somebody provided a one-line hack using "sorted" to emulate "shuffled". The statement basically is: shuffling is a special kind of sorting. So, I would expect the interface to work the same. That at least suffices for me to understand Arek's point of view. I would even go so far as to say: shuffled(my_list) # returns a new shuffled list my_list.shuffle() # shuffles in-place Allowing to plug-in the RNG, when needed. Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Wed Sep 7 18:11:17 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 8 Sep 2016 00:11:17 +0200 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On 8 September 2016 at 00:00, Paul Moore wrote: > > Type annotations are code, not tests. > > Not in Python they aren't. > Well, to a certain extent. One can try something like this in REPL: from typing import get_type_hints import __main__ s: str class C: x: int get_type_hints(C) get_type_hints(__main__) Although please remember that the PEP is still provisional and implementation details may change. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Sep 7 18:22:28 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 8 Sep 2016 00:22:28 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> Message-ID: <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> Maybe, there's a misunderstanding here and I hope I didn't waste too much of your and my time. Not sure where I got this from but my last status was that Arek would like to have a "shuffled" function that does exactly what you described. Maybe, that doesn't fit the reports description but his pull request will tell the truth. :) About the confusion of returning None. It's not confusing in the sense of "o my god, I did it wrong, I need to learn it", but more, like "I used shuffle, BECAUSE it's the only one I could find in a hurry" and then ascertain "why the heck is there no alternative returning the shuffled result instead of overwriting my list? I would expect this from Python as it already provides both alternatives for sorting". Sven On 08.09.2016 00:02, Tim Peters wrote: > [Sven R. Kunze ] >> I am not questioning experience which everyone in a team can benefit from. >> >> >> BUT experienced devs also need to recognize and respect the fact that >> younger/unexperienced developers are just better in detecting >> inconsistencies and bloody work-arounds. They simply haven't had to live >> with them for so long. Experienced devs just are stuck in a rut/are >> routine-blinded: "we've done that for years", "there's no better way". >> >> >> That's the way we do it in our teams. Employing the new guys as some sort of >> inconsistency detectors. This way, they learn to find their way around the >> code base and they can improve it by doing so. >> >> And I would never allow it in my team, to dismiss this kind of observation >> from new colleagues. It's invaluable as they will become routine-blinded as >> well. > I have been more than willing to discuss it, and I did not close the > issue report. I did say I was opposed to it, but that's simply > because I am, and I explained there too _why_ I was opposed. > > Do you have anything to say about the specific proposal? I doubt > either of us has found this meta-discussion useful. I'm still looking > for a compelling use case. The only concrete thing anyone has noted > in `shuffled()`'s favor so far is that sometimes they're surprised by > the behavior of random.shuffle(list) returning None in an interactive > shell (noted by you, and by another, and I cheerfully own up to being > a bit surprised by that too long ago). But that's an observation > about `random.shuffle()`, not about the proposed `shuffled()`. > > >>> [...] I would be far more annoyed if, e.g., >>> >>>>>> random.shuffle(some_million_element_list) >>> swamped my terminal with mountains of output. >> But you readily accept this behavior for "sorted"? That makes no sense at >> all. > Of course it does. The only analogy to random.shuffle(big_list) > returning None that makes a lick of sense here is that big_list.sort() > also returns None. IF a `shuffled()` function is introduced, then of > course it should return its result - just like `sorted()` returns its > result. > > >>> You can't both behaviors simultaneously, so the status quo wins. >>> Indeed, the venerable status quo ;-) >> Nobody said to change "shuffle". > A verbatim quote from the first message in this thread: > > "Also shuffle() should return self so mutating methods could be chained." From srkunze at mail.de Wed Sep 7 18:29:33 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 8 Sep 2016 00:29:33 +0200 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: <8aed0be1-3e64-1433-3061-3c04b05cb03a@mail.de> On 08.09.2016 00:00, Paul Moore wrote: > On 7 September 2016 at 22:31, Hugh Fisher wrote: >> The average programmer such as myself will expect that if I write code >> specifying the type of a variable or whatever it should *do >> something*. It's code, I wrote it, it should be interpreted. > Reading the documentation should correct that mistaken expectation. If you need to read the documentation in order to understand a product, something's wrong. Especially given the simplicity of the examples. Some smartphones are delivered without a manual because they are considered to be as intuitive as breathing or walking. >> Type annotations are code, not tests. > Not in Python they aren't. Unfortunately, that's not true. :) Sven From matt at getpattern.com Wed Sep 7 18:41:46 2016 From: matt at getpattern.com (Matt Gilson) Date: Wed, 7 Sep 2016 15:41:46 -0700 Subject: [Python-ideas] Shuffled In-Reply-To: <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> Message-ID: I lurk around here a lot more than I actually post -- But in this case I find myself weighing in firmly on the side of "please don't add this". It'd just add noise in the documentation and __builtins__ namespace. There are millions of useful functions that *could* be added to the `list` object (or to the `__builtins__` namespace). It's the job of the core dev team to decide which are the most useful for everybody and this is one where (in my opinion) the usefulness doesn't justify the additional clutter. It seems like the primary argument for this is that some users would like to have this functionality baked into an interactive session. If that's the case, then I'd like to point out that if you find yourself wanting this in an interactive session frequently there is an interactive startup file that you can modify so that this function will always be available to *you* whenever you start up an interactive session (and you can put whatever else in there that you like as well :-). I hope that helps. On Wed, Sep 7, 2016 at 3:22 PM, Sven R. Kunze wrote: > Maybe, there's a misunderstanding here and I hope I didn't waste too much > of your and my time. > > Not sure where I got this from but my last status was that Arek would like > to have a "shuffled" function that does exactly what you described. Maybe, > that doesn't fit the reports description but his pull request will tell the > truth. :) > > > About the confusion of returning None. It's not confusing in the sense of > "o my god, I did it wrong, I need to learn it", but more, like "I used > shuffle, BECAUSE it's the only one I could find in a hurry" and then > ascertain "why the heck is there no alternative returning the shuffled > result instead of overwriting my list? I would expect this from Python as > it already provides both alternatives for sorting". > > > Sven > > > > On 08.09.2016 00:02, Tim Peters wrote: > >> [Sven R. Kunze ] >> >>> I am not questioning experience which everyone in a team can benefit >>> from. >>> >>> >>> BUT experienced devs also need to recognize and respect the fact that >>> younger/unexperienced developers are just better in detecting >>> inconsistencies and bloody work-arounds. They simply haven't had to live >>> with them for so long. Experienced devs just are stuck in a rut/are >>> routine-blinded: "we've done that for years", "there's no better way". >>> >>> >>> That's the way we do it in our teams. Employing the new guys as some >>> sort of >>> inconsistency detectors. This way, they learn to find their way around >>> the >>> code base and they can improve it by doing so. >>> >>> And I would never allow it in my team, to dismiss this kind of >>> observation >>> from new colleagues. It's invaluable as they will become routine-blinded >>> as >>> well. >>> >> I have been more than willing to discuss it, and I did not close the >> issue report. I did say I was opposed to it, but that's simply >> because I am, and I explained there too _why_ I was opposed. >> >> Do you have anything to say about the specific proposal? I doubt >> either of us has found this meta-discussion useful. I'm still looking >> for a compelling use case. The only concrete thing anyone has noted >> in `shuffled()`'s favor so far is that sometimes they're surprised by >> the behavior of random.shuffle(list) returning None in an interactive >> shell (noted by you, and by another, and I cheerfully own up to being >> a bit surprised by that too long ago). But that's an observation >> about `random.shuffle()`, not about the proposed `shuffled()`. >> >> >> [...] I would be far more annoyed if, e.g., >>>> >>>> random.shuffle(some_million_element_list) >>>>>>> >>>>>> swamped my terminal with mountains of output. >>>> >>> But you readily accept this behavior for "sorted"? That makes no sense at >>> all. >>> >> Of course it does. The only analogy to random.shuffle(big_list) >> returning None that makes a lick of sense here is that big_list.sort() >> also returns None. IF a `shuffled()` function is introduced, then of >> course it should return its result - just like `sorted()` returns its >> result. >> >> >> You can't both behaviors simultaneously, so the status quo wins. >>>> Indeed, the venerable status quo ;-) >>>> >>> Nobody said to change "shuffle". >>> >> A verbatim quote from the first message in this thread: >> >> "Also shuffle() should return self so mutating methods could be >> chained." >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- [image: pattern-sig.png] Matt Gilson // SOFTWARE ENGINEER E: matt at getpattern.com // P: 603.892.7736 We?re looking for beta testers. Go here to sign up! -------------- next part -------------- An HTML attachment was scrubbed... URL: From arek.bulski at gmail.com Wed Sep 7 18:56:03 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Thu, 8 Sep 2016 00:56:03 +0200 Subject: [Python-ideas] Shuffled Message-ID: Thank you all for the meta discussion. I know we sidetracked a bit but please everyone remember, this is an open discussion floor. I can present you a use case, just didnt think I even needed one. This seems so obvious to me that it should just go in without much saying. In the project I maintain (construct) there are declarative testcases that look like a long list of (func, args, excepted output, error type) tuples. There is no way for me to call shuffle in there. I can only use shuffled. So here it is, the use case you requested. The argument that it can be easily implemented does not stick. In interactive sessions, indeed writing own functions is a pain in the ass. Expecting batteries included is rightful. Builtins like min() max() sum() are sooo easy to implement on your own. So are some methods in itertools, you can even find their sources in the docs. That doesnt mean batteries should not be included. shuffled = sorted(list, key=lambda x: randint) This is wrong because sorting is comparison based and must be nlogn bound. Shuffling is cheaper computationally. Also someone would have to come up with a formal proof that it provides a uniform distribution which probably was done for the shuffle implementaion. Another argument still remains: there is a missing analogy between sort-sorted and shuffle-shuffled. I did NOT suggest making it a builtin. It should be added to random module. > "Also shuffle() should return self so mutating methods could be chained." That was a side suggestion. I withdraw that part. Chaining methods is sometimes useful but we can go without it. pozdrawiam, Arkadiusz Bulski -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Sep 7 19:00:55 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 8 Sep 2016 01:00:55 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> Message-ID: <7a4480a3-2909-ff9b-4a14-d1fcb68fe70a@mail.de> On 08.09.2016 00:41, Matt Gilson wrote: > I lurk around here a lot more than I actually post -- But in this case > I find myself weighing in firmly on the side of "please don't add > this". It'd just add noise in the documentation and __builtins__ > namespace. There are millions of useful functions that /could/ be > added to the `list` object (or to the `__builtins__` namespace). It's > the job of the core dev team to decide which are the most useful for > everybody and this is one where (in my opinion) the usefulness doesn't > justify the additional clutter. > > It seems like the primary argument for this is that some users would > like to have this functionality baked into an interactive session. Maybe, it wasn't obvious (despite I thought so), but I also don't see it in the built-ins. Another function in the "random" module would just suffice. But still, this is not my proposal, so I am gonna wait for what Arek brings up. Sven > If that's the case, then I'd like to point out that if you find > yourself wanting this in an interactive session frequently there is an > interactive startup file > > that you can modify so that this function will always be available to > /you/ whenever you start up an interactive session (and you can put > whatever else in there that you like as well :-). > > I hope that helps. > > On Wed, Sep 7, 2016 at 3:22 PM, Sven R. Kunze > wrote: > > Maybe, there's a misunderstanding here and I hope I didn't waste > too much of your and my time. > > Not sure where I got this from but my last status was that Arek > would like to have a "shuffled" function that does exactly what > you described. Maybe, that doesn't fit the reports description but > his pull request will tell the truth. :) > > > About the confusion of returning None. It's not confusing in the > sense of "o my god, I did it wrong, I need to learn it", but more, > like "I used shuffle, BECAUSE it's the only one I could find in a > hurry" and then ascertain "why the heck is there no alternative > returning the shuffled result instead of overwriting my list? I > would expect this from Python as it already provides both > alternatives for sorting". > > > Sven > > > > On 08.09.2016 00:02, Tim Peters wrote: > > [Sven R. Kunze >] > > I am not questioning experience which everyone in a team > can benefit from. > > > BUT experienced devs also need to recognize and respect > the fact that > younger/unexperienced developers are just better in detecting > inconsistencies and bloody work-arounds. They simply > haven't had to live > with them for so long. Experienced devs just are stuck in > a rut/are > routine-blinded: "we've done that for years", "there's no > better way". > > > That's the way we do it in our teams. Employing the new > guys as some sort of > inconsistency detectors. This way, they learn to find > their way around the > code base and they can improve it by doing so. > > And I would never allow it in my team, to dismiss this > kind of observation > from new colleagues. It's invaluable as they will become > routine-blinded as > well. > > I have been more than willing to discuss it, and I did not > close the > issue report. I did say I was opposed to it, but that's simply > because I am, and I explained there too _why_ I was opposed. > > Do you have anything to say about the specific proposal? I doubt > either of us has found this meta-discussion useful. I'm still > looking > for a compelling use case. The only concrete thing anyone has > noted > in `shuffled()`'s favor so far is that sometimes they're > surprised by > the behavior of random.shuffle(list) returning None in an > interactive > shell (noted by you, and by another, and I cheerfully own up > to being > a bit surprised by that too long ago). But that's an observation > about `random.shuffle()`, not about the proposed `shuffled()`. > > > [...] I would be far more annoyed if, e.g., > > random.shuffle(some_million_element_list) > > swamped my terminal with mountains of output. > > But you readily accept this behavior for "sorted"? That > makes no sense at > all. > > Of course it does. The only analogy to random.shuffle(big_list) > returning None that makes a lick of sense here is that > big_list.sort() > also returns None. IF a `shuffled()` function is introduced, > then of > course it should return its result - just like `sorted()` > returns its > result. > > > You can't both behaviors simultaneously, so the status > quo wins. > Indeed, the venerable status quo ;-) > > Nobody said to change "shuffle". > > A verbatim quote from the first message in this thread: > > "Also shuffle() should return self so mutating methods > could be chained." > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > -- > > pattern-sig.png > > Matt Gilson// SOFTWARE ENGINEER > > E: matt at getpattern.com // P: 603.892.7736 > > We?re looking for beta testers. Go here > to sign up! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed Sep 7 19:14:12 2016 From: mertz at gnosis.cx (David Mertz) Date: Wed, 7 Sep 2016 16:14:12 -0700 Subject: [Python-ideas] Shuffled In-Reply-To: <0e641553-6d0f-5799-12ea-f9e0ca516ed3@mail.de> References: <20160906163257.GX26300@ando.pearwood.info> <0e641553-6d0f-5799-12ea-f9e0ca516ed3@mail.de> Message-ID: On Wed, Sep 7, 2016 at 3:10 PM, Sven R. Kunze wrote: > @David > Your idea of a PyPI package could almost work. However, in an interactive > python console, I expect as much batteries included as possible to make it > as quick as possible. And considering how simple such wrapper would be, it > almost does not warrant the download of a third-party package. > Of course I don't think it's worth having a PyPI package for this one function. That's why I suggested it could possibly live in `boltons` (that already contains a lot of other simple functions that aren't in the stdlib) or in `arek_utils` (which I haven't described except implicitly as a collection of "a bunch of useful things"). Tim argues that this functionality is even too simple for inclusion in boltons; that might be true. But either way, it's not a huge burden in the interactive console to type a first line of `from my_utils import *` before getting down to work. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Wed Sep 7 19:14:33 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 8 Sep 2016 00:14:33 +0100 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> Message-ID: <369aa2f3-f757-7e0c-e19c-c268db13d585@btinternet.com> On 07/09/2016 23:41, Matt Gilson wrote: > I lurk around here a lot more than I actually post Very wise. I am a tournament Bridge player, and one of the things I say to my partners is: "Bidding is very simple. When you have something to say, say it (unless it really seems too dangerous). When you have nothing to say, Pass". > -- But in this case I find myself weighing in firmly on the side of > "please don't add this". It'd just add noise in the documentation and > __builtins__ namespace. There are millions of useful functions that > /could/ be added to the `list` object (or to the `__builtins__` > namespace). It's the job of the core dev team to decide which are the > most useful for everybody and this is one where (in my opinion) the > usefulness doesn't justify the additional clutter. > > It seems like the primary argument for this is that some users would > like to have this functionality baked into an interactive session. If > that's the case, then I'd like to point out that if you find yourself > wanting this in an interactive session frequently there is an > interactive startup file > > that you can modify so that this function will always be available to > /you/ whenever you start up an interactive session (and you can put > whatever else in there that you like as well :-). > > I hope that helps. +1 And incidentally, thank you for reminding me of this. I do actually use |PYTHONSTARTUP| (mostly to pre-import modules that I use frequently: sys, time, datetime, os, etc. - YMMV). But your reminder may well prod me into finding new uses. Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Sep 7 19:28:57 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 7 Sep 2016 19:28:57 -0400 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: On Wed, Sep 7, 2016 at 6:56 PM, Arek Bulski wrote: > In the project I maintain (construct) there are declarative testcases that > look like a long list of (func, args, excepted output, error type) tuples. > There is no way for me to call shuffle in there. Can you explain why? Something like this can be easily done with pytest: In [1]: def foo(x): ...: return x + 1 ...: In [2]: import pytest In [3]: @pytest.mark.parametrize('x, y', [ ...: (100, 101), ...: (200, 201), ...: (300, 301),]) ...: def test_foo(x, y): ...: assert foo(x) == y ...: In [4]: test_foo.parametrize.args Out[4]: ('x, y', [(100, 101), (200, 201), (300, 301)]) In [5]: import random In [6]: random.shuffle(test_foo.parametrize.args[1]) In [7]: test_foo.parametrize.args Out[7]: ('x, y', [(200, 201), (100, 101), (300, 301)]) -------------- next part -------------- An HTML attachment was scrubbed... URL: From arek.bulski at gmail.com Wed Sep 7 20:14:36 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Thu, 8 Sep 2016 02:14:36 +0200 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: If you want to see the declarative tests, here it is. https://github.com/construct/construct/blob/master/tests/test_all.py pozdrawiam, Arkadiusz Bulski 2016-09-08 2:13 GMT+02:00 Arek Bulski : > See for yourself. There is a long list of declarative tests. > https://github.com/construct/construct/blob/master/tests/test_all.py > > pozdrawiam, > Arkadiusz Bulski > > > 2016-09-08 1:28 GMT+02:00 Alexander Belopolsky < > alexander.belopolsky at gmail.com>: > >> >> On Wed, Sep 7, 2016 at 6:56 PM, Arek Bulski >> wrote: >> >>> In the project I maintain (construct) there are declarative testcases >>> that look like a long list of (func, args, excepted output, error type) >>> tuples. There is no way for me to call shuffle in there. >> >> >> Can you explain why? Something like this can be easily done with pytest: >> >> In [1]: def foo(x): >> ...: return x + 1 >> ...: >> >> In [2]: import pytest >> >> In [3]: @pytest.mark.parametrize('x, y', [ >> ...: (100, 101), >> ...: (200, 201), >> ...: (300, 301),]) >> ...: def test_foo(x, y): >> ...: assert foo(x) == y >> ...: >> >> In [4]: test_foo.parametrize.args >> Out[4]: ('x, y', [(100, 101), (200, 201), (300, 301)]) >> >> In [5]: import random >> >> In [6]: random.shuffle(test_foo.parametrize.args[1]) >> >> In [7]: test_foo.parametrize.args >> Out[7]: ('x, y', [(200, 201), (100, 101), (300, 301)]) >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Sep 7 21:12:55 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 7 Sep 2016 21:12:55 -0400 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: On Wed, Sep 7, 2016 at 8:14 PM, Arek Bulski wrote: > > If you want to see the declarative tests, here it is. > https://github.com/construct/construct/blob/master/tests/test_all.py So, why can't you call random.shuffle(all_tests) if you want to run your tests in random order? If for some reason you prefer to stick shuffled in for i, (func, args, res, exctype) in enumerate(tests): I would say it is a bad idea because it is not clear whether you would want enumerate(shuffled(tests)) or shuffled(enumerate(tests)) and what the difference between the two constructs is. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Sep 7 21:22:00 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 7 Sep 2016 21:22:00 -0400 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: On Wed, Sep 7, 2016 at 9:12 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > On Wed, Sep 7, 2016 at 8:14 PM, Arek Bulski wrote: > > > > If you want to see the declarative tests, here it is. > > https://github.com/construct/construct/blob/master/tests/test_all.py > > > So, why can't you call random.shuffle(all_tests) if you want to run your > tests in random order? It may be instructive for you to see how this functionality is implemented in CPython's own test suit: https://github.com/python/cpython/blob/276f4ef97a434d4279a2d207daa34cafcf0994f7/Lib/test/libregrtest/main.py#L230 -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Sep 7 22:00:51 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Sep 2016 12:00:51 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> Message-ID: <20160908020050.GA22471@ando.pearwood.info> On Wed, Sep 07, 2016 at 11:43:59PM +0200, Sven R. Kunze wrote: > BUT experienced devs also need to recognize and respect the fact that > younger/unexperienced developers are just better in detecting > inconsistencies and bloody work-arounds. That is not a fact. It is the opposite of a fact -- inexperienced developers are WORSE at spotting inconsistencies, because they don't recognise deep consistencies. For example, the on-going controversy about mutable default arguments: def func(x=0, y=[]): x += 1 y.append(1) print(x, y) To the inexperienced Python developer this makes no sense. Why does Pythod reset the value of x by default, while y remembers the value it had from the previous call? That's an inconsistency. But the more experienced dev realises that there is no inconsistency. Python behaves exactly the same way in both cases. Both x and y are treated the same: they both are treated as "Early Binding". The only difference is that 0 is immutable, and so x += 1 binds a new value to x, while [] is mutable, and y.append(1) modifies that value in place. Try this instead: def func(x=0, y=[]): x += 1 y = y + [1] # don't use += print(x, y) The more experienced programmer might not *like* this behaviour, they may wish that Python used Late Binding instead, but it takes a very subtle and deep knowledge of the language to understand that this is not an inconsistency despite the superficial appearances. -- Steve From ncoghlan at gmail.com Wed Sep 7 22:25:41 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 12:25:41 +1000 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On 8 September 2016 at 07:31, Hugh Fisher wrote: > On Wed, Sep 7, 2016 at 9:56 PM, Nick Coghlan wrote: >> >> Exactly - in a dynamic language like Python, running a typechecker is >> a form of testing, rather than a necessary part of getting the code to >> work in the first place. >> > > The average programmer such as myself will expect that if I write code > specifying the type of a variable or whatever it should *do > something*. It's code, I wrote it, it should be interpreted. > > Assuming PEP 526 is implemented, suppose as a very new programmer I write > > foo: dict = 0 > > (If anyone thinks even new programmers wouldn't write that, you've > never taught an introductory programming course.) > > It takes very little programming experience to know that is flat out > wrong. I cannot think of any other programming language with type > notation where this would not be immediately flagged as an error. But > Python will silently accept it? It wouldn't surprise me in the least if education focused environments like the Mu text editor and potentially even Jupyter Notebooks decide to start running a typechecker implicitly, as many IDEs already do that. That way, if students do add type annotations (even if they're missing from all of the example code presented), they'll be enforced automatically. That's a decision to be made by instructors and development environment developers based on their assessment of their target audience though, it isn't something that needs to be baked into the core runtime design. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Wed Sep 7 22:31:38 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Sep 2016 12:31:38 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> Message-ID: <20160908023138.GB22471@ando.pearwood.info> On Thu, Sep 08, 2016 at 12:22:28AM +0200, Sven R. Kunze wrote: > About the confusion of returning None. It's not confusing in the sense > of "o my god, I did it wrong, I need to learn it", but more, like "I > used shuffle, BECAUSE it's the only one I could find in a hurry" Then don't be in such a hurry. Honestly, there is no substitute for professionalism. Outside of fantasy novels and movies, there has never been a "ticking time bomb" situation where a programmer has to write code to shuffle a list in six seconds or the world will be destroyed. There is *always* time to at least glance at the docs, and it is never more than a handful of keypresses away in the interactive interpreter: help(random.shuffle) And if you don't read the docs? Oh well, you find out soon enough once you actually test your code or have a code review. No harm done. If the programmer doesn't do *any* of those things -- doesn't read the docs, doesn't test their code, doesn't do code reviews -- then there's no hope for them. We could provide shuffled() and shuffle() and they'll still end up picking the wrong one fifty percent of the time. "My wild guess of how a function works was wrong, so now I'm cheesed off that the function doesn't do what I expect" is NOT a good reason to add things to the standard library. > I would expect this from > Python as it already provides both alternatives for sorting". Does that mean you expect Python to provide appended() as well as append(), extended() as well as extend(), popped() as well as pop(), cleared() as well as clear(), inserted() as well as insert(), removed() as well as remove()? If your response is to say "Why do we need all those? Why on earth would anyone need to use list.cleared()?" then perhaps you can now understand why shuffled() has to prove itself too. Functions don't get added just because they *can* be added. They have to be worth it. Every function has costs as well as benefits. Each function adds: - more code to download, compile, maintain - more tests - more documentation - more for the user to learn - more choices for the user to decide between - more complexity - more risk of bugs Unless the proposal can demonstrate that the benefit outweighs the costs, it should not be added. Merely stating that you need it doesn't demonstate that a four-line helper function at the top of your module isn't sufficient. For what's it is worth, if it were *my* call, I'd accept that the costs of adding this are low, but the benefits are just a *tiny* bit higher. But that's a judgement call, and if Raymond see the cost:benefit ratio going the other way, I have no objective argument to change his mind. -- Steve From danilo.bellini at gmail.com Wed Sep 7 22:47:49 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Wed, 7 Sep 2016 23:47:49 -0300 Subject: [Python-ideas] Shuffled In-Reply-To: <20160908023138.GB22471@ando.pearwood.info> References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> <20160908023138.GB22471@ando.pearwood.info> Message-ID: Though I agree with the argument that inexperienced developers are [usually] worse, that's not the case here, unless anyone here is really trying to say the ones arguing for or against "shuffled" are inexperienced. These ad hominem won't bring us anywhere. No one seem to be arguing if "shuffled" is inconsistent, but the "shuffle" consistency is a subject that didn't reach consensus. And the point now seem to be whether "shuffled" is useful or not. If I understood correctly, that proposal is about writing code in a more functional style (or a expression-oriented programming style), with a pure function for shuffling. I think there are other related subjects that can be said about sorted/shuffled. For example: why a list? Why not a generator? Something like random.choice is useful to get one single value, but if one just want to see a few random values from a large input without repeating the values, what's the best approach? Does that mean you expect Python to provide appended() [...] extended() > [...] > Doesn't the "+" operator do that? > [...] list.cleared()? Huh? OP is clearly talking about avoiding state, avoiding "in-place" operations. A clear list would be an empty list []. The idea of "popped", "inserted" and "removed" are also simply off-topic. 2016-09-07 23:31 GMT-03:00 Steven D'Aprano : > On Thu, Sep 08, 2016 at 12:22:28AM +0200, Sven R. Kunze wrote: > > > About the confusion of returning None. It's not confusing in the sense > > of "o my god, I did it wrong, I need to learn it", but more, like "I > > used shuffle, BECAUSE it's the only one I could find in a hurry" > > Then don't be in such a hurry. > > Honestly, there is no substitute for professionalism. Outside of fantasy > novels and movies, there has never been a "ticking time bomb" situation > where a programmer has to write code to shuffle a list in six seconds or > the world will be destroyed. There is *always* time to at least glance > at the docs, and it is never more than a handful of keypresses away in > the interactive interpreter: > > help(random.shuffle) > > And if you don't read the docs? Oh well, you find out soon enough once > you actually test your code or have a code review. No harm done. > > If the programmer doesn't do *any* of those things -- doesn't read the > docs, doesn't test their code, doesn't do code reviews -- then there's > no hope for them. We could provide shuffled() and shuffle() and they'll > still end up picking the wrong one fifty percent of the time. > > "My wild guess of how a function works was wrong, so now I'm cheesed off > that the function doesn't do what I expect" is NOT a good reason to add > things to the standard library. > > > I would expect this from > > Python as it already provides both alternatives for sorting". > > Does that mean you expect Python to provide appended() as well as > append(), extended() as well as extend(), popped() as well as pop(), > cleared() as well as clear(), inserted() as well as insert(), removed() > as well as remove()? > > If your response is to say "Why do we need all those? Why on earth would > anyone need to use list.cleared()?" then perhaps you can now understand > why shuffled() has to prove itself too. Functions don't get added just > because they *can* be added. They have to be worth it. Every function > has costs as well as benefits. Each function adds: > > - more code to download, compile, maintain > - more tests > - more documentation > - more for the user to learn > - more choices for the user to decide between > - more complexity > - more risk of bugs > > Unless the proposal can demonstrate that the benefit outweighs the > costs, it should not be added. Merely stating that you need it > doesn't demonstate that a four-line helper function at the top of > your module isn't sufficient. > > For what's it is worth, if it were *my* call, I'd accept that the costs > of adding this are low, but the benefits are just a *tiny* bit higher. > But that's a judgement call, and if Raymond see the cost:benefit ratio > going the other way, I have no objective argument to change his mind. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 7 23:02:33 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 13:02:33 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: <20160908020050.GA22471@ando.pearwood.info> References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <20160908020050.GA22471@ando.pearwood.info> Message-ID: On 8 September 2016 at 12:00, Steven D'Aprano wrote: > On Wed, Sep 07, 2016 at 11:43:59PM +0200, Sven R. Kunze wrote: > >> BUT experienced devs also need to recognize and respect the fact that >> younger/unexperienced developers are just better in detecting >> inconsistencies and bloody work-arounds. > > That is not a fact. It is the opposite of a fact -- inexperienced > developers are WORSE at spotting inconsistencies, because they don't > recognise deep consistencies. There are also cases where we're internally inconsistent or lacking internal integration because we're being consistent with different external environments. So we have both binary floating point and decimal floating point because there are inherent conflicts in the underlying computational models. We have both the abc module (which deals strictly with runtime classes) and the typing module (which deals strictly with type categories) as those things are similar, but not the same. We have both asyncio and traditional synchronous APIs as there are essential conflicts between blocking and non-blocking models of control flow. Cross-platform secure network connectivity handling is a pain because many of the underlying platform APIs are unreliably implemented and poorly documented moving targets. Cross-platform software distribution is a pain largely because major platform vendors are typically competing to lock in developers, so their incentives are aligned with making what we're trying to do difficult rather than offering to help out. For these kinds of cases, frustrations on the part of experienced developers arise when folks ask entirely reasonable questions "Why is so complicated?", but then refuse to accept the answer of "Because it's genuinely complex". Pretending complexity doesn't exist doesn't help anyone - it means they're left to their own devices to come up with answers, rather than receiving more explicit guidance at the language level. More directly relevant to the current discussion is the principle of "Not every three line function needs to be a builtin (or even in the standard library)". There are ways to collect objective data on the relative usage of particular constructs (especially as hosting platforms like GitHub team up with analytics platforms like Google Big Query to make their data searchable, and initiatives like Software Heritage aim to bring that kind of capability to all published open source software), but one of the *simplest* criteria to apply, and the one that experienced core developers can typically answer without further research, is "Would the standard library benefit from this?". When the answer to that is "No", then the default answer to requests for API additions that are relatively easy for people to implement for themselves is always going to be "No" - otherwise the stdlib will collapse under its own weight (and the persistent "up and to the right" trend on http://bugs.python.org/issue?@template=stats means that there's a solid argument to be made that the standard library is already bigger than we can effectively maintain at current maintenance funding levels). Cheers, Nick. P.S. I wrote http://www.curiousefficiency.org/posts/2011/04/musings-on-culture-of-python-dev.html about this a few years ago, and I still think it's an accurate description of the underlying causes of these conflicts -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Wed Sep 7 23:22:39 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Sep 2016 13:22:39 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> <20160908023138.GB22471@ando.pearwood.info> Message-ID: <20160908032239.GC22471@ando.pearwood.info> On Wed, Sep 07, 2016 at 11:47:49PM -0300, Danilo J. S. Bellini wrote: > Though I agree with the argument that inexperienced developers are > [usually] worse, that's not the case here, unless anyone here is really > trying to say the ones arguing for or against "shuffled" are inexperienced. > These ad hominem won't bring us anywhere. That is not what I said. Please be careful of accusing others of things they didn't say. It was Sven who made the claim that inexperienced developers are better at seeing inconsistancies. I was responding specifically to that claim. I'm not saying anything about the experience, or lack of, of either Sven or Arek. > No one seem to be arguing if "shuffled" is inconsistent, but the "shuffle" > consistency is a subject that didn't reach consensus. And the point now > seem to be whether "shuffled" is useful or not. If I understood correctly, > that proposal is about writing code in a more functional style (or a > expression-oriented programming style), with a pure function for shuffling. > > I think there are other related subjects that can be said about > sorted/shuffled. For example: why a list? Why not a generator? You cannot sort a sequence lazily. You have to sort the entire sequence before you can be sure which item comes first. Making sorted() an iterator by default doesn't give you any advantage: it still has to sort the entire sequence before it can yield the first item. > Something > like random.choice is useful to get one single value, but if one just want > to see a few random values from a large input without repeating the values, > what's the best approach? random.sample() > Does that mean you expect Python to provide appended() [...] extended() > > [...] > > > Doesn't the "+" operator do that? Of course it does. But the "consistency" argument would be "why is append a method not an operator? Why isn't + called appended?". That's just spelling. It isn't that important. > > [...] list.cleared()? > > Huh? OP is clearly talking about avoiding state, avoiding "in-place" > operations. A clear list would be an empty list []. Of course. We can see that. But again, that's "inconsistent": * the in-place operation is spelled `list.clear()` * the functional operation is spelled `[]` There is nothing wrong with that. The value of consistency is grossly exaggerated. Sometimes it really doesn't matter if similar operations are spelled differently. -- Steve From ncoghlan at gmail.com Wed Sep 7 23:23:54 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 13:23:54 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> <20160908023138.GB22471@ando.pearwood.info> Message-ID: On 8 September 2016 at 12:47, Danilo J. S. Bellini wrote: > Though I agree with the argument that inexperienced developers are [usually] > worse, that's not the case here, unless anyone here is really trying to say > the ones arguing for or against "shuffled" are inexperienced. These ad > hominem won't bring us anywhere. > > No one seem to be arguing if "shuffled" is inconsistent, but the "shuffle" > consistency is a subject that didn't reach consensus. And the point now seem > to be whether "shuffled" is useful or not. If I understood correctly, that > proposal is about writing code in a more functional style (or a > expression-oriented programming style), with a pure function for shuffling. A pure function for shuffling already exists - random.sample() So the question at hand boils down to whether it makes sense to provide: def shuffled(self, container): return self.sample(container, len(container)) as a method on Random objects, and equivalently as a top-level module function. I'd personally be in favour of that as a learning bridge between shuffling and sampling (shuffle in place -> shuffle out of place -> sample a subset rather than shuffling out of place and slicing), but I'm neither a professional educator nor a maintainer or heavy user of the module in question, so my preference in the matter should be weighted pretty low. > I think there are other related subjects that can be said about > sorted/shuffled. For example: why a list? Why not a generator? This is covered in the random.sample() docs - returning a list is useful, as you can then easily use slices and tuple unpacking to partition the result: winners = random.sample(participants, 10) first, second, *runners_up = winners Beyond that practical benefit, if you want random-sampling-with-replacement, then "map(random.choice, container)" already has you covered, while random-sampling-without-replacement inherently needs to maintain a set of already produced values so it can avoid repeating them, which makes the apparent memory efficiency of using a generator instead is somewhat illusory. > Something > like random.choice is useful to get one single value, but if one just want > to see a few random values from a large input without repeating the values, > what's the best approach? Use random.sample(), that's what it's for. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Sep 7 23:28:32 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 13:28:32 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> <20160908023138.GB22471@ando.pearwood.info> Message-ID: On 8 September 2016 at 13:23, Nick Coghlan wrote: > Beyond that practical benefit, if you want > random-sampling-with-replacement, then "map(random.choice, container)" Oops, that was supposed to be "map(random.choice, itertools.repeat(container))". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From danilo.bellini at gmail.com Wed Sep 7 23:33:42 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Thu, 8 Sep 2016 00:33:42 -0300 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> <20160908023138.GB22471@ando.pearwood.info> Message-ID: Nice to know about random.sample! =) I think what OP said can then be reduced to having the default k in random.sample to be the iterable size. The existance of random.sample is a very strong argument against "shuffled", and the only "feature" shuffled would have that random.sample doesn't have is that default size. You cannot sort a sequence lazily > You can, but it probably wouldn't be efficient if you need all the values. On the other hand, if you need just the 3 smaller values of a huge list... well, that's another topic. 2016-09-08 0:28 GMT-03:00 Nick Coghlan : > On 8 September 2016 at 13:23, Nick Coghlan wrote: > > > Beyond that practical benefit, if you want > > random-sampling-with-replacement, then "map(random.choice, container)" > > Oops, that was supposed to be "map(random.choice, > itertools.repeat(container))". > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 8 00:26:25 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 14:26:25 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> <20160908023138.GB22471@ando.pearwood.info> Message-ID: On 8 September 2016 at 13:33, Danilo J. S. Bellini wrote: > Nice to know about random.sample! =) > > I think what OP said can then be reduced to having the default k in > random.sample to be the iterable size. The existance of random.sample is a > very strong argument against "shuffled", and the only "feature" shuffled > would have that random.sample doesn't have is that default size. There are a few reasons I don't think defining a default for random.sample() would be a good answer to Arek's question: 1. The cognitive leap between shuffling and sampling isn't small 2. "shuffled" would be a more logical name for an out-of-place shuffle than "sample" 3. "Copy the whole container" would be a surprising default for a function called "sample" 4. With a default, random.sample becomes more easily confused with random.choice For the last two concerns, if I saw "result = random.sample(container)" and didn't already know about random.choice(), I'd expect it to behave like random.choice(). Even knowing they're different, I'd still need to do a double-take to make sure I was remembering which was which correctly. By contrast, "random.sample(container, 1)", "random.sample(container, k)", "random.sample(container, len(container))" are all clearly different from the single result primitive "random.choice(container)" One interesting (to me anyway) aspect of an out-of-place shuffle is that you can readily implement it based on *either* of the more primitive operations (in-place shuffle or random sample): def shuffled(container): result = list(container) random.shuffle(result) return result def shuffled(container): return random.sample(container, len(container)) Writing down those two examples does highlight a potential refactoring benefit to having "out-of-place shuffle" as an explicitly represented concept in the core Random API: it can be defined in terms of shuffle or sample *on the same Random instance*, and hence automatically benefit when code switches from using the global random number generator to using a purpose-specific Random instance that allows greater control over the reproducibility of results (which can be very important for testing, games, modeling & simulation). The above helper functions are both flawed on that front: they hardcode the use of the default global random number generator. To permit use of a specific random instance, they need to be changed to: def shuffled(container, random=random): result = list(container) random.shuffle(result) return result def shuffled(container, random=random): return random.sample(container, len(container)) and then used as "result = shuffled(original, my_random_instance)" if you decide to switch away from the global API. By contrast, a method based implementation could be refactored the exact same way as any other random.Random method: result = my_random_instance.shuffled(original) I'll reiterate that I don't have a use case for this myself, but I'll cite the key arguments I see in favour: - an out-of-place shuffle may be useful as a bridging concept between in place shuffling and out of place sampling (in either direction) - the presence of "shuffled" becomes a reminder that "shuffle" itself is an in-place operation - "write it yourself" isn't as simple as it first sounds due to the common migration away from the random module functions to a custom Random instance Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From danilo.bellini at gmail.com Thu Sep 8 01:05:17 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Thu, 8 Sep 2016 02:05:17 -0300 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> <20160908023138.GB22471@ando.pearwood.info> Message-ID: > > 1. The cognitive leap between shuffling and sampling isn't small > I don't think so, actually the Fisher-Yates shuffle algorithm algorithm is an iterative sampling algorithm: https://gist.github.com/danilo bellini/6384872 > 2. "shuffled" would be a more logical name for an out-of-place shuffle > than "sample" > Agreed. > 3. "Copy the whole container" would be a surprising default for a > function called "sample" > Perhaps "sample the whole population" sounds strange. Indeed, statisticians probably wouldn't be happy with that. That reminds me the variance estimator with the extra unbiasing "-1" in the denominator versus the population variance. > 4. With a default, random.sample becomes more easily confused with > random.choice > I don't think k=1 would be a good default sample size from a statistics point of view, but I get the point (I'm from a DSP background, where "a sample" means one single "value"). Controling the random function is required for the function to be really pure, else its output won't depend only on the inputs (and there would be some "state" in that "implicit input"). That would also be a great feature when non-uniform (or external) random number generators are to be used. This seem to be something that only shuffle gives some control (among the functions we're talking about), or am I missing something? 2016-09-08 1:26 GMT-03:00 Nick Coghlan : > On 8 September 2016 at 13:33, Danilo J. S. Bellini > wrote: > > Nice to know about random.sample! =) > > > > I think what OP said can then be reduced to having the default k in > > random.sample to be the iterable size. The existance of random.sample is > a > > very strong argument against "shuffled", and the only "feature" shuffled > > would have that random.sample doesn't have is that default size. > > There are a few reasons I don't think defining a default for > random.sample() would be a good answer to Arek's question: > > 1. The cognitive leap between shuffling and sampling isn't small > 2. "shuffled" would be a more logical name for an out-of-place shuffle > than "sample" > 3. "Copy the whole container" would be a surprising default for a > function called "sample" > 4. With a default, random.sample becomes more easily confused with > random.choice > > For the last two concerns, if I saw "result = > random.sample(container)" and didn't already know about > random.choice(), I'd expect it to behave like random.choice(). Even > knowing they're different, I'd still need to do a double-take to make > sure I was remembering which was which correctly. By contrast, > "random.sample(container, 1)", "random.sample(container, k)", > "random.sample(container, len(container))" are all clearly different > from the single result primitive "random.choice(container)" > > One interesting (to me anyway) aspect of an out-of-place shuffle is > that you can readily implement it based on *either* of the more > primitive operations (in-place shuffle or random sample): > > def shuffled(container): > result = list(container) > random.shuffle(result) > return result > > def shuffled(container): > return random.sample(container, len(container)) > > Writing down those two examples does highlight a potential refactoring > benefit to having "out-of-place shuffle" as an explicitly represented > concept in the core Random API: it can be defined in terms of shuffle > or sample *on the same Random instance*, and hence automatically > benefit when code switches from using the global random number > generator to using a purpose-specific Random instance that allows > greater control over the reproducibility of results (which can be very > important for testing, games, modeling & simulation). > > The above helper functions are both flawed on that front: they > hardcode the use of the default global random number generator. To > permit use of a specific random instance, they need to be changed to: > > def shuffled(container, random=random): > result = list(container) > random.shuffle(result) > return result > > def shuffled(container, random=random): > return random.sample(container, len(container)) > > and then used as "result = shuffled(original, my_random_instance)" if > you decide to switch away from the global API. > > By contrast, a method based implementation could be refactored the > exact same way as any other random.Random method: > > result = my_random_instance.shuffled(original) > > I'll reiterate that I don't have a use case for this myself, but I'll > cite the key arguments I see in favour: > > - an out-of-place shuffle may be useful as a bridging concept between > in place shuffling and out of place sampling (in either direction) > - the presence of "shuffled" becomes a reminder that "shuffle" itself > is an in-place operation > - "write it yourself" isn't as simple as it first sounds due to the > common migration away from the random module functions to a custom > Random instance > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 8 04:02:25 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 18:02:25 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <7b80de0f-ad1a-e666-90f8-0447e6c6ae38@mail.de> <20160908023138.GB22471@ando.pearwood.info> Message-ID: On 8 September 2016 at 15:05, Danilo J. S. Bellini wrote: >> 1. The cognitive leap between shuffling and sampling isn't small > > I don't think so, actually the Fisher-Yates shuffle algorithm algorithm is > an iterative sampling algorithm: > https://gist.github.com/danilobellini/6384872 I'm not talking about mathematical equivalence, I'm talking about making an unaided leap from "shuffle this deck of cards" (random.shuffle) and "pick a card, any card" (random.choice) to "choose a random sample from this population" (random.sample). I can see people following that logic given suitable instruction (since they really are closely related operations), but it's a tough connection to see on your own. >> 4. With a default, random.sample becomes more easily confused with >> random.choice > > I don't think k=1 would be a good default sample size from a statistics > point of view, but I get the point (I'm from a DSP background, where "a > sample" means one single "value"). Likewise - it isn't that I think "1" would be a reasonable default, it's that without the second argument being there, I lapse back into DSP terminology rather than statistical terminology. Requiring the second argument as random.sample() does today keeps everything nicely unambiguous. > Controling the random function is required for the function to be really > pure, else its output won't depend only on the inputs (and there would be > some "state" in that "implicit input"). That would also be a great feature > when non-uniform (or external) random number generators are to be used. This > seem to be something that only shuffle gives some control (among the > functions we're talking about), or am I missing something? The module level "functions" in random are just bound methods for a default global random.Random() instance, so they're not truly pure - there's interdependence there via the shared PRNG state. However, by creating your *own* Random instance, or another object that provides the same API, you can get a lot more control over things, including reproducible behaviour for a given seed. I'm not familiar with how shuffle works internally, but presumably passing a non-uniform distribution is a way let you bias the shuffle (the docs don't actually explain *why* you'd want to use a randomiser other than the default). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From arek.bulski at gmail.com Thu Sep 8 05:34:57 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Thu, 8 Sep 2016 11:34:57 +0200 Subject: [Python-ideas] Shuffled Message-ID: > So, why can't you call random.shuffle(all_tests) if you want to run your tests in random order? I dont randomize test order. People should stop assuming that they know better. I need to randomize some arguments for one particular test and I cannot call shuffle between tests. Its a continous list of declarative tests. Needs to be shuffled(). https://github.com/construct/construct/blob/master/tests/test_all.py See? No way to put imperative code between tests. > sample(container, len(container)) That wont work because I would have to type the expression that is used as argument twice in a test. I need shuffled. Enough said. > I'll reiterate that I don't have a use case for this myself... I dont have a use case for half of what the std library offers. Or for type annotations. Asynchronous comprehesions, what is that? Do you see me rejecting those? > (sample having default of entire list size) That would work but would not be pretty. shuffled() is self explanatory and has a nice ring to it. Randomized list is not a sample by definition. pozdrawiam, Arkadiusz Bulski -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.fisher at gmail.com Thu Sep 8 05:46:41 2016 From: hugo.fisher at gmail.com (Hugh Fisher) Date: Thu, 8 Sep 2016 19:46:41 +1000 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On Thu, Sep 8, 2016 at 8:00 AM, Paul Moore wrote: > >> Type annotations are code, not tests. > > Not in Python they aren't. > The interpreter certainly thinks they're code rather than comments or docstrings, even before PEP 526. I type this into my Python 3.4 interpreter: >>> def foo(x:itn): >>> ... return x and the interpreter raises a NameError because 'itn' is not defined. Annotations look like code, they're mixed in with names and operators and literals and keywords, and all the standard syntax and semantic checks are applied. -- cheers, Hugh Fisher From danilo.bellini at gmail.com Thu Sep 8 06:13:06 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Thu, 8 Sep 2016 07:13:06 -0300 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: > > That wont work because I would have to type the expression that is used as > argument twice in a test. (lambda data: random.sample(data, len(data)))(container) That lambda is actually your "shuffled"... 2016-09-08 6:34 GMT-03:00 Arek Bulski : > > So, why can't you call random.shuffle(all_tests) if you want to run > your tests in random order? > > I dont randomize test order. People should stop assuming that they know > better. I need to randomize some arguments for one particular test and I > cannot call shuffle between tests. Its a continous list of declarative > tests. Needs to be shuffled(). > > https://github.com/construct/construct/blob/master/tests/test_all.py > See? No way to put imperative code between tests. > > > sample(container, len(container)) > > That wont work because I would have to type the expression that is used as > argument twice in a test. I need shuffled. Enough said. > > > I'll reiterate that I don't have a use case for this myself... > > I dont have a use case for half of what the std library offers. Or for > type annotations. Asynchronous comprehesions, what is that? Do you see me > rejecting those? > > > (sample having default of entire list size) > > That would work but would not be pretty. shuffled() is self explanatory > and has a nice ring to it. Randomized list is not a sample by definition. > > > pozdrawiam, > Arkadiusz Bulski > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 8 06:19:14 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Sep 2016 20:19:14 +1000 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On 8 September 2016 at 19:46, Hugh Fisher wrote: > On Thu, Sep 8, 2016 at 8:00 AM, Paul Moore wrote: >> >>> Type annotations are code, not tests. >> >> Not in Python they aren't. > > The interpreter certainly thinks they're code rather than comments or > docstrings, even before PEP 526. I type this into my Python 3.4 > interpreter: > >>>> def foo(x:itn): >>>> ... return x > > and the interpreter raises a NameError because 'itn' is not defined. > > Annotations look like code, they're mixed in with names and operators > and literals and keywords, and all the standard syntax and semantic > checks are applied. This is why I quite like the phrase "type assertions", particularly for the variable annotation form - as with assert statements, annotations are tests that you can write inline with your code rather than necessarily putting them in a different file. That doesn't make them not tests - it just makes them inline self-tests instead of external ones. The fact that assertions are checked by default (but can be turned off), while annotations are ignored by default (but can be checked with the appropriate optional tooling) has more to do with the relative maturity of the typecheckers than it does anything else - at this point in history, you still need to be on the lookout for bugs in the typecheckers themselves, rather than assuming that because a typechecker complained, there must be something wrong with the code. Folks with the kinds of problems that typecheckers solve are likely to be prepared to put up with the inconvenience of also dealing with bugs in them. By contrast, folks that just want to run some Python code that they already know works well enough for their purposes will want Python to keep working the same way it always has. (Hence why all of the typechecker developers chiming into these discussions point out that they always err on the side of *not* complaining about *correct* code, even if that increases their chances of missing some situations where the code is actually wrong) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Thu Sep 8 06:31:22 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Sep 2016 11:31:22 +0100 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: On 8 September 2016 at 10:34, Arek Bulski wrote: > That wont work because I would have to type the expression that is used as > argument twice in a test. I need shuffled. Enough said. You've probably spent way more time debating shuffled here than you would have needed to add a shuffled function to your test file. Your justification for the amount of time you've spent here, and that you've persuaded others to spend (and are further asking to be spent on maintenance) *has* to be more significant than "I needed shuffled() for this one specific case". But it's hard to understand what the *general* use cases you've (presumably) identified are - everyone responding has been saying "but I don't personally have a use for this". So in order to warrant the addition of a function to the stdlib, you need to explain what benefits justify the cost - and you simply haven't done that. The one case you've stated could have been solved in 2 minutes by writing a one-off function. And no-one has said "now that you mention it, I find the problem you had comes up a lot", so we're still at the position where it's a specialised use. The cost is non-trivial. Implementation is a relatively small cost, design is slightly higher (see for example Nick's point that this should be a method on the random instance as well as a module-level function), documentation and tests need to be added, 3rd party training material needs to be updated, bug reports need to be managed and resolved, etc. The benefit? I'm not clear, beyond you not needing to write a custom function in your test suite, and a couple of people who weren't aware they could use PYTHONSTARTUP to add their own personal functions being less frustrated by how random.shuffle behaves. And some theoretical "people might find this useful" comments that no-one has backed up with code samples. Nick contributed some suggestions in terms of benefits around teachability, and the improved design noted above. But the point here is that you shouldn't expect others to justify your proposal for you - you need to justify it yourself, or be prepared to accept "we don't see the benefit" as a response. Paul From p.f.moore at gmail.com Thu Sep 8 06:53:39 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Sep 2016 11:53:39 +0100 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: On 8 September 2016 at 10:46, Hugh Fisher wrote: > Annotations look like code, they're mixed in with names and operators > and literals and keywords, and all the standard syntax and semantic > checks are applied. Apologies - I was thinking you were referring to variable annotations (and even there I was forgetting that the proposal had been changed to allow runtime introspection). What I should have said is that annotations do precisely that - they *annotate* the code objects with information. They don't in themselves have semantic consequences (and that's something the various PEPs have been at pains to establish - code that doesn't explicitly introspect the annotations will run the same with or without them). Paul From alexander.belopolsky at gmail.com Thu Sep 8 07:05:08 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 8 Sep 2016 07:05:08 -0400 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: > On Sep 8, 2016, at 5:34 AM, Arek Bulski wrote: > > > So, why can't you call random.shuffle(all_tests) if you want to run your tests in random order? > > I dont randomize test order. People should stop assuming that they know better. That's the best I could do given the minimal explanation that you provided. > I need to randomize some arguments for one particular test and I cannot call shuffle between tests. I still don't understand what you want. > Its a continous list of declarative tests. What is "Its"? > Needs to be shuffled(). This sentence would benefit from having a subject. > > https://github.com/construct/construct/blob/master/tests/test_all.py > See? No. Not without further assumptions. > No way to put imperative code between tests. Maybe you could show how you would write what you want with a "shuffled" function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Sep 8 07:47:40 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Sep 2016 21:47:40 +1000 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: <20160908114739.GD22471@ando.pearwood.info> This thread is just going around and around in circles, so I will make one final response and then I'm going to bow out. On Thu, Sep 08, 2016 at 11:34:57AM +0200, Arek Bulski wrote: > Needs to be shuffled(). > > https://github.com/construct/construct/blob/master/tests/test_all.py > See? No way to put imperative code between tests. > > I need shuffled. Enough said. Okay, okay. Suppose you convince us. You complain and stamp your feet and demand and insist and repeat "I need shuffled" ten thousand times until you wear down everyone and they give in. With the 3.6 feature freeze just a few days away, this new code is going into 3.7, which probably won't reach a stable version until March or April 2018. So what happens between now and then? I see that your package Construct supports Python versions 2.6, 2.7, 3.3, 3.4 and 3.5. Are you ready to abandon ALL of those and support only 3.7 and higher? If not, then you can't even use the std lib implementation of shuffled() **because it won't exist**. At best you can test for a std lib implementation, falling back to your own when it doesn't exist. So you have to write your own implementation regardless of what the std lib does. You cannot rely on the std lib version until you've dropped support for Python 3.6, which you haven't even got support for yet! If you support just *three* releases, the current release plus the two previous, you won't be able to rely on the std lib shuffled() until Python 3.9 comes out, which will probably be in 2021. Maybe you do need shuffled(). But you can't use the one in the std lib until (probably) 2021 or later. What are you going to do for the next five years? - Do without. Then you don't really need it. - Write your own. Then you don't need it to be in the std lib. Adding new functions to the std lib now is an investment for future code, not a necessity for current, existing code. If you need this shuffled function now, you know how to write it. Harsh, but true. -- Steve From k7hoven at gmail.com Thu Sep 8 10:10:33 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 8 Sep 2016 17:10:33 +0300 Subject: [Python-ideas] Three ways of paths canonization In-Reply-To: References: Message-ID: On Wed, Sep 7, 2016 at 12:20 PM, Serhiy Storchaka wrote: > Before removing provisional state from the pathlib module, we should resolve > the issue with Path.resolve(). It corresponds to os.path.realpath(), but > behaves differently in case of non-existent path. Actually we can't say that > any of these functions is wrong. Both behaviors make sense in different > situations. > Oh, almost missed this email. I won't be able to look into the issue you bring up at the moment. However, to be honest, I keep coming back to the thought that pathlib should have another provisional cycle. As the new fspath protocol (PEP 519) will be released in 3.6, the experimentation with pathlib has still been very limited. Since pathlib was not compatible with the rest of the stdlib, the limited experimentation that was done previously is not even completely valid, because PEP 519 makes the situation a little different. And there are some rough edges there, which make some things a little awkward. Removing the provisional status now might lead to the module becoming a mere burden, as third-party variations become significantly better. -- Koos > The readlink utility from GNU coreutils has three mode for resolving file > path: > > -f, --canonicalize > canonicalize by following every symlink in every component of > the given name recursively; all but the last component must exist > > -e, --canonicalize-existing > canonicalize by following every symlink in every component of > the given name recursively, all components must exist > > -m, --canonicalize-missing > canonicalize by following every symlink in every component of > the given name recursively, without requirements on components existence > > Current behavior of posixpath.realpath() is matches (besides one minor > detail) to `readlink -m`. The behavior of Path.resolve() matches `readlink > -e`. > > I have proposed a patch that adds three-state optional parameter to > posixpath.realpath() and I'm going to provide similar patch for > Path.resolve(). But I'm not sure this is good API. Are there better > variants? > > [1] http://bugs.python.org/issue19717 > [2] http://bugs.python.org/issue27002 > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- + Koos Zevenhoven + http://twitter.com/k7hoven + From arek.bulski at gmail.com Thu Sep 8 11:59:46 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Thu, 8 Sep 2016 17:59:46 +0200 Subject: [Python-ideas] Shuffled Message-ID: > You've probably spent way more time debating shuffled here than you would have needed to add a shuffled function to your test file. Another presupposition. I did add it to my utility. I am proposing to add this to the std library so that others dont have to. pozdrawiam, Arkadiusz Bulski ?? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Sep 8 12:13:03 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Sep 2016 17:13:03 +0100 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: On 8 September 2016 at 16:59, Arek Bulski wrote: >> You've probably spent way more time debating shuffled here than you would >> have needed to add a shuffled function to your test file. > > Another presupposition. I did add it to my utility. I am proposing to add > this to the std library so that others dont have to. OK, well (1) yet again you hadn't clarified that (it's not in the code you provided a link to) and (2) you haven't found or provided any evidence that anyone else needs it. I'm not going to comment further on this thread. You've been advised on what you'd need to do if you want to stand any chance of getting the decision changed, and if you don't feel it's worthwhile to do so that's fine. Paul. From guido at python.org Thu Sep 8 12:45:36 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 8 Sep 2016 09:45:36 -0700 Subject: [Python-ideas] Three ways of paths canonization In-Reply-To: <22480.10376.844689.468617@turnbull.sk.tsukuba.ac.jp> References: <22480.10376.844689.468617@turnbull.sk.tsukuba.ac.jp> Message-ID: I would prefer it if Path.resolve() resolved symlinks until it hits something that doesn't exist and then just keep the rest of the path unchanged. I think this is the equivalent of -m in the mentioned utility (which I had never heard of). It looks like os.path.realpath() already works this way. Steve Dower just mentioned to me that the Windows version of Path.resolve() uses a Windows API that opens the file and then asks the system for the pathname -- that doesn't work if the file doesn't exist, so it should be fixed to back off and try the same thing on the parent, etc. We should treat these things as bugs to fix before 3.6 (in 3.6b1 if possible) and make pathlib non-provisional as of 3.6.0.Probably these things should be fixed in 3.5.3 as well, since pathlib is still provisional there. --Guido On Wed, Sep 7, 2016 at 7:47 AM, Stephen J. Turnbull wrote: > Serhiy Storchaka writes: > > > The readlink utility from GNU coreutils has three mode for resolving > > file path: > > > > -f, --canonicalize > > canonicalize by following every symlink in every > > component of the given name recursively; all but the last component must > > exist > > > > -e, --canonicalize-existing > > canonicalize by following every symlink in every > > component of the given name recursively, all components must exist > > > > -m, --canonicalize-missing > > canonicalize by following every symlink in every > > component of the given name recursively, without requirements on > > components existence > > In Mac OS X (and I suppose other BSDs), realpath(3) implements -e. > glibc does none of these, instead: > > GNU extensions > If the call fails with either EACCES or ENOENT and > resolved_path is not NULL, then the prefix of path that is not > readable or does not exist is returned in resolved_path. > > I suppose this nonstandard behavior is controlled by a #define, but > the Linux manpage doesn't specify it. > > > Current behavior of posixpath.realpath() is matches (besides one minor > > detail) to `readlink -m`. The behavior of Path.resolve() matches > > `readlink -e`. > > This looks like a bug in posixpath, while Path.resolve follows POSIX. > http://pubs.opengroup.org/onlinepubs/009695399/functions/realpath.html > sez: > > RETURN VALUE > > Upon successful completion, realpath() shall return a pointer to > the resolved name. Otherwise, realpath() shall return a null > pointer and set errno to indicate the error, and the contents of > the buffer pointed to by resolved_name are undefined. > > ERRORS > > The realpath() function shall fail if: > > [...] > [ENOENT] A component of file_name does not name an existing file or > file_name points to an empty string. > [ENOTDIR] A component of the path prefix is not a directory. > > which corresponds to -e. > > > I have proposed a patch that adds three-state optional parameter to > > posixpath.realpath() and I'm going to provide similar patch for > > Path.resolve(). But I'm not sure this is good API. Are there better > > variants? > > Said parameter will almost always be a constant. Usually in those > cases Python prefers to use different functions. Eg, > > posixpath.realpath -e > posixpath.realpath_require_prefix -f > posixpath.realpath_allow_missing -m > posixpath.realpath_gnuext GNU extension > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From tim.peters at gmail.com Thu Sep 8 13:05:56 2016 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 8 Sep 2016 12:05:56 -0500 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: [Arek Bulski ] > I dont randomize test order. People should stop assuming that they know > better. They should, but, to be fair, you've left them little choice because you've been so telegraphic about why you think this is a compelling idea. Others are trying to fill in the blanks, and can only guess. > I need to randomize some arguments for one particular test and I > cannot call shuffle between tests. Its a continous list of declarative > tests. Needs to be shuffled(). I'm convinced you want to do that. Fine. The question is whether this is a _common enough_ need to justify all the ongoing costs of building it into a standard library. But in the only actual code you posted a link to, you don't even do it yourself. That just _enhances_ the perception that it's rarely needed. A question: do you know why `product()` isn't built in? Like product([2, 3, 4]) == 24? `sum()` is built in. So are `any()` and `all()` (other kinds of specialized reduction operations). Why does Python pick on multiplication as "unworthy" of the same kind of shortcut? The objections to product() were much the same as the ones being raised against shuffled(): sure, it would be useful at times in some contexts, but it's _rarely_ needed. It's also easy to write your own in the seemingly rare cases it is useful. > https://github.com/construct/construct/blob/master/tests/test_all.py > See? No way to put imperative code between tests. But there's also no example of anything in that code that requires shuffling. "But shuffled() isn't in the std lib!" isn't relevant: if you really _needed_ shuffling in test_all.py, you already know how to write such function, so if you really needed it you would have done so already. "Use cases" are about actual needs, not speculating about "maybe nice to have some day". >> sample(container, len(container)) > That wont work because I would have to type the expression that is used as > argument twice in a test. I need shuffled. Enough said. shuffled = lambda container: random.sample(container, len(container)) Now you have _a_ way to supply a `shuffled()` function (although I'd strongly recommend writing it in an obvious way building on random.shuffle()). But to justify becoming part of the standard distribution, it's not enough that _you_ "need" it: lots of people have to find it attractive. That's the same bar `sum()`, `any()`, `all()`, `sorted()` and `reversed()` (for examples) had to leap over. They were all popular ideas at the times they were proposed, and they indeed went on to become widely used in all kinds of code. I understand that you're proposing to make `shuffled()` a `Random` method instead of a builtin function, but that doesn't lower the bar much. I expect most people would say there's already "too much" rarely used stuff in Random. > I dont have a use case for half of what the std library offers. Or for type > annotations. Asynchronous comprehesions, what is that? Do you see me > rejecting those? I do not. Do you reject `product()`? You should ;-) But it's not about just you, and there's nothing personal about it. The examples you listed have many strong advocates, and "many" and "strong" both played roles in their acceptance. In contrast, the constituency for "shuffled()" appears small. >> (sample having default of entire list size) > That would work but would not be pretty. shuffled() is self explanatory and > has a nice ring to it. Randomized list is not a sample by definition. sample() in this context uses a different spelling of the same algorithm shuffle() uses, but wasn't _intended_ for sorting. Using sample() for this is "a trick", and I wouldn't use it either. But I would use: def shuffled(xs): xs = list(xs) random.shuffle(xs) return xs I don't care that there's an implicit dependence on a shared Random instance, and neither will 99+% of other users. The few who bother with creating their own Random instances can write their own `shuffled()` to use them instead. Assuming the intersection of two small sets isn't empty ;-) From mertz at gnosis.cx Thu Sep 8 13:32:22 2016 From: mertz at gnosis.cx (David Mertz) Date: Thu, 8 Sep 2016 12:32:22 -0500 Subject: [Python-ideas] Shuffled In-Reply-To: References: Message-ID: I also can't really get from "Arek has a need for this one line definition" to "it should be in the standard library". Sure, 'shuffled()' is a more obvious spelling than the slightly odd lambda. But I've yet to see the reason that alias can't just be defined at the top of the test suite. On Sep 8, 2016 3:14 AM, "Danilo J. S. Bellini" wrote: > That wont work because I would have to type the expression that is used as >> argument twice in a test. > > (lambda data: random.sample(data, len(data)))(container) > > That lambda is actually your "shuffled"... > > 2016-09-08 6:34 GMT-03:00 Arek Bulski : > >> > So, why can't you call random.shuffle(all_tests) if you want to run >> your tests in random order? >> >> I dont randomize test order. People should stop assuming that they know >> better. I need to randomize some arguments for one particular test and I >> cannot call shuffle between tests. Its a continous list of declarative >> tests. Needs to be shuffled(). >> >> https://github.com/construct/construct/blob/master/tests/test_all.py >> See? No way to put imperative code between tests. >> >> > sample(container, len(container)) >> >> That wont work because I would have to type the expression that is used >> as argument twice in a test. I need shuffled. Enough said. >> >> > I'll reiterate that I don't have a use case for this myself... >> >> I dont have a use case for half of what the std library offers. Or for >> type annotations. Asynchronous comprehesions, what is that? Do you see me >> rejecting those? >> >> > (sample having default of entire list size) >> >> That would work but would not be pretty. shuffled() is self explanatory >> and has a nice ring to it. Randomized list is not a sample by definition. >> >> >> pozdrawiam, >> Arkadiusz Bulski >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Danilo J. S. Bellini > --------------- > "*It is not our business to set up prohibitions, but to arrive at > conventions.*" (R. Carnap) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Sep 8 13:45:45 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 9 Sep 2016 03:45:45 +1000 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: <20160908174545.GE22471@ando.pearwood.info> On Wed, Sep 07, 2016 at 07:28:30PM +1000, Hugh Fisher wrote: > There's been discussion here and on python-dev regarding PEP 526 that > assumes there will be multiple type checkers for Python. I really > can't see this happening. As Ryan has already pointed out, that has already happened. > If typing annotations become common or maybe > considered best practice, the Zen of Python "there should be one-- and > preferably only one --obvious way to do it" will take effect. Like there is only one linter? PyFlakes, PyChecker, Jedi, PyLint... which is the One Obvious Way? But it doesn't really matter if the Python ecosystem eventually converges on one single linter or one single type checker. That's just a matter of market share. If there's one linter that is just so good that it completely dominates the community, well, that's fine. What matters the language allows the existence of multiple checkers competing on quality and features. > Firstly, the interpreter will need to have type checking built in. "Need" is very strong. I dare say that *eventually* some Python interpreters will have *some* soft of type checking built-in. Possibly even CPython itself. But it will always be an optional, off-by-default, feature, because if it is mandatory, the language you are using won't be Python, it will be some more-or-less close subset of Python. (Like StrongTalk compared to SmallTalk, or TrueScript compared to Javascript.) Fundamentally, Python will always be a dynamically-typed language, not a statically-typed one. Guido has been clear about that ever since he first proposed the idea of optional static typing more than a decade ago. Nothing has changed. http://www.artima.com/weblogs/viewpost.jsp?thread=85551 > Just about every intro book and tutorial for Python says how great it > is that you don't have an edit-save-compile cycle, just fire up the > Python interpreter and start typing. Right. And that won't change. > Having to run a separate type > checker will be considered as ridiculous as a C compiler that didn't > run the preprocessor itself. > > Secondly, PyPI will collapse if there isn't just one. How can we > express dependencies between packages that use different type > checkers? When type checkers themselves have versions? When a dev team > uses one type checker for 1.x and then switches to another for 2.x? > That's a special circle of hell. I fear that you haven't grasped the fundamental difference between gradual static typing of a dynamically-typed language like Python, and non-gradual typing of statically-typed languages like C, Haskell, Java, etc. Your statement seems like a reasonable fear if you think of static typing as a mandatory pre-compilation step which prevents the code from compiling or running if it doesn't pass. But it makes no sense in the context of an optional code analysis step which warns of things which may lead to runtime exceptions. I think that a good analogy here is that type checkers in the Python ecosystem are equivalent to static analysis tools like Coverity and BLAST in the C ecosystem. In the C ecosystem, it doesn't make sense to ask how you can express dependencies between packages that have been checked by Coverity versus those that have been checked by BLAST. The very question is irrelevant, and there's no reason not to check your package by *both*. Or neither. Or something else instead. -- Steve From steve at pearwood.info Thu Sep 8 14:08:45 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 9 Sep 2016 04:08:45 +1000 Subject: [Python-ideas] Typecheckers: there can be only one In-Reply-To: References: Message-ID: <20160908180844.GF22471@ando.pearwood.info> On Thu, Sep 08, 2016 at 07:46:41PM +1000, Hugh Fisher wrote: > The interpreter certainly thinks they're code rather than comments or > docstrings, even before PEP 526. I type this into my Python 3.4 > interpreter: > > >>> def foo(x:itn): > >>> ... return x > > and the interpreter raises a NameError because 'itn' is not defined. Right. Python has supported annotations since 3.0. Folks have been able to confuse themselves by "declaring" (annotating) the types of function parameters for almost a decade, and yet your fears of newbies being confused by the lack of type checks by default has not eventuated. But I'll grant you that you are partially right: I completely expect that some people will (through inexperience or ignorance) write x: dict = 1 and be surprised that Python doesn't flag it as an error. That's a matter for education. Beginners to Python are confused by many things, such as the (non)fact that "ints are passed by value and lists by reference". I daresay that some people will point to this as a Python "WAT?" moment, just as they already point to the fact that Python doesn't flag x = 1 + "foo" as an error until runtime as a WAT moment. Okay, fine. It is kinda funny that the Python compiler can't tell that numbers and strings can't be added together even when they are literals. Currently we can write: x = 1 assert isinstance(x, dict) and the interpreter will allow it. The only difference is that assertions are on by default and annotations are off by default. (To be clear: annotations will set the special __annotations__ variable, but by default the annotation is note checked for correctness.) Assertions can be used as checked comments in Python, and annotations can be seen as similar: they are *checked type comments* for third-party tools. And, just maybe, Python 5000 or thereabouts will mandate that every Python interpreter include a built-in type checker capable of at least flagging errors like `x: dict = 1`. But it will be off by default. > Annotations look like code, they're mixed in with names and operators > and literals and keywords, and all the standard syntax and semantic > checks are applied. I agree -- annotations are code. -- Steve From arek.bulski at gmail.com Thu Sep 8 14:35:14 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Thu, 8 Sep 2016 20:35:14 +0200 Subject: [Python-ideas] Shuffled Message-ID: Hey, I just did put my thoughts on the discussion floor. Dont yell at me for it not being the most interesting idea ever. There were few interesting responses after all. Lets move on. pozdrawiam, Arkadiusz Bulski -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Sep 8 17:28:22 2016 From: brett at python.org (Brett Cannon) Date: Thu, 08 Sep 2016 21:28:22 +0000 Subject: [Python-ideas] Three ways of paths canonization In-Reply-To: References: <22480.10376.844689.468617@turnbull.sk.tsukuba.ac.jp> Message-ID: On Thu, 8 Sep 2016 at 09:46 Guido van Rossum wrote: > I would prefer it if Path.resolve() resolved symlinks until it hits > something that doesn't exist and then just keep the rest of the path > unchanged. I think this is the equivalent of -m in the mentioned > utility (which I had never heard of). > > It looks like os.path.realpath() already works this way. > > Steve Dower just mentioned to me that the Windows version of > Path.resolve() uses a Windows API that opens the file and then asks > the system for the pathname -- that doesn't work if the file doesn't > exist, so it should be fixed to back off and try the same thing on the > parent, etc. > > We should treat these things as bugs to fix before 3.6 (in 3.6b1 if > possible) and make pathlib non-provisional as of 3.6.0.Probably these > things should be fixed in 3.5.3 as well, since pathlib is still > provisional there. > http://bugs.python.org/issue28031 to track this. -Brett > > --Guido > > On Wed, Sep 7, 2016 at 7:47 AM, Stephen J. Turnbull > wrote: > > Serhiy Storchaka writes: > > > > > The readlink utility from GNU coreutils has three mode for resolving > > > file path: > > > > > > -f, --canonicalize > > > canonicalize by following every symlink in every > > > component of the given name recursively; all but the last component > must > > > exist > > > > > > -e, --canonicalize-existing > > > canonicalize by following every symlink in every > > > component of the given name recursively, all components must exist > > > > > > -m, --canonicalize-missing > > > canonicalize by following every symlink in every > > > component of the given name recursively, without requirements on > > > components existence > > > > In Mac OS X (and I suppose other BSDs), realpath(3) implements -e. > > glibc does none of these, instead: > > > > GNU extensions > > If the call fails with either EACCES or ENOENT and > > resolved_path is not NULL, then the prefix of path that is not > > readable or does not exist is returned in resolved_path. > > > > I suppose this nonstandard behavior is controlled by a #define, but > > the Linux manpage doesn't specify it. > > > > > Current behavior of posixpath.realpath() is matches (besides one minor > > > detail) to `readlink -m`. The behavior of Path.resolve() matches > > > `readlink -e`. > > > > This looks like a bug in posixpath, while Path.resolve follows POSIX. > > http://pubs.opengroup.org/onlinepubs/009695399/functions/realpath.html > > sez: > > > > RETURN VALUE > > > > Upon successful completion, realpath() shall return a pointer to > > the resolved name. Otherwise, realpath() shall return a null > > pointer and set errno to indicate the error, and the contents of > > the buffer pointed to by resolved_name are undefined. > > > > ERRORS > > > > The realpath() function shall fail if: > > > > [...] > > [ENOENT] A component of file_name does not name an existing file or > > file_name points to an empty string. > > [ENOTDIR] A component of the path prefix is not a directory. > > > > which corresponds to -e. > > > > > I have proposed a patch that adds three-state optional parameter to > > > posixpath.realpath() and I'm going to provide similar patch for > > > Path.resolve(). But I'm not sure this is good API. Are there better > > > variants? > > > > Said parameter will almost always be a constant. Usually in those > > cases Python prefers to use different functions. Eg, > > > > posixpath.realpath -e > > posixpath.realpath_require_prefix -f > > posixpath.realpath_allow_missing -m > > posixpath.realpath_gnuext GNU extension > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at realpath.org Thu Sep 8 18:44:19 2016 From: sebastian at realpath.org (Sebastian Krause) Date: Fri, 09 Sep 2016 00:44:19 +0200 Subject: [Python-ideas] kw to be ordered dict In-Reply-To: (Arek Bulski's message of "Sun, 4 Sep 2016 22:06:44 +0200") References: Message-ID: Arek Bulski wrote: > I expected the **kw to reproduce the same order as actual keyword > arguments. The only right way to solve it, as it seems to me, would be to > make **kw an OrderedDict. FYI: A change that keeps the order of **kw was just committed, so 3.6 will have that behavior: https://mail.python.org/pipermail/python-dev/2016-September/146327.html Sebastian From xavier.combelle at gmail.com Thu Sep 8 19:36:00 2016 From: xavier.combelle at gmail.com (Xavier Combelle) Date: Fri, 9 Sep 2016 01:36:00 +0200 Subject: [Python-ideas] shuffled as a way to shuffle an iterable Message-ID: When thinking about the shuffled thread, it occurred to me that it was quite easy to pass an iterable and expect the iterable to be shuffled. but two mentioned implementation are close to success but fail by not taking this use case in account: def shuffled1(iterable): result = iterable[:] random.shuffle(result) return result This one might fail because an iterable don't have subscript, but in the case of range() it has. In this case it fail at the affectation place def shuffled2(iterable): random.sample(iterable,len(iterable)) This one perfectly work on range() but fail in a less specific iterable without len like shuffled2((i for i in range(5))) and eventually this one work in all case of iterable def shuffled3(iterable): result = list(iterable) random.shuffle(result) return result I don't think that the use case of shuffle an iterable is important enough to create a function just to take care about it but if one want to implement it in an analog way sorted is implemented this should be take in account. From rosuav at gmail.com Thu Sep 8 19:40:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 9 Sep 2016 09:40:56 +1000 Subject: [Python-ideas] shuffled as a way to shuffle an iterable In-Reply-To: References: Message-ID: On Fri, Sep 9, 2016 at 9:36 AM, Xavier Combelle wrote: > and eventually this one work in all case of iterable > > def shuffled3(iterable): > result = list(iterable) > random.shuffle(result) > return result > This is the one obvious way to do this. Not all iterables can logically be shuffled, so the most normal approach would be to turn it into a list, then shuffle. Which is exactly what this does. ChrisA From damien.p.george at gmail.com Fri Sep 9 00:35:23 2016 From: damien.p.george at gmail.com (Damien George) Date: Fri, 9 Sep 2016 14:35:23 +1000 Subject: [Python-ideas] Changing optimisation level from a script Message-ID: Hi all, When starting CPython from the command line you can pass the -O option to enable optimisations (eg `assert 0` won't raise an exception when -O is passed). But, AFAIK, there is no way to change the optimisation level after the interpreter has started up, ie there is no Python function call or variable that can change the optimisation. In MicroPython we want to be able to change the optimisation level from within a script because (on bare metal at least) there is no analog of passing options like -O. My idea would be to have a function like `sys.optimise(value)` that sets the optimisation level for all subsequent runs of the parser/compiler. For example: import sys import mymodule # no optimisations exec('assert 0') # will raise an exception sys.optimise(1) # enable optimisations import myothermodule # optimisations are enabled for this (unless it's already imported by mymodule) exec('assert 0') # will not raise an exception What do you think? Sorry if this has been discussed before! Cheers, Damien. From hugo.fisher at gmail.com Fri Sep 9 05:50:07 2016 From: hugo.fisher at gmail.com (Hugh Fisher) Date: Fri, 9 Sep 2016 19:50:07 +1000 Subject: [Python-ideas] Typecheckers: there can be only one Message-ID: > From: Steven D'Aprano > > I fear that you haven't grasped the fundamental difference between > gradual static typing of a dynamically-typed language like Python, and > non-gradual typing of statically-typed languages like C, Haskell, Java, > etc. Your statement seems like a reasonable fear if you think of static > typing as a mandatory pre-compilation step which prevents the code from > compiling or running if it doesn't pass. But it makes no sense in the > context of an optional code analysis step which warns of things which > may lead to runtime exceptions. > > I think that a good analogy here is that type checkers in the Python > ecosystem are equivalent to static analysis tools like Coverity and > BLAST in the C ecosystem. I guess time will show which of our analogies and expectations work out. I've made my points, other people have made theirs, so I'll shut up now. Thanks to everyone who responded. -- cheers, Hugh Fisher From yan12125 at gmail.com Fri Sep 9 06:23:48 2016 From: yan12125 at gmail.com (Chi Hsuan Yen) Date: Fri, 9 Sep 2016 18:23:48 +0800 Subject: [Python-ideas] Expose reasons for SSL/TLS cert verification failures Message-ID: Hi Python enthusiasts, Currently _ssl.c always reports CERTIFICATE_VERIFY_FAILED for any certification verification errors. In OpenSSL, it's possible to tell from different reasons that lead to CERTIFICATE_VERIFY_FAILED. For example, https://expired.badssl.com/ reports X509_V_ERR_CERT_HAS_EXPIRED, and https://self-signed.badssl.com/ reports X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT. Seems CPython does not expose such information yet? I hope it can be added to CPython. For example, creating a new exception class SSLCertificateError, which is a subclass of SSLError, that provides error codes like X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT. Any ideas? The attachment is a naive try to printf some information about a verification failure. It's just a proof-of-concept and does not provide any practical advantage :) Best, Yen Chi Hsuan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2770,16 +2770,24 @@ get_verify_mode(PySSLContext *self, void case SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT: return PyLong_FromLong(PY_SSL_CERT_REQUIRED); } PyErr_SetString(PySSLErrorObject, "invalid return value from SSL_CTX_get_verify_mode"); return NULL; } +static int verify_cb(int ok, X509_STORE_CTX *ctx) +{ + long x509_err = X509_STORE_CTX_get_error(ctx); + printf("ok = %d, err = %ld, errstr = %s\n", ok, x509_err, + X509_verify_cert_error_string(x509_err)); + return ok; +} + static int set_verify_mode(PySSLContext *self, PyObject *arg, void *c) { int n, mode; if (!PyArg_Parse(arg, "i", &n)) return -1; if (n == PY_SSL_CERT_NONE) mode = SSL_VERIFY_NONE; @@ -2794,16 +2802,22 @@ set_verify_mode(PySSLContext *self, PyOb } if (mode == SSL_VERIFY_NONE && self->check_hostname) { PyErr_SetString(PyExc_ValueError, "Cannot set verify_mode to CERT_NONE when " "check_hostname is enabled."); return -1; } SSL_CTX_set_verify(self->ctx, mode, NULL); + + { + X509_STORE *store = SSL_CTX_get_cert_store(self->ctx); + X509_STORE_set_verify_cb(store, verify_cb); + } + return 0; } static PyObject * get_verify_flags(PySSLContext *self, void *c) { X509_STORE *store; X509_VERIFY_PARAM *param; From christian at python.org Fri Sep 9 13:16:13 2016 From: christian at python.org (Christian Heimes) Date: Fri, 9 Sep 2016 19:16:13 +0200 Subject: [Python-ideas] Expose reasons for SSL/TLS cert verification failures In-Reply-To: References: Message-ID: On 2016-09-09 12:23, Chi Hsuan Yen wrote: > Hi Python enthusiasts, > > Currently _ssl.c always reports CERTIFICATE_VERIFY_FAILED for any > certification verification errors. In OpenSSL, it's possible to tell > from different reasons that lead to CERTIFICATE_VERIFY_FAILED. For > example, https://expired.badssl.com/ reports > X509_V_ERR_CERT_HAS_EXPIRED, and https://self-signed.badssl.com/ reports > X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT. Seems CPython does not expose > such information yet? I hope it can be added to CPython. For example, > creating a new exception class SSLCertificateError, which is a subclass > of SSLError, that provides error codes like > X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT. Any ideas? > > The attachment is a naive try to printf some information about a > verification failure. It's just a proof-of-concept and does not provide > any practical advantage :) I'm planning to add a proper validation hook to 3.7. I haven't had time to design and implement it for 3.6. Christian From guido at python.org Fri Sep 9 13:22:02 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Sep 2016 10:22:02 -0700 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: What is to stop you from adding this to micropython as a library extension? I've never felt the urge to do this and I don't think I've ever heard it requested before. If you think it should be in the stdlib, given the timing of the 3.6 feature freeze the earliest time it could land would be Python 3.7. Finally, let's not define APIs with British spelling. :-) On Thu, Sep 8, 2016 at 9:35 PM, Damien George wrote: > Hi all, > > When starting CPython from the command line you can pass the -O option > to enable optimisations (eg `assert 0` won't raise an exception when > -O is passed). But, AFAIK, there is no way to change the optimisation > level after the interpreter has started up, ie there is no Python > function call or variable that can change the optimisation. > > In MicroPython we want to be able to change the optimisation level > from within a script because (on bare metal at least) there is no > analog of passing options like -O. > > My idea would be to have a function like `sys.optimise(value)` that > sets the optimisation level for all subsequent runs of the > parser/compiler. For example: > > import sys > import mymodule # no optimisations > exec('assert 0') # will raise an exception > sys.optimise(1) # enable optimisations > import myothermodule # optimisations are enabled for this (unless it's > already imported by mymodule) > exec('assert 0') # will not raise an exception > > What do you think? Sorry if this has been discussed before! > > Cheers, > Damien. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From yan12125 at gmail.com Fri Sep 9 13:29:35 2016 From: yan12125 at gmail.com (Chi Hsuan Yen) Date: Sat, 10 Sep 2016 01:29:35 +0800 Subject: [Python-ideas] Expose reasons for SSL/TLS cert verification failures In-Reply-To: References: Message-ID: On Sat, Sep 10, 2016 at 1:16 AM, Christian Heimes wrote: > On 2016-09-09 12:23, Chi Hsuan Yen wrote: > > Hi Python enthusiasts, > > > > Currently _ssl.c always reports CERTIFICATE_VERIFY_FAILED for any > > certification verification errors. In OpenSSL, it's possible to tell > > from different reasons that lead to CERTIFICATE_VERIFY_FAILED. For > > example, https://expired.badssl.com/ reports > > X509_V_ERR_CERT_HAS_EXPIRED, and https://self-signed.badssl.com/ reports > > X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT. Seems CPython does not expose > > such information yet? I hope it can be added to CPython. For example, > > creating a new exception class SSLCertificateError, which is a subclass > > of SSLError, that provides error codes like > > X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT. Any ideas? > > > > The attachment is a naive try to printf some information about a > > verification failure. It's just a proof-of-concept and does not provide > > any practical advantage :) > > I'm planning to add a proper validation hook to 3.7. I haven't had time > to design and implement it for 3.6. > > Christian > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Thanks for the clarification. I know there are only a few hours before 3.6 feature freeze :) Is there already a bug? If not I can help creating one and paste related materials for easier tracking. Best, Yen Chi Hsuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Sep 9 13:20:29 2016 From: brett at python.org (Brett Cannon) Date: Fri, 09 Sep 2016 17:20:29 +0000 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: On Thu, 8 Sep 2016 at 21:36 Damien George wrote: > Hi all, > > When starting CPython from the command line you can pass the -O option > to enable optimisations (eg `assert 0` won't raise an exception when > -O is passed). But, AFAIK, there is no way to change the optimisation > level after the interpreter has started up, ie there is no Python > function call or variable that can change the optimisation. > > In MicroPython we want to be able to change the optimisation level > from within a script because (on bare metal at least) there is no > analog of passing options like -O. > > My idea would be to have a function like `sys.optimise(value)` that > sets the optimisation level for all subsequent runs of the > parser/compiler. For example: > > import sys > import mymodule # no optimisations > exec('assert 0') # will raise an exception > sys.optimise(1) # enable optimisations > import myothermodule # optimisations are enabled for this (unless it's > already imported by mymodule) > exec('assert 0') # will not raise an exception > > What do you think? Sorry if this has been discussed before! > I don't know if it's been discussed, but I have thought about it in context of PEP 511. The problem with swapping optimization levels post-start is that you end up with inconsistencies, e.g. asserts that depend on other asserts/__debug__ to function properly. If you let people jump around you potentially will break code in odd ways. Now obviously that's not necessarily a reason to not allow it, but it is something to consider. Where this does become a potential issue in the future is if we ever start to have optimizations that span modules, e.g. function inlining and the such. We don't have support for this now, but if we ever make it easier to do such things then the ability to change the optimization level mid-execution would break assumptions or flat-out ban cross-module optimizations in fear that too much code would break. So I'm not flat-out saying no to this idea, but there are some things to consider first. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Sep 9 15:02:19 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Sep 2016 12:02:19 -0700 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: I very much doubt that one assert might depend on another, and if they do, we can just tell people not to change the debug level. The API to change this should set __debug__ appropriately. On Fri, Sep 9, 2016 at 10:20 AM, Brett Cannon wrote: > > > On Thu, 8 Sep 2016 at 21:36 Damien George wrote: >> >> Hi all, >> >> When starting CPython from the command line you can pass the -O option >> to enable optimisations (eg `assert 0` won't raise an exception when >> -O is passed). But, AFAIK, there is no way to change the optimisation >> level after the interpreter has started up, ie there is no Python >> function call or variable that can change the optimisation. >> >> In MicroPython we want to be able to change the optimisation level >> from within a script because (on bare metal at least) there is no >> analog of passing options like -O. >> >> My idea would be to have a function like `sys.optimise(value)` that >> sets the optimisation level for all subsequent runs of the >> parser/compiler. For example: >> >> import sys >> import mymodule # no optimisations >> exec('assert 0') # will raise an exception >> sys.optimise(1) # enable optimisations >> import myothermodule # optimisations are enabled for this (unless it's >> already imported by mymodule) >> exec('assert 0') # will not raise an exception >> >> What do you think? Sorry if this has been discussed before! > > > I don't know if it's been discussed, but I have thought about it in context > of PEP 511. The problem with swapping optimization levels post-start is that > you end up with inconsistencies, e.g. asserts that depend on other > asserts/__debug__ to function properly. If you let people jump around you > potentially will break code in odd ways. Now obviously that's not > necessarily a reason to not allow it, but it is something to consider. > > Where this does become a potential issue in the future is if we ever start > to have optimizations that span modules, e.g. function inlining and the > such. We don't have support for this now, but if we ever make it easier to > do such things then the ability to change the optimization level > mid-execution would break assumptions or flat-out ban cross-module > optimizations in fear that too much code would break. > > So I'm not flat-out saying no to this idea, but there are some things to > consider first. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From arek.bulski at gmail.com Fri Sep 9 16:01:44 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Fri, 9 Sep 2016 22:01:44 +0200 Subject: [Python-ideas] Null coalescing operator Message-ID: Sometimes I find myself in need of this nice operator that I used back in the days when I was programming in .NET, essentially an expression >>> expr ?? instead should return expr when it `is not None` and `instead` otherwise. A piece of code that I just wrote, you can see a use case: def _sizeof(self, context): if self.totalsizeof is not None: return self.totalsizeof else: raise SizeofError("cannot calculate size") With the oprator it would just be def _sizeof(self, context): return self.totalsizeof ?? raise SizeofError("cannot calculate size") pozdrawiam, Arkadiusz Bulski -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Sep 9 16:06:27 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Sep 2016 06:06:27 +1000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: Message-ID: On Sat, Sep 10, 2016 at 6:01 AM, Arek Bulski wrote: > Sometimes I find myself in need of this nice operator that I used back in > the days when I was programming in .NET, essentially an expression > >>>> expr ?? instead > > should return expr when it `is not None` and `instead` otherwise. You can use 'or' for this, as long as you're okay with other falsey values being treated the same way. In a lot of cases, this isn't a problem. However, even if this is implemented, it would be in an expression context, so 'raise' would never work. For that, I'd just use the explicit statement form. ChrisA From zachary.ware+pyideas at gmail.com Fri Sep 9 16:06:37 2016 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Fri, 9 Sep 2016 15:06:37 -0500 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: Message-ID: On Fri, Sep 9, 2016 at 3:01 PM, Arek Bulski wrote: > Sometimes I find myself in need of this nice operator that I used back in > the days when I was programming in .NET, essentially an expression > >>>> expr ?? instead > > should return expr when it `is not None` and `instead` otherwise. > > A piece of code that I just wrote, you can see a use case: > > def _sizeof(self, context): > if self.totalsizeof is not None: > return self.totalsizeof > else: > raise SizeofError("cannot calculate size") > > With the oprator it would just be > > def _sizeof(self, context): > return self.totalsizeof ?? raise SizeofError("cannot calculate > size") This was proposed almost exactly a year ago, start reading here: https://mail.python.org/pipermail/python-ideas/2015-September/036289.html -- Zach From mertz at gnosis.cx Fri Sep 9 16:10:03 2016 From: mertz at gnosis.cx (David Mertz) Date: Fri, 9 Sep 2016 13:10:03 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: Message-ID: This idea has come up before. While I can see the use of it, to me at least that use doesn't feel nearly common enough to warrant dedicated syntax. In many cases, it is a "truthy" value you are looking for rather than `is not None` specifically. That has a convenient spelling: expr or instead If it really is the actual None-ness you are curious about, you need the slightly longer: expr if expr is not None else instead Your example seems to want to fall back to a statement suite rather than a value. To do that, you'd have to put the suite inside a function such as: def Raise(err): raise err And use it something like: self.totalsizeof or Raise(SizeofError(...)) On Fri, Sep 9, 2016 at 1:01 PM, Arek Bulski wrote: > Sometimes I find myself in need of this nice operator that I used back in > the days when I was programming in .NET, essentially an expression > > >>> expr ?? instead > > should return expr when it `is not None` and `instead` otherwise. > > A piece of code that I just wrote, you can see a use case: > > def _sizeof(self, context): > if self.totalsizeof is not None: > return self.totalsizeof > else: > raise SizeofError("cannot calculate size") > > With the oprator it would just be > > def _sizeof(self, context): > return self.totalsizeof ?? raise SizeofError("cannot calculate > size") > > > > pozdrawiam, > Arkadiusz Bulski > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Sep 9 16:13:51 2016 From: mertz at gnosis.cx (David Mertz) Date: Fri, 9 Sep 2016 13:13:51 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: Message-ID: I'd note you can also save 4 characters by writing: instead if expr is None else expr On Fri, Sep 9, 2016 at 1:10 PM, David Mertz wrote: > This idea has come up before. While I can see the use of it, to me at > least that use doesn't feel nearly common enough to warrant dedicated > syntax. > > In many cases, it is a "truthy" value you are looking for rather than `is > not None` specifically. That has a convenient spelling: > > expr or instead > > > If it really is the actual None-ness you are curious about, you need the > slightly longer: > > expr if expr is not None else instead > > > Your example seems to want to fall back to a statement suite rather than a > value. To do that, you'd have to put the suite inside a function such as: > > def Raise(err): > > raise err > > > And use it something like: > > self.totalsizeof or Raise(SizeofError(...)) > > > On Fri, Sep 9, 2016 at 1:01 PM, Arek Bulski wrote: > >> Sometimes I find myself in need of this nice operator that I used back in >> the days when I was programming in .NET, essentially an expression >> >> >>> expr ?? instead >> >> should return expr when it `is not None` and `instead` otherwise. >> >> A piece of code that I just wrote, you can see a use case: >> >> def _sizeof(self, context): >> if self.totalsizeof is not None: >> return self.totalsizeof >> else: >> raise SizeofError("cannot calculate size") >> >> With the oprator it would just be >> >> def _sizeof(self, context): >> return self.totalsizeof ?? raise SizeofError("cannot calculate >> size") >> >> >> >> pozdrawiam, >> Arkadiusz Bulski >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Sep 9 16:21:16 2016 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 9 Sep 2016 21:21:16 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: Message-ID: <9b317570-ea07-f63e-7c2f-9c5b22bfb5c3@mrabarnett.plus.com> On 2016-09-09 21:01, Arek Bulski wrote: > Sometimes I find myself in need of this nice operator that I used back > in the days when I was programming in .NET, essentially an expression > >>>> expr ?? instead > > should return expr when it `is not None` and `instead` otherwise. > > A piece of code that I just wrote, you can see a use case: > > def _sizeof(self, context): > if self.totalsizeof is not None: > return self.totalsizeof > else: > raise SizeofError("cannot calculate size") > > With the oprator it would just be > > def _sizeof(self, context): > return self.totalsizeof ?? raise SizeofError("cannot calculate > size") 'raise' is a statement, so it can't appear in an expression. This has been discussed before, so you might want to read this thread first: Null coalescing operators https://mail.python.org/pipermail/python-ideas/2015-September/036289.html From damien.p.george at gmail.com Fri Sep 9 20:04:46 2016 From: damien.p.george at gmail.com (Damien George) Date: Sat, 10 Sep 2016 10:04:46 +1000 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: > What is to stop you from adding this to micropython as a library > extension? That's what I would like to do, and usually we do just go ahead and add our own extensions, trying to be Pythonic as possible :) But we do that a lot and sometimes I think it would be good to discuss with upstream (ie python-dev/python-ideas) about adding new functions/classes so that MicroPython doesn't diverge too much (eg Paul Sokolovsky had a discussion about time.sleep_ms, time.sleep_us etc, which was fruitful). sys.optimize(value) is a pretty simple addition so I thought it would be a straight forward discussion to have here. I guess my main question to this list is: if CPython were to add a function to change the optimisation level at runtime, what would it look like? I don't want to push CPython to actually add such a thing if it's not seen as a useful addition. Instead I want to see how others would implement it if they needed to. A nice outcome would be that MicroPython adds the function now (eg sys.optimize), and in the future if CPython finds that it needs it, it also adds the same function with the same name/module/signature. Hence why I would like to add it correctly into MicroPython from the very beginning. Alternatively we can just use the micropython module and add micropython.optimize(value), and then if CPython does add it later on we'll have to change how we do it. > I've never felt the urge to do this and I don't think I've > ever heard it requested before. If you think it should be in the > stdlib, given the timing of the 3.6 feature freeze the earliest time > it could land would be Python 3.7. No, I definitely don't want to make a rush for 3.6. I guess the timing of this post was not the best considering the 3.6 release :) > Finally, let's not define APIs with > British spelling. :-) Haha, the spelling never even crossed my mind! Well, sys.opt() is definitely too short and confusing, so "optimize" would be the choice. Cheers, Damien. From python at mrabarnett.plus.com Fri Sep 9 20:24:04 2016 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 10 Sep 2016 01:24:04 +0100 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: On 2016-09-10 01:04, Damien George wrote: >> What is to stop you from adding this to micropython as a library >> extension? > > That's what I would like to do, and usually we do just go ahead and > add our own extensions, trying to be Pythonic as possible :) > > But we do that a lot and sometimes I think it would be good to discuss > with upstream (ie python-dev/python-ideas) about adding new > functions/classes so that MicroPython doesn't diverge too much (eg > Paul Sokolovsky had a discussion about time.sleep_ms, time.sleep_us > etc, which was fruitful). sys.optimize(value) is a pretty simple > addition so I thought it would be a straight forward discussion to > have here. > > I guess my main question to this list is: if CPython were to add a > function to change the optimisation level at runtime, what would it > look like? I don't want to push CPython to actually add such a thing > if it's not seen as a useful addition. Instead I want to see how > others would implement it if they needed to. > > A nice outcome would be that MicroPython adds the function now (eg > sys.optimize), and in the future if CPython finds that it needs it, it > also adds the same function with the same name/module/signature. > Hence why I would like to add it correctly into MicroPython from the > very beginning. > > Alternatively we can just use the micropython module and add > micropython.optimize(value), and then if CPython does add it later on > we'll have to change how we do it. > If I saw a function called 'optimize' which took an argument, I might initially think that it returns an optimised form of the argument, except that it's being called like a procedure, so it must be setting something. Then there's the question of how to get the current optimisation level. >> I've never felt the urge to do this and I don't think I've >> ever heard it requested before. If you think it should be in the >> stdlib, given the timing of the 3.6 feature freeze the earliest time >> it could land would be Python 3.7. > > No, I definitely don't want to make a rush for 3.6. I guess the > timing of this post was not the best considering the 3.6 release :) > >> Finally, let's not define APIs with >> British spelling. :-) > > Haha, the spelling never even crossed my mind! Well, sys.opt() is > definitely too short and confusing, so "optimize" would be the choice. > Perhaps: sys.opt_level = 1 From steve at pearwood.info Fri Sep 9 20:27:19 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 10 Sep 2016 10:27:19 +1000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: Message-ID: <20160910002719.GG22471@ando.pearwood.info> On Fri, Sep 09, 2016 at 10:01:44PM +0200, Arek Bulski wrote: > Sometimes I find myself in need of this nice operator that I used back in > the days when I was programming in .NET, essentially an expression > > >>> expr ?? instead > > should return expr when it `is not None` and `instead` otherwise. As Zach and MRAB mention, this was discussed last year. If I recall correctly, the discussion fizzled out without a solid conclusion. I think there's a PEP -- if not, there should be. I would be interested in revisiting this idea, but 3.6 feature freeze is only a day or two away and I won't have time to discuss this before then. So let's please drop this discussion until the 3.6 beta is released. -- Steve From steve at pearwood.info Fri Sep 9 21:01:26 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 10 Sep 2016 11:01:26 +1000 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: <20160910010125.GH22471@ando.pearwood.info> On Sat, Sep 10, 2016 at 10:04:46AM +1000, Damien George wrote: > I guess my main question to this list is: if CPython were to add a > function to change the optimisation level at runtime, what would it > look like? I don't think it would look like sys.optimize(flag). At the very least, it would have to take *at least* three values: - no optimizations (the default) - remove asserts (-O) - remove asserts and docstrings (-OO) but even that is very limiting. Why can't I keep asserts, which may be useful and important in running code, but remove docstrings, which mostly are used interactively? I think the idea of "optimization levels" has reached its use-by date and we should start considering separate flags for each individual optimization. In particular, I expect that some time soon somebody will propose at least one more optimization: - remove annotations and I seem to recall somebody requesting the ability to turn off constant folding, although I don't remember why. I can easily imagine Python implementations offering flags to control more complex/risky optimizations: - resolve built-in names at compile-time[1] - inline small functions - unroll loops etc. Victor Stinner is (I think) working on bringing some of these to CPython, and it seems to me that if they are to be under any user-control at all, a simple optimization level counter is not going to be sufficient. We're going to need a set of flags. But it seems to me that the ability to change optimizations at runtime would be rather confusing. Shouldn't optimizations, or at least some of them, apply on a per-module basis rather than globally? I don't even know how to reason about code optimizations if any function I run might have changed the optimization level without my knowledge. So I think this is a very dubious idea. [1] Yes, I know that will change the semantics of Python code. -- Steve From andrew.svetlov at gmail.com Sat Sep 10 02:56:29 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 10 Sep 2016 06:56:29 +0000 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: <20160910010125.GH22471@ando.pearwood.info> References: <20160910010125.GH22471@ando.pearwood.info> Message-ID: There are also peephole optimization. Removing it may prevent false positives for coverage tool http://bugs.python.org/issue2506 On Sat, Sep 10, 2016 at 4:01 AM Steven D'Aprano wrote: > On Sat, Sep 10, 2016 at 10:04:46AM +1000, Damien George wrote: > > > I guess my main question to this list is: if CPython were to add a > > function to change the optimisation level at runtime, what would it > > look like? > > I don't think it would look like sys.optimize(flag). At the very least, > it would have to take *at least* three values: > > - no optimizations (the default) > - remove asserts (-O) > - remove asserts and docstrings (-OO) > > but even that is very limiting. Why can't I keep asserts, which may be > useful and important in running code, but remove docstrings, which > mostly are used interactively? I think the idea of "optimization levels" > has reached its use-by date and we should start considering separate > flags for each individual optimization. > > In particular, I expect that some time soon somebody will propose at > least one more optimization: > > - remove annotations > > and I seem to recall somebody requesting the ability to turn off > constant folding, although I don't remember why. I can easily imagine > Python implementations offering flags to control more complex/risky > optimizations: > > - resolve built-in names at compile-time[1] > - inline small functions > - unroll loops > > etc. Victor Stinner is (I think) working on bringing some of these to > CPython, and it seems to me that if they are to be under any > user-control at all, a simple optimization level counter is not going to > be sufficient. We're going to need a set of flags. > > But it seems to me that the ability to change optimizations at runtime > would be rather confusing. Shouldn't optimizations, or at least some of > them, apply on a per-module basis rather than globally? > > I don't even know how to reason about code optimizations if any function > I run might have changed the optimization level without my knowledge. So > I think this is a very dubious idea. > > > > > > > [1] Yes, I know that will change the semantics of Python code. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Sep 10 03:17:11 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 10 Sep 2016 03:17:11 -0400 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: <20160910010125.GH22471@ando.pearwood.info> Message-ID: 2016-09-10 2:56 GMT-04:00 Andrew Svetlov : > There are also peephole optimization. > Removing it may prevent false positives for coverage tool > http://bugs.python.org/issue2506 My PEP 511 adds "-o noopt" to disable the peephole optimizer ;-) "PEP 511 -- API for code transformers" www.python.org/dev/peps/pep-0511/ Victor From ralph at ralphbroenink.net Sat Sep 10 03:12:22 2016 From: ralph at ralphbroenink.net (Ralph Broenink) Date: Sat, 10 Sep 2016 07:12:22 +0000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20160910002719.GG22471@ando.pearwood.info> References: <20160910002719.GG22471@ando.pearwood.info> Message-ID: It is PEP 505. I agree we should resume the discussion on this PEP though (for 3.7), I'm not completely sure why it stalled. Ralph On Sat, 10 Sep 2016, 02:50 Steven D'Aprano, wrote: > On Fri, Sep 09, 2016 at 10:01:44PM +0200, Arek Bulski wrote: > > Sometimes I find myself in need of this nice operator that I used back in > > the days when I was programming in .NET, essentially an expression > > > > >>> expr ?? instead > > > > should return expr when it `is not None` and `instead` otherwise. > > > As Zach and MRAB mention, this was discussed last year. If I recall > correctly, the discussion fizzled out without a solid conclusion. I > think there's a PEP -- if not, there should be. > > I would be interested in revisiting this idea, but 3.6 feature freeze is > only a day or two away and I won't have time to discuss this before > then. So let's please drop this discussion until the 3.6 beta is > released. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Sat Sep 10 04:23:10 2016 From: encukou at gmail.com (Petr Viktorin) Date: Sat, 10 Sep 2016 10:23:10 +0200 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: On 09/10/2016 02:04 AM, Damien George wrote: >> What is to stop you from adding this to micropython as a library >> extension? > > That's what I would like to do, and usually we do just go ahead and > add our own extensions, trying to be Pythonic as possible :) > > But we do that a lot and sometimes I think it would be good to discuss > with upstream (ie python-dev/python-ideas) about adding new > functions/classes so that MicroPython doesn't diverge too much (eg > Paul Sokolovsky had a discussion about time.sleep_ms, time.sleep_us > etc, which was fruitful). sys.optimize(value) is a pretty simple > addition so I thought it would be a straight forward discussion to > have here. > > I guess my main question to this list is: if CPython were to add a > function to change the optimisation level at runtime, what would it > look like? I don't want to push CPython to actually add such a thing > if it's not seen as a useful addition. Instead I want to see how > others would implement it if they needed to. Hello, The API you proposed here comes is similar to something I see a lot in MicroPython libraries: functions/methods that combine a getter and setter. For example, to set the value on a pin, you do: pin.value(1) and to read, you do: result = pin.value() If an API like this was added to the stdlib, I'd expect it to use a property, e.g. pin.value = 1 result = pin.value I was wondering, what's the story of this aspect of MicroPython API? Does it have hidden advantages? Were you inspired by another library? Or was it just the easiest way to get the functionality (I assume you implemented functions before properties), and then it stuck? From levkivskyi at gmail.com Sat Sep 10 05:49:34 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 10 Sep 2016 11:49:34 +0200 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: <20160910010125.GH22471@ando.pearwood.info> References: <20160910010125.GH22471@ando.pearwood.info> Message-ID: On 10 September 2016 at 03:01, Steven D'Aprano wrote: > In particular, I expect that some time soon somebody will propose at > least one more optimization: > > - remove annotations > > In process of developing the implementation of variable annotations this idea already appeared (I even did some benchmarks) but it was decided to postpone this idea. Maybe we could return to it when we will have more time. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Sep 10 07:53:11 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 10 Sep 2016 21:53:11 +1000 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: On 10 September 2016 at 03:20, Brett Cannon wrote: > I don't know if it's been discussed, but I have thought about it in context > of PEP 511. The problem with swapping optimization levels post-start is that > you end up with inconsistencies, e.g. asserts that depend on other > asserts/__debug__ to function properly. If you let people jump around you > potentially will break code in odd ways. Now obviously that's not > necessarily a reason to not allow it, but it is something to consider. > > Where this does become a potential issue in the future is if we ever start > to have optimizations that span modules, e.g. function inlining and the > such. We don't have support for this now, but if we ever make it easier to > do such things then the ability to change the optimization level > mid-execution would break assumptions or flat-out ban cross-module > optimizations in fear that too much code would break. > > So I'm not flat-out saying no to this idea, but there are some things to > consider first. We technically already have to deal with this problem, since folks can run compile() themselves with "optimize" set to something other than -1. "sys.flags.optimize" then gives the default setting used for "optimize" by the import system, eval, exec, etc. So if we did make this configurable, I'd suggest something along the lines of the other "buyer beware" settings in sys that can easily break the world, like setcheckinterval, setrecursionlimit, setswitchinterval, settrace, setprofile, and (our first PEP 8 compliant addition), set_coroutine_wrapper. Given sys.flags.optimize already exists to read the current setting from Python, we'd just need "sys.set_default_optimize()" to configure it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From srkunze at mail.de Sat Sep 10 08:09:49 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Sat, 10 Sep 2016 14:09:49 +0200 Subject: [Python-ideas] Inconsistencies (was: Shuffled) In-Reply-To: <20160908020050.GA22471@ando.pearwood.info> References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <20160908020050.GA22471@ando.pearwood.info> Message-ID: On 08.09.2016 04:00, Steven D'Aprano wrote: > On Wed, Sep 07, 2016 at 11:43:59PM +0200, Sven R. Kunze wrote: > >> >BUT experienced devs also need to recognize and respect the fact that >> >younger/unexperienced developers are just better in detecting >> >inconsistencies and bloody work-arounds. > That is not a fact. It is the opposite of a fact You might have heard of: "There are no such things as facts, just opinions. Everything, we see is a perspective not the truth." See below, why this applies here as well. > -- inexperienced > developers are WORSE at spotting inconsistencies, because they don't > recognise deep consistencies. Your example (default arguments) might make sense when going one or two level deeper. However: Is going deep really necessary at all? People program for different reasons, to have fun, to create value for others, to educate, or for reasons we both cannot even think of. Why should they leave their level? Because of you or me? Because they need to know what Turing-completeness means? What calling conventions are? I don't think so. They wanna solve problems and get things done whether or not they know every single bit of the language they use. If they decide to go deeper, that's wonderful, but if they don't, don't force them. So, an inconsistency at level 1 might be a **result of a consistency at level 0** BUT it nevertheless is and stays an inconsistency at level 1 no matter how sophisticated the consistency at level 0 is. And please note, some even want to go on level up, so inconsistencies at level 1 just plain suck then. People then simply don't care about how the current flows at cpu level or about your carefully handcrafted bits and pieces on level 0 which you are so proud of and which are so consistent there. Cheers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Sep 10 12:48:59 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 11 Sep 2016 01:48:59 +0900 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20160910002719.GG22471@ando.pearwood.info> References: <20160910002719.GG22471@ando.pearwood.info> Message-ID: <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > As Zach and MRAB mention, this was discussed last year. If I recall > correctly, the discussion fizzled out without a solid conclusion. I > think there's a PEP -- if not, there should be. The problem, as I recall, was that there was a conclusion that 'maybe_None if maybe_None is None else default' and the less precise 'maybe_falsy or default' are sufficient if you just want to provide a default if a value is "null". The proponents pointed out that other operators would benefit from null coalescing, and then discussion petered out because there was no remotely plausible suggestion except to use '?' to indicate null-coalescing versions. But Guido has historically resisted any use of '?' whatsoever in Python syntax, and there weren't any good alternatives or strong proponents of ?-based syntax. I forget if Guido was very sympathetic to null-coalescing operators, given somebody came up with a good syntax. There was also an issue of whether SQL NULLs and similar constructs "should" be spelled None in Python, and if not, how would null coalescence be defined. Steve From brett at python.org Sat Sep 10 12:57:00 2016 From: brett at python.org (Brett Cannon) Date: Sat, 10 Sep 2016 16:57:00 +0000 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: If people are curious as to where import makes its decision as to what bytecode to load based on the optimization level, see https://github.com/python/cpython/blob/ad3b9aeaab5122b22445f9120a6ccdc1987c15cc/Lib/importlib/_bootstrap_external.py#L292 . On Sat, 10 Sep 2016 at 04:53 Nick Coghlan wrote: > On 10 September 2016 at 03:20, Brett Cannon wrote: > > I don't know if it's been discussed, but I have thought about it in > context > > of PEP 511. The problem with swapping optimization levels post-start is > that > > you end up with inconsistencies, e.g. asserts that depend on other > > asserts/__debug__ to function properly. If you let people jump around you > > potentially will break code in odd ways. Now obviously that's not > > necessarily a reason to not allow it, but it is something to consider. > > > > Where this does become a potential issue in the future is if we ever > start > > to have optimizations that span modules, e.g. function inlining and the > > such. We don't have support for this now, but if we ever make it easier > to > > do such things then the ability to change the optimization level > > mid-execution would break assumptions or flat-out ban cross-module > > optimizations in fear that too much code would break. > > > > So I'm not flat-out saying no to this idea, but there are some things to > > consider first. > > We technically already have to deal with this problem, since folks can > run compile() themselves with "optimize" set to something other than > -1. > > "sys.flags.optimize" then gives the default setting used for > "optimize" by the import system, eval, exec, etc. > > So if we did make this configurable, I'd suggest something along the > lines of the other "buyer beware" settings in sys that can easily > break the world, like setcheckinterval, setrecursionlimit, > setswitchinterval, settrace, setprofile, and (our first PEP 8 > compliant addition), set_coroutine_wrapper. > > Given sys.flags.optimize already exists to read the current setting > from Python, we'd just need "sys.set_default_optimize()" to configure > it. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Sat Sep 10 13:14:46 2016 From: random832 at fastmail.com (Random832) Date: Sat, 10 Sep 2016 13:14:46 -0400 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> Message-ID: <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> On Sat, Sep 10, 2016, at 12:48, Stephen J. Turnbull wrote: > I forget if Guido was very sympathetic to null-coalescing operators, > given somebody came up with a good syntax. As I remember the discussion, I thought he'd more or less conceded on the use of ? but there was disagreement on how to implement it that never got resolved. Concerns like, you can't have a?.b return None because then a?.b() isn't callable, unless you want to use a?.b?() for this case, or some people wanted to have "a?" [where a is None] return a magic object whose attribute/call/getitem would give no error, but that would have to keep returning itself and never actually return None for chained operators. From guido at python.org Sat Sep 10 13:26:42 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2016 10:26:42 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> Message-ID: The way I recall it, we arrived at the perfect syntax (using ?) and semantics. The issue was purely strong hesitation about whether sprinkling ? all over your code is too ugly for Python, and in the end we couldn't get agreement on *that*. Another problem is PEP 505 -- it is full of discussion but its specification is unreadable due to the author's idea to defer the actual choice of operators and use a strange sequence of unicode characters instead. If someone wants to write a new, *short* PEP that defers to PEP 505 for motivation etc. and just writes up the spec for the syntax and semantics we'll have a better starting point. IMO the key syntax is simply one for accessing attributes returning None instead of raising AttributeError, so that e.g. `foo?.bar?.baz` is roughly equivalent to `foo.bar.baz if (foo is not None and foo.bar is not None) else None`, except evaluating foo and foo.bar only once. On Sat, Sep 10, 2016 at 10:14 AM, Random832 wrote: > On Sat, Sep 10, 2016, at 12:48, Stephen J. Turnbull wrote: >> I forget if Guido was very sympathetic to null-coalescing operators, >> given somebody came up with a good syntax. > > As I remember the discussion, I thought he'd more or less conceded on > the use of ? but there was disagreement on how to implement it that > never got resolved. Concerns like, you can't have a?.b return None > because then a?.b() isn't callable, unless you want to use a?.b?() for > this case, or some people wanted to have "a?" [where a is None] return a > magic object whose attribute/call/getitem would give no error, but that > would have to keep returning itself and never actually return None for > chained operators. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From srkunze at mail.de Sat Sep 10 13:40:22 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Sat, 10 Sep 2016 19:40:22 +0200 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> Message-ID: On 10.09.2016 19:14, Random832 wrote: > As I remember the discussion, I thought he'd more or less conceded on > the use of ? but there was disagreement on how to implement it that > never got resolved. Concerns like, you can't have a?.b return None > because then a?.b() isn't callable, unless you want to use a?.b?() for > this case, or some people wanted to have "a?" [where a is None] return a > magic object whose attribute/call/getitem would give no error, but that > would have to keep returning itself and never actually return None for > chained operators. That appeared to be one solution to make the ?-syntax class useful. But in the end, there were too many possibilities (operator?, nomad objects?, syntax expansion?, something else?), issues with all those "?" all over the place, hiding errors this way (which was the most serious one), and uncertainty about the overall benefit of this syntax compared to better designs like "how to not use None in the first place" didn't lead to a result so far. Let's see those can be resolved. Sven From p.f.moore at gmail.com Sat Sep 10 13:44:46 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 10 Sep 2016 18:44:46 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> Message-ID: On 10 September 2016 at 18:26, Guido van Rossum wrote: > IMO the key syntax is > simply one for accessing attributes returning None instead of raising > AttributeError, so that e.g. `foo?.bar?.baz` is roughly equivalent to > `foo.bar.baz if (foo is not None and foo.bar is not None) else None`, > except evaluating foo and foo.bar only once. If we're not looking to use all the other null-coalescing variants (?=, ?(), ...) - which is something I'm pleased about, as I do think that scattering that many ?'s about is likely to lead to ugly code - then it would probably be fine to just use ? for this operation, so we'd have foo?bar?baz rather than needing foo?.bar?.baz. Paul From python at mrabarnett.plus.com Sat Sep 10 14:09:36 2016 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 10 Sep 2016 19:09:36 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> Message-ID: On 2016-09-10 18:44, Paul Moore wrote: > On 10 September 2016 at 18:26, Guido van Rossum wrote: >> IMO the key syntax is >> simply one for accessing attributes returning None instead of raising >> AttributeError, so that e.g. `foo?.bar?.baz` is roughly equivalent to >> `foo.bar.baz if (foo is not None and foo.bar is not None) else None`, >> except evaluating foo and foo.bar only once. > > If we're not looking to use all the other null-coalescing variants > (?=, ?(), ...) - which is something I'm pleased about, as I do think > that scattering that many ?'s about is likely to lead to ugly code - > then it would probably be fine to just use ? for this operation, so > we'd have foo?bar?baz rather than needing foo?.bar?.baz. > I think that's not as clear; the "?." at least looks like a form of attribute access. It would also mean that it would be more difficult to add the other null-coalescing variants later, if the need arose. From mal at egenix.com Sat Sep 10 16:00:04 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 10 Sep 2016 22:00:04 +0200 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: <57D46644.5020006@egenix.com> In eGenix PyRun we expose a new helper in the sys module for this: --- ./Python/sysmodule.c 2015-12-07 02:39:11.000000000 +0100 +++ ./Python/sysmodule.c 2016-05-03 19:22:35.793193862 +0200 @@ -1205,6 +1205,50 @@ Return True if Python is exiting."); +/*** PyRun Extension ***************************************************/ + +#define SYS_SETFLAG(c_flag, flag_name) \ + if (strcmp(flagname, flag_name) == 0) { \ + old_value = c_flag; \ + if (value != -1) \ + c_flag = value; \ + } else + +static PyObject * +sys_setflag(PyObject* self, PyObject* args) +{ + char *flagname; + int value = -1, old_value = value; + + if (!PyArg_ParseTuple(args, "s|i", &flagname, &value)) + goto onError; + + SYS_SETFLAG(Py_DebugFlag, "debug") + SYS_SETFLAG(Py_OptimizeFlag, "optimize") + SYS_SETFLAG(Py_DontWriteBytecodeFlag, "dont_write_bytecode") + SYS_SETFLAG(Py_VerboseFlag, "verbose") + SYS_SETFLAG(Py_HashRandomizationFlag, "hash_randomization") + SYS_SETFLAG(Py_InspectFlag, "inspect") + { + PyErr_SetString(PyExc_ValueError, + "unknown flag name"); + goto onError; + } + return PyLong_FromLong((long)old_value); + + onError: + return NULL; +} + +#undef SYS_SETFLAG + +PyDoc_STRVAR(sys_setflag__doc__, +"_setflag(flagname, value) -> old_value\n\ +Set the given interpreter flag and return its previous value."); + +/*** End of PyRun Extension ***********************************************/ + + static PyMethodDef sys_methods[] = { /* Might as well keep this in alphabetic order */ {"callstats", (PyCFunction)PyEval_GetCallStats, METH_NOARGS, @@ -1268,6 +1312,7 @@ {"setdlopenflags", sys_setdlopenflags, METH_VARARGS, setdlopenflags_doc}, #endif + {"_setflag", sys_setflag, METH_VARARGS, sys_setflag__doc__}, {"setprofile", sys_setprofile, METH_O, setprofile_doc}, {"getprofile", sys_getprofile, METH_NOARGS, getprofile_doc}, {"setrecursionlimit", sys_setrecursionlimit, METH_VARARGS, We need this for similar reasons, since PyRun does the command line processing in Python rather than in C. The helper doesn't only expose the optimize flag, but also several other flags. On 10.09.2016 02:04, Damien George wrote: >> What is to stop you from adding this to micropython as a library >> extension? > > That's what I would like to do, and usually we do just go ahead and > add our own extensions, trying to be Pythonic as possible :) > > But we do that a lot and sometimes I think it would be good to discuss > with upstream (ie python-dev/python-ideas) about adding new > functions/classes so that MicroPython doesn't diverge too much (eg > Paul Sokolovsky had a discussion about time.sleep_ms, time.sleep_us > etc, which was fruitful). sys.optimize(value) is a pretty simple > addition so I thought it would be a straight forward discussion to > have here. > > I guess my main question to this list is: if CPython were to add a > function to change the optimisation level at runtime, what would it > look like? I don't want to push CPython to actually add such a thing > if it's not seen as a useful addition. Instead I want to see how > others would implement it if they needed to. > > A nice outcome would be that MicroPython adds the function now (eg > sys.optimize), and in the future if CPython finds that it needs it, it > also adds the same function with the same name/module/signature. > Hence why I would like to add it correctly into MicroPython from the > very beginning. > > Alternatively we can just use the micropython module and add > micropython.optimize(value), and then if CPython does add it later on > we'll have to change how we do it. > >> I've never felt the urge to do this and I don't think I've >> ever heard it requested before. If you think it should be in the >> stdlib, given the timing of the 3.6 feature freeze the earliest time >> it could land would be Python 3.7. > > No, I definitely don't want to make a rush for 3.6. I guess the > timing of this post was not the best considering the 3.6 release :) > >> Finally, let's not define APIs with >> British spelling. :-) > > Haha, the spelling never even crossed my mind! Well, sys.opt() is > definitely too short and confusing, so "optimize" would be the choice. > > Cheers, > Damien. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 10 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From guido at python.org Sat Sep 10 16:56:39 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2016 13:56:39 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> Message-ID: On Sat, Sep 10, 2016 at 11:09 AM, MRAB wrote: > On 2016-09-10 18:44, Paul Moore wrote: >> >> On 10 September 2016 at 18:26, Guido van Rossum wrote: >>> >>> IMO the key syntax is >>> simply one for accessing attributes returning None instead of raising >>> AttributeError, so that e.g. `foo?.bar?.baz` is roughly equivalent to >>> `foo.bar.baz if (foo is not None and foo.bar is not None) else None`, >>> except evaluating foo and foo.bar only once. >> >> >> If we're not looking to use all the other null-coalescing variants >> (?=, ?(), ...) - which is something I'm pleased about, as I do think >> that scattering that many ?'s about is likely to lead to ugly code - >> then it would probably be fine to just use ? for this operation, so >> we'd have foo?bar?baz rather than needing foo?.bar?.baz. >> > I think that's not as clear; the "?." at least looks like a form of > attribute access. > > It would also mean that it would be more difficult to add the other > null-coalescing variants later, if the need arose. Indeed. And ?. is how this is spelled in some other lanuages (C# and Dart). I forgot one detail that's in PEP 505: e.g. `foo?.bar.baz()` should be implemented as `foo.bar.baz() if foo is not None else None`. IOW if foo is None, the entire trailing section `.bar.baz()` should be skipped. (But this is a property of `?.` as an alternative attribute access operator; it doesn't mean `?` is a postfix operator on `foo`.) Another issue already discussed in PEP 505 is a conflict with IPython (Jupyter Notebook), which uses ? and ?? as custom syntax to request help. But maybe it can be taught to only recognize those when they're the last character(s) on the line? -- --Guido van Rossum (python.org/~guido) From alexander.belopolsky at gmail.com Sat Sep 10 17:05:55 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 10 Sep 2016 17:05:55 -0400 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> Message-ID: On Sat, Sep 10, 2016 at 4:56 PM, Guido van Rossum wrote: > Another issue already discussed in PEP 505 is a conflict with IPython > (Jupyter Notebook), which uses ? and ?? as custom syntax to request > help. But maybe it can be taught to only recognize those when they're > the last character(s) on the line? > I think this is already the case: In [1]: ?foo Object `foo` not found. In [2]: foo? Object `foo` not found. In [3]: foo?bar File "", line 1 foo?bar ^ SyntaxError: invalid syntax -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Sat Sep 10 18:15:27 2016 From: random832 at fastmail.com (Random832) Date: Sat, 10 Sep 2016 18:15:27 -0400 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> Message-ID: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> On Sat, Sep 10, 2016, at 13:26, Guido van Rossum wrote: > The way I recall it, we arrived at the perfect syntax (using ?) and > semantics. The issue was purely strong hesitation about whether > sprinkling ? all over your code is too ugly for Python I think that if there's "strong hesitation" about something being "too ugly" it can't really be described as "the perfect syntax". IIRC there were a couple alternatives being discussed that would have reduced the number of question marks to one [or one per object which might be None]. From mertz at gnosis.cx Sat Sep 10 18:21:25 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 15:21:25 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: I find the '?.' syntax very ugly, much more so in the examples of chained attributes. A much better way to handle the use case is to wrap objects in a class that gives this "propagating None" behavior with plain attribute access. A nice implementation was presented in this thread. On Sep 10, 2016 3:16 PM, "Random832" wrote: > On Sat, Sep 10, 2016, at 13:26, Guido van Rossum wrote: > > The way I recall it, we arrived at the perfect syntax (using ?) and > > semantics. The issue was purely strong hesitation about whether > > sprinkling ? all over your code is too ugly for Python > > I think that if there's "strong hesitation" about something being "too > ugly" it can't really be described as "the perfect syntax". IIRC there > were a couple alternatives being discussed that would have reduced the > number of question marks to one [or one per object which might be None]. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sat Sep 10 18:24:37 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sat, 10 Sep 2016 17:24:37 -0500 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: https://github.com/kirbyfan64/_frozensafemockobjectimplementation In all seriousness, though, I really feel like that would be the ultimate bug magnet, since it'd be easy to forget to un-wrap the object afterwards. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On Sep 10, 2016 5:21 PM, "David Mertz" wrote: > I find the '?.' syntax very ugly, much more so in the examples of chained > attributes. > > A much better way to handle the use case is to wrap objects in a class > that gives this "propagating None" behavior with plain attribute access. A > nice implementation was presented in this thread. > > On Sep 10, 2016 3:16 PM, "Random832" wrote: > >> On Sat, Sep 10, 2016, at 13:26, Guido van Rossum wrote: >> > The way I recall it, we arrived at the perfect syntax (using ?) and >> > semantics. The issue was purely strong hesitation about whether >> > sprinkling ? all over your code is too ugly for Python >> >> I think that if there's "strong hesitation" about something being "too >> ugly" it can't really be described as "the perfect syntax". IIRC there >> were a couple alternatives being discussed that would have reduced the >> number of question marks to one [or one per object which might be None]. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Sep 10 19:06:03 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 16:06:03 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: Actually, I guess the example I liked was from the year ago discussion. And it didn't do *exactly* what I think a wrapper should. What I'd want would be like this: class NoneCoalesce(object): "Standard operations on object for 'is not None'" def __init__(self, obj): self.obj = obj def __getattr__(self, name): try: return getattr(self.obj, name) except AttributeError: return NoneCoalesce(None) def __getitem__(self, item): try: return self.obj[item] except (TypeError, KeyError): return NoneCoalesce(None) def __call__(self, *args, **kwds): try: return self.obj(*args, **kwds) except TypeError: return NoneCoalesce(None) def __bool__(self): return self.obj is not None def __repr__(self): return "NoneCoalesce[%r]" % self.obj def __str__(self): return "NoneCoalesce[%r]" % self.obj def __len__(self): try: return len(self.obj) except TypeError: return 0 Then we might use it similar to this: >>> from boltons.dictutils import OrderedMultiDict >>> from NoneCoalesce import NoneCoalesce >>> omd = OrderedMultiDict() >>> omd['a'] = 1 >>> omd['b'] = 2 >>> omd.add('a', 3) >>> nc = NoneCoalesce(omd) >>> nc or "Spanish Inquisition" Out[8]: NoneCoalesce[OrderedMultiDict([('a', 1), ('b', 2), ('a', 3)])] >>> nc.spam or "Spam" Out[9]: 'Spam' >>> nc['nope'].bar.baz() Out[10]: NoneCoalesce[None] >>> nc['a'] Out[11]: 3 >>> nc.getlist('a') Out[12]: [1, 3] Nothing special about boltons' OrderedMultiDict here, just something I've been playing with that has some distinctive methods. The idea is that we can easily have both "regular" behavior and None coalescing just by wrapping any objects in a utility class... and WITHOUT adding ugly syntax. I might have missed some corners where we would want behavior wrapped, but those shouldn't be that hard to add in principle. On Sat, Sep 10, 2016 at 3:21 PM, David Mertz wrote: > I find the '?.' syntax very ugly, much more so in the examples of chained > attributes. > > A much better way to handle the use case is to wrap objects in a class > that gives this "propagating None" behavior with plain attribute access. A > nice implementation was presented in this thread. > > On Sep 10, 2016 3:16 PM, "Random832" wrote: > >> On Sat, Sep 10, 2016, at 13:26, Guido van Rossum wrote: >> > The way I recall it, we arrived at the perfect syntax (using ?) and >> > semantics. The issue was purely strong hesitation about whether >> > sprinkling ? all over your code is too ugly for Python >> >> I think that if there's "strong hesitation" about something being "too >> ugly" it can't really be described as "the perfect syntax". IIRC there >> were a couple alternatives being discussed that would have reduced the >> number of question marks to one [or one per object which might be None]. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Sep 10 19:10:08 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2016 16:10:08 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: So you're offering `NoneCoalesce(x).bar` as less-ugly alternative to `x?.bar`... Color me unconvinced. On Sat, Sep 10, 2016 at 4:06 PM, David Mertz wrote: > Actually, I guess the example I liked was from the year ago discussion. And > it didn't do *exactly* what I think a wrapper should. What I'd want would > be like this: > > class NoneCoalesce(object): > "Standard operations on object for 'is not None'" > def __init__(self, obj): > self.obj = obj > > def __getattr__(self, name): > try: > return getattr(self.obj, name) > except AttributeError: > return NoneCoalesce(None) > > def __getitem__(self, item): > try: > return self.obj[item] > except (TypeError, KeyError): > return NoneCoalesce(None) > > def __call__(self, *args, **kwds): > try: > return self.obj(*args, **kwds) > except TypeError: > return NoneCoalesce(None) > > def __bool__(self): > return self.obj is not None > > def __repr__(self): > return "NoneCoalesce[%r]" % self.obj > > def __str__(self): > return "NoneCoalesce[%r]" % self.obj > > def __len__(self): > try: > return len(self.obj) > except TypeError: > return 0 > > > Then we might use it similar to this: > >>>> from boltons.dictutils import OrderedMultiDict >>>> from NoneCoalesce import NoneCoalesce >>>> omd = OrderedMultiDict() >>>> omd['a'] = 1 >>>> omd['b'] = 2 >>>> omd.add('a', 3) >>>> nc = NoneCoalesce(omd) >>>> nc or "Spanish Inquisition" > Out[8]: NoneCoalesce[OrderedMultiDict([('a', 1), ('b', 2), ('a', 3)])] >>>> nc.spam or "Spam" > Out[9]: 'Spam' >>>> nc['nope'].bar.baz() > Out[10]: NoneCoalesce[None] >>>> nc['a'] > Out[11]: 3 >>>> nc.getlist('a') > Out[12]: [1, 3] > > > Nothing special about boltons' OrderedMultiDict here, just something I've > been playing with that has some distinctive methods. > > The idea is that we can easily have both "regular" behavior and None > coalescing just by wrapping any objects in a utility class... and WITHOUT > adding ugly syntax. I might have missed some corners where we would want > behavior wrapped, but those shouldn't be that hard to add in principle. > > On Sat, Sep 10, 2016 at 3:21 PM, David Mertz wrote: >> >> I find the '?.' syntax very ugly, much more so in the examples of chained >> attributes. >> >> A much better way to handle the use case is to wrap objects in a class >> that gives this "propagating None" behavior with plain attribute access. A >> nice implementation was presented in this thread. >> >> >> On Sep 10, 2016 3:16 PM, "Random832" wrote: >>> >>> On Sat, Sep 10, 2016, at 13:26, Guido van Rossum wrote: >>> > The way I recall it, we arrived at the perfect syntax (using ?) and >>> > semantics. The issue was purely strong hesitation about whether >>> > sprinkling ? all over your code is too ugly for Python >>> >>> I think that if there's "strong hesitation" about something being "too >>> ugly" it can't really be described as "the perfect syntax". IIRC there >>> were a couple alternatives being discussed that would have reduced the >>> number of question marks to one [or one per object which might be None]. >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From rosuav at gmail.com Sat Sep 10 19:27:39 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 11 Sep 2016 09:27:39 +1000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sun, Sep 11, 2016 at 9:10 AM, Guido van Rossum wrote: > So you're offering `NoneCoalesce(x).bar` as less-ugly alternative to > `x?.bar`... Color me unconvinced. As a syntactic form? Not interested. But what if it's the underlying implementation? We have "yield from X" as a tidy syntax for roughly a page of equivalent code. We could have "x?.bar" as syntactic sugar for "NoneCoalesce(x).bar". ChrisA From mertz at gnosis.cx Sat Sep 10 19:36:37 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 16:36:37 -0700 Subject: [Python-ideas] Fwd: Re: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sat, Sep 10, 2016 at 4:10 PM, Guido van Rossum wrote: > > So you're offering `NoneCoalesce(x).bar` as less-ugly alternative to > `x?.bar`... Color me unconvinced. No, I'm offering a more realistic use pattern: > for x in get_stuff(): > > x = NoneCoalesce(x) > > # ... bunch of stuff with x ... > # ... more stuff with nested keys or attributes ... > > x2 = x.foo > > x3 = x.bar.baz[x2] > > x4 = x(x.val) > > result = x3(x4) > > As a less ugly alternative in the fairly uncommon case that you want None coalescing as the behavior of getting attributes, keys, call values, etc. that may or may not be available (AND where you don't want to wrap all of those access patterns in one try/except block). In contrast, the ugly version of even this pretty simple toy code with the hypothetical syntax would be: > for x in get_stuff(): > > # ... bunch of stuff with x ... > > # ... more stuff with nested keys or attributes ... > > x2 = x?.foo > > x3 = x?.bar?.baz?[x2] > > x4 = x?(x?.val) > > result = x3?(x4) This second case looks absolutely awful to me. And real world uses, if implemented, would quickly get much worse than that. > > On Sat, Sep 10, 2016 at 4:06 PM, David Mertz wrote: > > Actually, I guess the example I liked was from the year ago discussion. And > > it didn't do *exactly* what I think a wrapper should. What I'd want would > > be like this: > > > > class NoneCoalesce(object): > > "Standard operations on object for 'is not None'" > > def __init__(self, obj): > > self.obj = obj > > > > def __getattr__(self, name): > > try: > > return getattr(self.obj, name) > > except AttributeError: > > return NoneCoalesce(None) > > > > def __getitem__(self, item): > > try: > > return self.obj[item] > > except (TypeError, KeyError): > > return NoneCoalesce(None) > > > > def __call__(self, *args, **kwds): > > try: > > return self.obj(*args, **kwds) > > except TypeError: > > return NoneCoalesce(None) > > > > def __bool__(self): > > return self.obj is not None > > > > def __repr__(self): > > return "NoneCoalesce[%r]" % self.obj > > > > def __str__(self): > > return "NoneCoalesce[%r]" % self.obj > > > > def __len__(self): > > try: > > return len(self.obj) > > except TypeError: > > return 0 > > > > > > Then we might use it similar to this: > > > >>>> from boltons.dictutils import OrderedMultiDict > >>>> from NoneCoalesce import NoneCoalesce > >>>> omd = OrderedMultiDict() > >>>> omd['a'] = 1 > >>>> omd['b'] = 2 > >>>> omd.add('a', 3) > >>>> nc = NoneCoalesce(omd) > >>>> nc or "Spanish Inquisition" > > Out[8]: NoneCoalesce[OrderedMultiDict([('a', 1), ('b', 2), ('a', 3)])] > >>>> nc.spam or "Spam" > > Out[9]: 'Spam' > >>>> nc['nope'].bar.baz() > > Out[10]: NoneCoalesce[None] > >>>> nc['a'] > > Out[11]: 3 > >>>> nc.getlist('a') > > Out[12]: [1, 3] > > > > > > Nothing special about boltons' OrderedMultiDict here, just something I've > > been playing with that has some distinctive methods. > > > > The idea is that we can easily have both "regular" behavior and None > > coalescing just by wrapping any objects in a utility class... and WITHOUT > > adding ugly syntax. I might have missed some corners where we would want > > behavior wrapped, but those shouldn't be that hard to add in principle. > > > > On Sat, Sep 10, 2016 at 3:21 PM, David Mertz wrote: > >> > >> I find the '?.' syntax very ugly, much more so in the examples of chained > >> attributes. > >> > >> A much better way to handle the use case is to wrap objects in a class > >> that gives this "propagating None" behavior with plain attribute access. A > >> nice implementation was presented in this thread. > >> > >> > >> On Sep 10, 2016 3:16 PM, "Random832" wrote: > >>> > >>> On Sat, Sep 10, 2016, at 13:26, Guido van Rossum wrote: > >>> > The way I recall it, we arrived at the perfect syntax (using ?) and > >>> > semantics. The issue was purely strong hesitation about whether > >>> > sprinkling ? all over your code is too ugly for Python > >>> > >>> I think that if there's "strong hesitation" about something being "too > >>> ugly" it can't really be described as "the perfect syntax". IIRC there > >>> were a couple alternatives being discussed that would have reduced the > >>> number of question marks to one [or one per object which might be None]. > >>> _______________________________________________ > >>> Python-ideas mailing list > >>> Python-ideas at python.org > >>> https://mail.python.org/mailman/listinfo/python-ideas > >>> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > > > > -- > > Keeping medicines from the bloodstreams of the sick; food > > from the bellies of the hungry; books from the hands of the > > uneducated; technology from the underdeveloped; and putting > > advocates of freedom in prisons. Intellectual property is > > to the 21st century what the slave trade was to the 16th. > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Sep 10 19:38:32 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 16:38:32 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: Sorry, I sent this accidentally as private reply, then tried to fix it on phone. The latter produced horrible formatting. Please just read this version. On Sat, Sep 10, 2016 at 4:10 PM, Guido van Rossum wrote: > So you're offering `NoneCoalesce(x).bar` as less-ugly alternative to > `x?.bar`... Color me unconvinced. > No, I'm offering a more realistic use pattern: for x in get_stuff(): x = NoneCoalesce(x) # ... bunch of stuff with x ... # ... more stuff with nested keys or attributes ... x2 = x.foo x3 = x.bar.baz[x2] x4 = x(x.val) result = x3(x4) As a less ugly alternative in the fairly uncommon case that you want None coalescing as the behavior of getting attributes, keys, call values, etc. that may or may not be available (AND where you don't want to wrap all of those access patterns in one try/except block). In contrast, the ugly version of even this pretty simple toy code with the hypothetical syntax would be: for x in get_stuff(): # ... bunch of stuff with x ... # ... more stuff with nested keys or attributes ... x2 = x?.foo x3 = x?.bar?.baz?[x2] x4 = x?(x?.val) result = x3?(x4) This second case looks absolutely awful to me. And real world uses, if implemented, would quickly get much worse than that. Yours, David... -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Sep 10 19:43:13 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2016 16:43:13 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sat, Sep 10, 2016 at 4:27 PM, Chris Angelico wrote: > On Sun, Sep 11, 2016 at 9:10 AM, Guido van Rossum wrote: >> So you're offering `NoneCoalesce(x).bar` as less-ugly alternative to >> `x?.bar`... Color me unconvinced. > > As a syntactic form? Not interested. But what if it's the underlying > implementation? We have "yield from X" as a tidy syntax for roughly a > page of equivalent code. We could have "x?.bar" as syntactic sugar for > "NoneCoalesce(x).bar". PEP 505 has an option for a way to customize the coalescing operation (https://www.python.org/dev/peps/pep-0505/#generalized-coalescing). Though I think I'd rather not do that. But it just occurs to me that the implementation given by David Mertz is not what I'd expect: it seems that `NoneCoalesce([]).flup` would catch the AttributeError (there' no `[].flup`) and return NoneCoalesce(None), whereas I would expect `?.` to only return None when the LHS is None, not when some other not-None object doesn't have the requested attribute. (And the "pile of poo" operator in PEP 505 agrees with me.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Sat Sep 10 19:44:54 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2016 16:44:54 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: There seems to be a major misunderstanding here. A None-coalescing operator is not for catching AttributeError, it's a shortcut similar to "a or b" except that it checks for "a is None" rather than bool(a). On Sat, Sep 10, 2016 at 4:38 PM, David Mertz wrote: > Sorry, I sent this accidentally as private reply, then tried to fix it on > phone. The latter produced horrible formatting. Please just read this > version. > > On Sat, Sep 10, 2016 at 4:10 PM, Guido van Rossum wrote: >> >> So you're offering `NoneCoalesce(x).bar` as less-ugly alternative to >> `x?.bar`... Color me unconvinced. > > > No, I'm offering a more realistic use pattern: > > for x in get_stuff(): > > x = NoneCoalesce(x) > > # ... bunch of stuff with x ... > # ... more stuff with nested keys or attributes ... > > x2 = x.foo > > x3 = x.bar.baz[x2] > > x4 = x(x.val) > > result = x3(x4) > > > > As a less ugly alternative in the fairly uncommon case that you want None > coalescing as the behavior of getting attributes, keys, call values, etc. > that may or may not be available (AND where you don't want to wrap all of > those access patterns in one try/except block). > > In contrast, the ugly version of even this pretty simple toy code with the > hypothetical syntax would be: > > for x in get_stuff(): > > # ... bunch of stuff with x ... > > # ... more stuff with nested keys or attributes ... > > x2 = x?.foo > > x3 = x?.bar?.baz?[x2] > > x4 = x?(x?.val) > > result = x3?(x4) > > > This second case looks absolutely awful to me. And real world uses, if > implemented, would quickly get much worse than that. > > Yours, David... > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From mertz at gnosis.cx Sat Sep 10 20:15:17 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 17:15:17 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sep 10, 2016 4:45 PM, "Guido van Rossum" wrote: > > There seems to be a major misunderstanding here. A None-coalescing > operator is not for catching AttributeError, it's a shortcut similar > to "a or b" except that it checks for "a is None" rather than bool(a). That's exactly what the wrapper does. Except it converts all the regular operators into their None-coalescing versions by putting the extra checks into the wrapped object itself. Now admittedly, this DOES mean that the behavior of operations is somewhat different depending on whether they are wrapped objects or not. False-like is a different thing than None-like. This really MUST BE essentially a way a catching AttributeErrors though. With the proposed syntax 'x?.foo?.bar' will resolve even if x has no 'foo'. So 'x?.'foo' has to be something special. I guess that special thing could be a 3.7-style None that responds to new operators, but that's essentially still *wrapping* a 3.6-style None. In my mind, as I say, the question marks look ugly, especially when repeated in chained operations (attribute, call, item get). But even if they didn't feel bad visually, I don't believe the use case is common enough to warrant dedicated syntax. Even if my keyboard had some character I thought was beautiful and intuitive for that meaning, it's still an extra cognitive burden to distinguish the plain from None-coalescing versions of every operation, especially for learners. Another problem is that the question mark still doesn't actually get the special 'a or b' behavior. For that you still need 'a if a is not None else b'. Or I guess, in concept, 'a ?or b'. For what it's worth, the wrapper gives you the special 'a or b' semantics by casting non-Nones as truthy... But again, 'a' has to have been wrapped first. > > On Sat, Sep 10, 2016 at 4:38 PM, David Mertz wrote: > > Sorry, I sent this accidentally as private reply, then tried to fix it on > > phone. The latter produced horrible formatting. Please just read this > > version. > > > > On Sat, Sep 10, 2016 at 4:10 PM, Guido van Rossum wrote: > >> > >> So you're offering `NoneCoalesce(x).bar` as less-ugly alternative to > >> `x?.bar`... Color me unconvinced. > > > > > > No, I'm offering a more realistic use pattern: > > > > for x in get_stuff(): > > > > x = NoneCoalesce(x) > > > > # ... bunch of stuff with x ... > > # ... more stuff with nested keys or attributes ... > > > > x2 = x.foo > > > > x3 = x.bar.baz[x2] > > > > x4 = x(x.val) > > > > result = x3(x4) > > > > > > > > As a less ugly alternative in the fairly uncommon case that you want None > > coalescing as the behavior of getting attributes, keys, call values, etc. > > that may or may not be available (AND where you don't want to wrap all of > > those access patterns in one try/except block). > > > > In contrast, the ugly version of even this pretty simple toy code with the > > hypothetical syntax would be: > > > > for x in get_stuff(): > > > > # ... bunch of stuff with x ... > > > > # ... more stuff with nested keys or attributes ... > > > > x2 = x?.foo > > > > x3 = x?.bar?.baz?[x2] > > > > x4 = x?(x?.val) > > > > result = x3?(x4) > > > > > > This second case looks absolutely awful to me. And real world uses, if > > implemented, would quickly get much worse than that. > > > > Yours, David... > > > > -- > > Keeping medicines from the bloodstreams of the sick; food > > from the bellies of the hungry; books from the hands of the > > uneducated; technology from the underdeveloped; and putting > > advocates of freedom in prisons. Intellectual property is > > to the 21st century what the slave trade was to the 16th. > > > > > > > > -- > > Keeping medicines from the bloodstreams of the sick; food > > from the bellies of the hungry; books from the hands of the > > uneducated; technology from the underdeveloped; and putting > > advocates of freedom in prisons. Intellectual property is > > to the 21st century what the slave trade was to the 16th. > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Sep 10 20:23:47 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2016 17:23:47 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: No. PEP 505 actually solves the problem without ever catching AttributeError. Please read it. On Sat, Sep 10, 2016 at 5:15 PM, David Mertz wrote: > On Sep 10, 2016 4:45 PM, "Guido van Rossum" wrote: >> >> There seems to be a major misunderstanding here. A None-coalescing >> operator is not for catching AttributeError, it's a shortcut similar >> to "a or b" except that it checks for "a is None" rather than bool(a). > > That's exactly what the wrapper does. Except it converts all the regular > operators into their None-coalescing versions by putting the extra checks > into the wrapped object itself. > > Now admittedly, this DOES mean that the behavior of operations is somewhat > different depending on whether they are wrapped objects or not. False-like > is a different thing than None-like. > > This really MUST BE essentially a way a catching AttributeErrors though. > With the proposed syntax 'x?.foo?.bar' will resolve even if x has no 'foo'. > So 'x?.'foo' has to be something special. I guess that special thing could > be a 3.7-style None that responds to new operators, but that's essentially > still *wrapping* a 3.6-style None. > > In my mind, as I say, the question marks look ugly, especially when repeated > in chained operations (attribute, call, item get). But even if they didn't > feel bad visually, I don't believe the use case is common enough to warrant > dedicated syntax. Even if my keyboard had some character I thought was > beautiful and intuitive for that meaning, it's still an extra cognitive > burden to distinguish the plain from None-coalescing versions of every > operation, especially for learners. > > Another problem is that the question mark still doesn't actually get the > special 'a or b' behavior. For that you still need 'a if a is not None else > b'. Or I guess, in concept, 'a ?or b'. For what it's worth, the wrapper > gives you the special 'a or b' semantics by casting non-Nones as truthy... > But again, 'a' has to have been wrapped first. > >> >> On Sat, Sep 10, 2016 at 4:38 PM, David Mertz wrote: >> > Sorry, I sent this accidentally as private reply, then tried to fix it >> > on >> > phone. The latter produced horrible formatting. Please just read this >> > version. >> > >> > On Sat, Sep 10, 2016 at 4:10 PM, Guido van Rossum >> > wrote: >> >> >> >> So you're offering `NoneCoalesce(x).bar` as less-ugly alternative to >> >> `x?.bar`... Color me unconvinced. >> > >> > >> > No, I'm offering a more realistic use pattern: >> > >> > for x in get_stuff(): >> > >> > x = NoneCoalesce(x) >> > >> > # ... bunch of stuff with x ... >> > # ... more stuff with nested keys or attributes ... >> > >> > x2 = x.foo >> > >> > x3 = x.bar.baz[x2] >> > >> > x4 = x(x.val) >> > >> > result = x3(x4) >> > >> > >> > >> > As a less ugly alternative in the fairly uncommon case that you want >> > None >> > coalescing as the behavior of getting attributes, keys, call values, >> > etc. >> > that may or may not be available (AND where you don't want to wrap all >> > of >> > those access patterns in one try/except block). >> > >> > In contrast, the ugly version of even this pretty simple toy code with >> > the >> > hypothetical syntax would be: >> > >> > for x in get_stuff(): >> > >> > # ... bunch of stuff with x ... >> > >> > # ... more stuff with nested keys or attributes ... >> > >> > x2 = x?.foo >> > >> > x3 = x?.bar?.baz?[x2] >> > >> > x4 = x?(x?.val) >> > >> > result = x3?(x4) >> > >> > >> > This second case looks absolutely awful to me. And real world uses, if >> > implemented, would quickly get much worse than that. >> > >> > Yours, David... >> > >> > -- >> > Keeping medicines from the bloodstreams of the sick; food >> > from the bellies of the hungry; books from the hands of the >> > uneducated; technology from the underdeveloped; and putting >> > advocates of freedom in prisons. Intellectual property is >> > to the 21st century what the slave trade was to the 16th. >> > >> > >> > >> > -- >> > Keeping medicines from the bloodstreams of the sick; food >> > from the bellies of the hungry; books from the hands of the >> > uneducated; technology from the underdeveloped; and putting >> > advocates of freedom in prisons. Intellectual property is >> > to the 21st century what the slave trade was to the 16th. >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido) From mertz at gnosis.cx Sat Sep 10 21:02:51 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 18:02:51 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sat, Sep 10, 2016 at 5:23 PM, Guido van Rossum wrote: > No. PEP 505 actually solves the problem without ever catching > AttributeError. Please read it. > I read it again (I did a year ago, but reviewed it now). I hadn't been thinking that the *mechanism* of a new None-coalescing operator would actually be catching an exception. It could (and should) work differently if it becomes syntax. What I was getting at with "essentially" was that it would *do the same thing* that an AttributeError does. That is, if `x.foo` can't be evaluated (i.e. x doesn't have an attribute 'foo'), then access is informally "an error." The hypothetical "x?.foo" catches that "error" and substitutes a different value. The particular implementation under-the-hood is less important for most programmers who might use the construct (and I think documentation would actually give an informal equivalent as something similar to what I put in the NoneCoalesce class). -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sat Sep 10 21:28:29 2016 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 11 Sep 2016 02:28:29 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: <9c5a8021-40c7-6078-d18a-9a35ddd62183@mrabarnett.plus.com> On 2016-09-11 02:02, David Mertz wrote: > On Sat, Sep 10, 2016 at 5:23 PM, Guido van Rossum > wrote: > > No. PEP 505 actually solves the problem without ever catching > AttributeError. Please read it. > > > I read it again (I did a year ago, but reviewed it now). I hadn't been > thinking that the *mechanism* of a new None-coalescing operator would > actually be catching an exception. It could (and should) work > differently if it becomes syntax. > > What I was getting at with "essentially" was that it would *do the same > thing* that an AttributeError does. That is, if `x.foo` can't be > evaluated (i.e. x doesn't have an attribute 'foo'), then access is > informally "an error." The hypothetical "x?.foo" catches that "error" > and substitutes a different value. The particular implementation > under-the-hood is less important for most programmers who might use the > construct (and I think documentation would actually give an informal > equivalent as something similar to what I put in the NoneCoalesce class). > x?.foo would lookup attribute 'foo' _unless_ x was None, in which case it would return None. It's simply: None if x is None else x.foo This means that None?.__str__() would return None, not 'None'. (None has an attribute called '__str__', and None.__str__() returns 'None', but it would not be looked up because None is, well, None.) From mertz at gnosis.cx Sat Sep 10 21:46:53 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 18:46:53 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: <9c5a8021-40c7-6078-d18a-9a35ddd62183@mrabarnett.plus.com> References: <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <9c5a8021-40c7-6078-d18a-9a35ddd62183@mrabarnett.plus.com> Message-ID: Ok, I have been thinking of the behavior too broadly. I realize now that `x?.foo` might still simply raise an AttributeError if x is neither None nor a thing with a foo attribute. The class I wrote is definitely too aggressive for the behavior described. On the other hand, by being narrower in behavior there feels like even less motivation for new syntax. On Sep 10, 2016 6:29 PM, "MRAB" wrote: > On 2016-09-11 02:02, David Mertz wrote: > >> On Sat, Sep 10, 2016 at 5:23 PM, Guido van Rossum > > wrote: >> >> No. PEP 505 actually solves the problem without ever catching >> AttributeError. Please read it. >> >> >> I read it again (I did a year ago, but reviewed it now). I hadn't been >> thinking that the *mechanism* of a new None-coalescing operator would >> actually be catching an exception. It could (and should) work >> differently if it becomes syntax. >> >> What I was getting at with "essentially" was that it would *do the same >> thing* that an AttributeError does. That is, if `x.foo` can't be >> evaluated (i.e. x doesn't have an attribute 'foo'), then access is >> informally "an error." The hypothetical "x?.foo" catches that "error" >> and substitutes a different value. The particular implementation >> under-the-hood is less important for most programmers who might use the >> construct (and I think documentation would actually give an informal >> equivalent as something similar to what I put in the NoneCoalesce class). >> >> x?.foo would lookup attribute 'foo' _unless_ x was None, in which case it > would return None. It's simply: > > None if x is None else x.foo > > This means that None?.__str__() would return None, not 'None'. (None has > an attribute called '__str__', and None.__str__() returns 'None', but it > would not be looked up because None is, well, None.) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Sep 10 22:10:06 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2016 19:10:06 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <9c5a8021-40c7-6078-d18a-9a35ddd62183@mrabarnett.plus.com> Message-ID: To the contrary. I read and write code that performs explicit checks for None all the time. Catching AttributeError is often a code smell or at least a measure of last resort. On Sat, Sep 10, 2016 at 6:46 PM, David Mertz wrote: > Ok, I have been thinking of the behavior too broadly. I realize now that > `x?.foo` might still simply raise an AttributeError if x is neither None nor > a thing with a foo attribute. > > The class I wrote is definitely too aggressive for the behavior described. > On the other hand, by being narrower in behavior there feels like even less > motivation for new syntax. > > > On Sep 10, 2016 6:29 PM, "MRAB" wrote: >> >> On 2016-09-11 02:02, David Mertz wrote: >>> >>> On Sat, Sep 10, 2016 at 5:23 PM, Guido van Rossum >> > wrote: >>> >>> No. PEP 505 actually solves the problem without ever catching >>> AttributeError. Please read it. >>> >>> >>> I read it again (I did a year ago, but reviewed it now). I hadn't been >>> thinking that the *mechanism* of a new None-coalescing operator would >>> actually be catching an exception. It could (and should) work >>> differently if it becomes syntax. >>> >>> What I was getting at with "essentially" was that it would *do the same >>> thing* that an AttributeError does. That is, if `x.foo` can't be >>> evaluated (i.e. x doesn't have an attribute 'foo'), then access is >>> informally "an error." The hypothetical "x?.foo" catches that "error" >>> and substitutes a different value. The particular implementation >>> under-the-hood is less important for most programmers who might use the >>> construct (and I think documentation would actually give an informal >>> equivalent as something similar to what I put in the NoneCoalesce class). >>> >> x?.foo would lookup attribute 'foo' _unless_ x was None, in which case it >> would return None. It's simply: >> >> None if x is None else x.foo >> >> This means that None?.__str__() would return None, not 'None'. (None has >> an attribute called '__str__', and None.__str__() returns 'None', but it >> would not be looked up because None is, well, None.) >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From mertz at gnosis.cx Sat Sep 10 22:15:12 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 19:15:12 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <9c5a8021-40c7-6078-d18a-9a35ddd62183@mrabarnett.plus.com> Message-ID: How much of the time is a branch of the None check a single fallback value or attribute access versus how often a suite of statements within the not-None branch? I definitely check for None very often also. I'm curious what the breakdown is in code I work with. On Sep 10, 2016 7:10 PM, "Guido van Rossum" wrote: > To the contrary. I read and write code that performs explicit checks > for None all the time. Catching AttributeError is often a code smell > or at least a measure of last resort. > > On Sat, Sep 10, 2016 at 6:46 PM, David Mertz wrote: > > Ok, I have been thinking of the behavior too broadly. I realize now that > > `x?.foo` might still simply raise an AttributeError if x is neither None > nor > > a thing with a foo attribute. > > > > The class I wrote is definitely too aggressive for the behavior > described. > > On the other hand, by being narrower in behavior there feels like even > less > > motivation for new syntax. > > > > > > On Sep 10, 2016 6:29 PM, "MRAB" wrote: > >> > >> On 2016-09-11 02:02, David Mertz wrote: > >>> > >>> On Sat, Sep 10, 2016 at 5:23 PM, Guido van Rossum >>> > wrote: > >>> > >>> No. PEP 505 actually solves the problem without ever catching > >>> AttributeError. Please read it. > >>> > >>> > >>> I read it again (I did a year ago, but reviewed it now). I hadn't been > >>> thinking that the *mechanism* of a new None-coalescing operator would > >>> actually be catching an exception. It could (and should) work > >>> differently if it becomes syntax. > >>> > >>> What I was getting at with "essentially" was that it would *do the same > >>> thing* that an AttributeError does. That is, if `x.foo` can't be > >>> evaluated (i.e. x doesn't have an attribute 'foo'), then access is > >>> informally "an error." The hypothetical "x?.foo" catches that "error" > >>> and substitutes a different value. The particular implementation > >>> under-the-hood is less important for most programmers who might use the > >>> construct (and I think documentation would actually give an informal > >>> equivalent as something similar to what I put in the NoneCoalesce > class). > >>> > >> x?.foo would lookup attribute 'foo' _unless_ x was None, in which case > it > >> would return None. It's simply: > >> > >> None if x is None else x.foo > >> > >> This means that None?.__str__() would return None, not 'None'. (None has > >> an attribute called '__str__', and None.__str__() returns 'None', but it > >> would not be looked up because None is, well, None.) > >> > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Sat Sep 10 22:36:20 2016 From: random832 at fastmail.com (Random832) Date: Sat, 10 Sep 2016 22:36:20 -0400 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: <1473561380.3622982.721930753.56AE282A@webmail.messagingengine.com> On Sat, Sep 10, 2016, at 19:38, David Mertz wrote: > x2 = x?.foo > > x3 = x?.bar?.baz?[x2] A. if you're doing three different things with x, why are you using this instead of wrapping it in an if statement? B. Under some of the proposals, unless x.bar might be None independently of x not being None, this would just be "x?.bar.baz[x2] > x4 = x?(x?.val) C. Under some of the proposals, the inner contents of the call brackets aren't evaluated if x is None, so you don't need to use x?.val here. > result = x3?(x4) From random832 at fastmail.com Sat Sep 10 22:38:59 2016 From: random832 at fastmail.com (Random832) Date: Sat, 10 Sep 2016 22:38:59 -0400 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: <1473561539.3623267.721931449.6893F2ED@webmail.messagingengine.com> On Sat, Sep 10, 2016, at 20:15, David Mertz wrote: > On Sep 10, 2016 4:45 PM, "Guido van Rossum" wrote: > > > > There seems to be a major misunderstanding here. A None-coalescing > > operator is not for catching AttributeError, it's a shortcut similar > > to "a or b" except that it checks for "a is None" rather than bool(a). To put it more explicitly, more similar to "a and a.b" > This really MUST BE essentially a way a catching AttributeErrors though. > With the proposed syntax 'x?.foo?.bar' will resolve even if x has no > 'foo'. Why? I think you're confusing this proposal for something else. This is for treating x is None specially, not simply catching AttributeError to deal with the fact that None.foo doesn't exist. From mertz at gnosis.cx Sat Sep 10 22:49:30 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 10 Sep 2016 19:49:30 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <9c5a8021-40c7-6078-d18a-9a35ddd62183@mrabarnett.plus.com> Message-ID: For kicks I looked at the code in a commercial product that we are open sourcing very soon at Continuum Analytics. It has about 27k lines of Python and Cython. I checked it just by running `grep -C3 'if.*None'` over the source code and eyeballing. Obviously, this code might be refactored if other syntax was available. And I personally wrote only a small part of the code, so I might write it differently either way. But looking through, the uses of 'is (not) None' I found fell into a few categories. Some were suites of statements under the 'foo is None' branch. Those would have to remain suites. Some are single line returns, other single line raise. Those I call 'value' are the ones that would lend themselves to the 'bar = foo ?? baz' style. The ones I call 'attribute' are the cases where we test for non-None before accessing an attribute of the object. The thing that I found surprisingly common was cases where a compound condition was used, e.g. 'if x is None and y > 37'. Those might be possible to refactor, but cannot be directly rewritten in the coalescing style. - Suite: 75 - Return: 21 - Raise: 13 - Value: 46 - Attribute: 2 - Compound condition: 25 On Sat, Sep 10, 2016 at 7:15 PM, David Mertz wrote: > How much of the time is a branch of the None check a single fallback value > or attribute access versus how often a suite of statements within the > not-None branch? > > I definitely check for None very often also. I'm curious what the > breakdown is in code I work with. > > On Sep 10, 2016 7:10 PM, "Guido van Rossum" wrote: > >> To the contrary. I read and write code that performs explicit checks >> for None all the time. Catching AttributeError is often a code smell >> or at least a measure of last resort. >> >> On Sat, Sep 10, 2016 at 6:46 PM, David Mertz wrote: >> > Ok, I have been thinking of the behavior too broadly. I realize now that >> > `x?.foo` might still simply raise an AttributeError if x is neither >> None nor >> > a thing with a foo attribute. >> > >> > The class I wrote is definitely too aggressive for the behavior >> described. >> > On the other hand, by being narrower in behavior there feels like even >> less >> > motivation for new syntax. >> > >> > >> > On Sep 10, 2016 6:29 PM, "MRAB" wrote: >> >> >> >> On 2016-09-11 02:02, David Mertz wrote: >> >>> >> >>> On Sat, Sep 10, 2016 at 5:23 PM, Guido van Rossum > >>> > wrote: >> >>> >> >>> No. PEP 505 actually solves the problem without ever catching >> >>> AttributeError. Please read it. >> >>> >> >>> >> >>> I read it again (I did a year ago, but reviewed it now). I hadn't >> been >> >>> thinking that the *mechanism* of a new None-coalescing operator would >> >>> actually be catching an exception. It could (and should) work >> >>> differently if it becomes syntax. >> >>> >> >>> What I was getting at with "essentially" was that it would *do the >> same >> >>> thing* that an AttributeError does. That is, if `x.foo` can't be >> >>> evaluated (i.e. x doesn't have an attribute 'foo'), then access is >> >>> informally "an error." The hypothetical "x?.foo" catches that "error" >> >>> and substitutes a different value. The particular implementation >> >>> under-the-hood is less important for most programmers who might use >> the >> >>> construct (and I think documentation would actually give an informal >> >>> equivalent as something similar to what I put in the NoneCoalesce >> class). >> >>> >> >> x?.foo would lookup attribute 'foo' _unless_ x was None, in which case >> it >> >> would return None. It's simply: >> >> >> >> None if x is None else x.foo >> >> >> >> This means that None?.__str__() would return None, not 'None'. (None >> has >> >> an attribute called '__str__', and None.__str__() returns 'None', but >> it >> >> would not be looked up because None is, well, None.) >> >> >> >> _______________________________________________ >> >> Python-ideas mailing list >> >> Python-ideas at python.org >> >> https://mail.python.org/mailman/listinfo/python-ideas >> >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leban.us Sat Sep 10 23:45:20 2016 From: bruce at leban.us (Bruce Leban) Date: Sat, 10 Sep 2016 20:45:20 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sat, Sep 10, 2016 at 6:02 PM, David Mertz wrote: > What I was getting at with "essentially" was that it would *do the same > thing* that an AttributeError does. That is, if `x.foo` can't be evaluated > (i.e. x doesn't have an attribute 'foo'), then access is informally "an > error." The hypothetical "x?.foo" catches that "error" and substitutes a > different value. The particular implementation under-the-hood is less > important for most programmers who might use the construct (and I think > documentation would actually give an informal equivalent as something > similar to what I put in the NoneCoalesce class). > That's not a good way to think about it. This new operator is only checking for None and not actually checking for AttributeErrors. Consider: (3).x # AttributeError {}.x # AttributeError None.x # AttributeError (3)?.x # still AttributeError {}?.x # still AttributeError None?.x # None And also: None.__class__ # None?.__class__ # None And it's certainly not the case that those values don't accept any attributes: (3).real # 3 {}.values # None.__class__ # --- Bruce Check out my puzzle book and get it free here: http://J.mp/ingToConclusionsFree (available on iOS) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Sep 11 00:15:15 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Sep 2016 14:15:15 +1000 Subject: [Python-ideas] Inconsistencies (was: Shuffled) In-Reply-To: References: <20160906163257.GX26300@ando.pearwood.info> <6ff619a4-773e-930f-a2a4-a2aa4eda2357@mail.de> <20160908020050.GA22471@ando.pearwood.info> Message-ID: On 10 September 2016 at 22:09, Sven R. Kunze wrote: > Is going deep really necessary at all? > > People program for different reasons, to have fun, to create value for > others, to educate, or for reasons we both cannot even think of. Why should > they leave their level? Because of you or me? Because they need to know what > Turing-completeness means? What calling conventions are? I don't think so. > They wanna solve problems and get things done whether or not they know every > single bit of the language they use. If they decide to go deeper, that's > wonderful, but if they don't, don't force them. If people don't want to actively learn about the enormous complexities of real world programming language design, then they need to be willing to defer to those that have put in that time and effort. Deliberately choosing to argue from a position of wilful ignorance is disrespectful of everyone else's time and attention, and has no place on python-ideas or python-dev. It's fine for newcomers not to understand those complexities yet - many of these problems don't have obvious answers, and explaining them to someone else is a great way to better formulate them in our own minds (hence the martial arts maxim: "If you want to learn, teach"). However, it's utterly unacceptable to refuse to acknowledge "It's more complicated than you currently realise" as a reasonable answer, especially when that answer is accompanied by explanations of some of those complexities. If folks aren't willing to adhere to those terms of participation, they can save themselves and everyone else a lot of irritation by unsubscribing voluntarily, rather than waiting until they annoy other list participants enough to earn a suspension of their posting privileges (and potentially even a permanent ban if they're persistent enough, although that step has only had to happen once to date). Regards, Nick. P.S. If folks want to vent about the arrogance and intransigence of the Python core development team because we collectively disagree with them about something, they have the entire rest of the internet to do that without interfering with the collaborative process. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greschd at gmx.ch Sun Sep 11 05:36:08 2016 From: greschd at gmx.ch (Dominik Gresch) Date: Sun, 11 Sep 2016 11:36:08 +0200 Subject: [Python-ideas] if-statement in for-loop Message-ID: Hi, I've recently found myself writing code similar to this: for i in range(10): if i == 5: continue "body" which I find a bit ugly. Obviously the same could be written as for i in range(10): if i != 5: "body" but here you would have to look at the end of the body to see if something happens when i==5. So I asked myself if a syntax as follows would be possible: for i in range(10) if i != 5: body Personally, I find this extremely intuitive since this kind of if-statement is already present in list comprehensions. What is your opinion on this? Sorry if this has been discussed before -- I didn't find anything in the archives. Best regards, Dominik Gresch From mafagafogigante at gmail.com Sun Sep 11 06:28:58 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Sun, 11 Sep 2016 07:28:58 -0300 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: Message-ID: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> On 09/11/2016 06:36 AM, Dominik Gresch wrote: > So I asked myself if a syntax as follows would be possible: > > for i in range(10) if i != 5: > body > > Personally, I find this extremely intuitive since this kind of > if-statement is already present in list comprehensions. > > What is your opinion on this? Sorry if this has been discussed before -- > I didn't find anything in the archives. > I find it interesting. I thing that this will likely take up too many columns in more convoluted loops such as for element in collection if is_pretty_enough(element) and ...: ... However, this "problem" is already faced by list comprehensions, so it is not a strong argument against your idea. I am still unsure about whether or not the pattern you describe is frequent enough to justify special syntax. Not to mention that the current way to do it is already **very** readable. Just notice how for e in l: if e != 2: ... and for e in l if e != 2: ... read essentially the same and take about the same number of keystrokes. From elazarg at gmail.com Sun Sep 11 07:13:16 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sun, 11 Sep 2016 11:13:16 +0000 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> Message-ID: This has come up before. It will be a special case of making "if" without "else" result in a special "empty" type that is not part of the iteration. As in `[1, (2 if False) ] == [1]`. ?????? ??? ??, 11 ????' 2016, 13:29, ??? Bernardo Sulzbach ?< mafagafogigante at gmail.com>: > On 09/11/2016 06:36 AM, Dominik Gresch wrote: > > So I asked myself if a syntax as follows would be possible: > > > > for i in range(10) if i != 5: > > body > > > > Personally, I find this extremely intuitive since this kind of > > if-statement is already present in list comprehensions. > > > > What is your opinion on this? Sorry if this has been discussed before -- > > I didn't find anything in the archives. > > > > I find it interesting. > > I thing that this will likely take up too many columns in more > convoluted loops such as > > for element in collection if is_pretty_enough(element) and ...: > ... > > However, this "problem" is already faced by list comprehensions, so it > is not a strong argument against your idea. > > I am still unsure about whether or not the pattern you describe is > frequent enough to justify special syntax. Not to mention that the > current way to do it is already **very** readable. Just notice how > > for e in l: > if e != 2: > ... > > and > > for e in l if e != 2: > ... > > read essentially the same and take about the same number of keystrokes. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Sep 11 07:59:50 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 Sep 2016 12:59:50 +0100 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: Message-ID: On 11 September 2016 at 10:36, Dominik Gresch wrote: > I've recently found myself writing code similar to this: > > for i in range(10): > if i == 5: > continue > "body" > > which I find a bit ugly. I write code like this quite frequently. However, unlike you I don't find it particularly ugly. It's essentially the primary purpose of the "continue" statement. [...] > So I asked myself if a syntax as follows would be possible: > > for i in range(10) if i != 5: > body > > Personally, I find this extremely intuitive since this kind of if-statement > is already present in list comprehensions. I find it less readable than the version using "continue". Maybe that's because the inverted condition doesn't match how I think of this - "go through all the integers but skip 5" rather than "go through all the integers which aren't 5". It's certainly subjective, though, and there's no objective reason for comparing one over the other. > What is your opinion on this? Sorry if this has been discussed before -- I > didn't find anything in the archives. My feeling is that it doesn't add enough to warrant dedicated syntax. The quest to compress everything into a single line doesn't feel in the spirit of Python to me - it reminds me of Perl, with its plethora of if, when and unless statements and statement modifiers. And personally, I tend to use "does it feel like Perl" as a test for "is it inappropriate for Python". If the logic for what to skip and what to keep was complicated, I'd probably end up writing a custom generator - so this proposed syntax would likely only be needed for simple cases, and in such cases, it doesn't seem worth it just to avoid a one or two line "continue". Looking at alternative formulations which we can use withing Python as it now stands, I can think of: # Your original for i in range(10): if i == 5: continue body() # Compressed in case you don't like the 2-line continue for i in range(10): if i == 5: continue body() # Hacky use of a generator expression: for i in (x for x in range(10) if x != 5): body() # Custom generator - looks a bit silly for something this simple # but quite reasonable for many "real life" examples def gen(): for i in range(10): if i != 5 yield i for i in gen(): body() # Functional style for i in filter(range(10), lambda n: n != 5): body() There's a lot of opportunity here for picking a style that you're comfortable with. So it's hard to see this issue as being worth additional syntax. And conversely, if we did have the proposed syntax, I suspect there would be a tendency for certain types of developer to try to cram far too much logic into the "if" clause, when one of the above approaches would be more readable. Of course, you can write bad code with or without extra syntax, but it just feels to me like we'd end up with the proposed syntax being used more often in cases where it damages readability, than in cases where it's appropriate. That's not to say that there may not be a good justification for the proposal - just that your example isn't compelling to me. Paul From rosuav at gmail.com Sun Sep 11 08:10:47 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 11 Sep 2016 22:10:47 +1000 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: Message-ID: On Sun, Sep 11, 2016 at 9:59 PM, Paul Moore wrote: > # Hacky use of a generator expression: > for i in (x for x in range(10) if x != 5): > body() This is what I'd like to compare the proposal against. It's perfectly legal but pretty ugly - why should you nest two 'for' loops just for the sake of filtering? > # Functional style > for i in filter(range(10), lambda n: n != 5): > body() And this one is very close (I'd use i instead of n in the lambda function), but still fairly verbose. That said, though, filtered iteration isn't common enough to demand its own syntax IMO. I do it fairly often, but it's usually fine to just have a condition on a separate line. (I do use ": continue" rather than making it two lines.) ChrisA From dmoisset at machinalis.com Sun Sep 11 15:44:29 2016 From: dmoisset at machinalis.com (Daniel Moisset) Date: Sun, 11 Sep 2016 20:44:29 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: Both this discussion, PEP 505, and the one a year ago, tend to mix up 2 related but separate proposals: w (A) Add a None-coalescing operator (like C# a ?? b, what you would write in Python as "a or b" if it didn't have the falsy gotcha) (B) Add some None-aware navigation operators ( The "?.", "?()", "?[]", or what you would write in python as "a and a.attribute" if it didn't have the falsy gotcha) Both are things that can be already done in python, so the purpose here is to add some convenience (aka "syntax sugar"). IMO, this kind of syntax sugar proposals should be weighed with the frequency of the coding pattern where the sugar can be applied. And from the stats presented in PEP-505 (B) is one order of magnitude less usual than (A); that matches most of the examples I see in the threads and FWIW my personal experience. So, as a counterproposal I would like to suggest: * Add an "a ?? b" operator which is equivalent to "a if a is None else b" (but evaluating a once) * Do not add none-aware navigation; in the less usual scenario where you need to do it AND ALSO the "and" operator is not usable (it frequently is, given that by default classes are truish), well, you can use a ternary operator * I don't care if it's an alternate syntax (I'm surprised nobody suggested "||" which is also used in other languages for similar purpose) Would this satisfy most of the people requesting this? (and, would it satisfy the people making the decision?) On Sun, Sep 11, 2016 at 4:45 AM, Bruce Leban wrote: > > On Sat, Sep 10, 2016 at 6:02 PM, David Mertz wrote: > >> What I was getting at with "essentially" was that it would *do the same >> thing* that an AttributeError does. That is, if `x.foo` can't be evaluated >> (i.e. x doesn't have an attribute 'foo'), then access is informally "an >> error." The hypothetical "x?.foo" catches that "error" and substitutes a >> different value. The particular implementation under-the-hood is less >> important for most programmers who might use the construct (and I think >> documentation would actually give an informal equivalent as something >> similar to what I put in the NoneCoalesce class). >> > > That's not a good way to think about it. This new operator is only > checking for None and not actually checking for AttributeErrors. Consider: > > (3).x # AttributeError > {}.x # AttributeError > > None.x # AttributeError > > > (3)?.x # still AttributeError > > {}?.x # still AttributeError > > None?.x # None > > > And also: > > None.__class__ # > > None?.__class__ # None > > > And it's certainly not the case that those values don't accept any > attributes: > > (3).real # 3 > {}.values # > None.__class__ # > > --- Bruce > Check out my puzzle book and get it free here: > http://J.mp/ingToConclusionsFree (available on iOS) > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Daniel F. Moisset - UK Country Manager www.machinalis.com Skype: @dmoisset -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony at xtfx.me Sun Sep 11 16:15:41 2016 From: anthony at xtfx.me (C Anthony Risinger) Date: Sun, 11 Sep 2016 15:15:41 -0500 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: Message-ID: On Sep 11, 2016 7:11 AM, "Chris Angelico" wrote: > > That said, though, filtered iteration isn't common enough to demand > its own syntax IMO. I do it fairly often, I do it often enough to want this. When I first started writing Python this struck me as an inconsistency... if it's useful in comprehensions, why not regular loops? I realize comprehensions are all about construction of the list itself, but the parallel still exists. Also feels similar to guard statements. I'd love to see Python gain more pattern matching and destructuring features because they are wonderful in Erlang/Elixir. > but it's usually fine to > just have a condition on a separate line. (I do use ": continue" > rather than making it two lines.) FWIW, code I write or review would mandate this be two lines followed by a blank line, so 3 total. I require any abrupt change or termination in the current flow of control to be followed by a blank line so the reader clearly sees the possible jump (continue, break, and return especially). -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Sun Sep 11 16:20:35 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Sun, 11 Sep 2016 22:20:35 +0200 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: Message-ID: <91f07bc1-853a-33f0-ac20-fdd39f1aae37@mail.de> On 11.09.2016 22:15, C Anthony Risinger wrote: > > On Sep 11, 2016 7:11 AM, "Chris Angelico" > wrote: > > > > That said, though, filtered iteration isn't common enough to demand > > its own syntax IMO. I do it fairly often, > > I do it often enough to want this. > Same here. Most of the time it's just a single condition which disturbs the coherence of the loop body. > When I first started writing Python this struck me as an > inconsistency... if it's useful in comprehensions, why not regular > loops? I realize comprehensions are all about construction of the list > itself, but the parallel still exists. > > Also feels similar to guard statements. I'd love to see Python gain > more pattern matching and destructuring features because they are > wonderful in Erlang/Elixir. > > > but it's usually fine to > > just have a condition on a separate line. (I do use ": continue" > > rather than making it two lines.) > > FWIW, code I write or review would mandate this be two lines followed > by a blank line, so 3 total. I require any abrupt change or > termination in the current flow of control to be followed by a blank > line so the reader clearly sees the possible jump (continue, break, > and return especially). > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun Sep 11 17:42:22 2016 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 11 Sep 2016 16:42:22 -0500 Subject: [Python-ideas] [Python-Dev] Drastically improving list.sort() for lists of strings/ints In-Reply-To: References: Message-ID: [redirected from python-dev, to python-ideas; please send followups only to python-ideas] [Elliot Gorokhovsky ] > ... > TL;DR: Should I spend time making list.sort() detect if it is sorting > lexicographically and, if so, use a much faster algorithm? It will be fun to find out ;-) As Mark, and especially Terry, pointed out, a major feature of the current sort is that it can exploit many kinds of pre-existing order. As the paper you referenced says, "Realistic sorting problems are usually far from random." But, although they did run some tests against data with significant order, they didn't test against any algorithms _aiming_ at exploiting uniformity. Just against their radix sort variants, and against a quicksort. That's where it's likely next to impossible to guess in advance whether radix sort _will_ have a real advantage. All the kinds of order the current sort can exploit are far from obvious, because the mechanisms it employs are low-level & very general. For example, consider arrays created by this function, for even `n`: def bn(n): r = [None] * n r[0::2] = range(0, n//2) r[1::2] = range(n//2, n) return r Then, e.g., >>> bn(16) [0, 8, 1, 9, 2, 10, 3, 11, 4, 12, 5, 13, 6, 14, 7, 15] This effectively takes range(n), cuts it in half, and does a "perfect shuffle" on the two halves. You'll find nothing in the current code looking for this as a special case, but it nevertheless sorts such arrays in "close to" O(n) time, and despite that there's no natural run in the input longer than 2 elements. That said, I'd encourage you to write your code as a new list method at first, to make it easiest to run benchmarks. If that proves promising, then you can worry about how to make a single method auto-decide which algorithm to use. Also use the two-array version. It's easier to understand and to code, and stability is crucial now. The extra memory burden doesn't bother me - an array of C pointers consumes little memory compared to the memory consumed by the Python objects they point at. Most of all, have fun! :-) From guido at python.org Sun Sep 11 20:26:53 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Sep 2016 17:26:53 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sun, Sep 11, 2016 at 12:44 PM, Daniel Moisset wrote: > Both this discussion, PEP 505, and the one a year ago, tend to mix up 2 > related but separate proposals: I don't think there's that much of a mix-up. PEP 505 clearly describes each proposal separately and even gives a choice to accept or reject each one separately. > (A) Add a None-coalescing operator (like C# a ?? b, what you would write in > Python as "a or b" if it didn't have the falsy gotcha) https://www.python.org/dev/peps/pep-0505/#none-coalescing-operator > (B) Add some None-aware navigation operators ( The "?.", "?()", "?[]", or > what you would write in python as "a and a.attribute" if it didn't have the > falsy gotcha) https://www.python.org/dev/peps/pep-0505/#none-aware-attribute-access-operator > Both are things that can be already done in python, so the purpose here is > to add some convenience (aka "syntax sugar"). IMO, this kind of syntax sugar > proposals should be weighed with the frequency of the coding pattern where > the sugar can be applied. And from the stats presented in PEP-505 (B) is one > order of magnitude less usual than (A); that matches most of the examples I > see in the threads and FWIW my personal experience. I can't argue here yet. Honestly I looked at some examples; for those where it wasn't instantly clear that a None-coalescing operator would *not* help, I found it hard to figure out how to rewrite it. E.g. this one -- quick, is there a better way? return "Illegal Argument" + (self.message is not None and (": " + self.message) or "") I think the answer is, if we had both None-coalescing (??) and None-severing (!!) it could be written as follows: return "Illegal Argument" + ((self.message !! (": " + self.message)) ?? "") but it took me way too long to prove that to myself. > So, as a counterproposal I would like to suggest: > > * Add an "a ?? b" operator which is equivalent to "a if a is None else b" > (but evaluating a once) > * Do not add none-aware navigation; in the less usual scenario where you > need to do it AND ALSO the "and" operator is not usable (it frequently is, > given that by default classes are truish), well, you can use a ternary > operator Honestly the one thing that makes `?.` attractive is that it's easier than the None-coalescing and -severing operators to grasp at an intuitive level. If "foo.bar" raises "AttributeError: 'NoneType' object has no attribute 'foo'" then try again with "foo?.bar". It's surprising how often that will work! > * I don't care if it's an alternate syntax (I'm surprised nobody suggested > "||" which is also used in other languages for similar purpose) Interestingly, after analyzing the above example I desperately want to write it as return "Illegal Argument" + (self.message && (": " + self.message) || "") Note that I already know the relative priorities of && and ||, so I can drop a set of parentheses. > Would this satisfy most of the people requesting this? (and, would it > satisfy the people making the decision?) Personally, after the above example, I'm less excited about ??/!! or ||/&&, and more excited about `?.` -- so it doesn't satisfy me. -- --Guido van Rossum (python.org/~guido) From mertz at gnosis.cx Sun Sep 11 21:00:42 2016 From: mertz at gnosis.cx (David Mertz) Date: Sun, 11 Sep 2016 18:00:42 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sun, Sep 11, 2016 at 12:44 PM, Daniel Moisset wrote: > Both this discussion, PEP 505, and the one a year ago, tend to mix up 2 > related but separate proposals: > w > (A) Add a None-coalescing operator (like C# a ?? b, what you would write > in Python as "a or b" if it didn't have the falsy gotcha) > (B) Add some None-aware navigation operators ( The "?.", "?()", "?[]", or > what you would write in python as "a and a.attribute" if it didn't have the > falsy gotcha) > I readily confess that my initial comments here had a think-o about just what was being discussed, notwithstanding having also followed the discussion a year ago. I somehow had in mind that "None-coalescing" meant something like "get to None on failure" (i.e. *coalesce* to) I think the reason I thought wrong was because the second set of syntax really encourages that wrong way of thinking. I don't find `a ?? b` particularly ugly, even if I don't entirely want it. Similarly if it were spelled `a || b`. Both of those feel easy to conceptualize and teach as "another kind of short-circuiting, similar to boolean operators." Maybe an actual word is better? `a ifNone b`? `a failto b`? Falsey is similar to None-like (I know it's actually None-identical), so the shortcut idea is familiar. So I'm only -0 on that idea. However, the None-aware navigation operators suggest something very different. `a?.foo` MIGHT BE an attribute access, or it might not be. Likewise, `a?[x]` might be an item get or it might not be. And `a?(y)` might or might not be a function call. Obviously, if some or all of those forms are not added they are simply syntax errors. What operation happens when we use these syntax forms is "it depends"... that answer feels less straightforward than "We call `a.__getattr__('foo')`" (or `a.__getitem__(x)` or `a.__call__(y)`, as the case may be). Even if there were some characters I found attractive for these "it depends" operations, that would introduce a needless non-uniformity into the semantics of Python. In contrast, a fully spelled out ternary doesn't give me this uneasiness. Of course the following expression might be one thing or the other, but that's more obvious in the ternary syntax (also more general, you can put *any* values on both branches of the ternary): None if a is None else a.foo -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sun Sep 11 21:06:47 2016 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 12 Sep 2016 02:06:47 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: <378993d5-a2b9-0624-d9c8-6ce9a0ca3631@mrabarnett.plus.com> On 2016-09-12 01:26, Guido van Rossum wrote: > On Sun, Sep 11, 2016 at 12:44 PM, Daniel Moisset > wrote: >> Both this discussion, PEP 505, and the one a year ago, tend to mix up 2 >> related but separate proposals: > > I don't think there's that much of a mix-up. PEP 505 clearly describes > each proposal separately and even gives a choice to accept or reject > each one separately. > >> (A) Add a None-coalescing operator (like C# a ?? b, what you would write in >> Python as "a or b" if it didn't have the falsy gotcha) > > https://www.python.org/dev/peps/pep-0505/#none-coalescing-operator > >> (B) Add some None-aware navigation operators ( The "?.", "?()", "?[]", or >> what you would write in python as "a and a.attribute" if it didn't have the >> falsy gotcha) > > https://www.python.org/dev/peps/pep-0505/#none-aware-attribute-access-operator > >> Both are things that can be already done in python, so the purpose here is >> to add some convenience (aka "syntax sugar"). IMO, this kind of syntax sugar >> proposals should be weighed with the frequency of the coding pattern where >> the sugar can be applied. And from the stats presented in PEP-505 (B) is one >> order of magnitude less usual than (A); that matches most of the examples I >> see in the threads and FWIW my personal experience. > > I can't argue here yet. Honestly I looked at some examples; for those > where it wasn't instantly clear that a None-coalescing operator would > *not* help, I found it hard to figure out how to rewrite it. E.g. this > one -- quick, is there a better way? > > return "Illegal Argument" + (self.message is not None and (": " + > self.message) or "") > I think this is better: return "Illegal Argument" + ("" if self.message is None else ": " + self.message) but it doesn't require a None-coalescing operator. From guido at python.org Sun Sep 11 21:11:29 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Sep 2016 18:11:29 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sun, Sep 11, 2016 at 6:00 PM, David Mertz wrote: > None if a is None else a.foo This is the crux of the matter to me. It's just too verbose, and the `if` and `else` keywords are lost in the noise of all the other words on the line. Plus the big win when it applies) is that if `a` is in fact something more complex, like `f(a)`, repeating it twice sounds like a performance penalty, and that's where `f(a)?.foo` really shines. -- --Guido van Rossum (python.org/~guido) From mertz at gnosis.cx Sun Sep 11 21:22:23 2016 From: mertz at gnosis.cx (David Mertz) Date: Sun, 11 Sep 2016 18:22:23 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: On Sun, Sep 11, 2016 at 6:11 PM, Guido van Rossum wrote: > On Sun, Sep 11, 2016 at 6:00 PM, David Mertz wrote: > > None if a is None else a.foo > > This is the crux of the matter to me. It's just too verbose, and the > `if` and `else` keywords are lost in the noise of all the other words > on the line. Plus the big win when it applies) is that if `a` is in > fact something more complex, like `f(a)`, repeating it twice sounds > like a performance penalty, and that's where `f(a)?.foo` really > shines. > The non-repetition is certain a plus, I readily confess. It's not only performance even; `f()` might not be a pure function. Silly example: >>> def f(a): .... if random() < .01: .... return None .... class V: pass .... v = V() .... v.foo = random()*a .... return v -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Sun Sep 11 21:53:04 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 11 Sep 2016 21:53:04 -0400 Subject: [Python-ideas] An exciting opportunity to update PEP 3156 Message-ID: Guido and I are looking for help with PEP 3156. Currently it's out of sync with the docs. It would be cool if someone could volunteer and update it (at least make sure that all APIs are up to date). Yury From ncoghlan at gmail.com Mon Sep 12 02:25:11 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Sep 2016 16:25:11 +1000 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: Message-ID: On 11 September 2016 at 19:36, Dominik Gresch wrote: > Hi, > > I've recently found myself writing code similar to this: > > for i in range(10): > if i == 5: > continue > "body" > > which I find a bit ugly. Obviously the same could be written as > > for i in range(10): > if i != 5: > "body" > > but here you would have to look at the end of the body to see if something > happens when i==5. > So I asked myself if a syntax as follows would be possible: > > for i in range(10) if i != 5: > body > > Personally, I find this extremely intuitive since this kind of if-statement > is already present in list comprehensions. > > What is your opinion on this? Sorry if this has been discussed before -- I > didn't find anything in the archives. Generally speaking, we only add new syntax in cases where we're prepared to say "In all cases where the new syntax applies, it should be used in preference to existing alternative spellings". Special casing a single "if-continue" in a loop body doesn't meet that (deliberately high) bar, with Paul Moore's email going into some more detail on the specifics of that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From andrew.svetlov at gmail.com Mon Sep 12 02:54:20 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Mon, 12 Sep 2016 06:54:20 +0000 Subject: [Python-ideas] An exciting opportunity to update PEP 3156 In-Reply-To: References: Message-ID: Should Task.current_task() be declared as a part of supported public API? Should we declare that every asyncio coroutine is executed in a task context? On Mon, Sep 12, 2016 at 4:53 AM Yury Selivanov wrote: > Guido and I are looking for help with PEP 3156. Currently it's out of > sync with the docs. It would be cool if someone could volunteer and > update it (at least make sure that all APIs are up to date). > > Yury > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Mon Sep 12 03:05:01 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 12 Sep 2016 09:05:01 +0200 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> Message-ID: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> I messed up my answer and replied to one person instead of the list, so I'll post it again. There is also an alternative to this operator, and it's allowing a shortcut to do: try: val = do_thing() except ThingError: val = "default" In the form of: val = do_thing() except ThingError: "default" I was debated, and rejected, but I feel like mentioning it again because it has some strong benefits. First, it handles the null coalescing very quite well: val = obj.foo.bar.hey except AttributeError: "default" But it also can deal with many common operations in Python without the need to add more operators or variants: val = my_list[0] except IndexError: "default" val = iterable[0] except TypeError: next(iter(iterable)) val = int(param) except ValueError: man.nan It's quite readable, in the same vein of val = foo if bar else "default", but also familiar since it's using known keyword. And it doesn't require to add a new operator support in the parser. Another serious benefits is that it fits current use cases, AND futur use cases. Indeed, since it leverages Python exception mechanism, any lib implementing a clean error model can immediately let the users benefit from it without having to implement, tests and document lots of helper methods with "default" keywords and the like, while not being limited to a set of values some operators would only care about, such as None. Plus, since EAFP is a popular and handy pattern, it makes sense. At last, it has the same characteristic as the null coalescing operator: it's lazy, and hence has a small performance interest too compared to functional equivalent. Did I mention it's also easy to expend to a full try/except when you need something more complicated ? And then you benefit from else and finally immediately. I already have many code that would benefit from such a syntax, and I'd like to hear again what you think about it. Le 11/09/2016 ? 21:44, Daniel Moisset a ?crit : > Both this discussion, PEP 505, and the one a year ago, tend to mix up 2 > related but separate proposals: > w > (A) Add a None-coalescing operator (like C# a ?? b, what you would write > in Python as "a or b" if it didn't have the falsy gotcha) > (B) Add some None-aware navigation operators ( The "?.", "?()", "?[]", > or what you would write in python as "a and a.attribute" if it didn't > have the falsy gotcha) > > Both are things that can be already done in python, so the purpose here > is to add some convenience (aka "syntax sugar"). IMO, this kind of > syntax sugar proposals should be weighed with the frequency of the > coding pattern where the sugar can be applied. And from the stats > presented in PEP-505 (B) is one order of magnitude less usual than (A); > that matches most of the examples I see in the threads and FWIW my > personal experience. > > So, as a counterproposal I would like to suggest: > > * Add an "a ?? b" operator which is equivalent to "a if a is None else > b" (but evaluating a once) > * Do not add none-aware navigation; in the less usual scenario where you > need to do it AND ALSO the "and" operator is not usable (it frequently > is, given that by default classes are truish), well, you can use a > ternary operator > * I don't care if it's an alternate syntax (I'm surprised nobody > suggested "||" which is also used in other languages for similar purpose) > > Would this satisfy most of the people requesting this? (and, would it > satisfy the people making the decision?) > > > > On Sun, Sep 11, 2016 at 4:45 AM, Bruce Leban > wrote: > > > On Sat, Sep 10, 2016 at 6:02 PM, David Mertz > wrote: > > What I was getting at with "essentially" was that it would *do > the same thing* that an AttributeError does. That is, if > `x.foo` can't be evaluated (i.e. x doesn't have an attribute > 'foo'), then access is informally "an error." The hypothetical > "x?.foo" catches that "error" and substitutes a different > value. The particular implementation under-the-hood is less > important for most programmers who might use the construct (and > I think documentation would actually give an informal equivalent > as something similar to what I put in the NoneCoalesce class). > > > That's not a good way to think about it. This new operator is only > checking for None and not actually checking for AttributeErrors. > Consider: > > (3).x # AttributeError > {}.x # AttributeError > > None.x # AttributeError > > > (3)?.x # still AttributeError > > {}?.x # still AttributeError > > None?.x # None > > > And also: > > None.__class__ # > > None?.__class__ # None > > > And it's certainly not the case that those values don't accept any > attributes: > > (3).real # 3 > {}.values # > None.__class__ # > > --- Bruce > Check out my puzzle book and get it free here: > http://J.mp/ingToConclusionsFree > (available on iOS) > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > -- > Daniel F. Moisset - UK Country Manager > www.machinalis.com > Skype: @dmoisset > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From ethan at stoneleaf.us Mon Sep 12 03:27:00 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 12 Sep 2016 00:27:00 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: <57D658C4.3000403@stoneleaf.us> On 09/12/2016 12:05 AM, Michel Desmoulin wrote: > There is also an alternative to this operator, and it's allowing a > shortcut to do: > > try: > val = do_thing() > except ThingError: > val = "default" > > In the form of: > > val = do_thing() except ThingError: "default" > > I was debated, and rejected, but I feel like mentioning it again because > it has some strong benefits. +1 There are many places in my code where this would clean things up a bit. Not having it is like not having list comps or not having the ternary if-else -- possible, but it would be much nicer to have it. -- ~Ethan~ From levkivskyi at gmail.com Mon Sep 12 06:01:53 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Mon, 12 Sep 2016 12:01:53 +0200 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On 12 September 2016 at 09:05, Michel Desmoulin wrote: > In the form of: > > val = do_thing() except ThingError: "default" > > [...] > > But it also can deal with many common operations in Python without the > need to add more operators or variants: > > val = my_list[0] except IndexError: "default" > > val = iterable[0] except TypeError: next(iter(iterable)) > > val = int(param) except ValueError: man.nan > I like this idea, I would propose a (maybe crazy) addition to it. What about a special exception NoneError, that will catch TypeError, AttributeError etc. but only when it was caused by None(), None.attr, None[1], etc. With this one can write: x = a.b()[0] except NoneError: 'default' without a risk of catching other (unrelated) exceptions. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From greschd at gmx.ch Mon Sep 12 05:52:06 2016 From: greschd at gmx.ch (Dominik Gresch) Date: Mon, 12 Sep 2016 11:52:06 +0200 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: Message-ID: Ok, I guess it's time to end this thread. Thank you all for your answers and the constructive discussion. Best, Dominik On 12.09.2016 08:25, Nick Coghlan wrote: > On 11 September 2016 at 19:36, Dominik Gresch wrote: >> Hi, >> >> I've recently found myself writing code similar to this: >> >> for i in range(10): >> if i == 5: >> continue >> "body" >> >> which I find a bit ugly. Obviously the same could be written as >> >> for i in range(10): >> if i != 5: >> "body" >> >> but here you would have to look at the end of the body to see if something >> happens when i==5. >> So I asked myself if a syntax as follows would be possible: >> >> for i in range(10) if i != 5: >> body >> >> Personally, I find this extremely intuitive since this kind of if-statement >> is already present in list comprehensions. >> >> What is your opinion on this? Sorry if this has been discussed before -- I >> didn't find anything in the archives. > Generally speaking, we only add new syntax in cases where we're > prepared to say "In all cases where the new syntax applies, it should > be used in preference to existing alternative spellings". > > Special casing a single "if-continue" in a loop body doesn't meet that > (deliberately high) bar, with Paul Moore's email going into some more > detail on the specifics of that. > > Cheers, > Nick. > From rob.cliffe at btinternet.com Mon Sep 12 06:52:39 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 12 Sep 2016 11:52:39 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On 12/09/2016 08:05, Michel Desmoulin wrote: > I messed up my answer and replied to one person instead of the list, so > I'll post it again. > > There is also an alternative to this operator, and it's allowing a > shortcut to do: > > try: > val = do_thing() > except ThingError: > val = "default" > > In the form of: > > val = do_thing() except ThingError: "default" > > I was debated, and rejected, but I feel like mentioning it again because > it has some strong benefits. > > First, it handles the null coalescing very quite well: > > val = obj.foo.bar.hey except AttributeError: "default" > > But it also can deal with many common operations in Python without the > need to add more operators or variants: > > val = my_list[0] except IndexError: "default" > > val = iterable[0] except TypeError: next(iter(iterable)) > > val = int(param) except ValueError: man.nan > > It's quite readable, in the same vein of val = foo if bar else > "default", but also familiar since it's using known keyword. And it > doesn't require to add a new operator support in the parser. > > Another serious benefits is that it fits current use cases, AND futur > use cases. Indeed, since it leverages Python exception mechanism, any > lib implementing a clean error model can immediately let the users > benefit from it without having to implement, tests and document lots of > helper methods with "default" keywords and the like, while not being > limited to a set of values some operators would only care about, such as > None. > > Plus, since EAFP is a popular and handy pattern, it makes sense. > > At last, it has the same characteristic as the null coalescing operator: > it's lazy, and hence has a small performance interest too compared to > functional equivalent. > > Did I mention it's also easy to expend to a full try/except when you > need something more complicated ? And then you benefit from else and > finally immediately. > > I already have many code that would benefit from such a syntax, and I'd > like to hear again what you think about it. > > +1, you're preaching to the converted. This is PEP 463, "Exception-catching expressions". Except that the PEP required parentheses around a whole exception-catching expression, for reasons that are not clear to me, i.e. val = (my_list[0] except IndexError: "default") rather than val = my_list[0] except IndexError: "default" Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Mon Sep 12 09:13:06 2016 From: random832 at fastmail.com (Random832) Date: Mon, 12 Sep 2016 09:13:06 -0400 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: <1473685986.2059521.723041985.67F387AF@webmail.messagingengine.com> On Mon, Sep 12, 2016, at 06:01, Ivan Levkivskyi wrote: > I like this idea, I would propose a (maybe crazy) addition to it. What > about a special exception NoneError, that will catch TypeError, > AttributeError etc. but only when it was caused by None(), > None.attr, None[1], etc. With this one can write: > > x = a.b()[0] except NoneError: 'default' > > without a risk of catching other (unrelated) exceptions. Unless there's another None somewhere inside b or b().__getitem__. From rob.cliffe at btinternet.com Mon Sep 12 10:03:20 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 12 Sep 2016 15:03:20 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On 12/09/2016 11:01, Ivan Levkivskyi wrote: > > > On 12 September 2016 at 09:05, Michel Desmoulin > > wrote: > > In the form of: > > val = do_thing() except ThingError: "default" > > [...] > > > But it also can deal with many common operations in Python without the > need to add more operators or variants: > > val = my_list[0] except IndexError: "default" > > val = iterable[0] except TypeError: next(iter(iterable)) > > val = int(param) except ValueError: man.nan > > > I like this idea, I would propose a (maybe crazy) addition to it. > What about a special exception NoneError, that will catch TypeError, > AttributeError etc. but only when it was caused by None(), > None.attr, None[1], etc. With this one can write: > > x = a.b()[0] except NoneError: 'default' > > without a risk of catching other (unrelated) exceptions. > > -- > Ivan Assuming you can't break existing code that already traps TypeError, AttributeError, etc., I don't see how you can do this without having separated kinds of NoneError which were subclasses of TypeError, AttributeError, etc. Rob Cliffe > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Sep 12 10:09:09 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Sep 2016 00:09:09 +1000 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On Tue, Sep 13, 2016 at 12:03 AM, Rob Cliffe wrote: > Assuming you can't break existing code that already traps TypeError, > AttributeError, etc., I don't see how you can do this without > having separated kinds of NoneError which were subclasses of TypeError, > AttributeError, etc. class NoneError(Exception): pass class TypeNoneError(TypeError, NoneError): pass class AttributeNoneError(AttributeError, NoneError): pass Now you can catch NoneError to catch None.xyz, or AttributeError to catch foo.xyz for any foo. I don't think it's good, but it's possible. ChrisA From guido at python.org Mon Sep 12 11:37:11 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Sep 2016 08:37:11 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: For the record, I still really don't like PEP 463. We should strive to catch fewer exceptions, not make it easier to catch them. On Mon, Sep 12, 2016 at 7:09 AM, Chris Angelico wrote: > On Tue, Sep 13, 2016 at 12:03 AM, Rob Cliffe wrote: >> Assuming you can't break existing code that already traps TypeError, >> AttributeError, etc., I don't see how you can do this without >> having separated kinds of NoneError which were subclasses of TypeError, >> AttributeError, etc. > > class NoneError(Exception): pass > class TypeNoneError(TypeError, NoneError): pass > class AttributeNoneError(AttributeError, NoneError): pass > > Now you can catch NoneError to catch None.xyz, or AttributeError to > catch foo.xyz for any foo. > > I don't think it's good, but it's possible. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From guido at python.org Mon Sep 12 15:45:44 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Sep 2016 12:45:44 -0700 Subject: [Python-ideas] An exciting opportunity to update PEP 3156 In-Reply-To: References: Message-ID: On Sun, Sep 11, 2016 at 11:54 PM, Andrew Svetlov wrote: > Should Task.current_task() be declared as a part of supported public API? Sure. > Should we declare that every asyncio coroutine is executed in a task > context? What importance does this have? Task.get_current_task() can return None. -- --Guido van Rossum (python.org/~guido) From ericsnowcurrently at gmail.com Mon Sep 12 16:47:43 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 12 Sep 2016 14:47:43 -0600 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On Mon, Sep 12, 2016 at 1:05 AM, Michel Desmoulin wrote: > There is also an alternative to this operator, and it's allowing a > shortcut to do: > > try: > val = do_thing() > except ThingError: > val = "default" > > In the form of: > > val = do_thing() except ThingError: "default" Note that there's a subtle difference here when multiple lookups are involved. Given: def f(spam): return spam().eggs().ham With null-coalescing: def f(spam): return spam()?.eggs()?.ham This is roughly equivalent to: def f(spam): _spam = spam() try: eggs = _spam.eggs except AttributeError: return None _eggs = eggs() try: return _eggs.ham except AttributeError: return None With PEP 463 it doesn't work out so well. The "obvious" spelling would be: def f(spam): return (spam().eggs().ham except AttributeError: None) This is roughly equivalent to: def f(spam): try: return spam().eggs().ham except AttributeError: return None Note how it's different. For one thing, it could mask AttributeError coming from the calls. For another, you no longer explicitly identify which lookups to handle. I would expect both to lead to subtle bugs, whereas with null-coalescing you don't have those problems. -eric From p.f.moore at gmail.com Mon Sep 12 17:13:29 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Sep 2016 22:13:29 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On 12 September 2016 at 21:47, Eric Snow wrote: > Note that there's a subtle difference here when multiple lookups are > involved. Given: > > def f(spam): > return spam().eggs().ham > > With null-coalescing: > > def f(spam): > return spam()?.eggs()?.ham > > This is roughly equivalent to: > > def f(spam): > _spam = spam() > try: > eggs = _spam.eggs > except AttributeError: > return None > _eggs = eggs() > try: > return _eggs.ham > except AttributeError: > return None >From previous explanations in this thread, that's *not* the behaviour at all. If I understand the proposal, f is actually intended to be equivalent to: def f(spam): spam_val = spam() if spam_val is None: return None eggs_val = spam_val.eggs() if eggs_val is None: return None return eggs_val.ham Personally, I find it pretty worrying that there's this persistent confusion over what the proposed syntax means. Sure, it's explained in the PEP and proposal, but if it gets implemented it'll be in the docs. And yet people don't seem to read any of those - and their intuition of what the construct does is wrong. IMO, the syntax may well be useful, but if we don't address the problem that what people *think* it means isn't what it actually means, then it'll do more harm than good. Paul From ethan at stoneleaf.us Mon Sep 12 17:28:24 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 12 Sep 2016 14:28:24 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: <57D71DF8.8060305@stoneleaf.us> On 09/12/2016 02:13 PM, Paul Moore wrote: > On 12 September 2016 at 21:47, Eric Snow wrote: >> Note that there's a subtle difference here when multiple lookups are >> involved. Given: >> >> def f(spam): >> return spam().eggs().ham >> >> With null-coalescing: >> >> def f(spam): >> return spam()?.eggs()?.ham >> >> This is roughly equivalent to: >> >> def f(spam): >> _spam = spam() >> try: >> eggs = _spam.eggs >> except AttributeError: >> return None >> _eggs = eggs() >> try: >> return _eggs.ham >> except AttributeError: >> return None > > From previous explanations in this thread, that's *not* the behaviour at all. > > If I understand the proposal, f is actually intended to be equivalent to: > > def f(spam): > spam_val = spam() > if spam_val is None: > return None > eggs_val = spam_val.eggs() > if eggs_val is None: > return None > return eggs_val.ham > > Personally, I find it pretty worrying that there's this persistent > confusion over what the proposed syntax means. Sure, it's explained in > the PEP and proposal, but if it gets implemented it'll be in the docs. > And yet people don't seem to read any of those - and their intuition > of what the construct does is wrong. > > IMO, the syntax may well be useful, but if we don't address the > problem that what people *think* it means isn't what it actually > means, then it'll do more harm than good. Perhaps the bumper-sticker explanation is: None-coalescing is not to /recover/ from an AttributeError, but to _prevent it_ in the first place -- but only when the base object is already None. Okay, several bumper stickers. ;) -- ~Ethan~ From ericsnowcurrently at gmail.com Mon Sep 12 17:45:36 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 12 Sep 2016 15:45:36 -0600 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On Mon, Sep 12, 2016 at 3:13 PM, Paul Moore wrote: > From previous explanations in this thread, that's *not* the behaviour at all. Gah, you're right. I knew better too. The diversion back to PEP 463 crossed some wires in my brain. :) > > If I understand the proposal, f is actually intended to be equivalent to: > > def f(spam): > spam_val = spam() > if spam_val is None: > return None > eggs_val = spam_val.eggs() > if eggs_val is None: > return None > return eggs_val.ham ...which is even less related to PEP 463. > > Personally, I find it pretty worrying that there's this persistent > confusion over what the proposed syntax means. Sure, it's explained in > the PEP and proposal, but if it gets implemented it'll be in the docs. > And yet people don't seem to read any of those - and their intuition > of what the construct does is wrong. The tricky bit is the syntactic association with attribute lookup. However, I don't think it's quite so confusing. As noted above, my confusion was mostly due to the recent mention of PEP 463. -eric From ned at nedbatchelder.com Mon Sep 12 18:05:34 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 12 Sep 2016 18:05:34 -0400 Subject: [Python-ideas] An exciting opportunity to update PEP 3156 In-Reply-To: References: Message-ID: <63013323-35cc-3fbb-07b9-5e05deed2910@nedbatchelder.com> I'm curious why it's important to update the PEP? I would think the docs should be the official source of truth about the library, and the PEP should be a record of the decisions that lead to the library. Trying to keep both in sync just seems like extra work. Why not put a notice at the top of the PEP indicating that it was accurate when it was accepted, but that changed have happened since then, see the docs for details? --Ned. On 9/11/16 9:53 PM, Yury Selivanov wrote: > Guido and I are looking for help with PEP 3156. Currently it's out of > sync with the docs. It would be cool if someone could volunteer and > update it (at least make sure that all APIs are up to date). > > Yury > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From senthil at uthcode.com Mon Sep 12 18:12:59 2016 From: senthil at uthcode.com (Senthil Kumaran) Date: Mon, 12 Sep 2016 15:12:59 -0700 Subject: [Python-ideas] An exciting opportunity to update PEP 3156 In-Reply-To: <63013323-35cc-3fbb-07b9-5e05deed2910@nedbatchelder.com> References: <63013323-35cc-3fbb-07b9-5e05deed2910@nedbatchelder.com> Message-ID: On Mon, Sep 12, 2016 at 3:05 PM, Ned Batchelder wrote: > Why not put a notice at > the top of the PEP indicating that it was accurate when it was accepted, > but that changed have happened since then, see the docs for details? > I see PEP as the design document and docs as user-friendly documentation. It helps to keep both of them up to date and in sync. If an "Accepted" PEP is not updated, then folks landing on it will feel misguided or confused if the read the docs later. Presenting a cohesive story here is probably more work than simply keeping them in sync. The history of decision changes is tracked in VCS, if that is desirable. -- Senthil -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Sep 12 18:15:04 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Sep 2016 15:15:04 -0700 Subject: [Python-ideas] An exciting opportunity to update PEP 3156 In-Reply-To: <63013323-35cc-3fbb-07b9-5e05deed2910@nedbatchelder.com> References: <63013323-35cc-3fbb-07b9-5e05deed2910@nedbatchelder.com> Message-ID: Part of it is that the official docs often aren't all that precise compared to the PEP. Part of it is that the PEP is still marked provisional, and we wish to declare it final as of 3.6.0 (or as of 3.6b1, depending on who you talk to). So I'd like to see the "final" version correspond with the API as of that point. But I agree it's not super important. On Mon, Sep 12, 2016 at 3:05 PM, Ned Batchelder wrote: > I'm curious why it's important to update the PEP? I would think the docs > should be the official source of truth about the library, and the PEP > should be a record of the decisions that lead to the library. Trying to > keep both in sync just seems like extra work. Why not put a notice at > the top of the PEP indicating that it was accurate when it was accepted, > but that changed have happened since then, see the docs for details? > > --Ned. > > > On 9/11/16 9:53 PM, Yury Selivanov wrote: >> Guido and I are looking for help with PEP 3156. Currently it's out of >> sync with the docs. It would be cool if someone could volunteer and >> update it (at least make sure that all APIs are up to date). >> >> Yury >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From rob.cliffe at btinternet.com Mon Sep 12 20:45:10 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 13 Sep 2016 01:45:10 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On 12/09/2016 16:37, Guido van Rossum wrote: > For the record, I still really don't like PEP 463. We should strive to > catch fewer exceptions, not make it easier to catch them. Can you please clarify what you are saying in the last sentence? The first time I read it, my brain parsed it as "People should raise fewer exceptions, i.e. use LBYL less". Later it seemed more likely that you meant "The set of exceptions we catch in a specific piece of code should not be too large; we should only catch those exceptions that we really mean to handle". But perhaps you mean something else altogether? Also I don't see the connection between the first sentence and the second. Exception-catching expressions as per PEP 463 just give you a more concise way of of doing something you can do with try+except. Do you mean it would make it easier to catch exceptions because you have to type fewer characters, and that this would be a bad thing? Thanks, Rob Cliffe From ethan at stoneleaf.us Mon Sep 12 21:14:33 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 12 Sep 2016 18:14:33 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: <57D752F9.6080700@stoneleaf.us> On 09/12/2016 08:37 AM, Guido van Rossum wrote: > For the record, I still really don't like PEP 463. We should strive to > catch fewer exceptions, not make it easier to catch them. I certainly agree with the first part, slightly reworded: we should strive to generate fewer exceptions that we have to catch. I disagree with the second part: being able to condense four lines of code (1 for try, 1 for except, 1 for the attempt, and 1 for recovery) in to one line of code seems like a win. I know I find it frustrating when my choice is between the 4-line boiler-plate try/except, or an equally verbose and ugly multi-line non-exception generating stanza. Anyway, my two cents worth. If you collect them all for this subject I think I owe you a dime. ;) -- ~Ethan~ From rob.cliffe at btinternet.com Mon Sep 12 21:14:20 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 13 Sep 2016 02:14:20 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: <895df305-c707-a093-5f22-1b258fb67889@btinternet.com> On 13/09/2016 01:45, Rob Cliffe wrote: > > > On 12/09/2016 16:37, Guido van Rossum wrote: >> For the record, I still really don't like PEP 463. We should strive to >> catch fewer exceptions, not make it easier to catch them. > Can you please clarify what you are saying in the last sentence? > The first time I read it, my brain parsed it as "People should raise > fewer exceptions, i.e. use LBYL less". Typo: I meant EAFP not LBYL - sorry! > Later it seemed more likely that you meant "The set of exceptions we > catch in a specific piece of code should not be too large; we should > only catch those exceptions that we really mean to handle". > But perhaps you mean something else altogether? > > Also I don't see the connection between the first sentence and the > second. Exception-catching expressions as per PEP 463 just give you a > more concise way of of doing something you can do with try+except. Do > you mean it would make it easier to catch exceptions because you have > to type fewer characters, and that this would be a bad thing? > Thanks, > Rob Cliffe > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From tim.peters at gmail.com Mon Sep 12 23:18:03 2016 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 12 Sep 2016 22:18:03 -0500 Subject: [Python-ideas] [Python-Dev] Drastically improving list.sort() for lists of strings/ints In-Reply-To: References: Message-ID: [Elliot Gorokhovsky ] > Wow, Tim himself! And Elliot himself! It's a party :-) > Regarding performance on semi-ordered data: we'll have to > benchmark to see, but intuitively I imagine radix would meet Timsort > because verifying that a list of strings is sorted takes Omega(nw) > (which gives a lower bound on Timsort), where w is the word length. Don't take that stuff so seriously. The number of character comparisons needed when comparing two strings is _typically_ small. Consecutive corresponding characters are compared only until the first non-equal pair is found. For example, if the alphabet is 256 characters, for random strings it only requires one character comparison 255 of 256 times on average (because there's only 1 chance in 256 that the first characters _are_ equal). > Radix sort is Theta(nw). So at least asymptotically it checks out. Analogously, most-significant-byte-first radix sort stops on the pass after the longest common prefix is processed. But the real devils are in the details. More on that below. > I think if one uses the two-array algorithm, other semi-sortings can also > be exploited, since the items get placed into their respective buckets > in the order in which they appear in the list. So, for the example you gave, > one pass would sort it correctly Can't know that unless you specify which algorithm you have in mind. If you're talking about the one in the paper, _any_ sequence with values all from range(256) will be sorted in the first pass, because each value would get its own bucket. > (since the list has the property if x_1 and x_2 are in bucket b, x1 comes > before x2 in the list, so x1 will also come before x2 in the bucket. Except > possibly for one "border bucket" that includes n/2). And then > it would just be Theta(nw/b) in each bucket to verify sorted. I don't think you want to go there. If you're keen on radix sorting, stick to radix sorting until it's as good as you can make it. "To verify sorted" requires the (potentially) arbitrarily expensive kinds of comparisons you're trying to _eliminate_. That's not saying there's no possible advantage from doing some of each; it's warning that pursuing every off-target idea that comes up will leave you spinning in circles. > I mean honestly the cool thing about radix is that the best case for > Timsort on strings, Omega(nw), is the worst case for radix! "Details": I wrote a prototype radix sort along the lines of the paper's 2-array version, in Python. When there's "a real" O() improvement to be had, one can usually demonstrate it by a Python program incorporating the improvement even though it's fighting against (unimproved) C code. For example, I helped someone here craft a Python radix sort for integers that runs faster than list.sort(), although it required a list over 10,000,000 elements and a radix of at least 4096 ;-) http://stackoverflow.com/questions/20207791/pushing-radix-sort-and-python-to-its-limits However, I'm not having that kind of luck with strings. Here's a detail that bites: as above, string comparisons _usually_ get out fast, after comparing just a few characters. So to construct a bad case for list.sort() in this respect, I want lots of strings with a long common prefix. For example, suppose we have a list of a million strings each of which starts with "A"*20000 (20 thousand "A"). String comparisons are then very expensive (each one requires at least 20001 character comparisons to resolve). But that also makes the radix sort excruciatingly slow! On the first pass, it copies the million elements, puts them all in the same bucket, then copies them all back to exactly where they started from. Pass 2, exactly the same, but looking at the 2nd occurrence of "A" in each string. Ditto for pass 3, pass 4, and ... pass 20,000. By then we've copied a million elements 40,000 times - but still have exactly the same array we started with. The data-copying costs are killing it (yes, it's only copying pointers, but 40 billion pointer copies aren't cheap - and on my 64-bit box, amounts to copying 320 billion bytes). On the same data, list.sort() is doing a great many character comparisons, but doing _far_ less pointer copying, and that leaves it running circles around the radix sort - list.sort() can't copy pointers more than about N*log2(N) times, and log2(1000000) is much smaller than 40,000. To be complete, I'll attach the code I used. Note that the paper is only concerned with "C strings": 0 bytes are special, acting as string terminators. 0 bytes aren't special in Python, so we need 257 buckets. I'm also using Python 3, where strings are always Unicode, and code points (`ord()` results) can be as large as 1,114,111. So this radix sort can't work directly on Python 3 strings; instead it expects a list of `bytes` (or `bytearray`) objects. def rsort(xs): count = [0] * 257 bindex = [None] * 257 stack = [(0, len(xs), 0)] # start index, length, byte index push = stack.append pop = stack.pop while stack: assert all(c == 0 for c in count) si, n, bi = pop() txs = xs[si: si + n] #if n < 30: # xs[si: si + n] = sorted(txs) # continue minbyte, maxbyte = 255, 0 for x in txs: b = x[bi] if bi < len(x) else -1 count[b] += 1 if count[b] == 1 and b >= 0: if b < minbyte: minbyte = b if b > maxbyte: maxbyte = b bindex[-1] = si sofar = si + count[-1] count[-1] = 0 for b in range(minbyte, maxbyte + 1): c = count[b] if c == 0: continue if c > 1: push((sofar, c, bi + 1)) bindex[b] = sofar sofar += c count[b] = 0 assert sofar == si + n for x in txs: b = x[bi] if bi < len(x) else -1 xs[bindex[b]] = x bindex[b] += 1 A little utility to generate "random" input may also be helpful: def build(n, m): from random import randrange xs = [] for _ in range(n): xs.append(bytes(randrange(256) for _ in range(randrange(m)))) return xs Then, e.g., >>> xs = build(1000000, 20) >>> ys = xs[:] >>> rsort(xs) >>> assert xs == sorted(ys) From guido at python.org Mon Sep 12 23:43:32 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Sep 2016 20:43:32 -0700 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On Mon, Sep 12, 2016 at 5:45 PM, Rob Cliffe wrote: > > > On 12/09/2016 16:37, Guido van Rossum wrote: >> >> For the record, I still really don't like PEP 463. We should strive to >> catch fewer exceptions, not make it easier to catch them. > > Can you please clarify what you are saying in the last sentence? > The first time I read it, my brain parsed it as "People should raise fewer > exceptions, i.e. use LBYL less". (If that means "Leap Before You Look", yes. :-) > Later it seemed more likely that you meant "The set of exceptions we catch > in a specific piece of code should not be too large; we should only catch > those exceptions that we really mean to handle". > But perhaps you mean something else altogether? > > Also I don't see the connection between the first sentence and the second. > Exception-catching expressions as per PEP 463 just give you a more concise > way of of doing something you can do with try+except. Do you mean it would > make it easier to catch exceptions because you have to type fewer > characters, and that this would be a bad thing? Yeah, that's exactly my point. PEP 463 gives you a shorter way to catch an exception, so it gives you less motivation to find a way to write your code (or define your API) that doesn't involve catching exceptions. But APIs involving exceptions are often inferior to APIs that don't require exception catching. (Yes, I am aware of __next__() raising StopIteration -- but that API design usually doesn't require you to catch it.) -- --Guido van Rossum (python.org/~guido) From damien.p.george at gmail.com Tue Sep 13 03:10:19 2016 From: damien.p.george at gmail.com (Damien George) Date: Tue, 13 Sep 2016 17:10:19 +1000 Subject: [Python-ideas] Changing optimisation level from a script Message-ID: Hi Petr, > The API you proposed here comes is similar to something I see a lot in > MicroPython libraries: functions/methods that combine a getter and setter. > For example, to set the value on a pin, you do: > pin.value(1) > and to read, you do: > result = pin.value() > > If an API like this was added to the stdlib, I'd expect it to use a > property, e.g. > pin.value = 1 > result = pin.value > > > I was wondering, what's the story of this aspect of MicroPython API? > Does it have hidden advantages? Were you inspired by another library? Or > was it just the easiest way to get the functionality (I assume you > implemented functions before properties), and then it stuck? Yes we do use this pattern a fair bit for things that are property-like, eg machine.freq() to get and machine.freq(42000000) to set the CPU frequency. The history reaches back to this issue: https://github.com/micropython/micropython/issues/378 . The main thing is that (for example) setting a frequency is a real action (a function if you will) that is doing lots of things behind the scenes, and which may fail with an exception. It therefore doesn't feel right to make this an attribute, but rather a proper function. To me, an attribute should only be used for things that are true constants, or that are conceptually just a "member variable of an object", to which you can assign any value, and reading it back gives you the same value. So, that's the rationale behind using functions. Cheers, Damien. From damien.p.george at gmail.com Tue Sep 13 03:34:29 2016 From: damien.p.george at gmail.com (Damien George) Date: Tue, 13 Sep 2016 17:34:29 +1000 Subject: [Python-ideas] Changing optimisation level from a script Message-ID: Thanks all for the input on this topic. Let me summarise what was said. There was discussion that the ability to change the "optimize" value is not enough and there should be more fine grained control over other flags/settings. Perhaps that is true, but at least for our case in MicroPython there's only one (runtime) optimisation variable and it mimics CPython's -O command line option (affecting __debug__, assert compilation, and whether line numbers are included in the bytecode). Suggestions for setting the opt level were: 1) sys.set_default_optimize(level), with the existing sys.flags.optimize attribute as the accessor. 2) sys.opt_level, being a read/write attribute 3) sys._setflag(name, value) as implemented by eGenix PyRun I think option 1 is the most consistent with the existing sys module, although the name is rather long and I think sys.set_optimize(level) would be better (after all, none of the other sys.set functions have the word default in them). As a second preference I like micropython.opt_level([value]), which can set and get the value, and keeps it specific to uPy. Cheers, Damien. From ncoghlan at gmail.com Tue Sep 13 07:11:11 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Sep 2016 21:11:11 +1000 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <1473545727.3581913.721692129.21833120@webmail.messagingengine.com> <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: On 13 September 2016 at 07:13, Paul Moore wrote: > If I understand the proposal, f is actually intended to be equivalent to: > > def f(spam): > spam_val = spam() > if spam_val is None: > return None > eggs_val = spam_val.eggs() > if eggs_val is None: > return None > return eggs_val.ham > > Personally, I find it pretty worrying that there's this persistent > confusion over what the proposed syntax means. Sure, it's explained in > the PEP and proposal, but if it gets implemented it'll be in the docs. > And yet people don't seem to read any of those - and their intuition > of what the construct does is wrong. Right, there are two quite reasonable interpretations for what a conditional attribute lookup does, and the "catch and ignore AttributeError" case is going to be more familiar to most Pythonistas, as that's the way getattr() works when given a default value. Consider the case of chained attribute lookup as a function using a "None" default to getattr(): def attr_chain_abort_on_missing(base, *attrs): result = base for attr in attrs: if result is None: break result = getattr(result, attr, None) return result vs the actually proposed semantics: def attr_chain_abort_only on_none(base, *attrs): result = base for attr in attrs: if result is None: break result = getattr(result, attr) return result In the latter version, AttibuteError escapes - it's only when the original value is None, or an attribute exists and is None that the iteration bails out early without raising an exception. > IMO, the syntax may well be useful, but if we don't address the > problem that what people *think* it means isn't what it actually > means, then it'll do more harm than good. Agreed, and I think a key indicator for that would be whether or not people that saw: result = obj?.first?.second?.third agreed on whether or not it could raise AttributeError or TypeError. Any prospective PhD students in the audience looking for a language usability question to study before 3.7b1 rolls around in 2018? :) Cheers, Nick. P.S. I'll note that the *upside* I see to (this part of) the proposal is that it would implement a *very* fine-grained "is not None" check, where the alternative in real code would often be an overly broad exception handling clause, like: try: result = obj.first().second.third except (TypeError, AttributeError): return None rather then actually spelling out the fine-grained checks properly. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rob.cliffe at btinternet.com Tue Sep 13 07:15:00 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 13 Sep 2016 12:15:00 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> Message-ID: <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> On 13/09/2016 04:43, Guido van Rossum wrote: > On Mon, Sep 12, 2016 at 5:45 PM, Rob Cliffe wrote: >> >> On 12/09/2016 16:37, Guido van Rossum wrote: >>> For the record, I still really don't like PEP 463. We should strive to >>> catch fewer exceptions, not make it easier to catch them. >> Can you please clarify what you are saying in the last sentence? >> The first time I read it, my brain parsed it as "People should raise fewer >> exceptions, i.e. use LBYL less". > (If that means "Leap Before You Look", yes. :-) I meant EAFP, but ! like your version! :-) Sorry for the confusion. > >> Later it seemed more likely that you meant "The set of exceptions we catch >> in a specific piece of code should not be too large; we should only catch >> those exceptions that we really mean to handle". >> But perhaps you mean something else altogether? >> >> Also I don't see the connection between the first sentence and the second. >> Exception-catching expressions as per PEP 463 just give you a more concise >> way of of doing something you can do with try+except. Do you mean it would >> make it easier to catch exceptions because you have to type fewer >> characters, and that this would be a bad thing? > Yeah, that's exactly my point. PEP 463 gives you a shorter way to > catch an exception, so it gives you less motivation to find a way to > write your code (or define your API) that doesn't involve catching > exceptions. But APIs involving exceptions are often inferior to APIs > that don't require exception catching. (Yes, I am aware of __next__() > raising StopIteration -- but that API design usually doesn't require > you to catch it.) > You surprise me. I thought LBYL and EAFP were both approved Python idioms, in some cases one being better, in some cases another, choice to be made on the merits of each case (or the author's preference). I certainly use both (and sometimes time both to see which is faster). Now it sounds as if you're trying to impose a style guide on the world by discouraging the EAFP. And wasn't the discussion general, not about APIs specifically? Best wishes, Rob Cliffe From ncoghlan at gmail.com Tue Sep 13 07:22:53 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Sep 2016 21:22:53 +1000 Subject: [Python-ideas] Changing optimisation level from a script In-Reply-To: References: Message-ID: On 13 September 2016 at 17:34, Damien George wrote: > Suggestions for setting the opt level were: > > 1) sys.set_default_optimize(level), with the existing > sys.flags.optimize attribute as the accessor. > > 2) sys.opt_level, being a read/write attribute > > 3) sys._setflag(name, value) as implemented by eGenix PyRun > > I think option 1 is the most consistent with the existing sys module, > although the name is rather long and I think sys.set_optimize(level) > would be better (after all, none of the other sys.set functions have > the word default in them). For CPython, where we have quite a few different runtime settings, something like the general purpose eGenix interface probably makes more sense, for the same reason we added sys.flags in the first place: reducing the level of API sprawl in the sys module. > As a second preference I like micropython.opt_level([value]), which > can set and get the value, and keeps it specific to uPy. And if you went that way, any future general purpose "runtime flag manipulation" API could just provide a then wrapper around that for the optimisation flag, while still exposing this API for use from MicroPython-specific boostrapping files. Given those two points, I think "micropython.opt_level([value])" would be the right way to go, with the official stance for CPython being that configuration is an environmental problem, but some environments (like PyRun and MicroPython) will expose a Python API rather than using an external mechanism. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 13 07:37:48 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Sep 2016 21:37:48 +1000 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> Message-ID: On 13 September 2016 at 21:15, Rob Cliffe wrote: > On 13/09/2016 04:43, Guido van Rossum wrote: >> Yeah, that's exactly my point. PEP 463 gives you a shorter way to >> catch an exception, so it gives you less motivation to find a way to >> write your code (or define your API) that doesn't involve catching >> exceptions. But APIs involving exceptions are often inferior to APIs >> that don't require exception catching. (Yes, I am aware of __next__() >> raising StopIteration -- but that API design usually doesn't require >> you to catch it.) >> > You surprise me. I thought LBYL and EAFP were both approved Python idioms, > in some cases one being better, in some cases another, choice to be made on > the merits of each case (or the author's preference). I certainly use both > (and sometimes time both to see which is faster). > Now it sounds as if you're trying to impose a style guide on the world by > discouraging the EAFP. And wasn't the discussion general, not about APIs > specifically? Which is preferable depends greatly on context of use, which is why you'll find a lot of Python APIs offer both forms - it's not *just* a matter of inheriting the exceptionless version from C, and then later adding a Python version that gives an exception instead of None or a user-supplied default value. It's similar to why IEEE754 defines both quiet NaN *and* signalling NaN - which one you want depends on what you're doing. In web servers, for example, you'll often have lots of fields where "not present" is a perfectly acceptable return value. For those, APIs that just return None for unknown entries are very handy, which is why SQL Alchemy offers both ".first()" and "one()", which mainly differ in how and when they throw an exception, rather than what they do when they succeed. However, blindly catching *all* exceptions from a complex subexpression is rarely the right thing to do, so APIs that only offer "this may throw exceptions during normal operation under these circumstances" without a convenience wrapper that does the exception handling for you can end up being a pain to work with. PEP 463 makes those APIs less painful to deal with, but at the cost of encouraging overly broad exception handlers. By contrast, fixing APIs on a case-by-case basis puts the design burden where it can do the most good: on the API designer, who can make the scope of the exception handling suitably narrow *inside* the API implementation, rather than being limited to wrapping the entire API call in try/except. Cheers, Nick. P.S. There are also some use cases where Look-Before-You-Leap is inherently subject to race conditions, and for those, exceptions are the only reliable signaling mechanism. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From desmoulinmichel at gmail.com Tue Sep 13 08:44:51 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 13 Sep 2016 14:44:51 +0200 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> Message-ID: <1a64a210-d0f0-46e5-c860-858b233a010d@gmail.com> I doubt very much it will be used for very complexe cases. Just like comprehensions or ternary expressions, they are a good fit for some specific use cases, and people will quickly catch on which one. You rarely see nested comprehensions or ternary expressions while it's possible to do so, because the Python communality values tend to limit abuses. It will be the same for this. You won't see very complex usages, mostly things like: val = foo[-1] except IndexError: "bar" doh = val.attr.other except AttributeError: "default" Those would already be coded with something similar (or maybe a chain of getattr()) or next(iter()). It's not a huge risk. But it's a huge convenience. Le 13/09/2016 ? 13:37, Nick Coghlan a ?crit : > On 13 September 2016 at 21:15, Rob Cliffe wrote: >> On 13/09/2016 04:43, Guido van Rossum wrote: >>> Yeah, that's exactly my point. PEP 463 gives you a shorter way to >>> catch an exception, so it gives you less motivation to find a way to >>> write your code (or define your API) that doesn't involve catching >>> exceptions. But APIs involving exceptions are often inferior to APIs >>> that don't require exception catching. (Yes, I am aware of __next__() >>> raising StopIteration -- but that API design usually doesn't require >>> you to catch it.) >>> >> You surprise me. I thought LBYL and EAFP were both approved Python idioms, >> in some cases one being better, in some cases another, choice to be made on >> the merits of each case (or the author's preference). I certainly use both >> (and sometimes time both to see which is faster). >> Now it sounds as if you're trying to impose a style guide on the world by >> discouraging the EAFP. And wasn't the discussion general, not about APIs >> specifically? > > Which is preferable depends greatly on context of use, which is why > you'll find a lot of Python APIs offer both forms - it's not *just* a > matter of inheriting the exceptionless version from C, and then later > adding a Python version that gives an exception instead of None or a > user-supplied default value. > > It's similar to why IEEE754 defines both quiet NaN *and* signalling > NaN - which one you want depends on what you're doing. > > In web servers, for example, you'll often have lots of fields where > "not present" is a perfectly acceptable return value. For those, APIs > that just return None for unknown entries are very handy, which is why > SQL Alchemy offers both ".first()" and "one()", which mainly differ in > how and when they throw an exception, rather than what they do when > they succeed. > > However, blindly catching *all* exceptions from a complex > subexpression is rarely the right thing to do, so APIs that only offer > "this may throw exceptions during normal operation under these > circumstances" without a convenience wrapper that does the exception > handling for you can end up being a pain to work with. > > PEP 463 makes those APIs less painful to deal with, but at the cost of > encouraging overly broad exception handlers. By contrast, fixing APIs > on a case-by-case basis puts the design burden where it can do the > most good: on the API designer, who can make the scope of the > exception handling suitably narrow *inside* the API implementation, > rather than being limited to wrapping the entire API call in > try/except. > > Cheers, > Nick. > > P.S. There are also some use cases where Look-Before-You-Leap is > inherently subject to race conditions, and for those, exceptions are > the only reliable signaling mechanism. > From random832 at fastmail.com Tue Sep 13 10:14:08 2016 From: random832 at fastmail.com (Random832) Date: Tue, 13 Sep 2016 10:14:08 -0400 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> Message-ID: <1473776048.2404546.724318953.17793F5D@webmail.messagingengine.com> On Tue, Sep 13, 2016, at 07:37, Nick Coghlan wrote: > P.S. There are also some use cases where Look-Before-You-Leap is > inherently subject to race conditions, Which use cases *aren't*? > and for those, exceptions are the only reliable signaling mechanism. It's entirely possible to design a non-exception interface without race conditions. dict.get is *almost* a suitable example of such a method. Remember, at the C level in CPython, throwing an exception *is* returning a value [typically (PyObject*)NULL for interfaces that otherwise return an object], along with setting the error indicator. The only missing piece is an "error value" outside the set of valid values. Guido's argument here seems to be that exception-based EAFP is not pythonic. From random832 at fastmail.com Tue Sep 13 10:22:43 2016 From: random832 at fastmail.com (Random832) Date: Tue, 13 Sep 2016 10:22:43 -0400 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: <1a64a210-d0f0-46e5-c860-858b233a010d@gmail.com> References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> <1a64a210-d0f0-46e5-c860-858b233a010d@gmail.com> Message-ID: <1473776563.2406584.724332561.7DED33BA@webmail.messagingengine.com> On Tue, Sep 13, 2016, at 08:44, Michel Desmoulin wrote: > You won't see very complex usages, mostly things like: > > val = foo[-1] except IndexError: "bar" > doh = val.attr.other except AttributeError: "default" It occurs to me that if lambda were more lightweight [whether it's the syntax or the implementation that's the problem is unclear], not only this but also the ternary operator could have been functions. val = iexcept(lambda: foo[-1], IndexError, lambda: "bar") From rosuav at gmail.com Tue Sep 13 10:24:35 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Sep 2016 00:24:35 +1000 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: <1473776048.2404546.724318953.17793F5D@webmail.messagingengine.com> References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> <1473776048.2404546.724318953.17793F5D@webmail.messagingengine.com> Message-ID: On Wed, Sep 14, 2016 at 12:14 AM, Random832 wrote: > On Tue, Sep 13, 2016, at 07:37, Nick Coghlan wrote: >> P.S. There are also some use cases where Look-Before-You-Leap is >> inherently subject to race conditions, > > Which use cases *aren't*? Ones in which no external force can affect things. For example: def frob(spam): if spam is None: print("Frobbing nothing") else: print("Frobbing some spam: ", spam) ... You can safely assume that locals won't be changed between the 'is None' check and the print. I suppose someone could mess around with thread call stacks, but that'd be seriously insane. ChrisA From rob.cliffe at btinternet.com Tue Sep 13 11:27:44 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 13 Sep 2016 16:27:44 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> Message-ID: On 13/09/2016 12:37, Nick Coghlan wrote: > On 13 September 2016 at 21:15, Rob Cliffe wrote: >> On 13/09/2016 04:43, Guido van Rossum wrote: >>> Yeah, that's exactly my point. PEP 463 gives you a shorter way to >>> catch an exception, so it gives you less motivation to find a way to >>> write your code (or define your API) that doesn't involve catching >>> exceptions. But APIs involving exceptions are often inferior to APIs >>> that don't require exception catching. (Yes, I am aware of __next__() >>> raising StopIteration -- but that API design usually doesn't require >>> you to catch it.) >>> >> You surprise me. I thought LBYL and EAFP were both approved Python idioms, >> in some cases one being better, in some cases another, choice to be made on >> the merits of each case (or the author's preference). I certainly use both >> (and sometimes time both to see which is faster). >> Now it sounds as if you're trying to impose a style guide on the world by >> discouraging the EAFP. And wasn't the discussion general, not about APIs >> specifically? > Which is preferable depends greatly on context of use, which is why > you'll find a lot of Python APIs offer both forms My point exactly. Random832 echoes my thoughts: "Guido's argument here seems to be that exception-based EAFP is not pythonic." [snip] > However, blindly catching *all* exceptions from a complex > subexpression is rarely the right thing to do, Of course. > so APIs that only offer > "this may throw exceptions during normal operation under these > circumstances" without a convenience wrapper that does the exception > handling for you can end up being a pain to work with. > > PEP 463 makes those APIs less painful to deal with, but at the cost of > encouraging overly broad exception handlers. Why? You can catch exactly the same (wide or narrow) range of exceptions in an exception-catching expression as you can in a try+except block. Of course result = (myList[0] except Exception: MyDefault) is poor code (presumably it should have been written "except IndexError"), but so is try: result = myList[0] except Exception: result = MyDefault ISTM you're giving an exception-catching dog a bad name and hanging him. Or have I missed something? And sorry to repeat myself, but we seemed to be having a *general* discussion about null-coalescing operators, which moved on to PEP 463, then suddenly Guido is talking about APIs. A lot of the time when I'm coding, I'm not writing an API, just a program to get a job done. No doubt, exceptions should be discouraged *in APIs*, but that doesn't make exceptions, or EAFP, a bad idea per se. I really don't mean this post to sound hostile, and I'm sorry if it comes across a bit that way. I'm just surprised at what I'm hearing, and to be honest, I have a soft spot for PEP 463. Best wishes Rob Cliffe From p.schellart at princeton.edu Tue Sep 13 12:16:37 2016 From: p.schellart at princeton.edu (Pim Schellart) Date: Tue, 13 Sep 2016 16:16:37 +0000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes Message-ID: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Dear Python Developers, we have a potential idea for enhancing Python. Below you will find what the PEP might look like. A reference implementation has been written and will be posted in a follow up message. We are look forward to hearing your feedback and ideas. Kind regards, Pim Schellart & Nate Lust Abstract ======== This PEP proposes the introduction of new syntax to create a community standard, readable way to continue a definition for classes which are already defined. Rationale ========= Extending existing classes with new members currently requires the members to be defined outside of class scope and then assigned to the class, as in:: def foo(self): pass A.foo = foo del foo This pattern frequently occurs when extending classes from extension modules written in other languages, but is also not uncommon in pure Python code. This syntax is however cumbersome and error prone for the following reasons: 1. The name of the new member might clash with an existing name at module scope. 2. The developer might forget the assignment. 3. The developer might forget to delete the temporary, leaving it at module scope where it is meaningless (or worse non functional as a stand alone function). Alternatives are to use inheritance, lambda or a decorator. Inheritance is not a good option for use with Python extension modules (written in for instance C / C++). The reason is that if in the other language the inheritance relation is A<-B, and this is exposed to Python, we can't add functionality to A in Python by subclassing A<-A' without also introducing B' that inherits from B and A'. Thus one change propagates all the way down the inheritance chain. Inheritance with pure Python classes exhibit the same issue. It might be tempting to avoid (1) and (2) by subclassing, reusing the name of the base class. However this approach suffers from two issues. The first issue, like above, has to do with the inheritance relation. If A subclasses A and B subclasses A (A<-A<-B), then calls to super in B may have unexpected behavior (anything using the method resolution order may exhibit the same unexpectedness). The second issue arises when class B inherits from class A before it is extended (A<-B) and then class A is extended though inheritance (A<-A). Class A will now have new methods and data members which are not present in B, as it is a subclass only of the original A. This would be confusing to anyone examining B as it would be apparently missing members of it's parent. Adding attributes to a class using lambda is not equivalent because it only allows for expressions and not statements. A class decorator (say "@continue_class(A)" that takes methods and attributes from the class and attaches them to the wrapped type) works, but is cumbersome and requires a new class name to be introduced for each extended type. Thus suffering from the same problems numbered (1) and (2) above. A function decorator could extend a class, but also requires defining a new name and suffers from problems (1) and (2). This proposal adds the keyword combination "continue class" which instructs the interpreter to add members to an existing class. This syntax requires no new keywords and allows all existing Python code to continue to function with no modification. By combining existing keywords it avoids name clashes. Precedents for this exists in "is not" and "yield from". Another big advantage of the "continue class" syntax is discoverability. For humans, the meaning of the syntax is evident, the class will be continued with the following block. Automated tools will also benefit from this syntax as parsing a file to discover the complete definition of a class will be easier than trying to detect 'monkey patching'. Semantics ========= The following two snippets are semantically identical:: continue class A: x = 5 def foo(self): pass def bar(self): pass def foo(self): pass def bar(self): pass A.x = 5 A.foo = foo A.bar = bar del foo del bar Alternatives ============ An alternative could be to allow a function definition to include a class specifier as in:: def A.foo(self): pass Implementation ============== Adapting Python's grammar to support the continue class keyword combination requires modifying the grammar file to parse for the new keyword combination. Additional changes to Python.asdl, ast.c, compile.c, symtable.c will be required to process the parsed syntax to byte code. A reference implementation (already written and posted separately) parses the continue class line for the name of the class to be modified and loads it. Next the body of the block is compiled into a code object scoped by the name of the class. The class and the code are used as arguments to a new built-in module, called __continue_class__. This function evaluates the code block, passing the results to setattr on the supplied class. From rosuav at gmail.com Tue Sep 13 12:22:28 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Sep 2016 02:22:28 +1000 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> Message-ID: On Wed, Sep 14, 2016 at 1:27 AM, Rob Cliffe wrote: > And sorry to repeat myself, but we seemed to be having a *general* > discussion about null-coalescing operators, which moved on to PEP 463, then > suddenly Guido is talking about APIs. A lot of the time when I'm coding, > I'm not writing an API, just a program to get a job done. No doubt, > exceptions should be discouraged *in APIs*, but that doesn't make > exceptions, or EAFP, a bad idea per se. PEP 463, by its nature, is talking about the language. Language features govern and advise API design, which then changes how you write "a program to get a job done", so it does actually have strong bearing - just via a couple of levels of indirection. ChrisA From rosuav at gmail.com Tue Sep 13 12:29:18 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Sep 2016 02:29:18 +1000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: On Wed, Sep 14, 2016 at 2:16 AM, Pim Schellart wrote: > Semantics > ========= > > The following two snippets are semantically identical:: > > continue class A: > x = 5 > def foo(self): > pass > def bar(self): > pass > > def foo(self): > pass > def bar(self): > pass > A.x = 5 > A.foo = foo > A.bar = bar > del foo > del bar Did you know that you can actually abuse decorators to do this with existing syntax? Check out this collection of evil uses of decorators: https://github.com/Rosuav/Decorators/blob/master/evil.py I call them "evil" because they're potentially VERY confusing, but they're not necessarily bad. The monkeypatch decorator does basically what you're doing here, but with this syntax: @monkeypatch class A: x = 5 def foo(self): pass def bar(self): pass It's a little bit magical, in that it looks up the original class using globals(); this is partly deliberate, as it means you can't accidentally monkey-patch something from the built-ins, which will either fail, or (far worse) succeed and confuse everyone. ChrisA From p.f.moore at gmail.com Tue Sep 13 13:15:31 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Sep 2016 18:15:31 +0100 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: On 13 September 2016 at 17:29, Chris Angelico wrote: > On Wed, Sep 14, 2016 at 2:16 AM, Pim Schellart > wrote: >> Semantics >> ========= >> >> The following two snippets are semantically identical:: >> >> continue class A: >> x = 5 >> def foo(self): >> pass >> def bar(self): >> pass >> >> def foo(self): >> pass >> def bar(self): >> pass >> A.x = 5 >> A.foo = foo >> A.bar = bar >> del foo >> del bar > > Did you know that you can actually abuse decorators to do this with > existing syntax? Check out this collection of evil uses of decorators: > > https://github.com/Rosuav/Decorators/blob/master/evil.py > > I call them "evil" because they're potentially VERY confusing, but > they're not necessarily bad. Also, it's worth noting that while Ruby has a tradition of "open" classes (as in, the ability to add methods to class definitions after those classes have been created) Python traditionally does not encourage such use. You claim in your proposal """ This pattern frequently occurs when extending classes from extension modules written in other languages, but is also not uncommon in pure Python code. """ but to be honest I don't recall seeing this pattern commonly used. That's not to say that the feature might not be useful, but I think you'll need a bit more justification, and examples of code that would be improved by this proposal, if you want to get it accepted. Paul From ethan at stoneleaf.us Tue Sep 13 13:21:28 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 13 Sep 2016 10:21:28 -0700 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: <57D83598.8090407@stoneleaf.us> On 09/13/2016 09:29 AM, Chris Angelico wrote: > On Wed, Sep 14, 2016 at 2:16 AM, Pim Schellart wrote: A well-written PEP that looked very interesting. > https://github.com/Rosuav/Decorators/blob/master/evil.py > @monkeypatch > class A: > x = 5 > def foo(self): > pass > def bar(self): > pass But this seems to solve the problem nicely. In other words, you now need to explain why this particular class decorator is not good enough to solve the general problem. -- ~Ethan~ From natelust at linux.com Tue Sep 13 12:22:52 2016 From: natelust at linux.com (nate lust) Date: Tue, 13 Sep 2016 16:22:52 +0000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: Hello developers, I have written a functional reference implementation for the proposed continue class syntax which can be found at https://raw.githubusercontent.com/natelust/continueClassCpythonPatch/master/continueClass.patch . This is a mercurial patch file created against python 3.5. An example of its functionality, and demonstration that it does not interfere with the existing continue keyword is as follows: # Define a class Foo, taking a message as an argument class Foo: def __init__(self, message): self.message = message # Continue the definition of the class, adding a method to print # the class message variable continue class Foo: def second(self): print(self.message) # Create an instance of the class inst = Foo('Hello World') # Demo the continued class inst.second() # Show that existing syntax still works for i in range(5): if i == 0: continue print(i) Thank you for your time and consideration, we will be happy to take any feedback or questions on the implementation. Nate Lust On Tue, Sep 13, 2016 at 12:17 PM Pim Schellart wrote: > Dear Python Developers, > > we have a potential idea for enhancing Python. > Below you will find what the PEP might look like. > A reference implementation has been written and will be posted in a follow > up message. > We are look forward to hearing your feedback and ideas. > > Kind regards, > > Pim Schellart & Nate Lust > > Abstract > ======== > > This PEP proposes the introduction of new syntax to create a community > standard, > readable way to continue a definition for classes which are already > defined. > > > Rationale > ========= > > Extending existing classes with new members currently requires the members > to be > defined outside of class scope and then assigned to the class, as in:: > > def foo(self): > pass > A.foo = foo > del foo > > This pattern frequently occurs when extending classes from extension > modules > written in other languages, but is also not uncommon in pure Python code. > > This syntax is however cumbersome and error prone for the following > reasons: > > 1. The name of the new member might clash with an existing name at module > scope. > 2. The developer might forget the assignment. > 3. The developer might forget to delete the temporary, leaving it at > module > scope where it is meaningless (or worse non functional as a stand > alone > function). > > Alternatives are to use inheritance, lambda or a decorator. > > Inheritance is not a good option for use with Python extension modules > (written > in for instance C / C++). The reason is that if in the other language the > inheritance relation is A<-B, and this is exposed to Python, we can't add > functionality to A in Python by subclassing A<-A' without also introducing > B' > that inherits from B and A'. Thus one change propagates all the way down > the > inheritance chain. Inheritance with pure Python classes exhibit the same > issue. > > It might be tempting to avoid (1) and (2) by subclassing, reusing the name > of > the base class. However this approach suffers from two issues. The first > issue, > like above, has to do with the inheritance relation. If A subclasses A > and B > subclasses A (A<-A<-B), then calls to super in B may have unexpected > behavior > (anything using the method resolution order may exhibit the same > unexpectedness). The second issue arises when class B inherits from class > A > before it is extended (A<-B) and then class A is extended though > inheritance > (A<-A). Class A will now have new methods and data members which are not > present in B, as it is a subclass only of the original A. This would be > confusing to anyone examining B as it would be apparently missing members > of > it's parent. > > Adding attributes to a class using lambda is not equivalent because it only > allows for expressions and not statements. > > A class decorator (say "@continue_class(A)" that takes methods and > attributes > from the class and attaches them to the wrapped type) works, but is > cumbersome > and requires a new class name to be introduced for each extended type. > Thus > suffering from the same problems numbered (1) and (2) above. > > A function decorator could extend a class, but also requires defining a > new name > and suffers from problems (1) and (2). > > This proposal adds the keyword combination "continue class" which > instructs the > interpreter to add members to an existing class. This syntax requires no > new > keywords and allows all existing Python code to continue to function with > no > modification. > > By combining existing keywords it avoids name clashes. Precedents for this > exists in "is not" and "yield from". > > Another big advantage of the "continue class" syntax is discoverability. > For > humans, the meaning of the syntax is evident, the class will be continued > with > the following block. Automated tools will also benefit from this syntax as > parsing a file to discover the complete definition of a class will be > easier > than trying to detect 'monkey patching'. > > > Semantics > ========= > > The following two snippets are semantically identical:: > > continue class A: > x = 5 > def foo(self): > pass > def bar(self): > pass > > def foo(self): > pass > def bar(self): > pass > A.x = 5 > A.foo = foo > A.bar = bar > del foo > del bar > > > Alternatives > ============ > > An alternative could be to allow a function definition to include a class > specifier as in:: > > def A.foo(self): > pass > > > Implementation > ============== > > Adapting Python's grammar to support the continue class keyword combination > requires modifying the grammar file to parse for the new keyword > combination. > Additional changes to Python.asdl, ast.c, compile.c, symtable.c will be > required > to process the parsed syntax to byte code. > > A reference implementation (already written and posted separately) parses > the > continue class line for the name of the class to be modified and loads > it. Next > the body of the block is compiled into a code object scoped by the name of > the > class. The class and the code are used as arguments to a new built-in > module, > called __continue_class__. This function evaluates the code block, > passing the > results to setattr on the supplied class. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Sep 13 14:18:17 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Sep 2016 04:18:17 +1000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: On Wed, Sep 14, 2016 at 2:22 AM, nate lust wrote: > I have written a functional reference implementation for the proposed > continue class syntax which can be found at > https://raw.githubusercontent.com/natelust/continueClassCpythonPatch/master/continueClass.patch > . AIUI, the central core to this code is: + /* Iterate over the dictionary which was populated in the eval call. + Set an attribute in the class corresponding to the key, item pair + generated by the eval statement */ + dict_size = PyDict_Size(newdict); + dictkeys = PyDict_Keys(newdict); + for (i = 0; i < dict_size; i++ ){ + key = PyList_GetItem(dictkeys, i); + item = PyDict_GetItem(newdict, key); + PyObject_SetAttr(aclass, key, item); + } How does this handle special elements such as __module__, __dict__, and __doc__? Does it overwrite them? ChrisA From python at mrabarnett.plus.com Tue Sep 13 14:24:17 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 13 Sep 2016 19:24:17 +0100 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> Message-ID: On 2016-09-13 16:27, Rob Cliffe wrote: > > > On 13/09/2016 12:37, Nick Coghlan wrote: >> On 13 September 2016 at 21:15, Rob Cliffe wrote: >>> On 13/09/2016 04:43, Guido van Rossum wrote: >>>> Yeah, that's exactly my point. PEP 463 gives you a shorter way to >>>> catch an exception, so it gives you less motivation to find a way to >>>> write your code (or define your API) that doesn't involve catching >>>> exceptions. But APIs involving exceptions are often inferior to APIs >>>> that don't require exception catching. (Yes, I am aware of __next__() >>>> raising StopIteration -- but that API design usually doesn't require >>>> you to catch it.) >>>> >>> You surprise me. I thought LBYL and EAFP were both approved Python idioms, >>> in some cases one being better, in some cases another, choice to be made on >>> the merits of each case (or the author's preference). I certainly use both >>> (and sometimes time both to see which is faster). >>> Now it sounds as if you're trying to impose a style guide on the world by >>> discouraging the EAFP. And wasn't the discussion general, not about APIs >>> specifically? >> Which is preferable depends greatly on context of use, which is why >> you'll find a lot of Python APIs offer both forms > My point exactly. Random832 echoes my thoughts: > > "Guido's argument here seems to be that exception-based EAFP is not > pythonic." > [snip] I think the point is that exceptions are for, well, exceptional processing, not normal processing. Code should be written on the assumption that everything works normally, e.g. division always returns a result. _Occasionally_ division will fail, such as attempting to divide by zero, but that should be uncommon. If something pretty much always succeeds, use an exception for failure, but if it sometimes succeeds and sometimes fails, check explicitly for success or failure. From brett at python.org Tue Sep 13 14:29:10 2016 From: brett at python.org (Brett Cannon) Date: Tue, 13 Sep 2016 18:29:10 +0000 Subject: [Python-ideas] Fwd: Null coalescing operator In-Reply-To: References: <831cda18-49c7-6e35-9041-5b3c816a60fb@gmail.com> <1b997a51-f2c8-1759-b049-31fb5e4a0f91@btinternet.com> Message-ID: This general API discussion is veering off-topic for the subject line. If people would like to continue to discuss it then please start a new thread (if you feel it impacts your view of a null-coalescing operator then please discuss in the new thread and come back to this one). On Tue, 13 Sep 2016 at 11:25 MRAB wrote: > On 2016-09-13 16:27, Rob Cliffe wrote: > > > > > > On 13/09/2016 12:37, Nick Coghlan wrote: > >> On 13 September 2016 at 21:15, Rob Cliffe > wrote: > >>> On 13/09/2016 04:43, Guido van Rossum wrote: > >>>> Yeah, that's exactly my point. PEP 463 gives you a shorter way to > >>>> catch an exception, so it gives you less motivation to find a way to > >>>> write your code (or define your API) that doesn't involve catching > >>>> exceptions. But APIs involving exceptions are often inferior to APIs > >>>> that don't require exception catching. (Yes, I am aware of __next__() > >>>> raising StopIteration -- but that API design usually doesn't require > >>>> you to catch it.) > >>>> > >>> You surprise me. I thought LBYL and EAFP were both approved Python > idioms, > >>> in some cases one being better, in some cases another, choice to be > made on > >>> the merits of each case (or the author's preference). I certainly use > both > >>> (and sometimes time both to see which is faster). > >>> Now it sounds as if you're trying to impose a style guide on the world > by > >>> discouraging the EAFP. And wasn't the discussion general, not about > APIs > >>> specifically? > >> Which is preferable depends greatly on context of use, which is why > >> you'll find a lot of Python APIs offer both forms > > My point exactly. Random832 echoes my thoughts: > > > > "Guido's argument here seems to be that exception-based EAFP is not > > pythonic." > > > [snip] > I think the point is that exceptions are for, well, exceptional > processing, not normal processing. > > Code should be written on the assumption that everything works normally, > e.g. division always returns a result. > > _Occasionally_ division will fail, such as attempting to divide by zero, > but that should be uncommon. > > If something pretty much always succeeds, use an exception for failure, > but if it sometimes succeeds and sometimes fails, check explicitly for > success or failure. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralph at ralphbroenink.net Tue Sep 13 15:46:06 2016 From: ralph at ralphbroenink.net (Ralph Broenink) Date: Tue, 13 Sep 2016 19:46:06 +0000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: Hi all, On Tue, 13 Sep 2016 at 18:30 Chris Angelico wrote: > Did you know that you can actually abuse decorators to do this with > existing syntax? Check out this collection of evil uses of decorators: > > https://github.com/Rosuav/Decorators/blob/master/evil.py > There's also this post from Guido a few years back in python-dev, which does something similar using metaclasses: https://mail.python.org/pipermail/python-dev/2008-January/076194.html I'm not sure it does exactly the same, but it is also an interesting approach. (Although I like the decorator syntax more.) The discussion that follows in that thread may also be of interest for this discussion. Ralph -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Sep 13 16:47:58 2016 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Sep 2016 08:47:58 +1200 Subject: [Python-ideas] An exciting opportunity to update PEP 3156 References: <63013323-35cc-3fbb-07b9-5e05deed2910@nedbatchelder.com> Message-ID: <20160914084758.1a35ee45@anarchist.wains.cybercomglobal.com> On Sep 12, 2016, at 03:12 PM, Senthil Kumaran wrote: >I see PEP as the design document and docs as user-friendly documentation. It >helps to keep both of them up to date and in sync. If an "Accepted" PEP is >not updated, then folks landing on it will feel misguided or confused if the >read the docs later. Presenting a cohesive story here is probably more work >than simply keeping them in sync. It's probably appropriate, once a PEP is accepted, implemented, and documented, to include a link from the PEP to the definitive documentation. That could be done in lieu of keeping the PEP strictly in sync as the feature changes. >The history of decision changes is tracked in VCS, if that is desirable. Not just desirable, an explicit purpose of PEPs! :) https://www.python.org/dev/peps/pep-0001/ Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From greg.ewing at canterbury.ac.nz Tue Sep 13 19:39:03 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Sep 2016 11:39:03 +1200 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: <57D88E17.8040406@canterbury.ac.nz> Pim Schellart wrote: > This PEP proposes the introduction of new syntax to create a community standard, > readable way to continue a definition for classes which are already defined. -1. Monkeyatching is something that should be disccouraged rather than encouraged, since it makes code very hard to follow. -- Greg From mertz at gnosis.cx Tue Sep 13 19:41:10 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 13 Sep 2016 16:41:10 -0700 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: Continuing the definition for a class feels to me like a strong anti-pattern for Python. If you absolutely must do this, I guess the presented decorator is available. It should be discouraged though, not be part of new syntax. I believe that if you find yourself doing this you should detect a bad choice smell, and think about expressing your goal with inheritance, delegation, or encapsulation instead. On Sep 13, 2016 9:17 AM, "Pim Schellart" wrote: > Dear Python Developers, > > we have a potential idea for enhancing Python. > Below you will find what the PEP might look like. > A reference implementation has been written and will be posted in a follow > up message. > We are look forward to hearing your feedback and ideas. > > Kind regards, > > Pim Schellart & Nate Lust > > Abstract > ======== > > This PEP proposes the introduction of new syntax to create a community > standard, > readable way to continue a definition for classes which are already > defined. > > > Rationale > ========= > > Extending existing classes with new members currently requires the members > to be > defined outside of class scope and then assigned to the class, as in:: > > def foo(self): > pass > A.foo = foo > del foo > > This pattern frequently occurs when extending classes from extension > modules > written in other languages, but is also not uncommon in pure Python code. > > This syntax is however cumbersome and error prone for the following > reasons: > > 1. The name of the new member might clash with an existing name at module > scope. > 2. The developer might forget the assignment. > 3. The developer might forget to delete the temporary, leaving it at > module > scope where it is meaningless (or worse non functional as a stand > alone > function). > > Alternatives are to use inheritance, lambda or a decorator. > > Inheritance is not a good option for use with Python extension modules > (written > in for instance C / C++). The reason is that if in the other language the > inheritance relation is A<-B, and this is exposed to Python, we can't add > functionality to A in Python by subclassing A<-A' without also introducing > B' > that inherits from B and A'. Thus one change propagates all the way down > the > inheritance chain. Inheritance with pure Python classes exhibit the same > issue. > > It might be tempting to avoid (1) and (2) by subclassing, reusing the name > of > the base class. However this approach suffers from two issues. The first > issue, > like above, has to do with the inheritance relation. If A subclasses A > and B > subclasses A (A<-A<-B), then calls to super in B may have unexpected > behavior > (anything using the method resolution order may exhibit the same > unexpectedness). The second issue arises when class B inherits from class > A > before it is extended (A<-B) and then class A is extended though > inheritance > (A<-A). Class A will now have new methods and data members which are not > present in B, as it is a subclass only of the original A. This would be > confusing to anyone examining B as it would be apparently missing members > of > it's parent. > > Adding attributes to a class using lambda is not equivalent because it only > allows for expressions and not statements. > > A class decorator (say "@continue_class(A)" that takes methods and > attributes > from the class and attaches them to the wrapped type) works, but is > cumbersome > and requires a new class name to be introduced for each extended type. > Thus > suffering from the same problems numbered (1) and (2) above. > > A function decorator could extend a class, but also requires defining a > new name > and suffers from problems (1) and (2). > > This proposal adds the keyword combination "continue class" which > instructs the > interpreter to add members to an existing class. This syntax requires no > new > keywords and allows all existing Python code to continue to function with > no > modification. > > By combining existing keywords it avoids name clashes. Precedents for this > exists in "is not" and "yield from". > > Another big advantage of the "continue class" syntax is discoverability. > For > humans, the meaning of the syntax is evident, the class will be continued > with > the following block. Automated tools will also benefit from this syntax as > parsing a file to discover the complete definition of a class will be > easier > than trying to detect 'monkey patching'. > > > Semantics > ========= > > The following two snippets are semantically identical:: > > continue class A: > x = 5 > def foo(self): > pass > def bar(self): > pass > > def foo(self): > pass > def bar(self): > pass > A.x = 5 > A.foo = foo > A.bar = bar > del foo > del bar > > > Alternatives > ============ > > An alternative could be to allow a function definition to include a class > specifier as in:: > > def A.foo(self): > pass > > > Implementation > ============== > > Adapting Python's grammar to support the continue class keyword combination > requires modifying the grammar file to parse for the new keyword > combination. > Additional changes to Python.asdl, ast.c, compile.c, symtable.c will be > required > to process the parsed syntax to byte code. > > A reference implementation (already written and posted separately) parses > the > continue class line for the name of the class to be modified and loads > it. Next > the body of the block is compiled into a code object scoped by the name of > the > class. The class and the code are used as arguments to a new built-in > module, > called __continue_class__. This function evaluates the code block, > passing the > results to setattr on the supplied class. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Sep 13 19:35:58 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Sep 2016 11:35:58 +1200 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: <57D88D5E.9020206@canterbury.ac.nz> Chris Angelico wrote: > It's a little bit magical, in that it looks up the original class > using globals(); The way you've written it, monkeypatch() will only work if you call it from the module it's defined in. -- Greg From rosuav at gmail.com Tue Sep 13 21:23:13 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Sep 2016 11:23:13 +1000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: <57D88D5E.9020206@canterbury.ac.nz> References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> <57D88D5E.9020206@canterbury.ac.nz> Message-ID: On Wed, Sep 14, 2016 at 9:35 AM, Greg Ewing wrote: > Chris Angelico wrote: > >> It's a little bit magical, in that it looks up the original class >> using globals(); > > > The way you've written it, monkeypatch() will only work > if you call it from the module it's defined in. Yes. You could tweak that by looking at cls.__module__ and grab it from sys.modules, but I kept the example simple. However, I would NOT use the standard name lookup mechanism. ChrisA From greg.ewing at canterbury.ac.nz Wed Sep 14 01:40:07 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Sep 2016 17:40:07 +1200 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> <57D88D5E.9020206@canterbury.ac.nz> Message-ID: <57D8E2B7.10206@canterbury.ac.nz> Chris Angelico wrote: > Yes. You could tweak that by looking at cls.__module__ and grab it > from sys.modules, but I kept the example simple. However, I would NOT > use the standard name lookup mechanism. Most classes in builtins won't let you monkeypatch them anyway, so I'm not sure it's worth the bother of trying to protect against that. -- Greg From bzvi7919 at gmail.com Wed Sep 14 07:20:02 2016 From: bzvi7919 at gmail.com (Bar Harel) Date: Wed, 14 Sep 2016 11:20:02 +0000 Subject: [Python-ideas] Naming convention for Enums Message-ID: Hey guys, This is something that has been discussed before several times, whether in Python mailing list and whether in stackoverflow, each providing different answers. I believe that we should add a naming convention in pep8 to solve this issue. Enums in Python are a unique creature - it's a class, the members aren't injected into module level, it can have methods, but it's attributes are somewhat consts (although they can be modified via __new__). The fact that currently every person names them differently defies the whole purpose of a convention document. Although I'm in favor of lower-case instead of upper-case, I believe that we need to make a decision as no decision is somewhat worse and causes a lot of confusion. I'm not entirely sure how pep-8 was built (was it a poll? was it a BDFL announcement? was it discovered in a fortune cookie just like pep 285?), regardless how it built I believe we should make that addition for a clearer future :-) Any thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Sep 14 07:51:32 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Sep 2016 21:51:32 +1000 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: Message-ID: On Wed, Sep 14, 2016 at 9:20 PM, Bar Harel wrote: > Enums in Python are a unique creature - it's a class, the members aren't > injected into module level, it can have methods, but it's attributes are > somewhat consts (although they can be modified via __new__). > > Although I'm in favor of lower-case instead of upper-case, I believe that we > need to make a decision as no decision is somewhat worse and causes a lot of > confusion. The trouble is that enums are a bit of a chimera - part constant, part class attribute, part class instance, part integer (often). You can treat them as "constant ints with nice reprs", or you can treat them as "representations of set membership and nothing more", and all manner of other uses. I would accept a PEP 8 ruling that they be in lower-case *only* if it's qualified with a blanket permission to name something the same as its C counterpart when one exists. >>> socket.AF_INET There is absolutely no value in lower-casing something like this. (Even leaving aside backward compatibility; imagine this situation with a brand-new third party module that provides Python bindings to a C library.) If they're going to be constants that match the C constants, they should have names that match the C names. Perhaps the advice needs to be along the lines of: Decide what the purpose of the enum is, and follow a naming convention accordingly. Uppercase if you're basically making constants; lowercase if you're not; etcetera. ChrisA From p.f.moore at gmail.com Wed Sep 14 07:58:53 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Sep 2016 12:58:53 +0100 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: Message-ID: On 14 September 2016 at 12:51, Chris Angelico wrote: > Perhaps the advice needs to be along the lines of: Decide what the > purpose of the enum is, and follow a naming convention accordingly. > Uppercase if you're basically making constants; lowercase if you're > not; etcetera. Agreed - it's not clear to me that a prescriptive rule has much benefit here. The OP said "I believe that we need to make a decision as no decision is somewhat worse and causes a lot of confusion" - personally, I don't agree, I think it's fine to leave this to the individual. And of course, no matter how many times we stress that any rules we add are "only recommendations", projects like pycodestyle will add checks (as they should) and then projects that mandate a clean run of a style checker will treat the rule as mandatory. So I'm basically -1 on a PEP 8 rule at this stage, but if we have to have one, I agree that it should say something along the lines of "depends on how you're using the enum". Paul PS FWIW, I tend to think of enums as named constants, and so would typically use uppercase, not lowercase. But I wouldn't like to impose that on everyone :-) From ncoghlan at gmail.com Wed Sep 14 08:02:11 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2016 22:02:11 +1000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: On 14 September 2016 at 05:46, Ralph Broenink wrote: > Hi all, > > On Tue, 13 Sep 2016 at 18:30 Chris Angelico wrote: >> >> Did you know that you can actually abuse decorators to do this with >> existing syntax? Check out this collection of evil uses of decorators: >> >> https://github.com/Rosuav/Decorators/blob/master/evil.py > > There's also this post from Guido a few years back in python-dev, which > does something similar using metaclasses: > > https://mail.python.org/pipermail/python-dev/2008-January/076194.html > > I'm not sure it does exactly the same, but it is also an interesting > approach. (Although I like the decorator syntax more.) The discussion that > follows in that thread may also be of interest for this discussion. PEP 422 and its reference implementation had a more fleshed out implementation of that concept: https://www.python.org/dev/peps/pep-0422/#new-ways-of-using-classes With class namespaces becoming ordered by default, I ended up withdrawing PEP 422 in favour of the simpler PEP 487 that's actually in 3.6: https://www.python.org/dev/peps/pep-0487/ I do think Pim's proposal is an excellent exemplar of what a PEP should be, and if we *did* want to make class extension easier than it already is, then the suggested "continue class ..." syntax would be an elegant way of spelling it. However, I also believe the proposal founders on: 1. Class extensions are a fundamentally bad idea from a maintainability perspective, as they make the original class definition incomplete with no local indicator of its lack of completeness (hence the "don't do this" caveat on the relevant example in PEP 422) 2. There are already ways to do this using metaclasses that provide a local indicator at the point of definition that the affected class is extensible (i.e. the use of the custom metaclass, or inheritance from a base class that uses it) and hence that the given definition may be incomplete Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Wed Sep 14 10:43:51 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Sep 2016 07:43:51 -0700 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: Message-ID: I recommend naming all enums UPPER_CASE. They're constants (within a namespace) and that's the rule for constants. It's helpful for the reader of the code to realize what they are when passed around -- they have a similar status to literal constants, you know they stand for a unique value and not for some computed quantity. On Wed, Sep 14, 2016 at 4:58 AM, Paul Moore wrote: > On 14 September 2016 at 12:51, Chris Angelico wrote: >> Perhaps the advice needs to be along the lines of: Decide what the >> purpose of the enum is, and follow a naming convention accordingly. >> Uppercase if you're basically making constants; lowercase if you're >> not; etcetera. > > Agreed - it's not clear to me that a prescriptive rule has much > benefit here. The OP said "I believe that we need to make a decision > as no decision is somewhat worse and causes a lot of confusion" - > personally, I don't agree, I think it's fine to leave this to the > individual. > > And of course, no matter how many times we stress that any rules we > add are "only recommendations", projects like pycodestyle will add > checks (as they should) and then projects that mandate a clean run of > a style checker will treat the rule as mandatory. > > So I'm basically -1 on a PEP 8 rule at this stage, but if we have to > have one, I agree that it should say something along the lines of > "depends on how you're using the enum". > > Paul > > PS FWIW, I tend to think of enums as named constants, and so would > typically use uppercase, not lowercase. But I wouldn't like to impose > that on everyone :-) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From p.schellart at princeton.edu Wed Sep 14 11:46:29 2016 From: p.schellart at princeton.edu (Pim Schellart) Date: Wed, 14 Sep 2016 15:46:29 +0000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> Message-ID: <9386B806-4DF1-4082-87CD-03028D9BD4EB@princeton.edu> Dear All, Thank you for your feedback! While we think the decorator solution is not as writable, readable or discoverable as our proposed syntax we understand the desire to discourage a potential paradigm switch to `open classes?. We recognise that in many cases refactoring the code to eliminate the need for monkey patching is the preferable solution. However, there are many instances where this is not the case, such as the one Guido described in https://mail.python.org/pipermail/python-dev/2008-January/076194.html. Personally, we have encountered the need for this in many projects each of us have worked on. The fact that monkey patching is common enough lexicon to be mentioned in Python programming books highlights the fact that programmers may encounter this in codebases they work on. In particular when using third party libraries or code written in other languages wrapped into Python, the existing syntaxes are less readable and non-standard. Each project using such codes is thus forced to come up with their own custom solution leading to a somewhat fractured appearance in otherwise similar Python codebases. We had hoped to come up with a standardised solution. And if new syntax is not desirable, perhaps including other solutions such as the proposed decorator in the standard library is an option? Or do you feel this is a problem that should be solved by better documentation and understanding instead? Kind regards, Pim Schellart & Nate Lust > On 14 Sep 2016, at 08:02, Nick Coghlan wrote: > > On 14 September 2016 at 05:46, Ralph Broenink wrote: >> Hi all, >> >> On Tue, 13 Sep 2016 at 18:30 Chris Angelico wrote: >>> >>> Did you know that you can actually abuse decorators to do this with >>> existing syntax? Check out this collection of evil uses of decorators: >>> >>> https://github.com/Rosuav/Decorators/blob/master/evil.py >> >> There's also this post from Guido a few years back in python-dev, which >> does something similar using metaclasses: >> >> https://mail.python.org/pipermail/python-dev/2008-January/076194.html >> >> I'm not sure it does exactly the same, but it is also an interesting >> approach. (Although I like the decorator syntax more.) The discussion that >> follows in that thread may also be of interest for this discussion. > > PEP 422 and its reference implementation had a more fleshed out > implementation of that concept: > https://www.python.org/dev/peps/pep-0422/#new-ways-of-using-classes > > With class namespaces becoming ordered by default, I ended up > withdrawing PEP 422 in favour of the simpler PEP 487 that's actually > in 3.6: https://www.python.org/dev/peps/pep-0487/ > > I do think Pim's proposal is an excellent exemplar of what a PEP > should be, and if we *did* want to make class extension easier than it > already is, then the suggested "continue class ..." syntax would be an > elegant way of spelling it. > > However, I also believe the proposal founders on: > > 1. Class extensions are a fundamentally bad idea from a > maintainability perspective, as they make the original class > definition incomplete with no local indicator of its lack of > completeness (hence the "don't do this" caveat on the relevant example > in PEP 422) > 2. There are already ways to do this using metaclasses that provide a > local indicator at the point of definition that the affected class is > extensible (i.e. the use of the custom metaclass, or inheritance from > a base class that uses it) and hence that the given definition may be > incomplete > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Wed Sep 14 12:29:08 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Sep 2016 09:29:08 -0700 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: <9386B806-4DF1-4082-87CD-03028D9BD4EB@princeton.edu> References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> <9386B806-4DF1-4082-87CD-03028D9BD4EB@princeton.edu> Message-ID: There shall be no standard syntax for monkey-patching. Decorators are fine. I recommend putting a monkey-patching package on PyPI and see if people use it. Maybe eventually it can be in the stdlib but I really don't want to encourage this idiom (even if it's sometimes useful). You should always consider other options (like petitioning upstream for a better API) before accepting the need to monkey-patch. On Wed, Sep 14, 2016 at 8:46 AM, Pim Schellart wrote: > Dear All, > > Thank you for your feedback! > While we think the decorator solution is not as writable, readable or discoverable as our > proposed syntax we understand the desire to discourage a potential > paradigm switch to `open classes?. > > We recognise that in many cases refactoring the code to eliminate the need for monkey patching > is the preferable solution. However, there are many instances where this is not the case, such as > the one Guido described in https://mail.python.org/pipermail/python-dev/2008-January/076194.html. > > Personally, we have encountered the need for this in many projects each of us have worked on. > The fact that monkey patching is common enough lexicon to be mentioned in Python programming > books highlights the fact that programmers may encounter this in codebases they work on. > > In particular when using third party libraries or code written in other languages wrapped into Python, > the existing syntaxes are less readable and non-standard. > Each project using such codes is thus forced to come up with their own custom solution leading > to a somewhat fractured appearance in otherwise similar Python codebases. > > We had hoped to come up with a standardised solution. And if new syntax is not desirable, perhaps > including other solutions such as the proposed decorator in the standard library is an option? > Or do you feel this is a problem that should be solved by better documentation and understanding > instead? > > Kind regards, > > Pim Schellart & Nate Lust >> On 14 Sep 2016, at 08:02, Nick Coghlan wrote: >> >> On 14 September 2016 at 05:46, Ralph Broenink wrote: >>> Hi all, >>> >>> On Tue, 13 Sep 2016 at 18:30 Chris Angelico wrote: >>>> >>>> Did you know that you can actually abuse decorators to do this with >>>> existing syntax? Check out this collection of evil uses of decorators: >>>> >>>> https://github.com/Rosuav/Decorators/blob/master/evil.py >>> >>> There's also this post from Guido a few years back in python-dev, which >>> does something similar using metaclasses: >>> >>> https://mail.python.org/pipermail/python-dev/2008-January/076194.html >>> >>> I'm not sure it does exactly the same, but it is also an interesting >>> approach. (Although I like the decorator syntax more.) The discussion that >>> follows in that thread may also be of interest for this discussion. >> >> PEP 422 and its reference implementation had a more fleshed out >> implementation of that concept: >> https://www.python.org/dev/peps/pep-0422/#new-ways-of-using-classes >> >> With class namespaces becoming ordered by default, I ended up >> withdrawing PEP 422 in favour of the simpler PEP 487 that's actually >> in 3.6: https://www.python.org/dev/peps/pep-0487/ >> >> I do think Pim's proposal is an excellent exemplar of what a PEP >> should be, and if we *did* want to make class extension easier than it >> already is, then the suggested "continue class ..." syntax would be an >> elegant way of spelling it. >> >> However, I also believe the proposal founders on: >> >> 1. Class extensions are a fundamentally bad idea from a >> maintainability perspective, as they make the original class >> definition incomplete with no local indicator of its lack of >> completeness (hence the "don't do this" caveat on the relevant example >> in PEP 422) >> 2. There are already ways to do this using metaclasses that provide a >> local indicator at the point of definition that the affected class is >> extensible (i.e. the use of the custom metaclass, or inheritance from >> a base class that uses it) and hence that the given definition may be >> incomplete >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From natelust at linux.com Wed Sep 14 12:41:47 2016 From: natelust at linux.com (nate lust) Date: Wed, 14 Sep 2016 16:41:47 +0000 Subject: [Python-ideas] New Python syntax for continuing definitions for existing classes In-Reply-To: References: <66EB9DAD-48B8-4486-B533-E1FF16E18F55@princeton.edu> <9386B806-4DF1-4082-87CD-03028D9BD4EB@princeton.edu> Message-ID: All, I must say It is still exciting to be at a point of getting an official declaration on a topic from Guido. We thank you for your time and feedback. It has been a good experience and hope to be able to contribute in some other way in the future. Nate and Pim On Wed, Sep 14, 2016 at 12:30 PM Guido van Rossum wrote: > There shall be no standard syntax for monkey-patching. > > Decorators are fine. I recommend putting a monkey-patching package on > PyPI and see if people use it. > > Maybe eventually it can be in the stdlib but I really don't want to > encourage this idiom (even if it's sometimes useful). You should > always consider other options (like petitioning upstream for a better > API) before accepting the need to monkey-patch. > > On Wed, Sep 14, 2016 at 8:46 AM, Pim Schellart > wrote: > > Dear All, > > > > Thank you for your feedback! > > While we think the decorator solution is not as writable, readable or > discoverable as our > > proposed syntax we understand the desire to discourage a potential > > paradigm switch to `open classes?. > > > > We recognise that in many cases refactoring the code to eliminate the > need for monkey patching > > is the preferable solution. However, there are many instances where this > is not the case, such as > > the one Guido described in > https://mail.python.org/pipermail/python-dev/2008-January/076194.html. > > > > Personally, we have encountered the need for this in many projects each > of us have worked on. > > The fact that monkey patching is common enough lexicon to be mentioned > in Python programming > > books highlights the fact that programmers may encounter this in > codebases they work on. > > > > In particular when using third party libraries or code written in other > languages wrapped into Python, > > the existing syntaxes are less readable and non-standard. > > Each project using such codes is thus forced to come up with their own > custom solution leading > > to a somewhat fractured appearance in otherwise similar Python codebases. > > > > We had hoped to come up with a standardised solution. And if new syntax > is not desirable, perhaps > > including other solutions such as the proposed decorator in the standard > library is an option? > > Or do you feel this is a problem that should be solved by better > documentation and understanding > > instead? > > > > Kind regards, > > > > Pim Schellart & Nate Lust > >> On 14 Sep 2016, at 08:02, Nick Coghlan wrote: > >> > >> On 14 September 2016 at 05:46, Ralph Broenink > wrote: > >>> Hi all, > >>> > >>> On Tue, 13 Sep 2016 at 18:30 Chris Angelico wrote: > >>>> > >>>> Did you know that you can actually abuse decorators to do this with > >>>> existing syntax? Check out this collection of evil uses of decorators: > >>>> > >>>> https://github.com/Rosuav/Decorators/blob/master/evil.py > >>> > >>> There's also this post from Guido a few years back in python-dev, > which > >>> does something similar using metaclasses: > >>> > >>> https://mail.python.org/pipermail/python-dev/2008-January/076194.html > >>> > >>> I'm not sure it does exactly the same, but it is also an interesting > >>> approach. (Although I like the decorator syntax more.) The discussion > that > >>> follows in that thread may also be of interest for this discussion. > >> > >> PEP 422 and its reference implementation had a more fleshed out > >> implementation of that concept: > >> https://www.python.org/dev/peps/pep-0422/#new-ways-of-using-classes > >> > >> With class namespaces becoming ordered by default, I ended up > >> withdrawing PEP 422 in favour of the simpler PEP 487 that's actually > >> in 3.6: https://www.python.org/dev/peps/pep-0487/ > >> > >> I do think Pim's proposal is an excellent exemplar of what a PEP > >> should be, and if we *did* want to make class extension easier than it > >> already is, then the suggested "continue class ..." syntax would be an > >> elegant way of spelling it. > >> > >> However, I also believe the proposal founders on: > >> > >> 1. Class extensions are a fundamentally bad idea from a > >> maintainability perspective, as they make the original class > >> definition incomplete with no local indicator of its lack of > >> completeness (hence the "don't do this" caveat on the relevant example > >> in PEP 422) > >> 2. There are already ways to do this using metaclasses that provide a > >> local indicator at the point of definition that the affected class is > >> extensible (i.e. the use of the custom metaclass, or inheritance from > >> a base class that uses it) and hence that the given definition may be > >> incomplete > >> > >> Cheers, > >> Nick. > >> > >> -- > >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 15 05:21:25 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 15 Sep 2016 09:21:25 +0000 Subject: [Python-ideas] Generics Syntax Message-ID: This suggestion is so obvious that it's likely has been discussed, but I can't find any reference (It's not what PEP-3124 talks about). Generic class syntax, now: _T_co = TypeVar('_T', covariant=True) class Container(Generic[_T_co]): @abstractmethod def __contains__(self, x: _T_co) -> bool: ... (yes it's object in reality) Generic class syntax, suggested: class Container[+T]: @abstractmethod def __contains__(self, x: T) -> bool: ... The + signifies covariant type, as in Scala. The need for underscore prefix and explicit name for covariant types is gone, since it's class-scoped. The + is a bit cryptic, but so are term "covariant" and Generic[_T_co]. The syntax by large is exactly what users coming from statically-typed languages will expect. Giving a bound can be done using "T <: AnyStr", "T: AnyStr" or any other syntax. Again, I assume it was discussed before, but my Google skills are failing me; a pointer will be helpful. ~Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Thu Sep 15 05:43:17 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 15 Sep 2016 11:43:17 +0200 Subject: [Python-ideas] Generics Syntax In-Reply-To: References: Message-ID: On 15 September 2016 at 11:21, ????? wrote: > This suggestion is so obvious that it's likely has been discussed, but I > can't find any reference (It's not what PEP-3124 talks about). > > You might want to read this tread https://github.com/python/typing/issues/211 and another tread mentioned there at the start. In short, this idea has been discussed, but nobody had seen it as good enough to sacrifice his time for implementing the changes. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 15 05:44:24 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 15 Sep 2016 09:44:24 +0000 Subject: [Python-ideas] Generics Syntax In-Reply-To: References: Message-ID: Does that mean that if I did, it would be reconsidered? On Thu, Sep 15, 2016 at 12:43 PM Ivan Levkivskyi wrote: > On 15 September 2016 at 11:21, ????? wrote: > >> This suggestion is so obvious that it's likely has been discussed, but I >> can't find any reference (It's not what PEP-3124 talks about). >> >> > You might want to read this tread > https://github.com/python/typing/issues/211 and another tread mentioned > there at the start. > In short, this idea has been discussed, but nobody had seen it as good > enough to sacrifice his time for implementing the changes. > > -- > Ivan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 15 05:46:02 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 15 Sep 2016 09:46:02 +0000 Subject: [Python-ideas] Generics Syntax In-Reply-To: References: Message-ID: And that thread is only about variance. What about the generic syntax? ?On Thu, Sep 15, 2016 at 12:44 PM ???????? wrote:? > Does that mean that if I did, it would be reconsidered? > > On Thu, Sep 15, 2016 at 12:43 PM Ivan Levkivskyi > wrote: > >> On 15 September 2016 at 11:21, ????? wrote: >> >>> This suggestion is so obvious that it's likely has been discussed, but I >>> can't find any reference (It's not what PEP-3124 talks about). >>> >>> >> You might want to read this tread >> https://github.com/python/typing/issues/211 and another tread mentioned >> there at the start. >> In short, this idea has been discussed, but nobody had seen it as good >> enough to sacrifice his time for implementing the changes. >> >> -- >> Ivan >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Thu Sep 15 05:53:54 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 15 Sep 2016 11:53:54 +0200 Subject: [Python-ideas] Generics Syntax In-Reply-To: References: Message-ID: On 15 September 2016 at 11:46, ????? wrote: > And that thread is only about variance. What about the generic syntax? > If you mean code like this: class Container[+T]: @abstractmethod def __contains__(self, x: T) -> bool: ... then there is little chance that this will be accepted because it requires changes to Python syntax. You have how much friction PEP 526 caused, here it will be even more. > ?On Thu, Sep 15, 2016 at 12:44 PM ???????? wrote:? > >> Does that mean that if I did, it would be reconsidered? >> > Maybe, Guido will decide on this. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 15 07:03:21 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Sep 2016 21:03:21 +1000 Subject: [Python-ideas] Generics Syntax In-Reply-To: References: Message-ID: On 15 September 2016 at 19:53, Ivan Levkivskyi wrote: > > > On 15 September 2016 at 11:46, ????? wrote: >> >> And that thread is only about variance. What about the generic syntax? > > > If you mean code like this: > > class Container[+T]: > @abstractmethod > def __contains__(self, x: T) -> bool: ... > > then there is little chance that this will be accepted because it requires > changes to Python syntax. If the proposed spelling is tweaked to be "class Container(Generic[+T]):", then it doesn't require a syntax change, as that's merely a matter of implementing unary plus on type vars: >>> +object() Traceback (most recent call last): File "", line 1, in TypeError: bad operand type for unary +: 'object' >>> class UnaryPlus: ... def __pos__(self): ... return self ... >>> +UnaryPlus() <__main__.UnaryPlus object at 0x7f5e0fe91c50> (I have no opinion on the value of providing a simpler spelling for covariance, I'm just noting that if you keep the "Generic[T]" aspect of the current spelling it wouldn't require any changes to Python's syntax and will work as far back as you care to support it) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From elazarg at gmail.com Thu Sep 15 07:11:43 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 15 Sep 2016 11:11:43 +0000 Subject: [Python-ideas] Generics Syntax In-Reply-To: References: Message-ID: Yes the "class A[T]:" syntax requires on the ability to express variance as an operator, but not the other way around. It might be an argument in favor of switching to the + syntax: to make possible future syntax change in class definition somewhat easier to swallow. ~Elazar On Thu, Sep 15, 2016 at 2:03 PM Nick Coghlan wrote: > On 15 September 2016 at 19:53, Ivan Levkivskyi > wrote: > > > > > > On 15 September 2016 at 11:46, ????? wrote: > >> > >> And that thread is only about variance. What about the generic syntax? > > > > > > If you mean code like this: > > > > class Container[+T]: > > @abstractmethod > > def __contains__(self, x: T) -> bool: ... > > > > then there is little chance that this will be accepted because it > requires > > changes to Python syntax. > > If the proposed spelling is tweaked to be "class > Container(Generic[+T]):", then it doesn't require a syntax change, as > that's merely a matter of implementing unary plus on type vars: > > >>> +object() > Traceback (most recent call last): > File "", line 1, in > TypeError: bad operand type for unary +: 'object' > >>> class UnaryPlus: > ... def __pos__(self): > ... return self > ... > >>> +UnaryPlus() > <__main__.UnaryPlus object at 0x7f5e0fe91c50> > > (I have no opinion on the value of providing a simpler spelling for > covariance, I'm just noting that if you keep the "Generic[T]" aspect > of the current spelling it wouldn't require any changes to Python's > syntax and will work as far back as you care to support it) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 15 07:41:35 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Sep 2016 21:41:35 +1000 Subject: [Python-ideas] Generics Syntax In-Reply-To: References: Message-ID: On 15 September 2016 at 21:11, ????? wrote: > Yes the "class A[T]:" syntax requires on the ability to express variance as > an operator, but not the other way around. > > It might be an argument in favor of switching to the + syntax: to make > possible future syntax change in class definition somewhat easier to > swallow. A future syntax change is very unlikely, as people encountering "class SomeName(Generic[+T]):" for the first time can easily google "python generic class" and hopefully find useful answers in the documentation for typing.Generic. By contrast, "class SomeName[+T]:" would have to be covered directly in the class statement documentation, which runs completely counter to the notion of type hints as executable code annotations (rather than being an unavoidable core part of the language the way they are in statically typed languages). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Thu Sep 15 13:03:43 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Sep 2016 03:03:43 +1000 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: Message-ID: <20160915170343.GT22471@ando.pearwood.info> On Wed, Sep 14, 2016 at 09:51:32PM +1000, Chris Angelico wrote: > Perhaps the advice needs to be along the lines of: Decide what the > purpose of the enum is, and follow a naming convention accordingly. > Uppercase if you're basically making constants; lowercase if you're > not; etcetera. I can't think of any situation where Enums would *not* be treated as constants. Can you give an example? -- Steve From ethan at stoneleaf.us Thu Sep 15 13:47:11 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 15 Sep 2016 10:47:11 -0700 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: <20160915170343.GT22471@ando.pearwood.info> References: <20160915170343.GT22471@ando.pearwood.info> Message-ID: <57DADE9F.9030909@stoneleaf.us> On 09/15/2016 10:03 AM, Steven D'Aprano wrote: > On Wed, Sep 14, 2016 at 09:51:32PM +1000, Chris Angelico wrote: >> Perhaps the advice needs to be along the lines of: Decide what the >> purpose of the enum is, and follow a naming convention accordingly. >> Uppercase if you're basically making constants; lowercase if you're >> not; etcetera. > > I can't think of any situation where Enums would *not* be treated as > constants. Can you give an example? Besides being CONSTANTS enum members are also Singletons, and since I find CamelCase easier to read than UPPERCASE I may use that instead; otherwise I leave it all lowercase. -- ~Ethan~ From rosuav at gmail.com Thu Sep 15 13:54:18 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 16 Sep 2016 03:54:18 +1000 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: <20160915170343.GT22471@ando.pearwood.info> References: <20160915170343.GT22471@ando.pearwood.info> Message-ID: On Fri, Sep 16, 2016 at 3:03 AM, Steven D'Aprano wrote: > On Wed, Sep 14, 2016 at 09:51:32PM +1000, Chris Angelico wrote: > >> Perhaps the advice needs to be along the lines of: Decide what the >> purpose of the enum is, and follow a naming convention accordingly. >> Uppercase if you're basically making constants; lowercase if you're >> not; etcetera. > > I can't think of any situation where Enums would *not* be treated as > constants. Can you give an example? Sometimes they function as integer constants (esp IntEnum), and sometimes more as just arbitrary values. See the examples in the docs [1] for Color and Mood, where the exact value is immaterial, just as long as Color.red is not Color.blue. Even though they happen to have integer values, they're not intended to be used as actual integers. For those cases, it's not as clear that they ought to be Color.RED and Color.BLUE - it's equally acceptable to have them in lowercase. But if one convention or the other had to be picked for all enums, I would go with all-caps, since they're most often going to be representing constant integers, per Guido's post. I don't know of _any_ real-world cases where they're not integers, though there are some examples in the docs. ChrisA [1] https://docs.python.org/3/library/enum.html From guido at python.org Thu Sep 15 14:06:37 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Sep 2016 11:06:37 -0700 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: <20160915170343.GT22471@ando.pearwood.info> Message-ID: On Thu, Sep 15, 2016 at 10:54 AM, Chris Angelico wrote: > On Fri, Sep 16, 2016 at 3:03 AM, Steven D'Aprano wrote: >> On Wed, Sep 14, 2016 at 09:51:32PM +1000, Chris Angelico wrote: >> >>> Perhaps the advice needs to be along the lines of: Decide what the >>> purpose of the enum is, and follow a naming convention accordingly. >>> Uppercase if you're basically making constants; lowercase if you're >>> not; etcetera. >> >> I can't think of any situation where Enums would *not* be treated as >> constants. Can you give an example? > > Sometimes they function as integer constants (esp IntEnum), and > sometimes more as just arbitrary values. See the examples in the docs > [1] for Color and Mood, where the exact value is immaterial, just as > long as Color.red is not Color.blue. Even though they happen to have > integer values, they're not intended to be used as actual integers. > For those cases, it's not as clear that they ought to be Color.RED and > Color.BLUE - it's equally acceptable to have them in lowercase. I disagree. The whole point of an enum is to define a new *kind* of constant. Clearly RED and BLUE are new constants. (The uppercase convention is not just for giving names to literals -- that's not what "constant" means.) > But if one convention or the other had to be picked for all enums, I > would go with all-caps, since they're most often going to be > representing constant integers, per Guido's post. I don't know of > _any_ real-world cases where they're not integers, though there are > some examples in the docs. Amen. > ChrisA > > [1] https://docs.python.org/3/library/enum.html -- --Guido van Rossum (python.org/~guido) From rosuav at gmail.com Thu Sep 15 14:14:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 16 Sep 2016 04:14:56 +1000 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: <20160915170343.GT22471@ando.pearwood.info> Message-ID: On Fri, Sep 16, 2016 at 4:06 AM, Guido van Rossum wrote: >> Sometimes they function as integer constants (esp IntEnum), and >> sometimes more as just arbitrary values. See the examples in the docs >> [1] for Color and Mood, where the exact value is immaterial, just as >> long as Color.red is not Color.blue. Even though they happen to have >> integer values, they're not intended to be used as actual integers. >> For those cases, it's not as clear that they ought to be Color.RED and >> Color.BLUE - it's equally acceptable to have them in lowercase. > > I disagree. The whole point of an enum is to define a new *kind* of > constant. Clearly RED and BLUE are new constants. (The uppercase > convention is not just for giving names to literals -- that's not what > "constant" means.) Well, my view was largely colored [1] by the official docs. Your view is broadly the same as my ideal preference, just without the caveats and flexibility of accepting it the other way. So should the docs be updated to upper-case everything? With or without an actual PEP 8 change, that would send a strong message to the community. I'm sure I'm not the only person who looks at official examples as a style guide as well as a syntax demo. ChrisA [1] Sorry [2], that pun was just terrible [2] Actually I'm not sorry. But the pun was still terrible. From guido at python.org Thu Sep 15 14:26:10 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Sep 2016 11:26:10 -0700 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: <20160915170343.GT22471@ando.pearwood.info> Message-ID: Please update the docs. On Thu, Sep 15, 2016 at 11:14 AM, Chris Angelico wrote: > On Fri, Sep 16, 2016 at 4:06 AM, Guido van Rossum wrote: >>> Sometimes they function as integer constants (esp IntEnum), and >>> sometimes more as just arbitrary values. See the examples in the docs >>> [1] for Color and Mood, where the exact value is immaterial, just as >>> long as Color.red is not Color.blue. Even though they happen to have >>> integer values, they're not intended to be used as actual integers. >>> For those cases, it's not as clear that they ought to be Color.RED and >>> Color.BLUE - it's equally acceptable to have them in lowercase. >> >> I disagree. The whole point of an enum is to define a new *kind* of >> constant. Clearly RED and BLUE are new constants. (The uppercase >> convention is not just for giving names to literals -- that's not what >> "constant" means.) > > Well, my view was largely colored [1] by the official docs. Your view > is broadly the same as my ideal preference, just without the caveats > and flexibility of accepting it the other way. So should the docs be > updated to upper-case everything? > > With or without an actual PEP 8 change, that would send a strong > message to the community. I'm sure I'm not the only person who looks > at official examples as a style guide as well as a syntax demo. > > ChrisA > > [1] Sorry [2], that pun was just terrible > [2] Actually I'm not sorry. But the pun was still terrible. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From steve at pearwood.info Thu Sep 15 14:32:07 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Sep 2016 04:32:07 +1000 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: <20160915170343.GT22471@ando.pearwood.info> Message-ID: <20160915183203.GU22471@ando.pearwood.info> On Fri, Sep 16, 2016 at 03:54:18AM +1000, Chris Angelico wrote: > On Fri, Sep 16, 2016 at 3:03 AM, Steven D'Aprano wrote: > > On Wed, Sep 14, 2016 at 09:51:32PM +1000, Chris Angelico wrote: > > > >> Perhaps the advice needs to be along the lines of: Decide what the > >> purpose of the enum is, and follow a naming convention accordingly. > >> Uppercase if you're basically making constants; lowercase if you're > >> not; etcetera. > > > > I can't think of any situation where Enums would *not* be treated as > > constants. Can you give an example? > > Sometimes they function as integer constants (esp IntEnum), and > sometimes more as just arbitrary values. See the examples in the docs > [1] for Color and Mood, where the exact value is immaterial, just as > long as Color.red is not Color.blue. Even though they happen to have > integer values, they're not intended to be used as actual integers. Yes, but when would you write something like this? Color.red = That's not to be confused with the case: text_colour = Color.RED # later... text_colour = Color.BLUE where *text_colour* is clearly used as a variable, not a constant. But the enumerations themselves are still constant. -- Steve From rosuav at gmail.com Thu Sep 15 15:02:00 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 16 Sep 2016 05:02:00 +1000 Subject: [Python-ideas] Naming convention for Enums In-Reply-To: References: <20160915170343.GT22471@ando.pearwood.info> Message-ID: On Fri, Sep 16, 2016 at 4:26 AM, Guido van Rossum wrote: > Please update the docs. Roger that. http://bugs.python.org/issue28172 ChrisA From elazarg at gmail.com Thu Sep 15 20:10:22 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 16 Sep 2016 00:10:22 +0000 Subject: [Python-ideas] typing.modifiers Message-ID: With the new variable annotation syntax, it is possible to implement a useful "modifiers" library, all used as superclasses. Possible modifiers: * Named / Struct: annotation-defined fields. No monkey patching. * Immutable / Const * Sealed / Final: unsubclassable class * Array: a mutable tuple. (No "+" or "*" operators) There of course may be others. Subclasses of the modifiers get default methods and behavior. The modifiers can used in combinations. For example: class Command(Immutable, Named): cmd : str = "NOP" value : int = None load = Command(cmd="load", value=1000) # no positional arguments The syntax for Array is less pretty: class Command(Array): _0 : str = "NOP" _1 : int = None load = Command("load", 1000) # no positional arguments Many other combinations are useful too. "Sealed" is almost orthogonal to the others, Obviously, some combinations already exists: * (Immutable, Array) is similar to Tuple * NamedTuple is another name for (Immutable, Named, Array). * Enum should be on this list too Some less promising modifiers: * Namespace / Static: uninstantiable class. A module. * Volatile, for externally facing classes. Hints the static checkers / jitters that they should not assume they know the values. * Abstract Alternatives and cons : I suggest base classes instead of decorators since NamedTuple and tuple go this way, and since it is static information. I am not sure which is better though. Metaclass parameters can be used (and actually used in my implementation, and in NamedTuple's), but the syntax is uglier. "array" is only for numeric values in Python, so this name is problematic.So is struct. I have a rough implementation for most of the above; much of it is not hard in general, though some details are hard to get right. --- Benefits of putting such a collection in stdlib (instead of as an external package) include: 1. This information can be used by typecheckers, and also by users, to reason about programs. If isinstance(x, ImmutableArray), then x is an instantiation of ImmutableArray. 2. A conventional syntax and a single answer for "How do I make my class immutable", "How do I make my class unsubclassable" 3. The syntax, especially for Struct as above, is pretty and clean. The Array syntax is less so. 4. I think that the array implementation can use internal CPython details to be implemented efficiently. I am not sure that typing.modifiers is the right place, since these are not exactly type hints; they generate methods, and are intended to be enforced at runtime. I think that even if this idea is not accepted, the general theme is something that might be useful to keep in mind, stdlib might accumulate such modifiers, and it will be nice to keep things uniform. Opinions? ~Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Sep 16 06:15:56 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Sep 2016 20:15:56 +1000 Subject: [Python-ideas] typing.modifiers In-Reply-To: References: Message-ID: <20160916101555.GW22471@ando.pearwood.info> On Fri, Sep 16, 2016 at 12:10:22AM +0000, ????? wrote: [...] > Benefits of putting such a collection in stdlib (instead of as an external > package) include: Slow down! Before getting all excited about adding these typing hints(?) into typing.modifiers, you first have to convince people that they are useful and deserve a place in the std library. What problem are these hints/classes supposed to solve? What solutions already exist? Why aren't those solutions good enough? > 1. This information can be used by typecheckers, and also by users, to > reason about programs. If isinstance(x, ImmutableArray), then x is an > instantiation of ImmutableArray. That's how type-checkers work. The class doesn't need to be in the std lib for a type-checker to reason about it. > 2. A conventional syntax and a single answer for "How do I make my class > immutable", "How do I make my class unsubclassable" Do we need syntax for those? > 3. The syntax, especially for Struct as above, is pretty and clean. The > Array syntax is less so. I don't even understand what the Array syntax is supposed to mean. > 4. I think that the array implementation can use internal CPython details > to be implemented efficiently. What happens to Jython, PyPy, IronPython etc? > I am not sure that typing.modifiers is the right place, since these are not > exactly type hints; they generate methods, and are intended to be enforced > at runtime. Then I think your answer is: no, typing.modifiers is NOT the right place. > I think that even if this idea is not accepted, the general theme is > something that might be useful to keep in mind, stdlib might accumulate > such modifiers, and it will be nice to keep things uniform. The stdlib might accumulate many things. Why should it accumulate these? -- Steve From songofacandy at gmail.com Fri Sep 16 06:41:25 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 16 Sep 2016 19:41:25 +0900 Subject: [Python-ideas] Adding bytes.frombuffer() constructor In-Reply-To: References: Message-ID: >> >> The main question to be answered here would be whether adding a dedicated >> spelling for "bytes(memoryview(bytearray)[:n])" actually smooths out the >> learning curve for memoryview in general, where folks would learn: >> >> 1. "bytes(mybytearray[:n])" copies the data twice for no good reason >> 2. "bytes.frombuffer(mybytearray, n)" avoids the double copy >> 3. "bytes(memoryview(mybytearray)[:n])" generalises to arbitrary slices >> >> With memoryview being a builtin, I'm not sure that argument can be made >> successfully - the transformation in going from step 1 direct to step 3 is >> just "wrap the original object with memoryview before slicing to avoid the >> double copy", and that's no more complicated than using a different >> constructor method. > > > I'm not sure, too. > memoryview may and may not be bytes-like object which os.write or > socket.send accepts. > > But memoryview is successor of buffer. So we should encourage to > use it for zero copy slicing. > Today, @socketpair found my bug relating to this. https://github.com/python/asyncio/pull/395#r79136785 For example, Bad (but works on CPython): packet_bytes = bytes(memoryview(buff)[:packet_len]) del buff[:packet_len] Good: with memoryview(buff) as m: packet_bytes = bytes(m[:packet_len]) del buff[:packet_len] There are two problem. 1. Avoiding bad code is difficult. It works without any warning on CPython 2. Good code has significant overhead. Slicing bytes from bytearray buffer is usually in low level code and performance may be important. So I feel dedicated API for slicing bytes from bytes-like is worth enough. Any thought about it? From elazarg at gmail.com Fri Sep 16 06:41:45 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 16 Sep 2016 10:41:45 +0000 Subject: [Python-ideas] typing.modifiers In-Reply-To: <20160916101555.GW22471@ando.pearwood.info> References: <20160916101555.GW22471@ando.pearwood.info> Message-ID: Thanks for the reply ?????? ??? ??, 16 ????' 2016, 13:16, ??? Steven D'Aprano ?< steve at pearwood.info>: > On Fri, Sep 16, 2016 at 12:10:22AM +0000, ????? wrote: > > [...] > > Benefits of putting such a collection in stdlib (instead of as an > external > > package) include: > > Slow down! Before getting all excited about adding these typing hints(?) > into typing.modifiers, you first have to convince people that they are > useful and deserve a place in the std library. > I was thinking this is the place to do this? What problem are these hints/classes supposed to solve? What solutions > already exist? Why aren't those solutions good enough? > I have addressed that only very briefly; I will try to elaborate later (I'm writing from the phone. Sorry) > > > 1. This information can be used by typecheckers, and also by users, to > > reason about programs. If isinstance(x, ImmutableArray), then x is an > > instantiation of ImmutableArray. > > That's how type-checkers work. The class doesn't need to be in the std > lib for a type-checker to reason about it. > No, it's not how they work, since it's not true. I meant the actual type, not a subtype. > > > > 2. A conventional syntax and a single answer for "How do I make my class > > immutable", "How do I make my class unsubclassable" > > Do we need syntax for those? > > > > 3. The syntax, especially for Struct as above, is pretty and clean. The > > Array syntax is less so. > > I don't even understand what the Array syntax is supposed to mean. > That's already a bad sign... Each underscore gives type to its index. The last index is the maximal. > > > 4. I think that the array implementation can use internal CPython details > > to be implemented efficiently. > > What happens to Jython, PyPy, IronPython etc? > Similarly or even more so. > > I am not sure that typing.modifiers is the right place, since these are > not > > exactly type hints; they generate methods, and are intended to be > enforced > > at runtime. > > Then I think your answer is: no, typing.modifiers is NOT the right > place. > Speculating it will make its way, where should it land then? > > > I think that even if this idea is not accepted, the general theme is > > something that might be useful to keep in mind, stdlib might accumulate > > such modifiers, and it will be nice to keep things uniform. > > The stdlib might accumulate many things. Why should it accumulate these? > In this part I wasn't talking about what should happen, but rather what might happen gradually, in which case it'll be nice to fit in a uniform structure. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Fri Sep 16 07:21:04 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 16 Sep 2016 11:21:04 +0000 Subject: [Python-ideas] typing.modifiers In-Reply-To: References: <20160916101555.GW22471@ando.pearwood.info> Message-ID: > > > 1. This information can be used by typecheckers, and also by users, to >> > reason about programs. If isinstance(x, ImmutableArray), then x is an >> > instantiation of ImmutableArray. >> >> That's how type-checkers work. The class doesn't need to be in the std >> lib for a type-checker to reason about it. >> > No, it's not how they work, since it's not true. I meant the actual type, > not a subtype. > I now see my error - I meant isinstance(x, SealedArray)... sorry. Similarly the reasoning about ImmutableStruct with constant propagation. ~Elazar > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Fri Sep 16 09:22:03 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 16 Sep 2016 13:22:03 +0000 Subject: [Python-ideas] typing.modifiers In-Reply-To: <20160916101555.GW22471@ando.pearwood.info> References: <20160916101555.GW22471@ando.pearwood.info> Message-ID: On Fri, Sep 16, 2016 at 1:16 PM Steven D'Aprano wrote: > On Fri, Sep 16, 2016 at 12:10:22AM +0000, ????? wrote: > > [...] > > Benefits of putting such a collection in stdlib (instead of as an > external > > package) include: > > Slow down! Before getting all excited about adding these typing hints(?) > into typing.modifiers, you first have to convince people that they are > useful and deserve a place in the std library. > > What problem are these hints/classes supposed to solve? What > solutions already exist? Why aren't those solutions good enough? > > It is partially described below: mainly, reasoning and readability. I will try to elaborate, and please forgive me if what I'm writing is obvious to everyone here. Note that by "readability" I mean "conventional, non-repetitive syntax, the allow human reasoning", not the subjective issue (which is also important). Q. Why standard? A. Python is not very good at allowing strict reasoning. The confidence in Python programs, insofar it exists, comes from its reliance on very strong conventions ("One Way To Do It") and mostly-explicit control flow and name binding. Data flow, however, is very much implicit since the language encourages mutation. Tools that employ conventions are generally linters; these are very useful in finding trivial bugs, but have zero guarantees about their absence - unless the inference is trivial. linters and IDEs can also use basic form of type information, but will not analyze it deeply; hence the need for convention for modifiers that themselves constrain mutability, subclassing, etc.As a gradually-typed language, constraints are opt-in at best, but there should be a way to opt-them-in in a conventional way. Type hints are very good, and the suggested modifiers complement them. --- Q. What solutions already exist? Why aren't those solutions good enough? A: * Named: There are no conventions and no standard way to prohibit monkey-patching, since this flexibility is often useful. That's fine. But what if I want to tell the other programmer that this class should not be monkey patched, but its fields are mutable? Currently the general solution is something like class Person: name = None id = None def __init__(self, name, id): self.name = name self.od = id Say we ignore the repetition. Will tools catch the last typo? Some might warn, but how should a tool know that it is not intentional? There's no standard way to express this intent. mypy will say that's an error (if there are type annotations), but we get the other problem of *allowing* monkey patching. StackOverflow suggests either NamedTuple (which is not what I want in this case, since it is immutable and indexable), another answer without __init__ (which requires useful default values, and no sign of the "non-mankey-patch" intention). another answer with arbitrary **kwargs assignment (very flexible, but zero reasoning). Yet another suggests `dict` which is useful but has nothing to do with reasoning. There are many solutions in pypi, most only address the boilerplate issue; but the "attrs" package seems nice; it might fit in the alternative decorator-oriented framework I mentioned. There isn't even a standard name for that feature. "Struct" means something else. * Immutable: The importance to reasoning is obvious. There are many alternative suggestions; again, most upvoted is NamedTuple, which is great but also indexable. This means one kind of type error that will not be caught. (actually two, since the __init__ method takes positional arguments that can be mixed). Additionally, inheriting classes can be monkey-patched, and there's no standard way to disallow inheritance. The above problems can be solved in an ad-hoc manner, but it won't give tool support. Other solutions involve messing up with __setattr__ etc. It Again, "attrs" seems has a solution in the form of "freeze". * Sealed: Useful for typechecker (prevent erroneus cast) and optimizers that need to know the actual implementation of method dispatch (I believe it can be used in CPython too, but that's a different topic). The only solution I've found is this answer http://stackoverflow.com/questions/16564198/pythons-equivalent-of-nets-sealed-class which gives a hand-crafeted metaclass. But if it's not conventional, it won't be used by public tools. * Array: If I want a tuple with mutable cells, I currently simply don't have any option. There's list, which is extendable. There's array.array and numpy's matrices, which are intended for numeric processing. There might be some package for that, but I can't find it - and of course none that allow type annotations. Besides, there's no reason to assume it will work together with some other "Named" modifier implementation. * Namespace (as a shorthand for Final+Abstract): "not instantiable" will help catch both erroneous instantiations (which is implementable by hand) and erroneous "isinstance" (which is not). It might be used as an equivalent to "object" in Scala. ---- My suggestion is (again) two steps mixed: 1. Add support for modifiers, by convention and library 2. Some concrete syntax, inspired by Enum and the new NamedTuple syntax. Alternative syntax, using "attrs"-like decorators, might be: @modifiers(Immutable, Named) class Command: cmd : str = "NOP" value : int = None load = Command(cmd="load", value=1000) But then `enum` and `NamedTuple` should be updated to match this convention. Again, even if this suggestion is not accepted right now, there should be some decision to avoid accumulating parts of it in a way that is not consistent and will not allow combinations. For example, If it the addition of modifiers was forced on the language by God, would it be in the form of decorators? if your answer is yes, then I believe the new NamedTuple form should become a decorator too, before it is too late (as in Enum). -- P.S. how do I change the name in my quotes? I believe ????? is not very easy to address... ~Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Sep 16 10:05:51 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 Sep 2016 15:05:51 +0100 Subject: [Python-ideas] typing.modifiers In-Reply-To: References: <20160916101555.GW22471@ando.pearwood.info> Message-ID: On 16 September 2016 at 14:22, ????? wrote: > P.S. how do I change the name in my quotes? I believe ????? is not very easy > to address... As you're posting from a gmail account, it's likely your name from personal info in your Google Accounts settings. Your mail client (if it's not the gmail web app) may have a way of changing it, too. Paul From rosuav at gmail.com Fri Sep 16 13:39:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 17 Sep 2016 03:39:08 +1000 Subject: [Python-ideas] typing.modifiers In-Reply-To: References: <20160916101555.GW22471@ando.pearwood.info> Message-ID: On Fri, Sep 16, 2016 at 11:22 PM, ????? wrote: > P.S. how do I change the name in my quotes? I believe ????? is not very easy > to address... > TBH I wouldn't worry about it. If people can't cite names using Hebrew script, that's their problem, not yours. :) ChrisA From alexander.belopolsky at gmail.com Fri Sep 16 13:55:53 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 16 Sep 2016 13:55:53 -0400 Subject: [Python-ideas] typing.modifiers In-Reply-To: References: <20160916101555.GW22471@ando.pearwood.info> Message-ID: On Fri, Sep 16, 2016 at 1:39 PM, Chris Angelico wrote: > If people can't cite names using Hebrew > script, that's their problem, not yours. :) > https://hg.python.org/cpython/rev/0308f97616a3 Enjoy! -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Sep 16 21:47:40 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 17 Sep 2016 11:47:40 +1000 Subject: [Python-ideas] typing.modifiers In-Reply-To: References: <20160916101555.GW22471@ando.pearwood.info> Message-ID: <20160917014739.GZ22471@ando.pearwood.info> On Sat, Sep 17, 2016 at 03:39:08AM +1000, Chris Angelico wrote: > On Fri, Sep 16, 2016 at 11:22 PM, ????? wrote: > > P.S. how do I change the name in my quotes? I believe ????? is not very easy > > to address... > > > > TBH I wouldn't worry about it. If people can't cite names using Hebrew > script, that's their problem, not yours. :) [in-joke, that Chris will get] But how do we know that ????? is his real name? It looks made up to me. [/in-joke] Still, thank you Elazar for also providing a Latin-1 compatible name which is readable and pronounceable by English speakers. -- Steve From spencerb21 at live.com Sat Sep 17 05:01:53 2016 From: spencerb21 at live.com (Spencer Brown) Date: Sat, 17 Sep 2016 09:01:53 +0000 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? Message-ID: Currently, calling divmod() on a class with __floordiv__ and __mod__ defined, but not __divmod__ raises a TypeError. Is there any reason why it doesn't fallback to (self // x, self % x)? The invariants described in the function documentation should still apply (with well behaved % and //). If divmod() doesn't make sense setting it to None should be permitted. - Spencer -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Sat Sep 17 06:51:16 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 17 Sep 2016 11:51:16 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL Message-ID: Hello, I would like to suggest adding a clear command (not function) to Python. It's simple purpose would be to clear the REPL screen, leaving the >>> prompt at the top left of the screen. This is something very basic but also very useful for newbies learning Python from the REPL. After some trial and errors it is best to start with a clean screen. Clearing the screen helps clear your mind. Historically it is a common command in interpreted languages. Best regards, JM From jcrmatos at gmail.com Sat Sep 17 06:55:42 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 17 Sep 2016 11:55:42 +0100 Subject: [Python-ideas] (Windows-only - calling Steve Dower) Consider adding a symlink to pip in the same location as the py launcher Message-ID: <751dab9f-8aa3-d6fa-2a5b-6b2956fb9527@gmail.com> Hello, If Py3.5 is installed in user mode instead of admin (all users) and we follow your advice that we shouldn't add it to the PATH env var, we can execute Python using the py launcher, but we can't use pip. Please consider adding a pip symlink in the same location as the py launcher. Best regards, JM From dickinsm at gmail.com Sat Sep 17 06:56:15 2016 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 17 Sep 2016 11:56:15 +0100 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2016 at 10:01 AM, Spencer Brown wrote: > Currently, calling divmod() on a class with __floordiv__ and __mod__ > defined, but not __divmod__ raises a TypeError. Is there any reason why it > doesn't fallback to (self // x, self % x)? It's an interesting idea. I wonder whether the falling back shouldn't be in the other direction, though: that is, if a class defines `__divmod__` but not `__floordiv__` or `__mod__`, perhaps the `//` and `%` operations should use `__divmod__`? That way, if you're writing a class that intends to support all three operations, you only have to write one method. And it might make sense from an efficiency perspective, too; it's common for a `divmod` computation to be cheaper than doing separate computations of the quotient and remainder. For the builtin int type, for example, in nontrivial cases Python computes both the quotient and remainder when asked to do a % or // operation, and then discards whichever part isn't needed. So in that case it would be wasteful to build up the divmod result from two separate % and // calls. -- Mark From jcrmatos at gmail.com Sat Sep 17 07:01:15 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 17 Sep 2016 12:01:15 +0100 Subject: [Python-ideas] (Windows-only - calling Steve Dower) Is Python for Windows using PGO? If not consider this a suggestion. Message-ID: Hello, Is Python for Windows using PGO (Profile Guided Optimizations)? If not consider this a suggestion. Best regards, JM From rosuav at gmail.com Sat Sep 17 07:07:27 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 17 Sep 2016 21:07:27 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: Message-ID: On Sat, Sep 17, 2016 at 8:51 PM, Jo?o Matos wrote: > I would like to suggest adding a clear command (not function) to Python. > It's simple purpose would be to clear the REPL screen, leaving the >>> > prompt at the top left of the screen. > > This is something very basic but also very useful for newbies learning > Python from the REPL. > After some trial and errors it is best to start with a clean screen. > Clearing the screen helps clear your mind. > I'm not sure that it _is_ helpful, given that you're starting with a clean screen but not restarting the session (so you'll still have all the state from your previous work). If you want a completely fresh start, just exit Python, clear the screen with a shell command, and re-enter. The easiest way to play around with this would be to create a pure Python clear() function in your usercustomize or sitecustomize, and then try it in your own workflow - see whether it annoys you that it doesn't change the interpreter state. Maybe it will, maybe it won't. ChrisA From phd at phdru.name Sat Sep 17 07:07:50 2016 From: phd at phdru.name (Oleg Broytman) Date: Sat, 17 Sep 2016 13:07:50 +0200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: Message-ID: <20160917110750.GA27126@phdru.name> Hi! On Sat, Sep 17, 2016 at 11:51:16AM +0100, Jo??o Matos wrote: > Hello, > > I would like to suggest adding a clear command (not function) to Python. Pressing [Ctrl]+[L] works for me. > Best regards, > JM Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From p.f.moore at gmail.com Sat Sep 17 07:10:05 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 17 Sep 2016 12:10:05 +0100 Subject: [Python-ideas] (Windows-only - calling Steve Dower) Consider adding a symlink to pip in the same location as the py launcher In-Reply-To: <751dab9f-8aa3-d6fa-2a5b-6b2956fb9527@gmail.com> References: <751dab9f-8aa3-d6fa-2a5b-6b2956fb9527@gmail.com> Message-ID: On 17 September 2016 at 11:55, Jo?o Matos wrote: > If Py3.5 is installed in user mode instead of admin (all users) and we > follow your advice that we shouldn't add it to the PATH env var, we can > execute Python using the py launcher, but we can't use pip. > Please consider adding a pip symlink in the same location as the py launcher One problem with this is that the py launcher can handle multiple versions of Python installed on the same machine, which the pip executable can't. To launch pip without having the Python directory on PATH, you can use "py -m pip". It's not quite as convenient as plain "pip", but it avoids the "which version?" problem. Paul From jcrmatos at gmail.com Sat Sep 17 07:11:51 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 17 Sep 2016 12:11:51 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160917110750.GA27126@phdru.name> References: <20160917110750.GA27126@phdru.name> Message-ID: Hello, Doesn't work on Windows. Best regards, JM On 17-09-2016 12:07, Oleg Broytman wrote: > Hi! > > On Sat, Sep 17, 2016 at 11:51:16AM +0100, Jo??o Matos wrote: >> Hello, >> >> I would like to suggest adding a clear command (not function) to Python. > Pressing [Ctrl]+[L] works for me. > >> Best regards, >> JM > Oleg. From jcrmatos at gmail.com Sat Sep 17 07:12:18 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 17 Sep 2016 12:12:18 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: Message-ID: <6fa94ec8-f651-cbd7-b42d-233a7b2969d0@gmail.com> Hello, In other interpreted programming languages the clear screen command (whatever it is) also does not clear the session. It just clears the screen clutter. As I said, this would be very useful for newbies, which don't know anything about usercustomize or sitecustomize. Best regards, JM On 17-09-2016 12:07, Chris Angelico wrote: > On Sat, Sep 17, 2016 at 8:51 PM, Jo?o Matos wrote: >> I would like to suggest adding a clear command (not function) to Python. >> It's simple purpose would be to clear the REPL screen, leaving the >>> >> prompt at the top left of the screen. >> >> This is something very basic but also very useful for newbies learning >> Python from the REPL. >> After some trial and errors it is best to start with a clean screen. >> Clearing the screen helps clear your mind. >> > I'm not sure that it _is_ helpful, given that you're starting with a > clean screen but not restarting the session (so you'll still have all > the state from your previous work). If you want a completely fresh > start, just exit Python, clear the screen with a shell command, and > re-enter. > > The easiest way to play around with this would be to create a pure > Python clear() function in your usercustomize or sitecustomize, and > then try it in your own workflow - see whether it annoys you that it > doesn't change the interpreter state. Maybe it will, maybe it won't. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From jcrmatos at gmail.com Sat Sep 17 07:18:36 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 17 Sep 2016 12:18:36 +0100 Subject: [Python-ideas] (Windows-only - calling Steve Dower) Consider adding a symlink to pip in the same location as the py launcher In-Reply-To: References: <751dab9f-8aa3-d6fa-2a5b-6b2956fb9527@gmail.com> Message-ID: Hello, OK, I didn't know that option. Thanks. May I suggest adding a notice to the Windows Python Installer alerting to use py instead of python and py -m pip instead of pip (only if the user doesn't choose the option to add it to the path, of course)? On the other hand, maybe instead of a pip symlink, could it be a pip executable that uses the same logic as py launcher to find the correct pip? Best regards, JM On 17-09-2016 12:10, Paul Moore wrote: > On 17 September 2016 at 11:55, Jo?o Matos wrote: >> If Py3.5 is installed in user mode instead of admin (all users) and we >> follow your advice that we shouldn't add it to the PATH env var, we can >> execute Python using the py launcher, but we can't use pip. >> Please consider adding a pip symlink in the same location as the py launcher > One problem with this is that the py launcher can handle multiple > versions of Python installed on the same machine, which the pip > executable can't. > > To launch pip without having the Python directory on PATH, you can use > "py -m pip". It's not quite as convenient as plain "pip", but it > avoids the "which version?" problem. > Paul From wes.turner at gmail.com Sat Sep 17 09:15:21 2016 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 17 Sep 2016 08:15:21 -0500 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <6fa94ec8-f651-cbd7-b42d-233a7b2969d0@gmail.com> References: <6fa94ec8-f651-cbd7-b42d-233a7b2969d0@gmail.com> Message-ID: With IPython, there are a number of ways to reset the terminal display: clear # %clear !cls #windows !reset - http://stackoverflow.com/questions/6892191/clearing-the-screen-in-ipython - Ctrl-L is a readline binding - http://pythonhosted.org/pyreadline/usage.html#pyreadline-with-python-interpreter - https://anaconda.org/anaconda/pyreadline - https://pypi.python.org/pypi/pyreadline - IPython >= 5 no longer uses pyreadline (instead, python prompt toolkit) - https://github.com/jonathanslenders/python-prompt-toolkit/blob/master/prompt_toolkit/shortcuts.py #def clear - http://unix.stackexchange.com/questions/124762/how-does-clear-command-work - http://urwid.org/reference/display_modules.html#urwid.raw_display.Screen.clear - https://rosettacode.org/wiki/Terminal_control/Clear_the_screen#Python On Saturday, September 17, 2016, Jo?o Matos wrote: > Hello, > > In other interpreted programming languages the clear screen command > (whatever it is) also does not clear the session. > It just clears the screen clutter. > > As I said, this would be very useful for newbies, which don't know > anything about usercustomize or sitecustomize. > > > Best regards, > > JM > > > On 17-09-2016 12:07, Chris Angelico wrote: > >> On Sat, Sep 17, 2016 at 8:51 PM, Jo?o Matos wrote: >> >>> I would like to suggest adding a clear command (not function) to Python. >>> It's simple purpose would be to clear the REPL screen, leaving the >>> >>> prompt at the top left of the screen. >>> >>> This is something very basic but also very useful for newbies learning >>> Python from the REPL. >>> After some trial and errors it is best to start with a clean screen. >>> Clearing the screen helps clear your mind. >>> >>> I'm not sure that it _is_ helpful, given that you're starting with a >> clean screen but not restarting the session (so you'll still have all >> the state from your previous work). If you want a completely fresh >> start, just exit Python, clear the screen with a shell command, and >> re-enter. >> >> The easiest way to play around with this would be to create a pure >> Python clear() function in your usercustomize or sitecustomize, and >> then try it in your own workflow - see whether it annoys you that it >> doesn't change the interpreter state. Maybe it will, maybe it won't. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eryksun at gmail.com Sat Sep 17 09:34:41 2016 From: eryksun at gmail.com (eryk sun) Date: Sat, 17 Sep 2016 13:34:41 +0000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160917110750.GA27126@phdru.name> Message-ID: On Sat, Sep 17, 2016 at 11:11 AM, Jo?o Matos wrote: >On 17-09-2016 12:07, Oleg Broytman wrote: >> >> Pressing [Ctrl]+[L] works for me. > > Doesn't work on Windows. Windows 10 added VT100 support to the console, so you can create a little cls() function to clear the screen: cls = lambda: print('\x1b[1J', end='', flush=True) VT100 support isn't enabled by default. However, cmd.exe always enables this mode. So run doskey.exe via os.system to enable VT100 mode while setting a convenient "cls" alias: import os os.system(r'doskey /exename=python.exe cls=cls()') This alias substitutes "cls()" for "cls" at the beginning of a line. This saves you from having to type "()" all the time. The alias isn't active for other programs; e.g. if you execute subprocess.call('powershell'), then "cls" will be the PowerShell alias for Clear-Host. You can also use ctypes instead of os.system and doskey.exe: import ctypes kernel32 = ctypes.WinDLL('kernel32') hStdOut = kernel32.GetStdHandle(-11) # enable VT100 mode mode = (ctypes.c_uint * 1)() kernel32.GetConsoleMode(hStdOut, mode) kernel32.SetConsoleMode(hStdOut, mode[0] | 4) # define a cls() function and set a console alias cls = lambda: print('\x1b[1J', end='', flush=True) kernel32.AddConsoleAliasW('cls', 'cls()', 'python.exe') For older versions of Windows you can use ANSICON or ConEmu, which use DLL injection to extend the console API. Python's colorama and pyreadline modules should also work for this. From jcrmatos at gmail.com Sat Sep 17 09:36:37 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 17 Sep 2016 14:36:37 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <6fa94ec8-f651-cbd7-b42d-233a7b2969d0@gmail.com> Message-ID: Hello, I know about those IPython commands and I searched and found several possible solutions to clear the screen in the CPython REPL, but all are, in my opinion, complex for a newbie. The existence of a clear command would be simple and obvious, therefore accessible to newbies. Best regards, JM On 17-09-2016 14:15, Wes Turner wrote: > With IPython, there are a number of ways to reset the terminal display: > > clear # %clear > !cls #windows > !reset > > - > http://stackoverflow.com/questions/6892191/clearing-the-screen-in-ipython > - Ctrl-L is a readline binding > - > http://pythonhosted.org/pyreadline/usage.html#pyreadline-with-python-interpreter > - https://anaconda.org/anaconda/pyreadline > - https://pypi.python.org/pypi/pyreadline > - IPython >= 5 no longer uses pyreadline (instead, python prompt > toolkit) > - > https://github.com/jonathanslenders/python-prompt-toolkit/blob/master/prompt_toolkit/shortcuts.py > #def clear > > > - > http://unix.stackexchange.com/questions/124762/how-does-clear-command-work > - > http://urwid.org/reference/display_modules.html#urwid.raw_display.Screen.clear > - https://rosettacode.org/wiki/Terminal_control/Clear_the_screen#Python > > On Saturday, September 17, 2016, Jo?o Matos > wrote: > > Hello, > > In other interpreted programming languages the clear screen > command (whatever it is) also does not clear the session. > It just clears the screen clutter. > > As I said, this would be very useful for newbies, which don't know > anything about usercustomize or sitecustomize. > > > Best regards, > > JM > > > On 17-09-2016 12:07, Chris Angelico wrote: > > On Sat, Sep 17, 2016 at 8:51 PM, Jo?o Matos > wrote: > > I would like to suggest adding a clear command (not > function) to Python. > It's simple purpose would be to clear the REPL screen, > leaving the >>> > prompt at the top left of the screen. > > This is something very basic but also very useful for > newbies learning > Python from the REPL. > After some trial and errors it is best to start with a > clean screen. > Clearing the screen helps clear your mind. > > I'm not sure that it _is_ helpful, given that you're starting > with a > clean screen but not restarting the session (so you'll still > have all > the state from your previous work). If you want a completely fresh > start, just exit Python, clear the screen with a shell > command, and > re-enter. > > The easiest way to play around with this would be to create a pure > Python clear() function in your usercustomize or > sitecustomize, and > then try it in your own workflow - see whether it annoys you > that it > doesn't change the interpreter state. Maybe it will, maybe it > won't. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Sat Sep 17 09:37:22 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 17 Sep 2016 14:37:22 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160917110750.GA27126@phdru.name> Message-ID: <7cb2f614-4697-6912-69a1-bfd7fbfc07a5@gmail.com> Hello, I searched and found several possible solutions to clear the screen in the CPython REPL, but all are, in my opinion, complex for a newbie. The existence of a clear command would be a simple and obvious, therefore accessible to newbies. Best regards, JM On 17-09-2016 14:34, eryk sun wrote: > On Sat, Sep 17, 2016 at 11:11 AM, Jo?o Matos wrote: >> On 17-09-2016 12:07, Oleg Broytman wrote: >>> Pressing [Ctrl]+[L] works for me. >> Doesn't work on Windows. > Windows 10 added VT100 support to the console, so you can create a > little cls() function to clear the screen: > > cls = lambda: print('\x1b[1J', end='', flush=True) > > VT100 support isn't enabled by default. However, cmd.exe always > enables this mode. So run doskey.exe via os.system to enable VT100 > mode while setting a convenient "cls" alias: > > import os > os.system(r'doskey /exename=python.exe cls=cls()') > > This alias substitutes "cls()" for "cls" at the beginning of a line. > This saves you from having to type "()" all the time. The alias isn't > active for other programs; e.g. if you execute > subprocess.call('powershell'), then "cls" will be the PowerShell alias > for Clear-Host. > > You can also use ctypes instead of os.system and doskey.exe: > > import ctypes > kernel32 = ctypes.WinDLL('kernel32') > hStdOut = kernel32.GetStdHandle(-11) > > # enable VT100 mode > mode = (ctypes.c_uint * 1)() > kernel32.GetConsoleMode(hStdOut, mode) > kernel32.SetConsoleMode(hStdOut, mode[0] | 4) > > # define a cls() function and set a console alias > cls = lambda: print('\x1b[1J', end='', flush=True) > kernel32.AddConsoleAliasW('cls', 'cls()', 'python.exe') > > For older versions of Windows you can use ANSICON or ConEmu, which use > DLL injection to extend the console API. Python's colorama and > pyreadline modules should also work for this. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From eryksun at gmail.com Sat Sep 17 10:13:33 2016 From: eryksun at gmail.com (eryk sun) Date: Sat, 17 Sep 2016 14:13:33 +0000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <6fa94ec8-f651-cbd7-b42d-233a7b2969d0@gmail.com> Message-ID: On Sat, Sep 17, 2016 at 1:15 PM, Wes Turner wrote: > !cls #windows cmd's built-in cls command doesn't clear just the screen, like a VT100 \x1b[1J. It clears the console's entire scrollback buffer. Unix `clear` may also work like that. With GNOME Terminal in Linux, `clear` leaves a single screen in the scrollback buffer. From elazarg at gmail.com Sat Sep 17 12:37:28 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sat, 17 Sep 2016 16:37:28 +0000 Subject: [Python-ideas] typing.modifiers In-Reply-To: <20160917014739.GZ22471@ando.pearwood.info> References: <20160916101555.GW22471@ando.pearwood.info> <20160917014739.GZ22471@ando.pearwood.info> Message-ID: Thank you all! ????? (AKA Elazar) On Sat, Sep 17, 2016 at 4:53 AM Steven D'Aprano wrote: > On Sat, Sep 17, 2016 at 03:39:08AM +1000, Chris Angelico wrote: > > On Fri, Sep 16, 2016 at 11:22 PM, ????? wrote: > > > P.S. how do I change the name in my quotes? I believe ????? is not > very easy > > > to address... > > > > > > > TBH I wouldn't worry about it. If people can't cite names using Hebrew > > script, that's their problem, not yours. :) > > [in-joke, that Chris will get] > But how do we know that ????? is his real name? It looks made up to me. > [/in-joke] > > Still, thank you Elazar for also providing a Latin-1 compatible name > which is readable and pronounceable by English speakers. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Sep 17 13:34:29 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 17 Sep 2016 10:34:29 -0700 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: References: Message-ID: It seems like this could be something similar to `functools.total_ordering` and decorate a class. In principle that transformation could go in either direction, but only if the decorator is used. On Sat, Sep 17, 2016 at 3:56 AM, Mark Dickinson wrote: > On Sat, Sep 17, 2016 at 10:01 AM, Spencer Brown > wrote: > > Currently, calling divmod() on a class with __floordiv__ and __mod__ > > defined, but not __divmod__ raises a TypeError. Is there any reason why > it > > doesn't fallback to (self // x, self % x)? > > It's an interesting idea. I wonder whether the falling back shouldn't > be in the other direction, though: that is, if a class defines > `__divmod__` but not `__floordiv__` or `__mod__`, perhaps the `//` and > `%` operations should use `__divmod__`? That way, if you're writing a > class that intends to support all three operations, you only have to > write one method. And it might make sense from an efficiency > perspective, too; it's common for a `divmod` computation to be cheaper > than doing separate computations of the quotient and remainder. > > For the builtin int type, for example, in nontrivial cases Python > computes both the quotient and remainder when asked to do a % or // > operation, and then discards whichever part isn't needed. So in that > case it would be wasteful to build up the divmod result from two > separate % and // calls. > > -- > Mark > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Sep 17 13:42:48 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 17 Sep 2016 10:42:48 -0700 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: References: Message-ID: <57DD8098.1030005@stoneleaf.us> On 09/17/2016 10:34 AM, David Mertz wrote: > On Sat, Sep 17, 2016 at 3:56 AM, Mark Dickinson wrote: >> On Sat, Sep 17, 2016 at 10:01 AM, Spencer Brown wrote: >>> Currently, calling divmod() on a class with __floordiv__ and __mod__ >>> defined, but not __divmod__ raises a TypeError. Is there any reason >>> why it doesn't fallback to (self // x, self % x)? >> >> It's an interesting idea. I wonder whether the falling back shouldn't >> be in the other direction, though: that is, if a class defines >> `__divmod__` but not `__floordiv__` or `__mod__`, perhaps the `//` and >> `%` operations should use `__divmod__`? That way, if you're writing a > > It seems like this could be something similar to `functools.total_ordering` >and decorate a class. In principle that transformation could go in either > direction, but only if the decorator is used. Not at all. Currently Python will fallback to `not ==` if a class does not define `!=`. Having `//` and `%` fall back to `__divmod__` would not be out of place. -- ~Ethan~ From mertz at gnosis.cx Sat Sep 17 13:57:49 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 17 Sep 2016 10:57:49 -0700 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: <57DD8098.1030005@stoneleaf.us> References: <57DD8098.1030005@stoneleaf.us> Message-ID: The fallback you described would be a change in the behavior of some working programs. Moreover, it would only affect custom classes where adding a decorator is an option (even in external code, you can use `MyThing = total_divmod(library.MyThing)` under this option. Showing that a recipe for a decorator is of wide use feels like a necessary step towards any future semantic change in the language to me. For example, '%' is fairly widely (ab)used for meanings other than modulo. E.g. string formatting. Probably not that many classes that respond to '%' to do something non-modulo simultaneously implement `.__divmod__()` ... but maybe some use case is not obvious to me. If those exist, your change would break that (well, depending whether methods of ancestors are used or not). On Sat, Sep 17, 2016 at 10:42 AM, Ethan Furman wrote: > On 09/17/2016 10:34 AM, David Mertz wrote: > >> On Sat, Sep 17, 2016 at 3:56 AM, Mark Dickinson wrote: >> >>> On Sat, Sep 17, 2016 at 10:01 AM, Spencer Brown wrote: >>> >> > Currently, calling divmod() on a class with __floordiv__ and __mod__ >>>> defined, but not __divmod__ raises a TypeError. Is there any reason >>>> why it doesn't fallback to (self // x, self % x)? >>>> >>> >>> It's an interesting idea. I wonder whether the falling back shouldn't >>> be in the other direction, though: that is, if a class defines >>> `__divmod__` but not `__floordiv__` or `__mod__`, perhaps the `//` and >>> `%` operations should use `__divmod__`? That way, if you're writing a >>> >> >> It seems like this could be something similar to >> `functools.total_ordering` >> and decorate a class. In principle that transformation could go in either >> direction, but only if the decorator is used. >> > > Not at all. Currently Python will fallback to `not ==` if a class does not > define `!=`. Having `//` and `%` fall back to `__divmod__` would not be > out of place. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Sep 17 14:05:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 18 Sep 2016 04:05:08 +1000 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: References: <57DD8098.1030005@stoneleaf.us> Message-ID: On Sun, Sep 18, 2016 at 3:57 AM, David Mertz wrote: > For example, '%' is fairly widely (ab)used for meanings other than modulo. > E.g. string formatting. Probably not that many classes that respond to '%' > to do something non-modulo simultaneously implement `.__divmod__()` ... but > maybe some use case is not obvious to me. If those exist, your change would > break that (well, depending whether methods of ancestors are used or not). So there'll be classes that define __mod__ but not __divmod__ - that's fine. Are there any that go the other way around? It might technically be a breaking change, but it's unlikely to cause very much breakage. So it'd be inappropriate for 3.6.1, but okay for 3.7. +1 on the fallback exactly as Ethan described. The One Obvious Way to implement arithmetic division would be to define __divmod__. Defining __truediv__ or __mod__ would be for times when you can implement it more efficiently by not calculating the other half, and of course the times when they're not implementing division (cf pathlib.Path and str, for / and % respectively). ChrisA From tjreedy at udel.edu Sat Sep 17 15:43:02 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 17 Sep 2016 15:43:02 -0400 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: Message-ID: On 9/17/2016 6:51 AM, Jo?o Matos wrote: > Hello, > > I would like to suggest adding a clear command (not function) to Python. > It's simple purpose would be to clear the REPL screen, leaving the >>> > prompt at the top left of the screen. > > This is something very basic but also very useful for newbies learning > Python from the REPL. > After some trial and errors it is best to start with a clean screen. > Clearing the screen helps clear your mind. > > Historically it is a common command in interpreted languages. Python is not an 'interpreted' language in the sense that you are using the word. It is a compiled language that includes its compile function as part of the runtime. This enables an interactive mode. Python does not have commands, only statements, expressions, and functions, etcetera. This is why the 'exit()' and 'quit()' 'commands' are functions and require the parentheses. In interactive mode, a statement consisting of a single indentifier (or expression) causes display of a representation of the resulting object. You could request that the site module also add a clear() function. But the next problem is that clearing the screen is not a Python function and the Python runtime does not normally know how to do so. The implementation of 'clear screen' is specific to the particular console or window or terminal emulation. The control sequence for a window that emulates VT100 does not work on Windows Command Prompt (excepting possibly a non-default CP on Win 10) or necessarily IDLE or IPython or PyCharm or ... . The only assumption that Python makes about the screen it is running on is that '\n' is interpreted as a newline. IDLE has a 'Restart Shell' menu command with key binding. It does not physically clear the screen but it draws a double line, which serves most of the 'clear the mind' purpose. If there is an editor window open, one can close and reopen the Shell window to start really fresh. At least on Windows, Select All (^A) + Del (or Backspace) is the common idiom for clearing an editor box in a GUI. Is this used on other OSes? This works in an IDLE editor also. It is disabled in the Shell because IDLE's original designers thought to protect beginners from doing such. A clear screen menu option has been requested, but it has never become a top priority. In any case, this, like other current clear screen commands, would work because the command would go to the window manager and not to python. -- Terry Jan Reedy From ethan at stoneleaf.us Sat Sep 17 18:06:29 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 17 Sep 2016 15:06:29 -0700 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: References: <57DD8098.1030005@stoneleaf.us> Message-ID: <57DDBE65.1090505@stoneleaf.us> On 09/17/2016 10:57 AM, David Mertz wrote: > The fallback you described would be a change in the behavior of some > working programs. Moreover, it would only affect custom classes > where adding a decorator is an option (even in external code, you > can use `MyThing = total_divmod(library.MyThing)` under this option. I'm really not sure what you're saying here, but it sounds like you're concerned about a __divmod__ overriding existing __mod__ and __floordiv__? That is not the case. Just like Python will use the defined __ne__ if it's present, or fall back to negating the result of __eq__ if __ne__ is not present, I see __divmod__ working the same way: - is __mod__ present? use it - is __floordiv__ present? use it - otherwise, use __divmod__ and return the needed piece I'm pretty sure __div__ should not fall back to __divmod__. -- ~Ethan~ From rosuav at gmail.com Sat Sep 17 18:14:36 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 18 Sep 2016 08:14:36 +1000 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: <57DDBE65.1090505@stoneleaf.us> References: <57DD8098.1030005@stoneleaf.us> <57DDBE65.1090505@stoneleaf.us> Message-ID: On Sun, Sep 18, 2016 at 8:06 AM, Ethan Furman wrote: > Just like Python will use the defined __ne__ if > it's present, or fall back to negating the result of __eq__ if __ne__ is > not present, I see __divmod__ working the same way: > > - is __mod__ present? use it > - is __floordiv__ present? use it > - otherwise, use __divmod__ and return the needed piece > > I'm pretty sure __div__ should not fall back to __divmod__. How does __mod__ fall back to __floordiv__? I'm lost. ChrisA From ethan at stoneleaf.us Sat Sep 17 19:35:38 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 17 Sep 2016 16:35:38 -0700 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: References: <57DD8098.1030005@stoneleaf.us> <57DDBE65.1090505@stoneleaf.us> Message-ID: <57DDD34A.6080306@stoneleaf.us> On 09/17/2016 03:14 PM, Chris Angelico wrote: > On Sun, Sep 18, 2016 at 8:06 AM, Ethan Furman wrote: >> Just like Python will use the defined __ne__ if >> it's present, or fall back to negating the result of __eq__ if __ne__ is >> not present, I see __divmod__ working the same way: >> >> - is __mod__ present? use it >> - is __floordiv__ present? use it >> - otherwise, use __divmod__ and return the needed piece >> >> I'm pretty sure __div__ should not fall back to __divmod__. > > How does __mod__ fall back to __floordiv__? I'm lost. Oops, sorry. Got my directions reversed when thinking about how __div__ should fit in. Bird's eye view: if the exact method needed is present, use it; otherwise if a fallback method is available, use that. Currently this is done for __ne__ --> not __eq__, and I seem to remember another case or two that was talked about but I don't remember what they were and I'm not sure if they got implemented to follow the fallback pattern. -- ~Ethan~ From arek.bulski at gmail.com Sat Sep 17 20:52:31 2016 From: arek.bulski at gmail.com (Arek Bulski) Date: Sun, 18 Sep 2016 02:52:31 +0200 Subject: [Python-ideas] Overloading operators for testing Message-ID: I am using declarative testing a lot and I found out why unit tests are so clunky. The reason why assertEquals(a,b) is used is because if we put `assert a==b` then nose can catch the AssertionError but wont find out what was returned or expected. This could be easily overcome if we allow oveloading == operator from outside. Right now == would have to be changed for every lefhand object that is compared in the tests, builtin types including. We could use a way to change it from above, so to speak. Consider this: def __glob_eq__(a,b): if not a == b: raise FoundInequalityError(a,b) return True assert obj1 == obj2 #<-- using eq above Nose could easily catch FoundInequalityError and print whatever assertEquals would. This goes very handy when you consider declarative unit testing that I use in my project. I have a unitest.TestCase derivative and the actual testcase has a method that yields individual comparisons, like this: class TestBinary(declarativeunittest.TestCase): def alltestsinteractive(self): yield [func(1) == 2] shuffle(alist) yield [sorted(alist) == [1,2,3]] Notice that this allows to put imperative statements in between declarative cases. So shuffled() is no longer necessary in this code. :) pozdrawiam, Arkadiusz Bulski -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sat Sep 17 20:58:59 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sat, 17 Sep 2016 19:58:59 -0500 Subject: [Python-ideas] Overloading operators for testing In-Reply-To: References: Message-ID: FYI, pytest already does this: http://doc.pytest.org/en/latest/ -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On Sep 17, 2016 7:55 PM, "Arek Bulski" wrote: > I am using declarative testing a lot and I found out why unit tests are so > clunky. The reason why assertEquals(a,b) is used is because if we put > `assert a==b` then nose can catch the AssertionError but wont find out what > was returned or expected. This could be easily overcome if we allow > oveloading == operator from outside. Right now == would have to be changed > for every lefhand object that is compared in the tests, builtin types > including. We could use a way to change it from above, so to speak. > Consider this: > > def __glob_eq__(a,b): > if not a == b: > raise FoundInequalityError(a,b) > return True > > assert obj1 == obj2 #<-- using eq above > > Nose could easily catch FoundInequalityError and print whatever > assertEquals would. This goes very handy when you consider declarative unit > testing that I use in my project. I have a unitest.TestCase derivative and > the actual testcase has a method that yields individual comparisons, like > this: > > class TestBinary(declarativeunittest.TestCase): > def alltestsinteractive(self): > > yield [func(1) == 2] > shuffle(alist) > yield [sorted(alist) == [1,2,3]] > > Notice that this allows to put imperative statements in between > declarative cases. So shuffled() is no longer necessary in this code. :) > > pozdrawiam, > Arkadiusz Bulski > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Sep 18 00:01:35 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 18 Sep 2016 16:01:35 +1200 Subject: [Python-ideas] Overloading operators for testing In-Reply-To: References: Message-ID: <57DE119F.9050206@canterbury.ac.nz> Arek Bulski wrote: > def __glob_eq__(a,b): > if not a == b: > raise FoundInequalityError(a,b) > return True > > assert obj1 == obj2 #<-- using eq above How would you ensure that this overriding only applied in the places you want it? You don't want to change the meaning of == in the code under test! Related to that, how would you prevent the use of == in the definition of __glob_eq__ above from triggering infinite recursion? -- Greg From steve.dower at python.org Sun Sep 18 08:50:59 2016 From: steve.dower at python.org (Steve Dower) Date: Sun, 18 Sep 2016 05:50:59 -0700 Subject: [Python-ideas] (Windows-only - calling Steve Dower) Consider addinga symlink to pip in the same location as the py launcher In-Reply-To: <751dab9f-8aa3-d6fa-2a5b-6b2956fb9527@gmail.com> References: <751dab9f-8aa3-d6fa-2a5b-6b2956fb9527@gmail.com> Message-ID: I'd like to add a launcher in the same style as py.exe, but that would upset people who manually configure their PATH appropriately. Personally, I find "py.exe -m pip" quite okay, but appreciate the idea. I'm thinking about this issue (also for other scripts). Top-posted from my Windows Phone -----Original Message----- From: "Jo?o Matos" Sent: ?9/?17/?2016 3:57 To: "python-ideas at python.org" Subject: [Python-ideas] (Windows-only - calling Steve Dower) Consider addinga symlink to pip in the same location as the py launcher Hello, If Py3.5 is installed in user mode instead of admin (all users) and we follow your advice that we shouldn't add it to the PATH env var, we can execute Python using the py launcher, but we can't use pip. Please consider adding a pip symlink in the same location as the py launcher. Best regards, JM _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sun Sep 18 08:51:43 2016 From: steve.dower at python.org (Steve Dower) Date: Sun, 18 Sep 2016 05:51:43 -0700 Subject: [Python-ideas] (Windows-only - calling Steve Dower) Is Python forWindows using PGO? If not consider this a suggestion. In-Reply-To: References: Message-ID: It was disable previously because of compiler bugs. 3.6.0b1 64-bit has PGO enabled, but we'll disable it again if there are any issues. Top-posted from my Windows Phone -----Original Message----- From: "Jo?o Matos" Sent: ?9/?17/?2016 4:02 To: "python-ideas at python.org" Subject: [Python-ideas] (Windows-only - calling Steve Dower) Is Python forWindows using PGO? If not consider this a suggestion. Hello, Is Python for Windows using PGO (Profile Guided Optimizations)? If not consider this a suggestion. Best regards, JM _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Sun Sep 18 08:55:32 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Matos?=) Date: Sun, 18 Sep 2016 05:55:32 -0700 (PDT) Subject: [Python-ideas] (Windows-only - calling Steve Dower) Is Python forWindows using PGO? If not consider this a suggestion. In-Reply-To: References: Message-ID: <39be7d8e-bea5-4e5b-9b44-ff4f97e152e9@googlegroups.com> Hello, Ok, thanks for the feedback. Best regards, JM domingo, 18 de Setembro de 2016 ?s 13:53:44 UTC+1, Steve Dower escreveu: > > It was disable previously because of compiler bugs. 3.6.0b1 64-bit has PGO > enabled, but we'll disable it again if there are any issues. > > Top-posted from my Windows Phone > ------------------------------ > From: Jo?o Matos > Sent: ?9/?17/?2016 4:02 > To: python... at python.org > Subject: [Python-ideas] (Windows-only - calling Steve Dower) Is Python > forWindows using PGO? If not consider this a suggestion. > > Hello, > > Is Python for Windows using PGO (Profile Guided Optimizations)? If not > consider this a suggestion. > > Best regards, > > JM > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Sun Sep 18 08:55:05 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Matos?=) Date: Sun, 18 Sep 2016 05:55:05 -0700 (PDT) Subject: [Python-ideas] (Windows-only - calling Steve Dower) Consider addinga symlink to pip in the same location as the py launcher In-Reply-To: References: <751dab9f-8aa3-d6fa-2a5b-6b2956fb9527@gmail.com> Message-ID: <65125f66-b489-42e6-8e0c-874ebfe123a3@googlegroups.com> Hello, Ok, thanks for the feedback. Best regards, JM domingo, 18 de Setembro de 2016 ?s 13:52:44 UTC+1, Steve Dower escreveu: > I'd like to add a launcher in the same style as py.exe, but that would > upset people who manually configure their PATH appropriately. > > Personally, I find "py.exe -m pip" quite okay, but appreciate the idea. > I'm thinking about this issue (also for other scripts). > > Top-posted from my Windows Phone > ------------------------------ > From: Jo?o Matos > Sent: ?9/?17/?2016 3:57 > To: python... at python.org > Subject: [Python-ideas] (Windows-only - calling Steve Dower) Consider > addinga symlink to pip in the same location as the py launcher > > Hello, > > If Py3.5 is installed in user mode instead of admin (all users) and we > follow your advice that we shouldn't add it to the PATH env var, we can > execute Python using the py launcher, but we can't use pip. > Please consider adding a pip symlink in the same location as the py > launcher. > > Best regards, > > JM > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Sep 18 22:04:12 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 19 Sep 2016 12:04:12 +1000 Subject: [Python-ideas] divmod(): fallback to __floordiv__ and __mod__? In-Reply-To: References: Message-ID: <20160919020411.GA22471@ando.pearwood.info> On Sat, Sep 17, 2016 at 09:01:53AM +0000, Spencer Brown wrote: > Currently, calling divmod() on a class with __floordiv__ and __mod__ > defined, but not __divmod__ raises a TypeError. Is there any reason > why it doesn't fallback to (self // x, self % x)? Because things get really complex, fast. Does the added complexity pay its own way? I may have some of the details wrong, but I understand the implementation of divmod() is something like this: # pseudo-code for divmod def divmod(a, b): A, B = type(a), type(b) if issubclass(B, A): # Give priority to the reverse dunder version if hasattr(B, '__rdivmod__'): result = B.__rdivmod__(b, a) if result is not NotImplemented: return result if hasattr(A, '__divmod__'): result = A.__divmod__(a, b) if result is not NotImplemented: return result raise TypeError if hasattr(A, '__divmod__'): result = A.__divmod__(a, b) if result is not NotImplemented: return result if hasattr(B, '__rdivmod__'): result = B.__rdivmod__(b, a) if result is not NotImplemented: return result raise TypeError only it will be in C, so probably three or five times as much code :-) Now consider adding a fallback to __floordiv__ and __mod__ as suggested. That adds a lot more complexity: - what if the object has __floordiv__, but not __mod__? Do you try the reversed method, __rmod__, instead? - likewise for NotImplemented? - what if they're None? So let's look at the complexity of the fallback: # reverse dunder version, for when b is a subclass of a rdiv = getattr(B, '__rfloordiv__, None) rmod = getattr(B, '__rmod__, None) if rdiv is not None and rmod is not None: x, y = rdiv(b, a), rmod(b, a) if x is NotImplemented: div = getattr(A, '__floordiv__', None) if div is not None: x = div(a, b) if x is NotImplemented: raise TypeError if y is NotImplemented: mod = getattr(A, '__mod__', None) if mod is not None: y = mod(a, b) if y is NotImplemented: raise TypeError assert NotImplemented not in (x, y) else: # try the __floordiv__ and __mod__ versions, without falling # back to reversed versions ... return (x, y) And then more or less the same for the "standard" case where a has priority over b (i.e. isn't a subclass). So, my estimate is that this will roughly triple the complexity of the divmod() function. Perhaps worse. Oh, and I forgot the case where __divmod__ exists but is None, but I'm not going back to add it in because I'm not even sure where that ought to be tested. Now I daresay we can refactor my naive pseudo-code above, but however you look at it, there's going to be a big jump in complexity for not a lot of benefit. In the case of __ne__ falling back on not __eq__, there is at least an utterly clear and obvious user-expectation that the two methods are linked, and there are no reversed __req__ and __rne__ methods to care about. That's not the case here. Going the other way (fall back to __divmod__ if __floordiv__ or __mod__ is not defined) will probably be not quite as complex, but it will still add a fair amount of complexity. So, *emotionally* I actually do quite like the idea of having divmod fall back to // and %, *and* the other way, but *intellectually* when I think about the complexity and the tests that will be needed to cover all the cases, I have to say "Uh uh, no way!" It may be that the actual C implementation is simpler than I think, in which case maybe this is a reasonable idea, but on the face of it, I think that it will be really hard to get right, add a lot of code, and provide very little benefit. -- Steve From steve at pearwood.info Sun Sep 18 22:32:01 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 19 Sep 2016 12:32:01 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: Message-ID: <20160919023201.GB22471@ando.pearwood.info> On Sat, Sep 17, 2016 at 11:51:16AM +0100, Jo?o Matos wrote: > Hello, > > I would like to suggest adding a clear command (not function) to Python. While technically "clear" could be a command, I think it should not be. First off, making clear a reserved keyword, and a statement, like print in Python 2, raise or import, would be a major backwards compatibility break. It would mean dict.clear() has to be renamed, and it would break lots of existing code. So making clear a keyword is not going to happen. If could be a pseudo-built-in, like help(), quit() and exit(), added to built-ins by the site module. In that case, it is *technically* possible to have it operate without the parentheses: class ClearType: def __repr__(self): # code to clear the screen here ... clear = ClearType() so that when you enter clear at the interactive interpreter, __repr__ is called and it clears the screen. But I would argue against it. Instead, it is better to use the same convention that executable code that has side-effects should be implemented as a function call. So I suggest following the design of exit() and quit(): py> exit Use exit() or Ctrl-D (i.e. EOF) to exit class ClearType: def __repr__(self): return "Use clear() or Ctrl-L (i.e. FF) to clear the screen" def __call__(self): # clear screen code goes here clear = ClearType() # or possibly cls ? That is, *if* we add this function at all. Personally, I agree with you. There are many different ways of clearing the screen, but they depend on the specific terminal used, whether readline is active or not, the operating system, etc. I think that interactive use is important enough that we should have a standard way of clearing the screen. I personally often find myself just holding down the Enter key until I have a blank screen. In this ticket: http://bugs.python.org/issue27771 Raymond Hettinger mentions that it is an often-requested feature by learners, and I believe that IDLE has an active task for this feature: http://bugs.python.org/issue6143 but I don't see any tasks for a clear screen command for the default REPL. I'm in favour of adding a clear() *function* to the site.py module, similar to exit/quit/help, but not making it "magical" or a keyword that doesn't require brackets. But I don't know how to implement it for the large variety of terminals and operating systems supported by Python. (The fallback if all else fails is easy: get the height of the terminal, in lines, and print that many blank lines.) -- Steve From rosuav at gmail.com Sun Sep 18 22:40:32 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 19 Sep 2016 12:40:32 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160919023201.GB22471@ando.pearwood.info> References: <20160919023201.GB22471@ando.pearwood.info> Message-ID: On Mon, Sep 19, 2016 at 12:32 PM, Steven D'Aprano wrote: > (The fallback if all else fails is easy: get the height of the terminal, > in lines, and print that many blank lines.) Assuming you can get the height in lines. Have you tried that in the default Windows shell? I don't think tcgetattr works on Windows. ChrisA From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Sep 18 23:51:58 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 19 Sep 2016 12:51:58 +0900 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> Message-ID: <22495.24798.542857.183546@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > On Mon, Sep 19, 2016 at 12:32 PM, Steven D'Aprano wrote: > > (The fallback if all else fails is easy: get the height of the terminal, > > in lines, and print that many blank lines.) > > Assuming you can get the height in lines. Have you tried that in the > default Windows shell? I don't think tcgetattr works on Windows. Since a command is out of the question (look what happened to print!), it's a matter of defining a callable with a repr that explains how to call it (like help and friends). So if someone really wants this to happen, I would say the thing to do is to define that callable, put it on PyPI, add support for all the platforms, fix all the bugs, and when there are a couple million downloads, suggest preloading it in interpreter then. Don't forget how to document how to add it to site.py. But I would think that nowadays we'd push in the opposite direction (as with print). That is, with the great improvements in IDLE (batteries included!), IPython, and now we have Jupyter, you could now argue that the built-in REPL should lose at least one of the two exit functions (since EOF is a sufficient reason to exit). (help() still makes sense as the public interface to __doc__.) In other words, the built-in REPL should just provide some line-editing features and otherwise simply read lines, incrementally compile and execute them, and print results. Leave UI conveniences to other applications that (gasp!) specialize in providing consistent, powerful UI that isn't going to feel like bat guano on Tim's keyboard. IMHO YMMV, of course. From p.f.moore at gmail.com Mon Sep 19 04:31:11 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 09:31:11 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> Message-ID: On 19 September 2016 at 03:40, Chris Angelico wrote: > On Mon, Sep 19, 2016 at 12:32 PM, Steven D'Aprano wrote: >> (The fallback if all else fails is easy: get the height of the terminal, >> in lines, and print that many blank lines.) > > Assuming you can get the height in lines. Have you tried that in the > default Windows shell? I don't think tcgetattr works on Windows. shutil.get_terminal_size() is available on all platforms. I don't think it would be unreasonable to add shutil.clear_terminal() (at the moment, get_terminal_size is the only "terminal handling" function in shutil, but it would be the obvious place to add others if we chose to do so). Paul From rosuav at gmail.com Mon Sep 19 04:38:00 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 19 Sep 2016 18:38:00 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> Message-ID: On Mon, Sep 19, 2016 at 6:31 PM, Paul Moore wrote: > On 19 September 2016 at 03:40, Chris Angelico wrote: >> On Mon, Sep 19, 2016 at 12:32 PM, Steven D'Aprano wrote: >>> (The fallback if all else fails is easy: get the height of the terminal, >>> in lines, and print that many blank lines.) >> >> Assuming you can get the height in lines. Have you tried that in the >> default Windows shell? I don't think tcgetattr works on Windows. > > shutil.get_terminal_size() is available on all platforms. > > I don't think it would be unreasonable to add shutil.clear_terminal() > (at the moment, get_terminal_size is the only "terminal handling" > function in shutil, but it would be the obvious place to add others if > we chose to do so). Sounds good to me. This is definitely sounding complicated and messy enough to justify (a) writing a function to clear the screen, and (b) testing that function thoroughly as a PyPI module before pushing anything into the stdlib. ChrisA From tjreedy at udel.edu Mon Sep 19 06:18:38 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 Sep 2016 06:18:38 -0400 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> Message-ID: On 9/19/2016 4:31 AM, Paul Moore wrote: > shutil.get_terminal_size() is available on all platforms. On windows, it works with Command Prompt and PowerShell but fails in IDLE, as it must. In the absence of knowledge, it guesses the default of 80 x 24 (as documented). AFAIK, there is no way whether an answer of 80 x 24 is correct or arbitrary nonsense. Returning 0 x 0 instead would be more informative. Terry Jan Reedy From me at dpk.io Mon Sep 19 07:05:27 2016 From: me at dpk.io (David Kendal) Date: Mon, 19 Sep 2016 13:05:27 +0200 Subject: [Python-ideas] ___repr__ on collections.UserDict, collections.UserList, collections.UserString Message-ID: Hi, the UserDict, UserList, and UserString classes in the collections module ought to provide __repr__ implementations along the following lines: def __repr__(self): return '%s(%r)' % (type(self).__name__, self.data) so that they give useful immediate information in the repl to the effect that they are not mere dicts/lists/strs but user-defined container types. dpk (David P. Kendal) ? Nassauische Str. 36, 10717 DE ? http://dpk.io/ The reason we had no idea how cats worked was because, since Newton, we had proceeded by the simple principle that essentially, to see how things work, we took them apart. If you try and take a cat apart to see how it works, the first thing you have on your hands is a non- working cat. -- Douglas Adams From steve at pearwood.info Mon Sep 19 07:56:48 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 19 Sep 2016 21:56:48 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> Message-ID: <20160919115647.GD22471@ando.pearwood.info> On Mon, Sep 19, 2016 at 06:38:00PM +1000, Chris Angelico wrote: > Sounds good to me. This is definitely sounding complicated and messy > enough to justify (a) writing a function to clear the screen, and (b) > testing that function thoroughly as a PyPI module before pushing > anything into the stdlib. This isn't Node.js where standard operating procedure is to rely on the third-party npm ecosystem even for tiny, ten line functions. And we should be *glad* that's not the case: http://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/ People don't often install minor or small packages off PyPI as a matter of course: they either roll their own, or do without, depending on which is more painful, or whether they are even allowed to download third-party software (not everybody is), or whether managing the dependency is more or less work than writing their own. I get it that not everything belongs in the std lib, and I get it that for *some* people (but not all) its easy to rely on packages in PyPI. But there's quite considerable friction in getting *small* things off PyPI. For starters, you have to know it exists, you have to trust that its functional and well-written. There is a huge amount of junk, abandoned and "version 0.1" packages on PyPI that nobody in their right mind should use. Dependencies don't come for free, they add complexity and cost to a project. Not everything belongs as a package on PyPI either. If the cost of dealing with the added dependency is greater than the benefit gained, people won't use third-party packages on PyPI. For small but useful functionality, saying "Put it on PyPI" is effectively just a dismissal. For relatively small pieces of functionality, if it is useful enough, we should just add it to the std lib, and if it isn't, we should just say it isn't useful enough. We shouldn't condemn supporters of the idea to this false hope that if they can convince a thousand, or a million, people to download their package, it will be added to PyPI. -- Steve From p.f.moore at gmail.com Mon Sep 19 08:10:34 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 13:10:34 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160919115647.GD22471@ando.pearwood.info> References: <20160919023201.GB22471@ando.pearwood.info> <20160919115647.GD22471@ando.pearwood.info> Message-ID: On 19 September 2016 at 12:56, Steven D'Aprano wrote: > > For relatively small pieces of functionality, if it is useful enough, we > should just add it to the std lib, and if it isn't, we should just say > it isn't useful enough. We shouldn't condemn supporters of the idea to > this false hope that if they can convince a thousand, or a million, > people to download their package, it will be added to PyPI "... to the stdlib". Agreed. However, there are libraries already on PyPI that try to act in the role of "useful bits that aren't in the stdlib". The boltons project is one that I know of. Maybe a middle ground between stdlib inclusion and outright rejection would be the suggestion to submit the code to one of those 3rd party projects? There's no implication that doing so means that there's any better chance of getting accepted for the stdlib, but it does offer an option for people who want to publish their idea for wider use. For this particular suggestion, though, I don't think that's the case. I think it's going to either be something that's accepted into the stdlib, or something that's rejected as too platform-specific or messy to standardise, and people should roll their own implementation. I'm inclined to think there may be some scope for a blog entry or HOWTO on customising your personal Python environment - what facilities there are to do so, what other options (such as different REPLs) are available, and how to judge whether it's worth doing or not. Lots of people (myself included at times!) seem to be unaware of the options available. It's not something I'm likely to ever write, but if anyone is interested in doing so, it might be useful background for this type of discussion. Paul From rosuav at gmail.com Mon Sep 19 08:14:30 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 19 Sep 2016 22:14:30 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160919115647.GD22471@ando.pearwood.info> References: <20160919023201.GB22471@ando.pearwood.info> <20160919115647.GD22471@ando.pearwood.info> Message-ID: On Mon, Sep 19, 2016 at 9:56 PM, Steven D'Aprano wrote: > On Mon, Sep 19, 2016 at 06:38:00PM +1000, Chris Angelico wrote: > >> Sounds good to me. This is definitely sounding complicated and messy >> enough to justify (a) writing a function to clear the screen, and (b) >> testing that function thoroughly as a PyPI module before pushing >> anything into the stdlib. > > This isn't Node.js where standard operating procedure > is to rely on the third-party npm ecosystem even for tiny, ten line > functions. And we should be *glad* that's not the case: I agree; however, the bar for getting something onto PyPI is far lower than the stdlib, and it makes it far easier to get a bit of testing (hey, folks, here's what I'm playing with, can you try pip-installing it and see if it works on your platform please). So the "first on PyPI, then in the stdlib" route does have a lot of merit. But you're quite right that we don't want to depend on a ton of packages.... ChrisA From p.f.moore at gmail.com Mon Sep 19 09:12:48 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 14:12:48 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> <20160919115647.GD22471@ando.pearwood.info> Message-ID: On 19 September 2016 at 13:10, Paul Moore wrote: > > For this particular suggestion, though, I don't think that's the case. > I think it's going to either be something that's accepted into the > stdlib, or something that's rejected as too platform-specific or messy > to standardise, and people should roll their own implementation. Note, by the way, that in the form of a Python function, this capability is already available in lots of places - colorama and click include it, for a start. And even the stdlib has a version in the curses module (albeit not available on Windows). In the light of this, there seems little reason to water down this proposal to "provide a clear screen function in the stdlib". Taken in its original form of "add a command to the REPL to clear the screen", the main issue is that the standard Python REPL simply doesn't have a concept of "REPL commands", so there's nowhere really to add that functionality without a relatively major overhaul. People who want a richer REPL can look at other options like IDLE, or IPython/Jupyter. By the way - if you're on a system with readline support included with Python, GNU readline apparently has a binding for clear-screen (CTRL-L) so you may well have this functionality already (I don;'t use Unix or readline, so I can't comment for sure). So the proposal becomes limited to: """ Add a capability to the standard REPL to recognise and support "commands" (as distinct from executable Python code) and provide a "clear screen" command. """ This feature is only needed for users of the standard Python REPL, on systems without readline support (i.e. Windows). I don't think that's a significant enough benefit to warrant the change (even if we take into account the potential opportunity for additional commands that we'd get from adding command support to the REPL). Paul From eryksun at gmail.com Mon Sep 19 10:45:53 2016 From: eryksun at gmail.com (eryk sun) Date: Mon, 19 Sep 2016 14:45:53 +0000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> <20160919115647.GD22471@ando.pearwood.info> Message-ID: On Mon, Sep 19, 2016 at 1:12 PM, Paul Moore wrote: > By the way - if you're on a system with readline support included with > Python, GNU readline apparently has a binding for clear-screen > (CTRL-L) so you may well have this functionality already (I don;'t use > Unix or readline, so I can't comment for sure). Hooking Ctrl+L to clear the screen can be implemented for Windows Vista and later via the ReadConsole pInputControl parameter, as called by PyOS_StdioReadline. It should be possible to match how GNU readline works -- i.e. clear the screen, reprint the prompt, flush the input buffer, and write the current line's input back to the input buffer. The pInputControl parameter can also be used to implement Unix-style Ctrl+D to end a read anywhere on a line, whereas the classic [Ctrl+Z][Enter] has to be entered at the start of a line. From phd at phdru.name Mon Sep 19 11:07:20 2016 From: phd at phdru.name (Oleg Broytman) Date: Mon, 19 Sep 2016 17:07:20 +0200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> <20160919115647.GD22471@ando.pearwood.info> Message-ID: <20160919150720.GA27644@phdru.name> On Mon, Sep 19, 2016 at 02:45:53PM +0000, eryk sun wrote: > The pInputControl parameter can also be used to implement Unix-style > Ctrl+D to end a read anywhere on a line, whereas the classic > [Ctrl+Z][Enter] has to be entered at the start of a line. [Ctrl+D] also recognized as EOF only at the start of an input. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From zachary.ware+pyideas at gmail.com Mon Sep 19 10:49:28 2016 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Mon, 19 Sep 2016 09:49:28 -0500 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> <20160919115647.GD22471@ando.pearwood.info> Message-ID: On Mon, Sep 19, 2016 at 9:45 AM, eryk sun wrote: > On Mon, Sep 19, 2016 at 1:12 PM, Paul Moore wrote: >> By the way - if you're on a system with readline support included with >> Python, GNU readline apparently has a binding for clear-screen >> (CTRL-L) so you may well have this functionality already (I don;'t use >> Unix or readline, so I can't comment for sure). > > Hooking Ctrl+L to clear the screen can be implemented for Windows > Vista and later via the ReadConsole pInputControl parameter, as called > by PyOS_StdioReadline. It should be possible to match how GNU readline > works -- i.e. clear the screen, reprint the prompt, flush the input > buffer, and write the current line's input back to the input buffer. > > The pInputControl parameter can also be used to implement Unix-style > Ctrl+D to end a read anywhere on a line, whereas the classic > [Ctrl+Z][Enter] has to be entered at the start of a line. That sounds lovely, any chance you could work up a patch? :) -- Zach From desmoulinmichel at gmail.com Mon Sep 19 11:24:10 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 19 Sep 2016 17:24:10 +0200 Subject: [Python-ideas] Make partial a built-in Message-ID: While lambda are a great tools that can do the job of partial, partial is always a superior solution when trying to curry a Python function because it's explicit and easier to debug. Having to import it everytime means I usually go for a lambda out of lazyness, but also means that most people don't know about it. If think having it in the builtins would promote cleaner, faster code, but would also facilitate and encourage coding patterns using curried functions. From eryksun at gmail.com Mon Sep 19 11:39:15 2016 From: eryksun at gmail.com (eryk sun) Date: Mon, 19 Sep 2016 15:39:15 +0000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160919150720.GA27644@phdru.name> References: <20160919023201.GB22471@ando.pearwood.info> <20160919115647.GD22471@ando.pearwood.info> <20160919150720.GA27644@phdru.name> Message-ID: On Mon, Sep 19, 2016 at 3:07 PM, Oleg Broytman wrote: > [Ctrl+D] also recognized as EOF only at the start of an input. You're right. I was mixing it up with sys.stdin.buffer.raw.read(), for which Ctrl+D can end the read anywhere if entered twice. Having to enter it twice may be a bug, because os.read(0, 100) works fine. So making it compatible with GNU readline is more complicated, but I think it's possible. If the Ctrl+D isn't at the start of the line, it could write the result up to the "\x04" back to the input buffer and resume the read. From elazarg at gmail.com Mon Sep 19 11:59:45 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 15:59:45 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: If at all, it should be function.bind(). It was discussed and dropped; I don't remember why, but one problem is that it looks like an in-place modification. Personally I think that higher order programming should not be encouraged (as opposed to functional programming), and from the lambda syntax I understand it isn't. Elazar ?????? ??? ??, 19 ????' 2016, 18:24, ??? Michel Desmoulin ?< desmoulinmichel at gmail.com>: > While lambda are a great tools that can do the job of partial, partial > is always a superior solution when trying to curry a Python function > because it's explicit and easier to debug. > > Having to import it everytime means I usually go for a lambda out of > lazyness, but also means that most people don't know about it. > > If think having it in the builtins would promote cleaner, faster code, > but would also facilitate and encourage coding patterns using curried > functions. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Mon Sep 19 12:25:49 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 16:25:49 +0000 Subject: [Python-ideas] from __pip__ import Message-ID: Many proposals to add something to stdlib are rejected here with the suggestion to add such library to pypi first. As noted by someone, pypi is not as reachable as stdlib, and one should install that package first, which many people don't know how. Additionally, there is no natural distinction between 3rd party dependencies and in-project imports (at least in tiny projects). This can be made easier if the first line of the program will declare the required library, and executing it will try to download and install that library if it is not installed yet. Additionally, the 3rd party dependencies will be more explicit, and editors can then allow you to search for them as you type. Of course it is *not* an alternative for real dependency management, but it will ease the burden on small scripts and tiny projects - which today simply break with errors that many users does not understand, instead of simply asking permission to install the dependency. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Sep 19 12:38:12 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Sep 2016 09:38:12 -0700 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: Message-ID: <57E01474.8060309@stoneleaf.us> On 09/19/2016 09:25 AM, ????? wrote: > Many proposals to add something to stdlib are rejected here with the suggestion to add such library to pypi first. As noted by someone, pypi is not as reachable as stdlib, and one should install that package first, which many people don't know how. Additionally, there is no natural distinction between 3rd party dependencies and in-project imports (at least in tiny projects). > > This can be made easier if the first line of the program will declare the required library, and executing it will try to download and install that library if it is not installed yet. Additionally, the 3rd party dependencies will be more explicit, and editors can then allow you to search for them as you type. > > Of course it is *not* an alternative for real dependency management, but it will ease the burden on small scripts and tiny projects - which today simply break with errors that many users does not understand, instead of simply asking permission to install the dependency. This should start out as a library on PyPI. (Sorry, couldn't resist. ;) Actually, it should. Perhaps a name of "import_pip" would make sense? Any hurdles faced by this library would be (mostly) the same as a stdlib version. -- ~Ethan~ From elazarg at gmail.com Mon Sep 19 12:55:26 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 16:55:26 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <57E01474.8060309@stoneleaf.us> References: <57E01474.8060309@stoneleaf.us> Message-ID: A library in PyPi still requires installing it, which undermine many of the benefits. It won't help me with my gist/activestate recipe, code that I send to a friend, etc. I want to lower the barrier of inexperienced users. As a documentation of dependencies it will suffice indeed. Elazar On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman wrote: > On 09/19/2016 09:25 AM, ????? wrote: > > > Many proposals to add something to stdlib are rejected here with the > suggestion to add such library to pypi first. As noted by someone, pypi is > not as reachable as stdlib, and one should install that package first, > which many people don't know how. Additionally, there is no natural > distinction between 3rd party dependencies and in-project imports (at least > in tiny projects). > > > > This can be made easier if the first line of the program will declare > the required library, and executing it will try to download and install > that library if it is not installed yet. Additionally, the 3rd party > dependencies will be more explicit, and editors can then allow you to > search for them as you type. > > > > Of course it is *not* an alternative for real dependency management, but > it will ease the burden on small scripts and tiny projects - which today > simply break with errors that many users does not understand, instead of > simply asking permission to install the dependency. > > This should start out as a library on PyPI. (Sorry, couldn't resist. ;) > > Actually, it should. Perhaps a name of "import_pip" would make sense? > Any hurdles faced by this library would be (mostly) the same as a stdlib > version. > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From liik.joonas at gmail.com Mon Sep 19 12:59:18 2016 From: liik.joonas at gmail.com (Joonas Liik) Date: Mon, 19 Sep 2016 19:59:18 +0300 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> Message-ID: On 19 September 2016 at 19:55, ????? wrote: > A library in PyPi still requires installing it, which undermine many of the > benefits. It won't help me with my gist/activestate recipe, code that I send > to a friend, etc. I want to lower the barrier of inexperienced users. > > As a documentation of dependencies it will suffice indeed. > > Elazar > > On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman wrote: >> >> On 09/19/2016 09:25 AM, ????? wrote: >> >> > Many proposals to add something to stdlib are rejected here with the >> > suggestion to add such library to pypi first. As noted by someone, pypi is >> > not as reachable as stdlib, and one should install that package first, which >> > many people don't know how. Additionally, there is no natural distinction >> > between 3rd party dependencies and in-project imports (at least in tiny >> > projects). >> > >> > This can be made easier if the first line of the program will declare >> > the required library, and executing it will try to download and install that >> > library if it is not installed yet. Additionally, the 3rd party dependencies >> > will be more explicit, and editors can then allow you to search for them as >> > you type. >> > >> > Of course it is *not* an alternative for real dependency management, but >> > it will ease the burden on small scripts and tiny projects - which today >> > simply break with errors that many users does not understand, instead of >> > simply asking permission to install the dependency. >> >> This should start out as a library on PyPI. (Sorry, couldn't resist. ;) >> >> Actually, it should. Perhaps a name of "import_pip" would make sense? >> Any hurdles faced by this library would be (mostly) the same as a stdlib >> version. >> >> -- >> ~Ethan~ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ using pip from within python is not that difficult already. as can be seen with a glance to: http://stackoverflow.com/questions/12332975/installing-python-module-within-code From elazarg at gmail.com Mon Sep 19 13:01:42 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 17:01:42 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> Message-ID: Thanks Joonas. I withdraw my proposal - nothing more is strictly needed. It should be idiomatic somehow, but I don't have any specific suggestion. On Mon, Sep 19, 2016 at 7:59 PM Joonas Liik wrote: > On 19 September 2016 at 19:55, ????? wrote: > > A library in PyPi still requires installing it, which undermine many of > the > > benefits. It won't help me with my gist/activestate recipe, code that I > send > > to a friend, etc. I want to lower the barrier of inexperienced users. > > > > As a documentation of dependencies it will suffice indeed. > > > > Elazar > > > > On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman wrote: > >> > >> On 09/19/2016 09:25 AM, ????? wrote: > >> > >> > Many proposals to add something to stdlib are rejected here with the > >> > suggestion to add such library to pypi first. As noted by someone, > pypi is > >> > not as reachable as stdlib, and one should install that package > first, which > >> > many people don't know how. Additionally, there is no natural > distinction > >> > between 3rd party dependencies and in-project imports (at least in > tiny > >> > projects). > >> > > >> > This can be made easier if the first line of the program will declare > >> > the required library, and executing it will try to download and > install that > >> > library if it is not installed yet. Additionally, the 3rd party > dependencies > >> > will be more explicit, and editors can then allow you to search for > them as > >> > you type. > >> > > >> > Of course it is *not* an alternative for real dependency management, > but > >> > it will ease the burden on small scripts and tiny projects - which > today > >> > simply break with errors that many users does not understand, instead > of > >> > simply asking permission to install the dependency. > >> > >> This should start out as a library on PyPI. (Sorry, couldn't resist. ;) > >> > >> Actually, it should. Perhaps a name of "import_pip" would make sense? > >> Any hurdles faced by this library would be (mostly) the same as a stdlib > >> version. > >> > >> -- > >> ~Ethan~ > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > using pip from within python is not that difficult already. > as can be seen with a glance to: > > http://stackoverflow.com/questions/12332975/installing-python-module-within-code > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Mon Sep 19 13:15:57 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 19 Sep 2016 19:15:57 +0200 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> Message-ID: <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> I can definitely understand your point. The only issue with it (besides that it doesn't seem to be a good way for dependency management) is how do you manage the syntax involved here? Pip provides distributions. Each distribution contains a set of packages and modules. The latter can be imported, the former not. That's also due to the fact that the name of distribution can contain minuses: from __pip__ import nova-lxd # would this work? What about versions? from __pip__ import run-lambda>=0.1.0 # would this work? Maybe, I thinking too complicated here but if it works for, say, "requests" people tend to want it for special cases as well. :) Cheers, Sven On 19.09.2016 18:55, ????? wrote: > A library in PyPi still requires installing it, which undermine many > of the benefits. It won't help me with my gist/activestate recipe, > code that I send to a friend, etc. I want to lower the barrier of > inexperienced users. > > As a documentation of dependencies it will suffice indeed. > > Elazar > > On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman > wrote: > > On 09/19/2016 09:25 AM, ????? wrote: > > > Many proposals to add something to stdlib are rejected here with > the suggestion to add such library to pypi first. As noted by > someone, pypi is not as reachable as stdlib, and one should > install that package first, which many people don't know how. > Additionally, there is no natural distinction between 3rd party > dependencies and in-project imports (at least in tiny projects). > > > > This can be made easier if the first line of the program will > declare the required library, and executing it will try to > download and install that library if it is not installed yet. > Additionally, the 3rd party dependencies will be more explicit, > and editors can then allow you to search for them as you type. > > > > Of course it is *not* an alternative for real dependency > management, but it will ease the burden on small scripts and tiny > projects - which today simply break with errors that many users > does not understand, instead of simply asking permission to > install the dependency. > > This should start out as a library on PyPI. (Sorry, couldn't > resist. ;) > > Actually, it should. Perhaps a name of "import_pip" would make > sense? Any hurdles faced by this library would be (mostly) the > same as a stdlib version. > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Sep 19 13:16:20 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 18:16:20 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> Message-ID: On 19 September 2016 at 17:59, Joonas Liik wrote: > using pip from within python is not that difficult already. > as can be seen with a glance to: > http://stackoverflow.com/questions/12332975/installing-python-module-within-code Note that pip does not officially support any use of its internal API - it is purely a command line tool, and should only be invoked as a subprocess. Having said that, simple use of pip's internal API isn't difficult and if you keep to the higher-level APIs, things *shouldn't* break (but if they do, you get to keep both pieces :-)). For the use case of running simple scripts with dependencies, rwt (https://pypi.python.org/pypi/rwt) might be a reasonable solution. You can declare your dependencies in your script, then run it with rwt which will install those dependencies to a temporary location, then run the script with those dependencies active, and then tidy up afterwards. Paul From elazarg at gmail.com Mon Sep 19 13:20:13 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 17:20:13 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: Obviously from __pip__ import "run-lambda>=0.1.0" Which is ugly but not my fault :) On Mon, Sep 19, 2016 at 8:16 PM Sven R. Kunze wrote: > I can definitely understand your point. > > The only issue with it (besides that it doesn't seem to be a good way for > dependency management) is how do you manage the syntax involved here? > > Pip provides distributions. Each distribution contains a set of packages > and modules. The latter can be imported, the former not. That's also due to > the fact that the name of distribution can contain minuses: > > > from __pip__ import nova-lxd # would this work? > > What about versions? > > from __pip__ import run-lambda>=0.1.0 # would this work? > > > Maybe, I thinking too complicated here but if it works for, say, > "requests" people tend to want it for special cases as well. :) > > > Cheers, > > Sven > > On 19.09.2016 18:55, ????? wrote: > > A library in PyPi still requires installing it, which undermine many of > the benefits. It won't help me with my gist/activestate recipe, code that I > send to a friend, etc. I want to lower the barrier of inexperienced users. > > As a documentation of dependencies it will suffice indeed. > > Elazar > > On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman wrote: > >> On 09/19/2016 09:25 AM, ????? wrote: >> >> > Many proposals to add something to stdlib are rejected here with the >> suggestion to add such library to pypi first. As noted by someone, pypi is >> not as reachable as stdlib, and one should install that package first, >> which many people don't know how. Additionally, there is no natural >> distinction between 3rd party dependencies and in-project imports (at least >> in tiny projects). >> > >> > This can be made easier if the first line of the program will declare >> the required library, and executing it will try to download and install >> that library if it is not installed yet. Additionally, the 3rd party >> dependencies will be more explicit, and editors can then allow you to >> search for them as you type. >> > >> > Of course it is *not* an alternative for real dependency management, >> but it will ease the burden on small scripts and tiny projects - which >> today simply break with errors that many users does not understand, instead >> of simply asking permission to install the dependency. >> >> This should start out as a library on PyPI. (Sorry, couldn't resist. ;) >> >> Actually, it should. Perhaps a name of "import_pip" would make sense? >> Any hurdles faced by this library would be (mostly) the same as a stdlib >> version. >> >> -- >> ~Ethan~ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing listPython-ideas at python.orghttps://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Sep 19 13:27:34 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 18:27:34 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On 19 September 2016 at 18:20, ????? wrote: > Obviously > > from __pip__ import "run-lambda>=0.1.0" If we were going to go down this route (which I'm not at all convinced we should) it should be "from __pypi__ import" not "from __pip__ import" (as PyPI is where the code is coming from, not pip...) With importlib, it would probably be perfectly possible to write a module loader that worked like this. Of course, it's not accessible until you manually install it (with pip) and activate it, but for a prototype that demonstrates how the final version would work, and to thrash out how the various corner cases would be handled, that's a perfectly acceptable compromise. If the design of a 3rd-party implementation looks clean and useful, it would be much easier to debate whether it belongs in the stdlib (if for no other reason than the fact that many of the "how would X work" questions could be answered by reference to the prototype). Paul From srkunze at mail.de Mon Sep 19 13:26:07 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 19 Sep 2016 19:26:07 +0200 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: <4dd7d633-06d1-cbc3-4c7c-ee36127a50bf@mail.de> Which is still not allowed syntax IIRC? ;) On 19.09.2016 19:20, ????? wrote: > Obviously > from __pip__ import "run-lambda>=0.1.0" > > Which is ugly but not my fault :) > > On Mon, Sep 19, 2016 at 8:16 PM Sven R. Kunze > wrote: > > I can definitely understand your point. > > The only issue with it (besides that it doesn't seem to be a good > way for dependency management) is how do you manage the syntax > involved here? > > Pip provides distributions. Each distribution contains a set of > packages and modules. The latter can be imported, the former not. > That's also due to the fact that the name of distribution can > contain minuses: > > > from __pip__ import nova-lxd # would this work? > > What about versions? > > from __pip__ import run-lambda>=0.1.0 # would this work? > > > Maybe, I thinking too complicated here but if it works for, say, > "requests" people tend to want it for special cases as well. :) > > > Cheers, > > Sven > > > On 19.09.2016 18:55, ????? wrote: >> A library in PyPi still requires installing it, which undermine >> many of the benefits. It won't help me with my gist/activestate >> recipe, code that I send to a friend, etc. I want to lower the >> barrier of inexperienced users. >> >> As a documentation of dependencies it will suffice indeed. >> >> Elazar >> >> On Mon, Sep 19, 2016 at 7:38 PM Ethan Furman > > wrote: >> >> On 09/19/2016 09:25 AM, ????? wrote: >> >> > Many proposals to add something to stdlib are rejected here >> with the suggestion to add such library to pypi first. As >> noted by someone, pypi is not as reachable as stdlib, and one >> should install that package first, which many people don't >> know how. Additionally, there is no natural distinction >> between 3rd party dependencies and in-project imports (at >> least in tiny projects). >> > >> > This can be made easier if the first line of the program >> will declare the required library, and executing it will try >> to download and install that library if it is not installed >> yet. Additionally, the 3rd party dependencies will be more >> explicit, and editors can then allow you to search for them >> as you type. >> > >> > Of course it is *not* an alternative for real dependency >> management, but it will ease the burden on small scripts and >> tiny projects - which today simply break with errors that >> many users does not understand, instead of simply asking >> permission to install the dependency. >> >> This should start out as a library on PyPI. (Sorry, couldn't >> resist. ;) >> >> Actually, it should. Perhaps a name of "import_pip" would >> make sense? Any hurdles faced by this library would be >> (mostly) the same as a stdlib version. >> >> -- >> ~Ethan~ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct:http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Sep 19 13:34:52 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 19 Sep 2016 12:34:52 -0500 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: Message-ID: This sounds like a great idea, but I've seen stuff like this done before, and it never ends well. You end up with a gargantuan-sized rabbit hole, since running a basic script could now involve using an internet connection and potentially root permissions. IF one were to go this route, I prefer bundling a `requirements.txt` into the distribution and then doing something like this in the script: import pkgcheck pkgcheck.check_requirements('requirements.txt') If any of the requirements were missing, then something like this would happen: Traceback (most recent call last): ... MissingPackageError: This script requires package(s) which are not installed: mypackage>=1.0, other_package That way, you don't get the weird import errors, but you don't have to worry about all the subtleties of automatic downloading. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On Sep 19, 2016 11:26 AM, "?????" wrote: > Many proposals to add something to stdlib are rejected here with the > suggestion to add such library to pypi first. As noted by someone, pypi is > not as reachable as stdlib, and one should install that package first, > which many people don't know how. Additionally, there is no natural > distinction between 3rd party dependencies and in-project imports (at least > in tiny projects). > > This can be made easier if the first line of the program will declare the > required library, and executing it will try to download and install that > library if it is not installed yet. Additionally, the 3rd party > dependencies will be more explicit, and editors can then allow you to > search for them as you type. > > Of course it is *not* an alternative for real dependency management, but > it will ease the burden on small scripts and tiny projects - which today > simply break with errors that many users does not understand, instead of > simply asking permission to install the dependency. > > Elazar > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Mon Sep 19 13:43:18 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 17:43:18 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: Message-ID: Ryan I assume any error could be translated to your version through the `from ` statement too? Regarding the rabbit hole, the internet connection and privileges were required for running this script in the first place. I only suggest giving the developer - perhaps not a software engineer at all - an easy way not to require the (possibly inexperienced) user to do it manually. Put it another way, you a copy-pasted code from SO is not needed to use only stdlib now. On Mon, Sep 19, 2016 at 8:34 PM Ryan Gonzalez wrote: > This sounds like a great idea, but I've seen stuff like this done before, > and it never ends well. You end up with a gargantuan-sized rabbit hole, > since running a basic script could now involve using an internet connection > and potentially root permissions. > > IF one were to go this route, I prefer bundling a `requirements.txt` into > the distribution and then doing something like this in the script: > > import pkgcheck > pkgcheck.check_requirements('requirements.txt') > > If any of the requirements were missing, then something like this would > happen: > > Traceback (most recent call last): > ... > MissingPackageError: This script requires package(s) which are not > installed: mypackage>=1.0, other_package > > That way, you don't get the weird import errors, but you don't have to > worry about all the subtleties of automatic downloading. > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > On Sep 19, 2016 11:26 AM, "?????" wrote: > >> Many proposals to add something to stdlib are rejected here with the >> suggestion to add such library to pypi first. As noted by someone, pypi is >> not as reachable as stdlib, and one should install that package first, >> which many people don't know how. Additionally, there is no natural >> distinction between 3rd party dependencies and in-project imports (at least >> in tiny projects). >> >> This can be made easier if the first line of the program will declare the >> required library, and executing it will try to download and install that >> library if it is not installed yet. Additionally, the 3rd party >> dependencies will be more explicit, and editors can then allow you to >> search for them as you type. >> >> Of course it is *not* an alternative for real dependency management, but >> it will ease the burden on small scripts and tiny projects - which today >> simply break with errors that many users does not understand, instead of >> simply asking permission to install the dependency. >> >> Elazar >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Mon Sep 19 14:43:41 2016 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 19 Sep 2016 19:43:41 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On 2016-09-19 18:20, ????? wrote: > Obviously > > from __pip__ import "run-lambda>=0.1.0" > > Which is ugly but not my fault :) > [snip] One possible problem I can see is that if it's quoted you might think that it's an expression and that you could also write: package = "run-lambda>=0.1.0" from __pip__ import package From rosuav at gmail.com Mon Sep 19 14:48:12 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Sep 2016 04:48:12 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On Tue, Sep 20, 2016 at 4:43 AM, MRAB wrote: > On 2016-09-19 18:20, ????? wrote: >> >> Obviously >> >> from __pip__ import "run-lambda>=0.1.0" >> >> Which is ugly but not my fault :) >> > [snip] > One possible problem I can see is that if it's quoted you might think that > it's an expression and that you could also write: > > package = "run-lambda>=0.1.0" > from __pip__ import package One surprisingly effective option is this: import blargle.spam # Solution: pip install blarg Comes out looking like this: Traceback (most recent call last): File "useblarg.py", line 1, in import blargle.spam # Solution: pip install blarg ModuleNotFoundError: No module named 'blargle' (in older Pythons, that says "ImportError" rather than the more specific subclass, but same diff) You can put whatever you like onto that command line. ChrisA From elazarg at gmail.com Mon Sep 19 14:48:42 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 18:48:42 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: Then you'll get SyntaxError: from __pip__ imports must occur at the beginning of the file Or "must be string literal or an identifier" if it's allowed outside the beginning of the file. On Mon, Sep 19, 2016 at 9:44 PM MRAB wrote: > On 2016-09-19 18:20, ????? wrote: > > Obviously > > > > from __pip__ import "run-lambda>=0.1.0" > > > > Which is ugly but not my fault :) > > > [snip] > One possible problem I can see is that if it's quoted you might think > that it's an expression and that you could also write: > > package = "run-lambda>=0.1.0" > from __pip__ import package > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Mon Sep 19 14:52:09 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 18:52:09 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: Wow chris, that's a nice one! Of course it doesn't address the issues - for example, running a python script from the file manager will result in a failed execution, unexplained. On Mon, Sep 19, 2016 at 9:48 PM Chris Angelico wrote: > On Tue, Sep 20, 2016 at 4:43 AM, MRAB wrote: > > On 2016-09-19 18:20, ????? wrote: > >> > >> Obviously > >> > >> from __pip__ import "run-lambda>=0.1.0" > >> > >> Which is ugly but not my fault :) > >> > > [snip] > > One possible problem I can see is that if it's quoted you might think > that > > it's an expression and that you could also write: > > > > package = "run-lambda>=0.1.0" > > from __pip__ import package > > One surprisingly effective option is this: > > import blargle.spam # Solution: pip install blarg > > Comes out looking like this: > > Traceback (most recent call last): > File "useblarg.py", line 1, in > import blargle.spam # Solution: pip install blarg > ModuleNotFoundError: No module named 'blargle' > > (in older Pythons, that says "ImportError" rather than the more > specific subclass, but same diff) > > You can put whatever you like onto that command line. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Sep 19 14:57:30 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Sep 2016 04:57:30 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On Tue, Sep 20, 2016 at 4:52 AM, ????? wrote: > Wow chris, that's a nice one! > > Of course it doesn't address the issues - for example, running a python > script from the file manager will result in a failed execution, unexplained. True, it doesn't; but if anything else goes wrong, you have the same problem. Python is built around the use of exceptions. So I would just add something around the outside - if you're telling people to download and run this script, tell them that if it doesn't work first time, they should open up a terminal/console and run it from there. That'd cope with any sort of error (including that Python itself isn't installed, which is something you can't fix from within Python). ChrisA From p.f.moore at gmail.com Mon Sep 19 16:24:22 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 21:24:22 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On 19 September 2016 at 19:52, ????? wrote: > Of course it doesn't address the issues - for example, running a python > script from the file manager will result in a failed execution, unexplained. What you're talking about here is deployment of Python "applications" (where in your case, you're focusing on single-file scripts, but the problem is broader). Beyond a certain point, as I'm sure you realise, you can't solve this issue - someone tries to run a Unix-only script on Windows, for example. So really, we're looking at ways of improving the current state of affairs. As things stand, deploying "standalone applications" in Python is not particularly easy, with dependency management being only one of the issues. Some options for deployment come to mind - I'm sure none of them seem quite as easy as "just run this script", but you should remember that even "just run this script" doesn't always work as well as you'd like - on Unix you need to make the script executable and have a shebang line, on Windows the file extension needs to be right and registered. Not all users have the right (or sometimes any) Python version on their PATH, etc. So even "just run this script" isn't always simple even if the script has no dependencies outside the stdlib. So, some options: 1. The previously-mentioned rwt makes "just run this script" work as you want, as long as you have rwt installed. Conceded, it isn't available by default, but it's not hard to install as a one-off task. And there's a discussion ongoing about including it with pip as "pip run" so "pip run myscript.py" (or "py -m pip run myscript.py" on Windows - remember what I said about PATH? ;-)) will run a script with its dependencies. 2. You can bundle your script with its dependencies using zipapp. On Unix, you can even make the resulting file executable, and on Windows, the .pyz extension is registered for this. 3. Someone could write a tool to scan a script for a specially-formatted comment at the top that described its dependencies, and installed them. That (in effect) adds an "install" step before the script can be run, but lots of utilities have that, so it's not completely unheard of. And if it's GUI users you're concerned about, you could make that a GUI utility. Drag the script onto it, it lists the dependencies and confirms you want them installed, then offers to run the script. 4. There are existing platform-specific options like py2exe or cx_Freeze. Probably a bit heavyweight for a "simple script", though. Of course, none of these options cover the use case of "I saw this nice script on Stack Overflow, so I copied it, and now I want to run it on my PC". But honestly, if it has some dependencies, and the script author didn't say "you need to install FOO and BAR", or the user doesn't know how to install those dependencies, you're probably out of luck anyway. Even if the proposed "from __pypi__ import foo" syntax existed, I doubt many scripts posted on the web would use it - if only because then, Python 3.5 users wouldn't be able to run the scripts! IMO, for command line use, rwt is 99% of the way to what you want. For GUI use, there's nothing specific to this problem right now, but someone could certainly write something. A syntax change to the language would likely be *less* useful, as people would not be able to use it unless they were deliberately only targeting Python 3.7 and later, which seems optimistic at best... Paul From python at mrabarnett.plus.com Mon Sep 19 16:26:01 2016 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 19 Sep 2016 21:26:01 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On 2016-09-19 19:48, Chris Angelico wrote: > On Tue, Sep 20, 2016 at 4:43 AM, MRAB wrote: >> On 2016-09-19 18:20, ????? wrote: >>> >>> Obviously >>> >>> from __pip__ import "run-lambda>=0.1.0" >>> >>> Which is ugly but not my fault :) >>> >> [snip] >> One possible problem I can see is that if it's quoted you might think that >> it's an expression and that you could also write: >> >> package = "run-lambda>=0.1.0" >> from __pip__ import package > > One surprisingly effective option is this: > > import blargle.spam # Solution: pip install blarg > > Comes out looking like this: > > Traceback (most recent call last): > File "useblarg.py", line 1, in > import blargle.spam # Solution: pip install blarg > ModuleNotFoundError: No module named 'blargle' > > (in older Pythons, that says "ImportError" rather than the more > specific subclass, but same diff) > > You can put whatever you like onto that command line. > One further suggestion: import blargle.spam in "blarg" The syntax would be: "import" module_name ["in" quoted_pypi_name] From elazarg at gmail.com Mon Sep 19 16:34:41 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 20:34:41 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: Paul, I understand what you say, except the part of compatibility with Python3.5. Of course such a change is targeting future version more than the current and previous versions. If we have this syntax for Python3.6, users of Python3.9 will find many copy-pastable answers written this way, and other answers will have comments saying "just add this statement to the beginning of your script". On Mon, Sep 19, 2016 at 11:24 PM Paul Moore wrote: > On 19 September 2016 at 19:52, ????? wrote: > > Of course it doesn't address the issues - for example, running a python > > script from the file manager will result in a failed execution, > unexplained. > > What you're talking about here is deployment of Python "applications" > (where in your case, you're focusing on single-file scripts, but the > problem is broader). Beyond a certain point, as I'm sure you realise, > you can't solve this issue - someone tries to run a Unix-only script > on Windows, for example. So really, we're looking at ways of improving > the current state of affairs. As things stand, deploying "standalone > applications" in Python is not particularly easy, with dependency > management being only one of the issues. > > Some options for deployment come to mind - I'm sure none of them seem > quite as easy as "just run this script", but you should remember that > even "just run this script" doesn't always work as well as you'd like > - on Unix you need to make the script executable and have a shebang > line, on Windows the file extension needs to be right and registered. > Not all users have the right (or sometimes any) Python version on > their PATH, etc. So even "just run this script" isn't always simple > even if the script has no dependencies outside the stdlib. > > So, some options: > > 1. The previously-mentioned rwt makes "just run this script" work as > you want, as long as you have rwt installed. Conceded, it isn't > available by default, but it's not hard to install as a one-off task. > And there's a discussion ongoing about including it with pip as "pip > run" so "pip run myscript.py" (or "py -m pip run myscript.py" on > Windows - remember what I said about PATH? ;-)) will run a script with > its dependencies. > 2. You can bundle your script with its dependencies using zipapp. On > Unix, you can even make the resulting file executable, and on Windows, > the .pyz extension is registered for this. > 3. Someone could write a tool to scan a script for a > specially-formatted comment at the top that described its > dependencies, and installed them. That (in effect) adds an "install" > step before the script can be run, but lots of utilities have that, so > it's not completely unheard of. And if it's GUI users you're concerned > about, you could make that a GUI utility. Drag the script onto it, it > lists the dependencies and confirms you want them installed, then > offers to run the script. > 4. There are existing platform-specific options like py2exe or > cx_Freeze. Probably a bit heavyweight for a "simple script", though. > > Of course, none of these options cover the use case of "I saw this > nice script on Stack Overflow, so I copied it, and now I want to run > it on my PC". But honestly, if it has some dependencies, and the > script author didn't say "you need to install FOO and BAR", or the > user doesn't know how to install those dependencies, you're probably > out of luck anyway. Even if the proposed "from __pypi__ import foo" > syntax existed, I doubt many scripts posted on the web would use it - > if only because then, Python 3.5 users wouldn't be able to run the > scripts! > > IMO, for command line use, rwt is 99% of the way to what you want. For > GUI use, there's nothing specific to this problem right now, but > someone could certainly write something. A syntax change to the > language would likely be *less* useful, as people would not be able to > use it unless they were deliberately only targeting Python 3.7 and > later, which seems optimistic at best... > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Mon Sep 19 16:35:53 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Matos?=) Date: Mon, 19 Sep 2016 13:35:53 -0700 (PDT) Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160919023201.GB22471@ando.pearwood.info> References: <20160919023201.GB22471@ando.pearwood.info> Message-ID: <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> Hello, I don't see why creating a clear command would interfere with dict.clear() which is a function/method. Although my first idea was a clear command, I have no problem if it is a clear() function from site.py. I didn't suggest cls because it is normally used to mean class. I use Windows and tested a simple (possibly not the best of course) solution that seems to work in REPL (but not in IDLE). import os import sys def clear(): if sys.platform == 'win32': os.system('cls') else: os.system('clear') Best regards, JM segunda-feira, 19 de Setembro de 2016 ?s 03:33:45 UTC+1, Steven D'Aprano escreveu: > On Sat, Sep 17, 2016 at 11:51:16AM +0100, Jo?o Matos wrote: > > Hello, > > > > I would like to suggest adding a clear command (not function) to Python. > > While technically "clear" could be a command, I think it should not be. > > First off, making clear a reserved keyword, and a statement, like print > in Python 2, raise or import, would be a major backwards compatibility > break. It would mean dict.clear() has to be renamed, and it would break > lots of existing code. > > So making clear a keyword is not going to happen. > > If could be a pseudo-built-in, like help(), quit() and exit(), added to > built-ins by the site module. In that case, it is *technically* possible > to have it operate without the parentheses: > > class ClearType: > def __repr__(self): > # code to clear the screen here > ... > > clear = ClearType() > > so that when you enter clear at the interactive interpreter, __repr__ is > called and it clears the screen. But I would argue against it. Instead, > it is better to use the same convention that executable code that has > side-effects should be implemented as a function call. > > So I suggest following the design of exit() and quit(): > > py> exit > Use exit() or Ctrl-D (i.e. EOF) to exit > > > class ClearType: > def __repr__(self): > return "Use clear() or Ctrl-L (i.e. FF) to clear the screen" > def __call__(self): > # clear screen code goes here > > clear = ClearType() # or possibly cls ? > > > > That is, *if* we add this function at all. > > Personally, I agree with you. There are many different ways of clearing > the screen, but they depend on the specific terminal used, whether > readline is active or not, the operating system, etc. I think that > interactive use is important enough that we should have a standard way > of clearing the screen. I personally often find myself just holding down > the Enter key until I have a blank screen. > > In this ticket: > > http://bugs.python.org/issue27771 > > Raymond Hettinger mentions that it is an often-requested feature by > learners, and I believe that IDLE has an active task for this feature: > > http://bugs.python.org/issue6143 > > but I don't see any tasks for a clear screen command for the default > REPL. > > I'm in favour of adding a clear() *function* to the site.py module, > similar to exit/quit/help, but not making it "magical" or a keyword that > doesn't require brackets. But I don't know how to implement it for the > large variety of terminals and operating systems supported by Python. > > (The fallback if all else fails is easy: get the height of the terminal, > in lines, and print that many blank lines.) > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Sep 19 17:23:11 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 22:23:11 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On 19 September 2016 at 21:34, ????? wrote: > Paul, I understand what you say, except the part of compatibility with > Python3.5. Of course such a change is targeting future version more than the > current and previous versions. If we have this syntax for Python3.6, users > of Python3.9 will find many copy-pastable answers written this way, and > other answers will have comments saying "just add this statement to the > beginning of your script". Agreed. Sorry, I was being pessimistic. Blame it on spending too much time writing code that needs to retain compatibility with 2.x, when I haven't used 2.x myself for years... Paul From p.f.moore at gmail.com Mon Sep 19 17:34:08 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 22:34:08 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> References: <20160919023201.GB22471@ando.pearwood.info> <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> Message-ID: On 19 September 2016 at 21:35, Jo?o Matos wrote: > Hello, > > I don't see why creating a clear command would interfere with dict.clear() > which is a function/method. > > Although my first idea was a clear command, I have no problem if it is a > clear() function from site.py. > > I didn't suggest cls because it is normally used to mean class. > > I use Windows and tested a simple (possibly not the best of course) solution > that seems to work in REPL (but not in IDLE). > > import os > import sys > > def clear(): > if sys.platform == 'win32': > os.system('cls') > else: > os.system('clear') Ah, I think people had misunderstood your proposal of a "command" as something where you didn't have to type the parentheses. I know I did. If you're OK with what you type at the REPL being >>> clear() then that's much easier to achieve. For example, if you install click (https://pypi.python.org/pypi/click) you can just add "from click import clear" at the start of your REPL session (or add it to your PYTHONSTARTUP if you want it always available) and you're done. Adding similar functionality to the stdlib somewhere (for example os or shutil - I doubt it's going to get accepted as a builtin) isn't inconceivable. I'm still not sure the benefit is sufficient to be worthwhile (it might be more useful to just bite the bullet and bundle one of the many curses variants for Windows into the distribution, and make the curses module cross-platform once and for all, rather than implementing individual bits of functionality) but if someone wanted to contribute a patch on the tracker, it might be accepted. Paul From elazarg at gmail.com Mon Sep 19 17:53:13 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 21:53:13 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: Another use case, though I admit not the top priority of anyone here, is that of assignment checkers. In most courses I took at the university, the person who checks the assignments says something like "you are allowed to use only this this and this libraries", in order not to mess with unknown dependencies from tens of students (I am talking about advanced courses, where the method I use to solve the problem is unimportant or only requires explanation). With this statement they can simply state "you can import pip". Of course it still requires privileges and network connection, etc. And yes it can be solved in other ways automatically, but the fact is, it isn't. We get the side benefit of making more people aware of pypi and actually using it; many people are not aware and/or don't bother using it. In other words, the language will encourage people to use pypi. It might has its downsides (of introducing possibly unneeded dependencies) but I am under the impression that using pypi is something that is considered A Good Thing, in general. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Mon Sep 19 18:13:06 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 22:13:06 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: Alternative syntax might be that of import decorators @from_pip("attrs") import attr It's only worthwhile if there are any other uses for import decorators. One possibility is for other online modules, probably by giving a url. I can come up with more, but it's making a solution without having a problem; it looks like a handy tool, but I guess such suggestions were already raised and rejected in the past. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Sep 19 18:40:40 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Sep 2016 23:40:40 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On 19 September 2016 at 23:13, ????? wrote: > Alternative syntax might be that of import decorators > > @from_pip("attrs") > import attr At which point you may as well just (insert disclaimer about "using pip's internal API isn't supported" here, and glossing over the fact that pip doesn't yet have an install function with this API) do import pip pip.install('attrs') import attr It's not as if that's functionally any different, or in practical terms any harder to read. And it needs no change to Python. Paul From elazarg at gmail.com Mon Sep 19 18:46:12 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 22:46:12 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On Tue, Sep 20, 2016 at 1:40 AM Paul Moore wrote: > On 19 September 2016 at 23:13, ????? wrote: > > Alternative syntax might be that of import decorators > > > > @from_pip("attrs") > > import attr > > At which point you may as well just (insert disclaimer about "using > pip's internal API isn't supported" here, and glossing over the fact > that pip doesn't yet have an install function with this API) do > > import pip > pip.install('attrs') > import attr > Please forgive me for my ignorance, but it doesn't work as written - what's the actual method? Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Mon Sep 19 19:02:41 2016 From: mertz at gnosis.cx (David Mertz) Date: Mon, 19 Sep 2016 16:02:41 -0700 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: >>> import pip >>> pip.main(['install', 'boltons']) Requirement already satisfied (use --upgrade to upgrade): boltons in ./anaconda/lib/python3.5/site-packages 0 This is easy, short, and uses existing syntax and an existing module. Yes, I know technically the API isn't promised; I wouldn't object to making that API official. But PLEASE, no new syntax for doing this! Btw. On failure: >>> pip.main(['install', 'nonesuch']) Collecting nonesuch Could not find a version that satisfies the requirement nonesuch (from versions: ) No matching distribution found for nonesuch 1 On Mon, Sep 19, 2016 at 3:46 PM, ????? wrote: > On Tue, Sep 20, 2016 at 1:40 AM Paul Moore wrote: > >> On 19 September 2016 at 23:13, ????? wrote: >> > Alternative syntax might be that of import decorators >> > >> > @from_pip("attrs") >> > import attr >> >> At which point you may as well just (insert disclaimer about "using >> pip's internal API isn't supported" here, and glossing over the fact >> that pip doesn't yet have an install function with this API) do >> >> import pip >> pip.install('attrs') >> import attr >> > > Please forgive me for my ignorance, but it doesn't work as written - > what's the actual method? > > Elazar > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Mon Sep 19 19:09:53 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 23:09:53 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: Sounds good, thanks. So I'll suggest adding the method mentioned in Paul's code as a promised API, either as pip.install or otherwise. `pip.main(['install', 'boltons'])` is not the clearest it can get. Elazar On Tue, Sep 20, 2016 at 2:02 AM David Mertz wrote: > >>> import pip > >>> pip.main(['install', 'boltons']) > Requirement already satisfied (use --upgrade to upgrade): boltons in > ./anaconda/lib/python3.5/site-packages > 0 > > This is easy, short, and uses existing syntax and an existing module. > Yes, I know technically the API isn't promised; I wouldn't object to making > that API official. But PLEASE, no new syntax for doing this! > > Btw. On failure: > > >>> pip.main(['install', 'nonesuch']) > Collecting nonesuch > Could not find a version that satisfies the requirement nonesuch (from > versions: ) > No matching distribution found for nonesuch > 1 > > > On Mon, Sep 19, 2016 at 3:46 PM, ????? wrote: > >> On Tue, Sep 20, 2016 at 1:40 AM Paul Moore wrote: >> >>> On 19 September 2016 at 23:13, ????? wrote: >>> > Alternative syntax might be that of import decorators >>> > >>> > @from_pip("attrs") >>> > import attr >>> >>> At which point you may as well just (insert disclaimer about "using >>> pip's internal API isn't supported" here, and glossing over the fact >>> that pip doesn't yet have an install function with this API) do >>> >>> import pip >>> pip.install('attrs') >>> import attr >>> >> >> Please forgive me for my ignorance, but it doesn't work as written - >> what's the actual method? >> >> Elazar >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Sep 19 19:20:40 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 20 Sep 2016 08:20:40 +0900 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> ????? writes: > Another use case, though I admit not the top priority of anyone here, is > that of assignment checkers. In most courses I took at the university, the > person who checks the assignments says something like "you are allowed to > use only this this and this libraries", in order not to mess with unknown > dependencies from tens of students (I am talking about advanced courses, > where the method I use to solve the problem is unimportant or only requires > explanation). With this statement they can simply state "you can import > pip". In other words, you're advocating a feature that allows script writers to download, install, and execute arbitrary, unsandboxed code on any machine where the script is run. That sounds ... *scary*, when put that way. Remember, you're advocating this on behalf of people who by assumption are infants years below the age of consent. From xavier.combelle at gmail.com Mon Sep 19 19:27:57 2016 From: xavier.combelle at gmail.com (Xavier Combelle) Date: Tue, 20 Sep 2016 01:27:57 +0200 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: Message-ID: <68dc67c3-47dc-ffda-b768-df8bd650daff@gmail.com> Le 19/09/2016 ? 18:25, ????? a ?crit : > Many proposals to add something to stdlib are rejected here with the > suggestion to add such library to pypi first. As noted by someone, > pypi is not as reachable as stdlib, and one should install that > package first, which many people don't know how. Additionally, there > is no natural distinction between 3rd party dependencies and > in-project imports (at least in tiny projects). > > This can be made easier if the first line of the program will declare > the required library, and executing it will try to download and > install that library if it is not installed yet. Additionally, the 3rd > party dependencies will be more explicit, and editors can then allow > you to search for them as you type. > > Of course it is *not* an alternative for real dependency management, > but it will ease the burden on small scripts and tiny projects - which > today simply break with errors that many users does not understand, > instead of simply asking permission to install the dependency. > > Elazar > I find the idea of tracking the dependencies in the script might be a good idea. However, magically downloading without warning the user is in my point of view for sure a bad idea. I would far prefer that pip could scan a script to know the dependencies. (A little bit like a requirements.txt but inside the script) A special comment or docstring would do the job. for example """ pip_requirements: - requests >0.0 - asyncio """" to run the script it would be at the first time a two step process for example: python3 -m pip --script-dependencies [--user] my_script.py python3 my_script.py From elazarg at gmail.com Mon Sep 19 19:28:56 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 23:28:56 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Sep 20, 2016 at 2:20 AM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > ????? writes: > > > Another use case, though I admit not the top priority of anyone here, is > > that of assignment checkers. In most courses I took at the university, > the > > person who checks the assignments says something like "you are allowed > to > > use only this this and this libraries", in order not to mess with > unknown > > dependencies from tens of students (I am talking about advanced courses, > > where the method I use to solve the problem is unimportant or only > requires > > explanation). With this statement they can simply state "you can import > > pip". > > In other words, you're advocating a feature that allows script writers > to download, install, and execute arbitrary, unsandboxed code on any > machine where the script is run. That sounds ... *scary*, when put > that way. Remember, you're advocating this on behalf of people who > by assumption are infants years below the age of consent. > > Let me understand. Your argument is "installing pip modules is unsafe, and therefore we should make it less usable, where the appropriate amount of (un)usability is running cmd and then `pip install unsafe`" ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Sep 19 19:34:12 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Sep 2016 09:34:12 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Sep 20, 2016 at 9:20 AM, Stephen J. Turnbull wrote: > ????? writes: > > > Another use case, though I admit not the top priority of anyone here, is > > that of assignment checkers. In most courses I took at the university, the > > person who checks the assignments says something like "you are allowed to > > use only this this and this libraries", in order not to mess with unknown > > dependencies from tens of students (I am talking about advanced courses, > > where the method I use to solve the problem is unimportant or only requires > > explanation). With this statement they can simply state "you can import > > pip". > > In other words, you're advocating a feature that allows script writers > to download, install, and execute arbitrary, unsandboxed code on any > machine where the script is run. That sounds ... *scary*, when put > that way. Remember, you're advocating this on behalf of people who by > assumption are infants years below the age of consent. It's more than that, though. When a student is given an assignment, it's usually to help said student to learn to *write code*, not to learn how to walk into the plumbing store of PyPI and look for something that approximates to the job being done. Maybe it's different at university, but with my students, it's always been "no external libraries" (and in some cases, a common-sense avoidance of obvious solutions from the standard library - if you teach someone how to implement a sort and the response is simply "lst.sort()", it's not exactly implementing anything). So the use-case for this isn't nearly as big as it might be. By the time you get to writing large applications (again, taking it from my students' work: a Flask-based web app), it's not illogical to have a requirements.txt and standard pip incantations. ChrisA From elazarg at gmail.com Mon Sep 19 19:35:39 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 23:35:39 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: Xavier, how is connecting and installing different from a windows popup "this software requires changes to your firewall settings" or "requires elevated privileges" which already happens. I am all for a two-step process, but I think it should be more user friendly, and it can be done as a Python command from inside the script. If I send you a small script, it should be treated in the same way as if I send you a program - an installer - not as a code that you should incorporate into your already existing code base. ?On Tue, Sep 20, 2016 at 2:28 AM ???????? wrote:? > On Tue, Sep 20, 2016 at 2:20 AM Stephen J. Turnbull < > turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > >> ????? writes: >> >> > Another use case, though I admit not the top priority of anyone here, >> is >> > that of assignment checkers. In most courses I took at the university, >> the >> > person who checks the assignments says something like "you are allowed >> to >> > use only this this and this libraries", in order not to mess with >> unknown >> > dependencies from tens of students (I am talking about advanced >> courses, >> > where the method I use to solve the problem is unimportant or only >> requires >> > explanation). With this statement they can simply state "you can import >> > pip". >> >> In other words, you're advocating a feature that allows script writers >> to download, install, and execute arbitrary, unsandboxed code on any >> machine where the script is run. That sounds ... *scary*, when put >> that way. Remember, you're advocating this on behalf of people who >> by assumption are infants years below the age of consent. >> >> Let me understand. Your argument is "installing pip modules is unsafe, > and therefore we should make it less usable, where the appropriate amount > of (un)usability is running cmd and then `pip install unsafe`" ? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Mon Sep 19 19:38:54 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 23:38:54 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Sep 20, 2016 at 2:34 AM Chris Angelico wrote: > On Tue, Sep 20, 2016 at 9:20 AM, Stephen J. Turnbull > wrote: > > ????? writes: > > > > > Another use case, though I admit not the top priority of anyone here, > is > > > that of assignment checkers. In most courses I took at the > university, the > > > person who checks the assignments says something like "you are > allowed to > > > use only this this and this libraries", in order not to mess with > unknown > > > dependencies from tens of students (I am talking about advanced > courses, > > > where the method I use to solve the problem is unimportant or only > requires > > > explanation). With this statement they can simply state "you can > import > > > pip". > It's more than that, though. When a student is given an assignment, > it's usually to help said student to learn to *write code*, not to > learn how to walk into the plumbing store of PyPI and look for > something that approximates to the job being done. Maybe it's > different at university, but with my students, it's always been "no > external libraries" (and in some cases, a common-sense avoidance of > obvious solutions from the standard library - if you teach someone how > to implement a sort and the response is simply "lst.sort()", it's not > exactly implementing anything). So the use-case for this isn't nearly > as big as it might be. By the time you get to writing large > applications (again, taking it from my students' work: a Flask-based > web app), it's not illogical to have a requirements.txt and standard > pip incantations. > > ChrisA > I was talking specifically about advanced courses, in which an assignment is "implement a side-channel attack using data" and you can use whatever library you like. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Sep 19 19:40:49 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Sep 2016 09:40:49 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <68dc67c3-47dc-ffda-b768-df8bd650daff@gmail.com> References: <68dc67c3-47dc-ffda-b768-df8bd650daff@gmail.com> Message-ID: On Tue, Sep 20, 2016 at 9:27 AM, Xavier Combelle wrote: > I find the idea of tracking the dependencies in the script might be a > good idea. > However, magically downloading without warning the user is in my point > of view for sure a bad idea. > I would far prefer that pip could scan a script to know the dependencies. > (A little bit like a requirements.txt but inside the script) > A special comment or docstring would do the job. for example > > """ > pip_requirements: > - requests >0.0 > - asyncio > """" > > to run the script it would be at the first time a two step process > for example: > > python3 -m pip --script-dependencies [--user] my_script.py > python3 my_script.py How about this: python3 -m pip install -r requirements.txt python3 my_script.py That already works, but the requirements have to be in a separate text file, not in the .py file itself. If you want a docstring-based solution, I'd piggy-back it on requirements.txt, and use the same format (eg if the docstring starts "pip_requirements:", the rest of it is a requirements.txt file). Downside: Docstrings can be easily parsed and read *IF* you can successfully import the module. Which you can't, if you still need all those dependencies. So there would have to be some kind of import hook that crafts a dummy package with an infinite number of dummy subpackages, to allow all forms of import to trivially succeed - "import X", "import X.Y", "from X.Y import Z", etc. This may well be more hassle than it's worth. ChrisA From elazarg at gmail.com Mon Sep 19 19:43:56 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 19 Sep 2016 23:43:56 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <68dc67c3-47dc-ffda-b768-df8bd650daff@gmail.com> Message-ID: But grepping and piping could work I assume? On Tue, Sep 20, 2016 at 2:41 AM Chris Angelico wrote: > On Tue, Sep 20, 2016 at 9:27 AM, Xavier Combelle > wrote: > > I find the idea of tracking the dependencies in the script might be a > > good idea. > > However, magically downloading without warning the user is in my point > > of view for sure a bad idea. > > I would far prefer that pip could scan a script to know the dependencies. > > (A little bit like a requirements.txt but inside the script) > > A special comment or docstring would do the job. for example > > > > """ > > pip_requirements: > > - requests >0.0 > > - asyncio > > """" > > > > to run the script it would be at the first time a two step process > > for example: > > > > python3 -m pip --script-dependencies [--user] my_script.py > > python3 my_script.py > > How about this: > > python3 -m pip install -r requirements.txt > python3 my_script.py > > That already works, but the requirements have to be in a separate text > file, not in the .py file itself. If you want a docstring-based > solution, I'd piggy-back it on requirements.txt, and use the same > format (eg if the docstring starts "pip_requirements:", the rest of it > is a requirements.txt file). Downside: Docstrings can be easily parsed > and read *IF* you can successfully import the module. Which you can't, > if you still need all those dependencies. So there would have to be > some kind of import hook that crafts a dummy package with an infinite > number of dummy subpackages, to allow all forms of import to trivially > succeed - "import X", "import X.Y", "from X.Y import Z", etc. This may > well be more hassle than it's worth. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Sep 19 20:06:50 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Sep 2016 17:06:50 -0700 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: <57E07D9A.4030305@stoneleaf.us> On 09/19/2016 04:38 PM, ????? wrote: > I was talking specifically about advanced courses, in which an assignment > is "implement a side-channel attack using data" and you can use whatever > library you like. Am I misunderstanding, or did you just say you want this new functionality in order to implement attacks on people's computers? -- ~Ethan~ From elazarg at gmail.com Mon Sep 19 20:15:13 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 00:15:13 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <57E07D9A.4030305@stoneleaf.us> References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> <57E07D9A.4030305@stoneleaf.us> Message-ID: On Tue, Sep 20, 2016 at 3:06 AM Ethan Furman wrote: > On 09/19/2016 04:38 PM, ????? wrote: > > > I was talking specifically about advanced courses, in which an assignment > > is "implement a side-channel attack using data" and you can use whatever > > library you like. > > Am I misunderstanding, or did you just say you want this new functionality > in order to implement attacks on people's computers? > This is completely off topic (the feature is irrelevant to the contents of the course). But - you are misunderstanding :) And yes I took a course in which I was expected to demonstrate attacks on people's computers using e.g. correlation power analysis. It was based on fake data but yes. They also give courses in which you learn that there's these things called man in the middle, buffer overflows and return-oriented programming! Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Sep 19 20:15:29 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Sep 2016 10:15:29 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: <20160920001529.GF22471@ando.pearwood.info> On Mon, Sep 19, 2016 at 08:34:41PM +0000, ????? wrote: > If we have this syntax for Python3.6 3.6 now is under feature-freeze. The earliest any new features can be added is 3.7. -- Steve From steve at pearwood.info Mon Sep 19 20:42:18 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Sep 2016 10:42:18 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: <20160920004218.GG22471@ando.pearwood.info> On Tue, Sep 20, 2016 at 09:34:12AM +1000, Chris Angelico wrote: > It's more than that, though. When a student is given an assignment, > it's usually to help said student to learn to *write code*, not to > learn how to walk into the plumbing store of PyPI and look for > something that approximates to the job being done. That might be sufficient for theoretical computer science courses, but it is *far* from sufficient for learning to be a professional programmer. I think that programmer education lets students down hugely. Knowing how to "walk into the plumbing store" to get a standard plumbing fixture is a skill programmers desperately need. (We also need better standard plumbing fixtures, but that's another story.) Failure to teach appropriate use of external dependencies, including when to use them and when not to, is part of the reason so many programmers suffer under Not Invented Here syndrome. It is not always appropriate to use PyPI, but being able to use it is an essential skill for programmers. To be a professional programmer, you need more skills than just writing code. You need to be able to estimate costs, write and use tests, write documentation, etc, and most relevant to this discussion, manage dependencies. Notice I didn't say "install dependencies". There is more to dependency management than just automatically installing any failed import. > Maybe it's > different at university, but with my students, it's always been "no > external libraries" (and in some cases, a common-sense avoidance of > obvious solutions from the standard library - if you teach someone how > to implement a sort and the response is simply "lst.sort()", it's not > exactly implementing anything). This is off-topic, but unless you are specifically doing an algorithms course, why would you bother teaching anyone to implement a sort? Do you teach them to implement their own float multiplication as well? > So the use-case for this isn't nearly > as big as it might be. By the time you get to writing large > applications (again, taking it from my students' work: a Flask-based > web app), it's not illogical to have a requirements.txt and standard > pip incantations. Indeed. -- Steve From steve at pearwood.info Mon Sep 19 21:16:21 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Sep 2016 11:16:21 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> Message-ID: <20160920011621.GH22471@ando.pearwood.info> On Mon, Sep 19, 2016 at 04:55:26PM +0000, ????? wrote: > A library in PyPi still requires installing it, which undermine many of > the benefits. It won't help me with my gist/activestate recipe, code that I > send to a friend, etc. I want to lower the barrier of inexperienced users. When you say users, are you talking about completely non-technical end-users who are not programmers at all? In their case, you should bundle the application and all its dependencies into a single frozen .exe file (for Windows), or equivalent for other platforms. Otherwise, the users you are talking about are *programmers*, or at least "technical people" (sys admin, etc), who can be expected to run a couple of commands to install needed dependencies. Perhaps they are inexperienced, and still learning. In that case, teaching them to use pip is a good thing, especially since it isn't that hard: python3 -m ensurepip pip install foo bar baz # whatever your dependencies are (In principle at least. In practice, I have found pip to be not quite as foolproof as advertised. But that's not something the std lib can fix.) https://docs.python.org/3/library/ensurepip.html -- Steve From steve at pearwood.info Mon Sep 19 21:38:28 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Sep 2016 11:38:28 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: <20160920013828.GI22471@ando.pearwood.info> On Mon, Sep 19, 2016 at 11:35:39PM +0000, ????? wrote: > Xavier, how is connecting and installing different from a windows popup > "this software requires changes to your firewall settings" or "requires > elevated privileges" which already happens. I am all for a two-step > process, but I think it should be more user friendly, and it can be done as > a Python command from inside the script. Installing dependencies must be a separate step from running the code. You personally might not care, but some people *do* care. They may have policies about downloading, or even legal requirements about what software they install and run, and so need to vet dependencies, not just blindly install whatever packages are required by a module. They may need authority to install. I don't mean account privileges, I mean they may need their manager's approval. Perhaps the legal department needs to check the licence terms. Perhaps they need to pay for a licence, or get approval to spend the money on a licence. Or they may have a policy of "no unapproved software" because they are legally required to run a specific, known set of software which has been audited, not just any old rubbish they've downloaded off the internet. Or maybe they just don't trust any old rubbish available on the internet and want the choice of whether or not to install it. I know places where it is a firing offence, with no warnings or second chances, to download and install unapproved software on work computers. Your suggestion would make it unsafe to use Python in such an environment. (Of course any Python script *could* try to reach out to the internet to download code, but the risk of this is low. But if the Python language had a built-in command to do this, the risk would be magnified.) > If I send you a small script, it should be treated in the same way as if I > send you a program - an installer - not as a code that you should > incorporate into your already existing code base. Some scripts are installers. Some scripts are not. You cannot assume that all scripts should be treated as installers. I normally run scripts as an unprivileged user. Even if I don't trust the code, the worst that happens is limited by the privileges of that user. But installers generally require greater trust and greater privileges -- I might run them as root, or using sudo, otherwise the installation will fail. Keeping installation and execution as separate steps is a security measure. -- Steve From rosuav at gmail.com Mon Sep 19 21:45:53 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Sep 2016 11:45:53 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <20160920004218.GG22471@ando.pearwood.info> References: <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> <20160920004218.GG22471@ando.pearwood.info> Message-ID: On Tue, Sep 20, 2016 at 10:42 AM, Steven D'Aprano wrote: > On Tue, Sep 20, 2016 at 09:34:12AM +1000, Chris Angelico wrote: > >> It's more than that, though. When a student is given an assignment, >> it's usually to help said student to learn to *write code*, not to >> learn how to walk into the plumbing store of PyPI and look for >> something that approximates to the job being done. > > That might be sufficient for theoretical computer science courses, but > it is *far* from sufficient for learning to be a professional > programmer. I think that programmer education lets students down hugely. > Knowing how to "walk into the plumbing store" to get a standard plumbing > fixture is a skill programmers desperately need. > > (We also need better standard plumbing fixtures, but that's another > story.) > > Failure to teach appropriate use of external dependencies, including > when to use them and when not to, is part of the reason so many > programmers suffer under Not Invented Here syndrome. It is not always > appropriate to use PyPI, but being able to use it is an essential skill > for programmers. This is true, but by the time we're teaching people how to manage dependencies, we've already taught them to manage a *project*. That means a git repository, usually a README, a clear entry point, etc, etc, etc. With that sort of groundwork, it's easy to instruct them to use requirements.txt, rather than try to have a single self-contained .py file that identifies all its deps. Maybe it's different elsewhere, eg if you're starting people off in a scientific field and there's a single obvious library to use. >> Maybe it's >> different at university, but with my students, it's always been "no >> external libraries" (and in some cases, a common-sense avoidance of >> obvious solutions from the standard library - if you teach someone how >> to implement a sort and the response is simply "lst.sort()", it's not >> exactly implementing anything). > > This is off-topic, but unless you are specifically doing an algorithms > course, why would you bother teaching anyone to implement a sort? Do you > teach them to implement their own float multiplication as well? Yes we are (at least in that instance). With more real-world tests, they're allowed to use all the standard library - this was the "in some cases" bit. (We don't teach float multiplication, because it's very complication, but we do teach some stuff about how bitwise operations work. So yes, we are teaching low level stuff in high level languages.) ChrisA From steve at pearwood.info Mon Sep 19 21:50:40 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Sep 2016 11:50:40 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <57E07D9A.4030305@stoneleaf.us> References: <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> <57E07D9A.4030305@stoneleaf.us> Message-ID: <20160920015040.GJ22471@ando.pearwood.info> On Mon, Sep 19, 2016 at 05:06:50PM -0700, Ethan Furman wrote: > On 09/19/2016 04:38 PM, ????? wrote: > > >I was talking specifically about advanced courses, in which an assignment > > is "implement a side-channel attack using data" and you can use whatever > > library you like. > > Am I misunderstanding, or did you just say you want this new functionality > in order to implement attacks on people's computers? I'm pretty sure you misinterpreted Elazar. He was talking about an advanced programming course where the course allows the students to use any library they like. The "side-channel attack" part just happened to be an example (I think a particularly poorly thought out example, but nevermind). The idea is that the student writes: from __pip__ import somepackage in their script, and the examiner merely needs to run their script in order to have somepackage automatically installed. (I really, really, REALLY hope that the examiner runs their code in a sandbox.) I'm not sure how this reconciles with Elazar's earlier use-case that this feature is for the benefit of *beginners*. Presumably the examiner of an advanced programming course should be able to cope with installing a package. I would expect that for such an advanced course, being able to deal with external dependencies should be part of the assignment: "If your lecturer cannot work out what dependencies are needed by your project, because you haven't documented it or provided an installer, then you will fail the assignment." -- Steve From steve at pearwood.info Mon Sep 19 21:55:29 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Sep 2016 11:55:29 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> References: <20160919023201.GB22471@ando.pearwood.info> <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> Message-ID: <20160920015529.GK22471@ando.pearwood.info> On Mon, Sep 19, 2016 at 01:35:53PM -0700, Jo?o Matos wrote: > Hello, > > I don't see why creating a clear command would interfere with dict.clear() > which is a function/method. For the same reason that you can't have a method called foo.while or foo.if or foo.raise. If clear is a "command" (a statement) it would need to be a keyword, like while, if and raise. -- Steve From tim.mitchell at leapfrog3d.com Mon Sep 19 23:02:22 2016 From: tim.mitchell at leapfrog3d.com (Tim Mitchell) Date: Tue, 20 Sep 2016 15:02:22 +1200 Subject: [Python-ideas] singledispatch for instance methods Message-ID: Hi All, We have a modified version of singledispatch at work which works for methods as well as functions. We have open-sourced it as methoddispatch (pypi: https://pypi.python.org/pypi/methoddispatch). IMHO I thought it would make a nice addition to python stdlib. What does everyone else think? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Sep 19 23:32:54 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Sep 2016 20:32:54 -0700 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> <20160920004218.GG22471@ando.pearwood.info> Message-ID: <57E0ADE6.6090100@stoneleaf.us> On 09/19/2016 06:45 PM, Chris Angelico wrote: > Yes we are (at least in that instance). With more real-world tests, > they're allowed to use all the standard library - this was the "in > some cases" bit. (We don't teach float multiplication, because it's > very complication, but we do teach some stuff about how bitwise > operations work. So yes, we are teaching low level stuff in high level > languages.) Good. Just remember that "bin(x)" does not print 2s-complement for negative numbers. Had me pulling my hair out for a while. -- ~Ethan~ From stefan_ml at behnel.de Tue Sep 20 01:23:43 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 20 Sep 2016 07:23:43 +0200 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: ????? schrieb am 19.09.2016 um 17:59: > If at all, it should be function.bind(). It was discussed and dropped; I > don't remember why, but one problem is that it looks like an in-place > modification. IMHO, the main drawback of that solution is that it only works for functions and not for any other callable in Python. It would introduce an inconsistency without solving the problem in general. Stefan From elazarg at gmail.com Tue Sep 20 03:01:41 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 07:01:41 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: But the foo() finds the function to call, so foo.bind() could be made to find it too. ?????? ??? ??, 20 ????' 2016, 08:24, ??? Stefan Behnel ?: > ????? schrieb am 19.09.2016 um 17:59: > > If at all, it should be function.bind(). It was discussed and dropped; I > > don't remember why, but one problem is that it looks like an in-place > > modification. > > IMHO, the main drawback of that solution is that it only works for > functions and not for any other callable in Python. It would introduce an > inconsistency without solving the problem in general. > > Stefan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Tue Sep 20 03:02:18 2016 From: stephanh42 at gmail.com (Stephan Houben) Date: Tue, 20 Sep 2016 09:02:18 +0200 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: Hi all, I would like to add that I don't believe that discoverability is always better in the `builtins' module. I personally had the experience of going over itertools, where I found 'zip_longest', but I couldn't find a 'zip_shortest'. Only after some googling I found out it was called `zip' and it lived in `builtins'. Stephan 2016-09-20 7:23 GMT+02:00 Stefan Behnel : > ????? schrieb am 19.09.2016 um 17:59: > > If at all, it should be function.bind(). It was discussed and dropped; I > > don't remember why, but one problem is that it looks like an in-place > > modification. > > IMHO, the main drawback of that solution is that it only works for > functions and not for any other callable in Python. It would introduce an > inconsistency without solving the problem in general. > > Stefan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Tue Sep 20 03:04:25 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Matos?=) Date: Tue, 20 Sep 2016 00:04:25 -0700 (PDT) Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160920015529.GK22471@ando.pearwood.info> References: <20160919023201.GB22471@ando.pearwood.info> <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> <20160920015529.GK22471@ando.pearwood.info> Message-ID: Hello, You are correct. Thanks for the explanation. Best regards, JM ter?a-feira, 20 de Setembro de 2016 ?s 02:56:57 UTC+1, Steven D'Aprano escreveu: > On Mon, Sep 19, 2016 at 01:35:53PM -0700, Jo?o Matos wrote: > > Hello, > > > > I don't see why creating a clear command would interfere with > dict.clear() > > which is a function/method. > > For the same reason that you can't have a method called foo.while or > foo.if or foo.raise. If clear is a "command" (a statement) it would need > to be a keyword, like while, if and raise. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Sep 20 03:53:39 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Sep 2016 17:53:39 +1000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 5:01 PM, ????? wrote: > But the foo() finds the function to call, so foo.bind() could be made to > find it too. class Demo: def __init__(self): self.bind = 42 def __call__(self): print("I got called!") foo = Demo() You can most certainly call foo(), but foo.bind() will bite you. With a stand-alone function bind(foo), it can use protocols like __call__. ChrisA From elazarg at gmail.com Tue Sep 20 03:57:28 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 07:57:28 +0000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160920015529.GK22471@ando.pearwood.info> References: <20160919023201.GB22471@ando.pearwood.info> <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> <20160920015529.GK22471@ando.pearwood.info> Message-ID: On Tue, Sep 20, 2016 at 4:56 AM Steven D'Aprano wrote: > On Mon, Sep 19, 2016 at 01:35:53PM -0700, Jo?o Matos wrote: > > Hello, > > > > I don't see why creating a clear command would interfere with > dict.clear() > > which is a function/method. > > For the same reason that you can't have a method called foo.while or > foo.if or foo.raise. If clear is a "command" (a statement) it would need > to be a keyword, like while, if and raise. > > > This is since in Python there are no contextual keywords (like "override" and "final" in C++). I remember encountering error in a Django project where accessing u.pass was a syntax error, but there *was* a field "pass" in u and they had to resort to getattr(u, "pass"). What is the reasoning behind that decision? Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Tue Sep 20 04:09:47 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 08:09:47 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 10:54 AM Chris Angelico wrote: > On Tue, Sep 20, 2016 at 5:01 PM, ????? wrote: > > But the foo() finds the function to call, so foo.bind() could be made to > > find it too. > > class Demo: > def __init__(self): > self.bind = 42 > def __call__(self): > print("I got called!") > > foo = Demo() > > You can most certainly call foo(), but foo.bind() will bite you. > > With a stand-alone function bind(foo), it can use protocols like __call__. > > I meant something like making it a "__bind__" (just a strawman suggestion) and do the same lookup as foo() does, and using a (wrong) functional-programming-inspired syntax foo 5 () Such a syntax will have the side benefit of allowing calling print in a similar way to Python2, which people seem to love. print "hello" () This strawman proposal has many downsides I guess. My point being, this can be made to work, but it's probably not worth it. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Sep 20 04:17:15 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Sep 2016 18:17:15 +1000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 6:09 PM, ????? wrote: > I meant something like making it a "__bind__" (just a strawman suggestion) > and do the same lookup as foo() does, and using a (wrong) > functional-programming-inspired syntax > > foo 5 () > > Such a syntax will have the side benefit of allowing calling print in a > similar way to Python2, which people seem to love. > > print "hello" () > Python has a rule that syntax shouldn't look like grit on Tim's screen. In this case, it looks like the *absence of* grit, which is even worse :) You're giving meaning to the abuttal of two tokens, the first of which must be callable but the second can be anything. And it creates the scary situation of giving valid-but-useless meaning to something all too common in Py2 code: print "hello" This would now create a function that, if called, would print "hello", but then abandons it without a second thought. So it'd work in 2.7, fail with an opaque error in 3.3, fail with a more informative error in 3.5, and silently do nothing in 3.7. No thank you! :) ChrisA From p.f.moore at gmail.com Tue Sep 20 04:20:00 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Sep 2016 09:20:00 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> <20160920015529.GK22471@ando.pearwood.info> Message-ID: On 20 September 2016 at 08:57, ????? wrote: > This is since in Python there are no contextual keywords (like "override" > and "final" in C++). I remember encountering error in a Django project where > accessing u.pass was a syntax error, but there *was* a field "pass" in u and > they had to resort to getattr(u, "pass"). > What is the reasoning behind that decision? Python's grammar is deliberately not context sensitive, as that makes it easier to parse (for both humans and computers). In particular, this means that things *other* than the Python interpreter can parse Python easily (think editor syntax highlighting, linters, etc). There have been occasional deviations from this (for example, the "as" in "import foo as bar" was, for a time, only a keyword in that specific context) but I don't believe any of them survived long-term. Guido has always avoided constructs that need lookahead or other contextual information to parse correctly, for this specific reason. Paul From elazarg at gmail.com Tue Sep 20 04:22:10 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 08:22:10 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: Yeah I did say it was a strawman :) On Tue, Sep 20, 2016 at 11:17 AM Chris Angelico wrote: > On Tue, Sep 20, 2016 at 6:09 PM, ????? wrote: > > I meant something like making it a "__bind__" (just a strawman > suggestion) > > and do the same lookup as foo() does, and using a (wrong) > > functional-programming-inspired syntax > > > > foo 5 () > > > > Such a syntax will have the side benefit of allowing calling print in a > > similar way to Python2, which people seem to love. > > > > print "hello" () > > > > Python has a rule that syntax shouldn't look like grit on Tim's > screen. In this case, it looks like the *absence of* grit, which is > even worse :) You're giving meaning to the abuttal of two tokens, the > first of which must be callable but the second can be anything. And it > creates the scary situation of giving valid-but-useless meaning to > something all too common in Py2 code: > > print "hello" > > This would now create a function that, if called, would print "hello", > but then abandons it without a second thought. So it'd work in 2.7, > fail with an opaque error in 3.3, fail with a more informative error > in 3.5, and silently do nothing in 3.7. No thank you! :) > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Sep 20 04:26:21 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Sep 2016 18:26:21 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> <4b05826b-fe06-450a-b5f7-b2ce670c4268@googlegroups.com> <20160920015529.GK22471@ando.pearwood.info> Message-ID: On Tue, Sep 20, 2016 at 6:20 PM, Paul Moore wrote: > There > have been occasional deviations from this (for example, the "as" in > "import foo as bar" was, for a time, only a keyword in that specific > context) but I don't believe any of them survived long-term. async and await are similarly context-sensitive for a couple of versions, to make the breakage simpler. But again, it's not intended to be long-term. ChrisA From desmoulinmichel at gmail.com Tue Sep 20 05:42:50 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 20 Sep 2016 11:42:50 +0200 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: <45479549-2e8d-e2a4-edd0-abf31a0a7b52@gmail.com> (Sorry, answered to one person again. Reposting) I like the bind() idea because it doesn't clutter the builtin namespace. It solves the import problem and feels very natural IMO. The only issue is the name. In my mind, bind() convey the idea you are modifying the function it self, while partial() convey the idea you return new function. It could also be written in C for performances. Le 20/09/2016 ? 10:22, ????? a ?crit : > Yeah I did say it was a strawman :) > > On Tue, Sep 20, 2016 at 11:17 AM Chris Angelico > wrote: > > On Tue, Sep 20, 2016 at 6:09 PM, ????? > wrote: > > I meant something like making it a "__bind__" (just a strawman > suggestion) > > and do the same lookup as foo() does, and using a (wrong) > > functional-programming-inspired syntax > > > > foo 5 () > > > > Such a syntax will have the side benefit of allowing calling print > in a > > similar way to Python2, which people seem to love. > > > > print "hello" () > > > > Python has a rule that syntax shouldn't look like grit on Tim's > screen. In this case, it looks like the *absence of* grit, which is > even worse :) You're giving meaning to the abuttal of two tokens, the > first of which must be callable but the second can be anything. And it > creates the scary situation of giving valid-but-useless meaning to > something all too common in Py2 code: > > print "hello" > > This would now create a function that, if called, would print "hello", > but then abandons it without a second thought. So it'd work in 2.7, > fail with an opaque error in 3.3, fail with a more informative error > in 3.5, and silently do nothing in 3.7. No thank you! :) > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From desmoulinmichel at gmail.com Tue Sep 20 05:43:52 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 20 Sep 2016 11:43:52 +0200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <6fa94ec8-f651-cbd7-b42d-233a7b2969d0@gmail.com> References: <6fa94ec8-f651-cbd7-b42d-233a7b2969d0@gmail.com> Message-ID: <03f6a9ea-3779-fa44-6e44-5e46cca68136@gmail.com> +1 for this. I regularly miss this feature. Le 17/09/2016 ? 13:12, Jo?o Matos a ?crit : > Hello, > > In other interpreted programming languages the clear screen command > (whatever it is) also does not clear the session. > It just clears the screen clutter. > > As I said, this would be very useful for newbies, which don't know > anything about usercustomize or sitecustomize. > > > Best regards, > > JM > > > On 17-09-2016 12:07, Chris Angelico wrote: >> On Sat, Sep 17, 2016 at 8:51 PM, Jo?o Matos wrote: >>> I would like to suggest adding a clear command (not function) to Python. >>> It's simple purpose would be to clear the REPL screen, leaving the >>> >>> prompt at the top left of the screen. >>> >>> This is something very basic but also very useful for newbies learning >>> Python from the REPL. >>> After some trial and errors it is best to start with a clean screen. >>> Clearing the screen helps clear your mind. >>> >> I'm not sure that it _is_ helpful, given that you're starting with a >> clean screen but not restarting the session (so you'll still have all >> the state from your previous work). If you want a completely fresh >> start, just exit Python, clear the screen with a shell command, and >> re-enter. >> >> The easiest way to play around with this would be to create a pure >> Python clear() function in your usercustomize or sitecustomize, and >> then try it in your own workflow - see whether it annoys you that it >> doesn't change the interpreter state. Maybe it will, maybe it won't. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From elazarg at gmail.com Tue Sep 20 05:56:48 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 09:56:48 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: <45479549-2e8d-e2a4-edd0-abf31a0a7b52@gmail.com> References: <45479549-2e8d-e2a4-edd0-abf31a0a7b52@gmail.com> Message-ID: foo.__call__.partial() solves most of the problem I think. ?????? ??? ??, 20 ????' 2016, 12:43, ??? Michel Desmoulin ?< desmoulinmichel at gmail.com>: > (Sorry, answered to one person again. Reposting) > > I like the bind() idea because it doesn't clutter the builtin namespace. > It solves the import problem and feels very natural IMO. > > The only issue is the name. In my mind, bind() convey the idea you are > modifying the function it self, while partial() convey the idea you > return new function. > > It could also be written in C for performances. > > > Le 20/09/2016 ? 10:22, ????? a ?crit : > > Yeah I did say it was a strawman :) > > > > On Tue, Sep 20, 2016 at 11:17 AM Chris Angelico > > wrote: > > > > On Tue, Sep 20, 2016 at 6:09 PM, ????? > > wrote: > > > I meant something like making it a "__bind__" (just a strawman > > suggestion) > > > and do the same lookup as foo() does, and using a (wrong) > > > functional-programming-inspired syntax > > > > > > foo 5 () > > > > > > Such a syntax will have the side benefit of allowing calling print > > in a > > > similar way to Python2, which people seem to love. > > > > > > print "hello" () > > > > > > > Python has a rule that syntax shouldn't look like grit on Tim's > > screen. In this case, it looks like the *absence of* grit, which is > > even worse :) You're giving meaning to the abuttal of two tokens, the > > first of which must be callable but the second can be anything. And > it > > creates the scary situation of giving valid-but-useless meaning to > > something all too common in Py2 code: > > > > print "hello" > > > > This would now create a function that, if called, would print > "hello", > > but then abandons it without a second thought. So it'd work in 2.7, > > fail with an opaque error in 3.3, fail with a more informative error > > in 3.5, and silently do nothing in 3.7. No thank you! :) > > > > ChrisA > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Sep 20 06:35:30 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Sep 2016 11:35:30 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On 19 September 2016 at 23:46, ????? wrote: >> import pip >> pip.install('attrs') >> import attr > > Please forgive me for my ignorance, but it doesn't work as written - what's > the actual method? As David Mertz said, pip.main(['install', 'attrs']) works right now, but it is NOT a supported use of pip[1]. To be 100% explicit, the only supported way of doing this is import sys from subprocess import run run([sys.executable, '-m', 'pip', 'install', 'attrs']) I suggested a hypothetical "pip.install" method as there is currently some discussion on the pip tracker about providing a supported install method. But it doesn't exist yet. Sorry for being confusing. While on the whole subject of this, I should also point out that there are a lot of potential issues with installing new packages while a Python program is running. They are all low-probability, and easy to avoid if you're not doing weird things, but for a generally-promoted mechanism, we need to explain the corner cases[2], and an approach with a list of caveats longer than the main documentation is problematic. 1. If the install fails, you need to catch that and report it to the user, in a more friendly manner than pip's output. For example if the user has no C compiler and you need a C extension to be built. 2. You quite possibly want to suppress pip's output if it's *not* a failure, as it'll clutter up the program's real output. 3. If the code has already imported foo.bar, then you install a new version of foo (there are discussions as to whether pip install foo should automatically imply --upgrade, so even if it won't do that by default now, it might in the future), and maybe that new version doesn't have a bar submodule. So now you have a weird mix of old and new code in your process. 4. The install mechanism sometimes (I can't recall the details) caches the fact that it couldn't import a module. If it does that and then later you pip install that module, imports will still fail because the information is cached. I'm still not at all clear why any of this is so much better than a comment at the top of the script # To run this script, you need to "pip install attrs" first Paul. [1] We've had people report issues where pip breaks their logging config, for example, because pip uses logging but doesn't expect to be run from user code that also does so. [2] That "run([sys.executable, ...])" invocation doesn't work in an embedded program, for example, where sys.executable isn't "python". From steve at pearwood.info Tue Sep 20 06:31:03 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Sep 2016 20:31:03 +1000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: <45479549-2e8d-e2a4-edd0-abf31a0a7b52@gmail.com> Message-ID: <20160920103103.GL22471@ando.pearwood.info> On Tue, Sep 20, 2016 at 09:56:48AM +0000, ????? wrote: > foo.__call__.partial() solves most of the problem I think. There are two problems with that, one obvious and one subtle. The obvious one is that __call__ is a dunder method, which means that accessing it directly from outside of the object's class is a code smell. It's not necessarily *wrong*, but its a bit... smelly. A bit suspicious. Something we should think *very* carefully about before encouraging. The subtle one is that foo.__call__ may not actually be the method that is used when you call foo(). py> class X(object): ... def __call__(self): ... return "from X" ... py> x = X() py> x.__call__ = lambda: "surprise!" py> x() 'from X' py> x.__call__() 'surprise!' -- Steve From elazarg at gmail.com Tue Sep 20 06:40:45 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 10:40:45 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: I believe that at least some of these problems can be addressed given that pip *knows* that this import is an in-script import. So the list of corner cases will be shorter. Elazar On Tue, Sep 20, 2016 at 1:35 PM Paul Moore wrote: > On 19 September 2016 at 23:46, ????? wrote: > >> import pip > >> pip.install('attrs') > >> import attr > > > > Please forgive me for my ignorance, but it doesn't work as written - > what's > > the actual method? > > As David Mertz said, pip.main(['install', 'attrs']) works right now, > but it is NOT a supported use of pip[1]. To be 100% explicit, the only > supported way of doing this is > > import sys > from subprocess import run > run([sys.executable, '-m', 'pip', 'install', 'attrs']) > > I suggested a hypothetical "pip.install" method as there is currently > some discussion on the pip tracker about providing a supported install > method. But it doesn't exist yet. Sorry for being confusing. > > While on the whole subject of this, I should also point out that there > are a lot of potential issues with installing new packages while a > Python program is running. They are all low-probability, and easy to > avoid if you're not doing weird things, but for a generally-promoted > mechanism, we need to explain the corner cases[2], and an approach > with a list of caveats longer than the main documentation is > problematic. > > 1. If the install fails, you need to catch that and report it to the > user, in a more friendly manner than pip's output. For example if the > user has no C compiler and you need a C extension to be built. > 2. You quite possibly want to suppress pip's output if it's *not* a > failure, as it'll clutter up the program's real output. > 3. If the code has already imported foo.bar, then you install a new > version of foo (there are discussions as to whether pip install foo > should automatically imply --upgrade, so even if it won't do that by > default now, it might in the future), and maybe that new version > doesn't have a bar submodule. So now you have a weird mix of old and > new code in your process. > 4. The install mechanism sometimes (I can't recall the details) caches > the fact that it couldn't import a module. If it does that and then > later you pip install that module, imports will still fail because the > information is cached. > > I'm still not at all clear why any of this is so much better than a > comment at the top of the script > > # To run this script, you need to "pip install attrs" first > > Paul. > > [1] We've had people report issues where pip breaks their logging > config, for example, because pip uses logging but doesn't expect to be > run from user code that also does so. > [2] That "run([sys.executable, ...])" invocation doesn't work in an > embedded program, for example, where sys.executable isn't "python". > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Sep 20 06:42:23 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Sep 2016 11:42:23 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: On 20 September 2016 at 00:28, ????? wrote: > On Tue, Sep 20, 2016 at 2:20 AM Stephen J. Turnbull > wrote: >> >> ????? writes: >> >> > Another use case, though I admit not the top priority of anyone here, >> is >> > that of assignment checkers. In most courses I took at the university, >> the >> > person who checks the assignments says something like "you are allowed >> to >> > use only this this and this libraries", in order not to mess with >> unknown >> > dependencies from tens of students (I am talking about advanced >> courses, >> > where the method I use to solve the problem is unimportant or only >> requires >> > explanation). With this statement they can simply state "you can import >> > pip". >> >> In other words, you're advocating a feature that allows script writers >> to download, install, and execute arbitrary, unsandboxed code on any >> machine where the script is run. That sounds ... *scary*, when put >> that way. Remember, you're advocating this on behalf of people who by >> assumption are infants years below the age of consent. >> > Let me understand. Your argument is "installing pip modules is unsafe, and > therefore we should make it less usable, where the appropriate amount of > (un)usability is running cmd and then `pip install unsafe`" ? The argument is that if someone posts a script that says it does something innocuous, for example "benchmark showing that X is faster than Y", people will scan it, see that it looks OK, and run it. They have a reasonable expectation that it's not a security risk. If it requires a benchmarking module from PyPI, they may not immediately notice that "from __pypi__ import benchmark" opens up a security risk. On the other hand, being explicitly told to run a command whose sole purpose is to download and install an external tool clearly indicates to them that they need to be aware of what's happening. Likely they will simply do so and it's no big deal. But in certain environments they may have to pause and possibly even check with their security team as to whether that tool has been approved. It's not about "making it less usable", it's about ensuring that the implications are clear - explicit is better than implicit, in effect. Which is a particularly important principle when security risks such as "downloading arbitrary code from the internet" is involved. Paul From elazarg at gmail.com Tue Sep 20 06:46:01 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 10:46:01 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Sep 20, 2016 at 1:42 PM Paul Moore wrote: > On 20 September 2016 at 00:28, ????? wrote: > > On Tue, Sep 20, 2016 at 2:20 AM Stephen J. Turnbull > > wrote: > >> > >> ????? writes: > >> > >> > Another use case, though I admit not the top priority of anyone here, > >> is > >> > that of assignment checkers. In most courses I took at the > university, > >> the > >> > person who checks the assignments says something like "you are > allowed > >> to > >> > use only this this and this libraries", in order not to mess with > >> unknown > >> > dependencies from tens of students (I am talking about advanced > >> courses, > >> > where the method I use to solve the problem is unimportant or only > >> requires > >> > explanation). With this statement they can simply state "you can > import > >> > pip". > >> > >> In other words, you're advocating a feature that allows script writers > >> to download, install, and execute arbitrary, unsandboxed code on any > >> machine where the script is run. That sounds ... *scary*, when put > >> that way. Remember, you're advocating this on behalf of people who by > >> assumption are infants years below the age of consent. > >> > > Let me understand. Your argument is "installing pip modules is unsafe, > and > > therefore we should make it less usable, where the appropriate amount of > > (un)usability is running cmd and then `pip install unsafe`" ? > > The argument is that if someone posts a script that says it does > something innocuous, for example "benchmark showing that X is faster > than Y", people will scan it, see that it looks OK, and run it. They > have a reasonable expectation that it's not a security risk. > > If it requires a benchmarking module from PyPI, they may not > immediately notice that "from __pypi__ import benchmark" opens up a > security risk. On the other hand, being explicitly told to run a > command whose sole purpose is to download and install an external tool > clearly indicates to them that they need to be aware of what's > happening. Likely they will simply do so and it's no big deal. But in > certain environments they may have to pause and possibly even check > with their security team as to whether that tool has been approved. > > It's not about "making it less usable", it's about ensuring that the > implications are clear - explicit is better than implicit, in effect. > Which is a particularly important principle when security risks such > as "downloading arbitrary code from the internet" is involved. > > So it should be something like from unsafe.__pip__ import benchmark Where unsafe is the hypothetical namespace in which exec(), eval() and subprocess.run() would have reside given your concerns. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Sep 20 06:47:16 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Sep 2016 11:47:16 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: On 20 September 2016 at 11:40, ????? wrote: > I believe that at least some of these problems can be addressed given that > pip *knows* that this import is an in-script import. Not with pip as it is now. There's talk of detecting problem cases and reporting back "you need to restart your Python session before proceeding" for use in cases like an IDLE "install packages" menu item, but that's not been implemented yet. That may help. Paul From p.f.moore at gmail.com Tue Sep 20 06:56:11 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Sep 2016 11:56:11 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: On 20 September 2016 at 11:46, ????? wrote: > So it should be something like > > from unsafe.__pip__ import benchmark > > Where unsafe is the hypothetical namespace in which exec(), eval() and > subprocess.run() would have reside given your concerns. In my opinion, it should be # Please install benchmark using pip to run this script Or you should run the script using a dedicated runner like rwt. Or you can depend on a custom import hook that makes "from __pip__ install..." work as you want. I'm just saying that I don't want core Python to implicitly install packages for me. But that's simply a personal opinion. I'm not trying to persuade you you're wrong, just trying to explain my position. We can agree to differ. It certainly doesn't seem to me that there's any need for you to modify your proposal to suit me, it's unlikely I'll like any variation you're going to be happy with, which is fine (you're under no obligation to convince me). Paul From elazarg at gmail.com Tue Sep 20 07:02:55 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 11:02:55 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: I think I generally understand concerns, and partially agree. I'm certainly not dismissing them. I only try to understand what are the precise problems and why the current situation - with dangerous functions at reach, easily buried deep in the code instead of marked on the top of the script - is so much better. Elazar On Tue, Sep 20, 2016 at 1:56 PM Paul Moore wrote: > On 20 September 2016 at 11:46, ????? wrote: > > So it should be something like > > > > from unsafe.__pip__ import benchmark > > > > Where unsafe is the hypothetical namespace in which exec(), eval() and > > subprocess.run() would have reside given your concerns. > > In my opinion, it should be > > # Please install benchmark using pip to run this script > > Or you should run the script using a dedicated runner like rwt. Or you > can depend on a custom import hook that makes "from __pip__ > install..." work as you want. I'm just saying that I don't want core > Python to implicitly install packages for me. But that's simply a > personal opinion. I'm not trying to persuade you you're wrong, just > trying to explain my position. We can agree to differ. It certainly > doesn't seem to me that there's any need for you to modify your > proposal to suit me, it's unlikely I'll like any variation you're > going to be happy with, which is fine (you're under no obligation to > convince me). > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Tue Sep 20 07:12:29 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 11:12:29 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: Moreover, being able to do it programmatically is a security risk, since it requires elevated privileges that I don't know how to drop, and most people will not think about doing, but a library implementation will. So if someone uses subprocess.run(), and the system asks the user for elevated privileges, a bug in later code can easily cause serious harm instead of failing. Yes, untrusted code should be sandboxed - but it isn't, more often than not. Elazar ?On Tue, Sep 20, 2016 at 2:02 PM ???????? wrote:? > I think I generally understand concerns, and partially agree. I'm > certainly not dismissing them. I only try to understand what are the > precise problems and why the current situation - with dangerous functions > at reach, easily buried deep in the code instead of marked on the top of > the script - is so much better. > > Elazar > > On Tue, Sep 20, 2016 at 1:56 PM Paul Moore wrote: > >> On 20 September 2016 at 11:46, ????? wrote: >> > So it should be something like >> > >> > from unsafe.__pip__ import benchmark >> > >> > Where unsafe is the hypothetical namespace in which exec(), eval() and >> > subprocess.run() would have reside given your concerns. >> >> In my opinion, it should be >> >> # Please install benchmark using pip to run this script >> >> Or you should run the script using a dedicated runner like rwt. Or you >> can depend on a custom import hook that makes "from __pip__ >> install..." work as you want. I'm just saying that I don't want core >> Python to implicitly install packages for me. But that's simply a >> personal opinion. I'm not trying to persuade you you're wrong, just >> trying to explain my position. We can agree to differ. It certainly >> doesn't seem to me that there's any need for you to modify your >> proposal to suit me, it's unlikely I'll like any variation you're >> going to be happy with, which is fine (you're under no obligation to >> convince me). >> >> Paul >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xavier.combelle at gmail.com Tue Sep 20 07:52:36 2016 From: xavier.combelle at gmail.com (Xavier Combelle) Date: Tue, 20 Sep 2016 13:52:36 +0200 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> Message-ID: Le 20/09/2016 ? 12:35, Paul Moore a ?crit : > > While on the whole subject of this, I should also point out that there > are a lot of potential issues with installing new packages while a > Python program is running. They are all low-probability, and easy to > avoid if you're not doing weird things, but for a generally-promoted > mechanism, we need to explain the corner cases[2], and an approach > with a list of caveats longer than the main documentation is > problematic. > > 1. If the install fails, you need to catch that and report it to the > user, in a more friendly manner than pip's output. For example if the > user has no C compiler and you need a C extension to be built. > 2. You quite possibly want to suppress pip's output if it's *not* a > failure, as it'll clutter up the program's real output. > 3. If the code has already imported foo.bar, then you install a new > version of foo (there are discussions as to whether pip install foo > should automatically imply --upgrade, so even if it won't do that by > default now, it might in the future), and maybe that new version > doesn't have a bar submodule. So now you have a weird mix of old and > new code in your process. > 4. The install mechanism sometimes (I can't recall the details) caches > the fact that it couldn't import a module. If it does that and then > later you pip install that module, imports will still fail because the > information is cached. > > you forget the use of a linux where it will fail with high probability because by default, pip need to have super user right From p.f.moore at gmail.com Tue Sep 20 08:29:42 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Sep 2016 13:29:42 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: On 20 September 2016 at 12:12, ????? wrote: > Moreover, being able to do it programmatically is a security risk, since it > requires elevated privileges that I don't know how to drop, and most people > will not think about doing, but a library implementation will. > > So if someone uses subprocess.run(), and the system asks the user for > elevated privileges, a bug in later code can easily cause serious harm > instead of failing. Yes, untrusted code should be sandboxed - but it isn't, > more often than not. It's not possible to gain elevated privileges without asking the user (certainly not on Windows, and I don't believe so on Unix). So what you're talking about is getting people used to the idea that running a script they grabbed off the internet would ask them to run it elevated, and they should agree. That sounds to me like a very dangerous lesson to be teaching. (rwt gets round this by installing dependencies to a temporary location for the duration of the script. I *really* recommend that you look into it if you haven't already). Paul From random832 at fastmail.com Tue Sep 20 08:58:37 2016 From: random832 at fastmail.com (Random832) Date: Tue, 20 Sep 2016 08:58:37 -0400 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: <1474376317.2700684.731358161.7067CDC8@webmail.messagingengine.com> On Tue, Sep 20, 2016, at 07:12, ????? wrote: > Moreover, being able to do it programmatically is a security risk, > since it requires elevated privileges that I don't know how to drop, > and most people will not think about doing, but a library > implementation will. Maybe we should be thinking about why pip requires elevated privileges. From bzvi7919 at gmail.com Tue Sep 20 10:49:46 2016 From: bzvi7919 at gmail.com (Bar Harel) Date: Tue, 20 Sep 2016 14:49:46 +0000 Subject: [Python-ideas] singledispatch for instance methods In-Reply-To: References: Message-ID: At last! Haven't used single dispatch exactly because of that. Thank you savior! +1 On Tue, Sep 20, 2016, 6:03 AM Tim Mitchell wrote: > Hi All, > > We have a modified version of singledispatch at work which works for > methods as well as functions. We have open-sourced it as methoddispatch > (pypi: https://pypi.python.org/pypi/methoddispatch). > > IMHO I thought it would make a nice addition to python stdlib. > > What does everyone else think? > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Sep 20 11:09:23 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Sep 2016 16:09:23 +0100 Subject: [Python-ideas] from __pip__ import In-Reply-To: <1474376317.2700684.731358161.7067CDC8@webmail.messagingengine.com> References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> <1474376317.2700684.731358161.7067CDC8@webmail.messagingengine.com> Message-ID: On 20 September 2016 at 13:58, Random832 wrote: > On Tue, Sep 20, 2016, at 07:12, ????? wrote: >> Moreover, being able to do it programmatically is a security risk, >> since it requires elevated privileges that I don't know how to drop, >> and most people will not think about doing, but a library >> implementation will. > > Maybe we should be thinking about why pip requires elevated privileges. I'm not sure to what extent this was a rhetorical question, but basically because, by default pip installs into the Python installation directory, and if the user is running a system Python, that directory is only modifiable by an admin. You can use --user to make pip install into the user's site-packages. But that's not the default, and the proposal didn't discuss supplying any non-default options to pip. Pip could be changed to make the default --user, but that's not happened yet (and there are some compatibility issues holding it up). And even ignoring that, what about *other* pip options that might be needed (for example, specifying a proxy, or a non-default certificate store)? There's no capability to specify them in the proposal. Paul. From gvanrossum at gmail.com Tue Sep 20 11:19:18 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 20 Sep 2016 08:19:18 -0700 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: I am radically opposed to this proposal. Every time I see a partial application I wonder endlessly about what's going on. --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Tue Sep 20 11:29:36 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 15:29:36 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: Guido, can you please elaborate? "What's going on" is usually that the same arguments are going to be passed over and over again, and the programmer wanted to avoid this repetition. The other option is adjusting the function to a predefined interface. The alternative to partial is writing a closure in the form of a function, that needs to be carefully inspected to verify that it is indeed just a partial application and not something more complex. It has more opportunity for introducing an error. And it's longer and adds distance between related parts of the code. Elazar ?????? ??? ??, 20 ????' 2016, 18:20, ??? Guido van Rossum ?< gvanrossum at gmail.com>: > I am radically opposed to this proposal. Every time I see a partial > application I wonder endlessly about what's going on. > > --Guido (mobile) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Sep 20 09:31:18 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Sep 2016 23:31:18 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> Message-ID: <20160920133118.GM22471@ando.pearwood.info> On Mon, Sep 19, 2016 at 06:18:38AM -0400, Terry Reedy wrote: > On 9/19/2016 4:31 AM, Paul Moore wrote: > > >shutil.get_terminal_size() is available on all platforms. > > On windows, it works with Command Prompt and PowerShell but fails in > IDLE, as it must. Why "must" it fail? What is it about the IDLE terminal that prevents it from emulating an actual terminal, like all(?) the other terminal emulation programs people use? I just opened my Konqueror file & web browser, gave the command "Show Terminal Emulator", and ran Python: py> import shutil py> shutil.get_terminal_size() os.terminal_size(columns=85, lines=10) That's spot on perfectly correct. I then resized the window and tried it again: py> shutil.get_terminal_size() os.terminal_size(columns=108, lines=13) I then quit Python, and ran idle3.3 from the Konqueror terminal. Unfortunately, get_terminal_size() returned the size of the *Konqueror* terminal, instead of the IDLE terminal. I think that's a bug. I don't know if its a bug in IDLE or get_terminal_size() or both. -- Steve From gvanrossum at gmail.com Tue Sep 20 11:51:59 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 20 Sep 2016 08:51:59 -0700 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 8:29 AM, ????? wrote: > Guido, can you please elaborate? > > "What's going on" is usually that the same arguments are going to be > passed over and over again, and the programmer wanted to avoid this > repetition. The other option is adjusting the function to a predefined > interface. > I did a little searching on a large Python code base I have access to, and that's not what I found. The partial calls don't seem to produce code that's in any way clearer than the equivalent use of lambda would. My conclusion is that most people who are, um, partial to partial do so out of a habit (or perhaps a belief that it's more performant), not to make their code clearer. (FWIW I found an order of magnitude more uses of lambda than of partial in the same code base.) > The alternative to partial is writing a closure in the form of a function, > that needs to be carefully inspected to verify that it is indeed just a > partial application and not something more complex. It has more opportunity > for introducing an error. And it's longer and adds distance between related > parts of the code. > We seem to have fundamentally different ideas of what sort of code is most readable. The alternative to partial is usually a lambda, and the signature of the lambda helps me understand how it is being called. I don't see how the lambda is longer or creates more distance. There are some exceptions, when the function has a complex signature that should mostly be preserved (and a few other, rare exceptions). But most of the uses of partial that I found had exactly one call site and the call site was not calling a complex signature. A big problem I have reading code that uses partial is that *unless I already know the function being partialed*, the partial() call itself gives me no clue about the signature of that function. So then I have to look for the call site of the partial (what do you call that?) and then in my head I have to combine that with the partial parameters and figure out what is really happening. A lambda would have given me a simple stepping stone. In effect it's a little bit of "typing" right there, showing me the signature that's being called, which is useful for reading the code quickly. Also, I once timed it and could not show that partial was faster. This surprised me but it wa what I measured (in one particular case). > Elazar > > ?????? ??? ??, 20 ????' 2016, 18:20, ??? Guido van Rossum ?< > gvanrossum at gmail.com>: > >> I am radically opposed to this proposal. Every time I see a partial >> application I wonder endlessly about what's going on. >> >> --Guido (mobile) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Sep 20 12:00:57 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 20 Sep 2016 12:00:57 -0400 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 11:51 AM, Guido van Rossum wrote: > Also, I once timed it and could not show that partial was faster. This > surprised me but it was what I measured (in one particular case). I did similar timings on several occasions in the past and was also surprised by the results. Lambdas are often faster than equivalent partials. It looks like at least recent versions of Python have ways to optimize calls to pure Python functions that are not available to functions implemented in C or called from C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Tue Sep 20 12:42:04 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 20 Sep 2016 11:42:04 -0500 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On Sep 20, 2016 10:52 AM, "Guido van Rossum" wrote: > > On Tue, Sep 20, 2016 at 8:29 AM, ????? wrote: >> *snip* >> The alternative to partial is writing a closure in the form of a function, that needs to be carefully inspected to verify that it is indeed just a partial application and not something more complex. It has more opportunity for introducing an error. And it's longer and adds distance between related parts of the code. > > > We seem to have fundamentally different ideas of what sort of code is most readable. The alternative to partial is usually a lambda, and the signature of the lambda helps me understand how it is being called. I don't see how the lambda is longer or creates more distance. There are some exceptions, when the function has a complex signature that should mostly be preserved (and a few other, rare exceptions). But most of the uses of partial that I found had exactly one call site and the call site was not calling a complex signature. > > A big problem I have reading code that uses partial is that *unless I already know the function being partialed*, the partial() call itself gives me no clue about the signature of that function. So then I have to look for the call site of the partial (what do you call that?) and then in my head I have to combine that with the partial parameters and figure out what is really happening. A lambda would have given me a simple stepping stone. In effect it's a little bit of "typing" right there, showing me the signature that's being called, which is useful for reading the code quickly. Most often, when I see lambdas used for this, it looks like: lambda *args, **kw: myfunc(partial_arg, *args, **kw) which isn't more readable than just: partial(myfunc, partial_func) Doing something like: lambda x, y: myfunc(partial_arg, x, y) is more error-prone to changes in myfunc's signature. Also, I don't quite understand the readability issue here; all partial is doing is just...err...partializing the function (I should totally patent that phrase). I find it easy to mentally expand it: partial(f, ...) == lambda *args, **kw: f(..., *args, **kw) > Also, I once timed it and could not show that partial was faster. This surprised me but it wa what I measured (in one particular case). > >> >> Elazar >> >> >> ?????? ??? ??, 20 ????' 2016, 18:20, ??? Guido van Rossum ?< gvanrossum at gmail.com>: >>> >>> I am radically opposed to this proposal. Every time I see a partial application I wonder endlessly about what's going on. >>> >>> --Guido (mobile) >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at slenders.be Tue Sep 20 13:04:39 2016 From: jonathan at slenders.be (Jonathan Slenders) Date: Tue, 20 Sep 2016 19:04:39 +0200 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: Le 20 sept. 2016 18:42, "Ryan Gonzalez" a ?crit : > Doing something like: > > lambda x, y: myfunc(partial_arg, x, y) > > is more error-prone to changes in myfunc's signature. No, if the signature of the function changes, then the signature of the partial would also change. The risk is the same. Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Sep 20 13:48:23 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 20 Sep 2016 10:48:23 -0700 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: I find myself "partializing" in ways partial() doesn't support more often than not. E.g. lambda first, third: myfunc(first, 42, third) I think it's good to have partial() in functools, but it's two orders of magnitude less common than things that should be in builtins. On Sep 20, 2016 9:42 AM, "Ryan Gonzalez" wrote: > lambda *args, **kw: myfunc(partial_arg, *args, **kw) > > which isn't more readable than just: > > partial(myfunc, partial_func) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Tue Sep 20 14:17:33 2016 From: stephanh42 at gmail.com (Stephan Houben) Date: Tue, 20 Sep 2016 20:17:33 +0200 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: I must admit I am a bit partial to partial, you can do fun things like this: >>> from functools import partial >>> @partial(partial, partial) ... def add(x, y): ... return x+y ... >>> add(3)(4) 7 I suppose that isn't exactly going to convince Guide to put it in builtins, though. Stephan 2016-09-20 19:48 GMT+02:00 David Mertz : > I find myself "partializing" in ways partial() doesn't support more often > than not. E.g. > > lambda first, third: myfunc(first, 42, third) > > I think it's good to have partial() in functools, but it's two orders of > magnitude less common than things that should be in builtins. > > On Sep 20, 2016 9:42 AM, "Ryan Gonzalez" wrote: > > lambda *args, **kw: myfunc(partial_arg, *args, **kw) > > > > which isn't more readable than just: > > > > partial(myfunc, partial_func) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Sep 20 14:35:54 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 20 Sep 2016 11:35:54 -0700 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: I actually wrote the book _ Functional Programming in Python_, yet I haven't the foggiest idea how to parse that code. A decorator parameterized buy two instances of the decorator function itself?! I know I could read the implementation and figure out why that works. But I don't want to encourage that code in real use. On Sep 20, 2016 11:17 AM, "Stephan Houben" wrote: > I must admit I am a bit partial to partial, you can do fun things like > this: > > >>> from functools import partial > >>> @partial(partial, partial) > ... def add(x, y): > ... return x+y > ... > >>> add(3)(4) > 7 > > I suppose that isn't exactly going to convince Guide to put it in > builtins, though. > > Stephan > > > > 2016-09-20 19:48 GMT+02:00 David Mertz : > >> I find myself "partializing" in ways partial() doesn't support more often >> than not. E.g. >> >> lambda first, third: myfunc(first, 42, third) >> >> I think it's good to have partial() in functools, but it's two orders of >> magnitude less common than things that should be in builtins. >> >> On Sep 20, 2016 9:42 AM, "Ryan Gonzalez" wrote: >> > lambda *args, **kw: myfunc(partial_arg, *args, **kw) >> > >> > which isn't more readable than just: >> > >> > partial(myfunc, partial_func) >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Tue Sep 20 14:45:44 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 18:45:44 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 9:18 PM Stephan Houben wrote: > I must admit I am a bit partial to partial, you can do fun things like > this: > > >>> from functools import partial > >>> @partial(partial, partial) > ... def add(x, y): > ... return x+y > ... > >>> add(3)(4) > 7 > > I suppose that isn't exactly going to convince Guide to put it in > builtins, though. > > I quietly LOLed, but note that for three arguments you'll need @partial(partial, partial(partial, partial)) def add(a, b, c): return a + b + c >>> add(1)(2)(3) 6 So @curry is the transitive closure or something :) Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Sep 20 14:48:29 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 21 Sep 2016 03:48:29 +0900 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> Message-ID: <22497.33917.616919.671799@turnbull.sk.tsukuba.ac.jp> Paul Moore writes: > I'm just saying that I don't want core Python to implicitly install > packages for me. +1 > But that's simply a personal opinion. +0.5 It's not *purely* a personal opinion, though. The original proposal is to include the facility and allow *script authors*, rather than script users, to invoke it. Since the (original) motivation is for it to work for naive users, you'll want it on by default to support them (they're going to click on an icon, so command line options and environment variables are right out, no?) I have no objection to people who want to install this in their personal environments, but that doesn't serve the original use case of distributing code that depends on non-stdlib modules to people who aren't handy with pip. To serve those use cases, we all need to get it by default. That I personally don't want (for me or for the students I supervise), and I'm pretty sure my employer will take a very dim view of it. Steve From elazarg at gmail.com Tue Sep 20 14:58:03 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 18:58:03 +0000 Subject: [Python-ideas] from __pip__ import In-Reply-To: <22497.33917.616919.671799@turnbull.sk.tsukuba.ac.jp> References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> <22497.33917.616919.671799@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Sep 20, 2016 at 9:48 PM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Paul Moore writes: > > > I'm just saying that I don't want core Python to implicitly install > > packages for me. > > +1 > > > But that's simply a personal opinion. > > +0.5 > > It's not *purely* a personal opinion, though. The original proposal > is to include the facility and allow *script authors*, rather than > script users, to invoke it. Since the (original) motivation is for it > to work for naive users, you'll want it on by default to support them > (they're going to click on an icon, so command line options and > environment variables are right out, no?) > > I have no objection to people who want to install this in their > personal environments, but that doesn't serve the original use case of > distributing code that depends on non-stdlib modules to people who > aren't handy with pip. To serve those use cases, we all need to get > it by default. That I personally don't want (for me or for the > students I supervise), and I'm pretty sure my employer will take a > very dim view of it. > > I think that combining user convenience and security considerations, there should be some way to invoke a GUI version of pip with flashing screen asking for permissions to install the library. In situations where interaction with the user is not trivial (i.e. when you don't have GUI accessible) we can assume that the user is knowledgeable enough to install the dependencies by herself. The import statement will be self explanatory in this case. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Sep 20 16:59:14 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 21 Sep 2016 06:59:14 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> <22497.33917.616919.671799@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Sep 21, 2016 at 4:58 AM, ????? wrote: > I think that combining user convenience and security considerations, there > should be some way to invoke a GUI version of pip with flashing screen > asking for permissions to install the library. In situations where > interaction with the user is not trivial (i.e. when you don't have GUI > accessible) we can assume that the user is knowledgeable enough to install > the dependencies by herself. The import statement will be self explanatory > in this case. There have been talks of linking pip with Idle, which might do what you want. I don't know how that has progressed, but it'd be something to look into. ChrisA From tjreedy at udel.edu Tue Sep 20 17:03:27 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Sep 2016 17:03:27 -0400 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On 9/20/2016 11:51 AM, Guido van Rossum wrote: > > We seem to have fundamentally different ideas of what sort of code is > most readable. The alternative to partial is usually a lambda, Partial and lambda are different in that partial captures variable values immediately, whereas lambda does not, unless the default argument trick, with its attendant disadvantages, is used. I consider this one reason to chose one or the other. (The greater flexibility of lambda, pointed out by David Mertz, is another.) # Demonstration from functools import partial def f(a): print(a) # lambda fl = [] for i in range(3): fl.append(lambda: f(i)) fl[0]() # 2 - bug flc = [lambda: f(i) for i in range(3)] flc[0]() # 2 - bug flcd = [lambda i=i: f(i) for i in range(3)] flcd[0]() # 0 - correct # partial fp = [] for i in range(3): fp.append(partial(f, i)) fp[0]() # 0 - correct fpc = [partial(f, i) for i in range(3)] fpc[0]() # 0 - correct > and the > signature of the lambda helps me understand how it is being called. The 'i=i' signature is typically left out by beginners creating multiple functions in a loop, leading to one of *the* most frequent of FAQs. https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result When added 'i=i', better written '_i=i', is misleading, as '_i' is not really a parameter and 'i' is not a replaceable default value, but is supposed to be a fixed value, as with partial. I believe in a previous thread about creating functions in a loop, it was generally agreed that using partial is better. (This alternative has not made it to the FAQ yet, but I thing it should.) I otherwise generally agree with what you wrote. -- Terry Jan Reedy From elazarg at gmail.com Tue Sep 20 17:26:35 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 20 Sep 2016 21:26:35 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On Wed, Sep 21, 2016 at 12:04 AM Terry Reedy wrote: > On 9/20/2016 11:51 AM, Guido van Rossum wrote: > > ... (The greater flexibility of lambda, pointed out by David Mertz, is > another.) > > I just wanted to point out that the greater flexibility of lambda is a very good reason *not* to choose it: more flexibility implies being harder to reason about. I recommend reading this article about "Principle of Least Power" http://www.lihaoyi.com/post/StrategicScalaStylePrincipleofLeastPower.html Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Sep 20 19:14:03 2016 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Sep 2016 16:14:03 -0700 (PDT) Subject: [Python-ideas] Overloading operators for testing In-Reply-To: References: Message-ID: <6192ba93-30b8-44f3-8834-2146c7f662af@googlegroups.com> As Ryan points out, pytest does this right. The way I understand it, pytest is actively maintained and nose isn't. You should switch to pytest as soon as possible. Best, Neil On Saturday, September 17, 2016 at 8:55:43 PM UTC-4, Arek Bulski wrote: > > I am using declarative testing a lot and I found out why unit tests are so > clunky. The reason why assertEquals(a,b) is used is because if we put > `assert a==b` then nose can catch the AssertionError but wont find out what > was returned or expected. This could be easily overcome if we allow > oveloading == operator from outside. Right now == would have to be changed > for every lefhand object that is compared in the tests, builtin types > including. We could use a way to change it from above, so to speak. > Consider this: > > def __glob_eq__(a,b): > if not a == b: > raise FoundInequalityError(a,b) > return True > > assert obj1 == obj2 #<-- using eq above > > Nose could easily catch FoundInequalityError and print whatever > assertEquals would. This goes very handy when you consider declarative unit > testing that I use in my project. I have a unitest.TestCase derivative and > the actual testcase has a method that yields individual comparisons, like > this: > > class TestBinary(declarativeunittest.TestCase): > def alltestsinteractive(self): > > yield [func(1) == 2] > shuffle(alist) > yield [sorted(alist) == [1,2,3]] > > Notice that this allows to put imperative statements in between > declarative cases. So shuffled() is no longer necessary in this code. :) > > pozdrawiam, > Arkadiusz Bulski > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Sep 20 19:16:18 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Sep 2016 19:16:18 -0400 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160920133118.GM22471@ando.pearwood.info> References: <20160919023201.GB22471@ando.pearwood.info> <20160920133118.GM22471@ando.pearwood.info> Message-ID: On 9/20/2016 9:31 AM, Steven D'Aprano wrote: > On Mon, Sep 19, 2016 at 06:18:38AM -0400, Terry Reedy wrote: >> On 9/19/2016 4:31 AM, Paul Moore wrote: >> >>> shutil.get_terminal_size() is available on all platforms. >> >> On windows, it works with Command Prompt and PowerShell but fails in >> IDLE, as it must. > > Why "must" it fail? Good question. When I wrote that, I had in mind the qualification 'in IDLE's default mode, as IDLE is currently structured'. In the default mode with user code executed in a separate no-window process, there is currently no way for the child process to know the current size of Shell's tk text window in the parent process. But this is not the real story. See below. > What is it about the IDLE terminal that prevents it from emulating an > actual terminal, like all(?) the other terminal emulation programs > people use? IDLE's Shell is based on a tk Text and by design is not a terminal emulation. (Which terminal would it emulate?). Note that with proportional fonts, a text editor does not really have character columns, though it can be sized according to some 'average' character width. Also, IDLE is a development environment, not a run environment, and this impacts a few decisions. > I just opened my Konqueror file & web browser, gave the command "Show > Terminal Emulator", and ran Python: > > py> import shutil > py> shutil.get_terminal_size() > os.terminal_size(columns=85, lines=10) > > That's spot on perfectly correct. I then resized the window and tried it > again: > > py> shutil.get_terminal_size() > os.terminal_size(columns=108, lines=13) This is what I did and equivalently got with Command Prompt. > I then quit Python, and ran idle3.3 from the Konqueror terminal. > Unfortunately, get_terminal_size() returned the size of the *Konqueror* > terminal, instead of the IDLE terminal. I think that's a bug. I don't > know if its a bug in IDLE or get_terminal_size() or both. If I run IDLE from Command Prompt, without or with the currently deprecated -n option, to run user code in the original IDLE process instead of a separate subprocess, I get the same -- the current size of the parent Command Prompt. The reason is that shutil.get_terminal_size first looks for int(os.environ['COLUMNS']) (or 'LINES') and if that fails, calls os.get_terminal_size(sys.__stdout__.fileno()). In both cases, the latter refer to the parent console. The 'obvious' possible solution is for Shell to set the environment variables when created or modified. On creation would be easy. Unfortunately, there is not, AFAIK, an application-accessible Resize event. A feature of tk is that geometry managers handle resizing automatically once the user sets the parameters of which widgets can be resized, and if so, min sizes and relative weights. But maybe there is a workaround. I am also not sure if modified environ is propagated to subprocesses. On Posix, subprocess.Popen uses os.execvp, so the parent environment is inherited by the new process. I don't know if this means that subsequent changes are inherited. On Windows, Popen used the system CreateProcess and I don't know what this does. A workaround would be to monkeypatch the function. In the rarely used -n mode, this would be easy. In normal mode, this would require the execution server to send an rpc message to the Shell client asking for its dimensions so the server can format the string to send to the client. I do not know if the current rpc mechanism is set up for requests from server to client. If not fixed soon, I should add the limitation to the IDLE doc. -- Terry Jan Reedy From steve at pearwood.info Tue Sep 20 10:12:23 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 21 Sep 2016 00:12:23 +1000 Subject: [Python-ideas] Overloading operators for testing In-Reply-To: References: Message-ID: <20160920141223.GN22471@ando.pearwood.info> On Sun, Sep 18, 2016 at 02:52:31AM +0200, Arek Bulski wrote: > I am using declarative testing a lot and I found out why unit tests are so > clunky. I don't think unit tests are clunky. > The reason why assertEquals(a,b) is used is because if we put > `assert a==b` then nose can catch the AssertionError but wont find out what > was returned or expected. That's not the only reason. assert* methods can take an arbitrary number of arguments. == cannot. It is easy to extend the collection of assert* methods, to use inheritance to modify them, and so on. Not so easy if you have only a single, global function or operator. assert* methods can run even when assertions are disabled. > This could be easily overcome if we allow > oveloading == operator from outside. Right now == would have to be changed > for every lefhand object that is compared in the tests, builtin types > including. We could use a way to change it from above, so to speak. Having global state that controls the behaviour of == is not a good idea. Look at your proposed code: > Consider this: > > def __glob_eq__(a,b): > if not a == b: > raise FoundInequalityError(a,b) > return True > > assert obj1 == obj2 #<-- using eq above That can't work as you have written it, because it will recurse forever. obj1 == obj2 will call the global eq, which calls == which calls the global eq, which calls == which calls the global eq. So let's re-write it: def __glob_eq__(a,b): saved_state = get_global_eq() delete_global_eq() try: if not a == b: # calls the real == raise FoundInequalityError(a,b) return True finally: set_global_eq(saved_state) Okay, that's pretty yucky, and its not thread-safe, but at least its only in one place, right? Except it isn't. Every single call to == (and != ) will have to be re-written to do the same thing. And of course, that just replaces assertEquals. What about all the other assert* methods? But there's another, more fundamental problem with using assert in this way. You cannot run your unit tests with assertions turned off. That is why I think the nose style of using "assert a == b" for unit testing is seriously broken. That is an abuse of the assert statement. It's good enough for a handful of quick and dirty tests, but is not good enough for a full-sized unit test suite. -- Steve From dan at tombstonezero.net Tue Sep 20 22:31:18 2016 From: dan at tombstonezero.net (Dan Sommers) Date: Wed, 21 Sep 2016 02:31:18 +0000 (UTC) Subject: [Python-ideas] Make partial a built-in References: Message-ID: On Tue, 20 Sep 2016 15:29:36 +0000, ????? wrote: > The alternative to partial is writing a closure in the form of a > function, that needs to be carefully inspected to verify that it is > indeed just a partial application and not something more complex. It > has more opportunity for introducing an error. And it's longer and > adds distance between related parts of the code. While I'm not usually one to promote object oriented programming, another Python alternative is a class with a __call__ method; e.g.: class Adder: def __init__(self, addend_one): self.addend_one = addend_one def __call__(self, addend_two): return self.addend_one + addend_two add_5 = Adder(5) print(add_5(4)) I like the way the whole thing is bundled up into a class. Yes, it's more verbose than a lambda expression or a partial function application, but I find it very readable and its intent is usually pretty obvious. Instances are closures in disguise (and they're all just different ways of hiding state of one kind or another). From elazarg at gmail.com Tue Sep 20 22:37:55 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Wed, 21 Sep 2016 02:37:55 +0000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: This class should be put somewhere. If you need it from inside a method, you have to choose where to put it. If it is inside the method, it forces the reader to scan it thoroughly to verify there's no "real" closure there, and it also makes your methods significantly longer. If it is outside the whole class, it might be very far away from your logic, doing a very simple thing that is not needed anywhere else. Elazar On Wed, Sep 21, 2016 at 5:32 AM Dan Sommers wrote: > On Tue, 20 Sep 2016 15:29:36 +0000, ????? wrote: > > > The alternative to partial is writing a closure in the form of a > > function, that needs to be carefully inspected to verify that it is > > indeed just a partial application and not something more complex. It > > has more opportunity for introducing an error. And it's longer and > > adds distance between related parts of the code. > > While I'm not usually one to promote object oriented programming, > another Python alternative is a class with a __call__ method; e.g.: > > class Adder: > def __init__(self, addend_one): > self.addend_one = addend_one > def __call__(self, addend_two): > return self.addend_one + addend_two > > add_5 = Adder(5) > print(add_5(4)) > > I like the way the whole thing is bundled up into a class. Yes, it's > more verbose than a lambda expression or a partial function application, > but I find it very readable and its intent is usually pretty obvious. > > Instances are closures in disguise (and they're all just different ways > of hiding state of one kind or another). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Sep 20 22:53:12 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 21 Sep 2016 12:53:12 +1000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: <20160921025311.GU22471@ando.pearwood.info> Folks, There are pros and cons of partial over lambda over classes, and which you prefer may be at least in part a matter of subjective taste. But Guido has spoken that he is virulently against making partial a builtin. It really isn't hard to put "from functools import partial" at the top of your modules. Can we let this thread die a natural death now please? -- Steve From breamoreboy at yahoo.co.uk Wed Sep 21 00:40:40 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 21 Sep 2016 05:40:40 +0100 Subject: [Python-ideas] Overloading operators for testing In-Reply-To: <6192ba93-30b8-44f3-8834-2146c7f662af@googlegroups.com> References: <6192ba93-30b8-44f3-8834-2146c7f662af@googlegroups.com> Message-ID: On 21/09/2016 00:14, Neil Girdhar wrote: > As Ryan points out, pytest does this right. The way I understand it, > pytest is actively maintained and nose isn't. You should switch to > pytest as soon as possible. > > Best, > > Neil Nose is no longer maintained but long live nose2 https://pypi.python.org/pypi/nose2/0.6.5 https://github.com/nose-devs/nose2 -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From greg.ewing at canterbury.ac.nz Wed Sep 21 01:44:36 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Sep 2016 17:44:36 +1200 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: <57E21E44.4010202@canterbury.ac.nz> ????? wrote: > @partial(partial, partial(partial, partial)) > def add(a, b, c): return a + b + c For large numbers of arguments, it's much clearer if you write it this way: >>> from functools import partial as badger, partial as mushroom >>> @badger(badger, badger(badger, badger(badger, mushroom))) ... def add(a,b,c,d): ... return a+b+c+d ... >>> add(1)(2)(3)(4) 10 -- Greg From greg.ewing at canterbury.ac.nz Wed Sep 21 01:56:07 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Sep 2016 17:56:07 +1200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <20160919023201.GB22471@ando.pearwood.info> <20160920133118.GM22471@ando.pearwood.info> Message-ID: <57E220F7.8010803@canterbury.ac.nz> Terry Reedy wrote: > In the default > mode with user code executed in a separate no-window process, there is > currently no way for the child process to know the current size of > Shell's tk text window in the parent process. On unix it should be possible to let the child know if it's connected through a pty rather than a pipe. I don't know whether Windows has any equivalent notion, though. -- Greg From ncoghlan at gmail.com Wed Sep 21 11:46:01 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Sep 2016 01:46:01 +1000 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: <57E01474.8060309@stoneleaf.us> <4dedc126-65e8-4d5a-e2f4-7ea7029a6187@mail.de> <22496.29384.554391.213319@turnbull.sk.tsukuba.ac.jp> <22497.33917.616919.671799@turnbull.sk.tsukuba.ac.jp> Message-ID: On 21 September 2016 at 06:59, Chris Angelico wrote: > On Wed, Sep 21, 2016 at 4:58 AM, ????? wrote: >> I think that combining user convenience and security considerations, there >> should be some way to invoke a GUI version of pip with flashing screen >> asking for permissions to install the library. In situations where >> interaction with the user is not trivial (i.e. when you don't have GUI >> accessible) we can assume that the user is knowledgeable enough to install >> the dependencies by herself. The import statement will be self explanatory >> in this case. > > There have been talks of linking pip with Idle, which might do what > you want. I don't know how that has progressed, but it'd be something > to look into. The issue for that is here: http://bugs.python.org/issue27051 It turns out the fact IDLE is sometimes used to teach *complete* novices creates a problem for that concept, as learning to safely navigate the free-for-all that is PyPI is actually a pretty advanced development skill vs using a more curated collection like a Linux distro or conda (where there are entities behind them that promise that the code you download will be both non-hostile and at least arguably useful, as opposed to PyPI where the only promise we make is "the code you download will be the code the publisher uploaded" without any attestation, good or otherwise, regarding the trustworthiness of the publisher). A whitelist where we pre-approve a bunch of known-safe components (perhaps even populated automatically from the conda ecosystem) might resolve that, but we really do need some form of curation if we're proposing to offer this as a default capability to learners that aren't even familar with their system command line yet. There's also an open issue at https://github.com/pypa/python-packaging-user-guide/issues/267 regarding improving the documentation on packaging.python.org that points folks towards the various bundling utilities that can given them installers and self-contained scripts for execution on end user systems. In that domain, a potentially useful addition would be a clear recipe for how to combine a script and a PyPI requirements file, into a zipapp archive that bundles all those dependencies along with the script as __main__.py. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Sep 21 12:15:08 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Sep 2016 02:15:08 +1000 Subject: [Python-ideas] Make partial a built-in In-Reply-To: References: Message-ID: On 21 September 2016 at 02:42, Ryan Gonzalez wrote: > Most often, when I see lambdas used for this, it looks like: > > lambda *args, **kw: myfunc(partial_arg, *args, **kw) Wrapper functions like that are almost always more readable when written as named functions: def my_modified_func(*args, **kwds): return myfunc(partial_arg, *args, **kwds) > which isn't more readable than just: > > partial(myfunc, partial_func) It's significantly less clear that it's defining a new callable though, especially for folks that have never heard the phrase "higher order function". It sometimes the right answer in particular contexts (hence the availability of functools.partial), but it does immediately raise an additional barrier to entry for future mainteners of that code (not a particularly *high* barrier as these things go, but a barrier nonetheless). > Doing something like: > > lambda x, y: myfunc(partial_arg, x, y) > > is more error-prone to changes in myfunc's signature. This form of lambda usage is more commonly seen when the lambda is being used to adapt to a particular callback protocol that passes a certain number of positional arguments to the callbacks (think things like "key" arguments to sorting functions, error handlers, event handlers, etc) In those cases, having the lambda definition right there may help the *reader* remember (or learn!) the callback signature. Consider: mylist = sorted(original, key=(lambda item: measure(item, setting=2)) Even if this is the first time you've seen sorted(), and you've never seen measure(), you know immediately the key function takes one parameter, and can probably make a decent guess as to how to change the behaviour of that sorting operation. By contrast: mylist = sorted(original, key=partial(measure, setting=2)) doesn't give you any hint about the signature of the key parameter unless you already know the signature of measure(). While many folks will already recognise the sorted/key combination specifically, the same can't be said for callback parameters in general. Making as few prior assumptions as we can about the reader's prior knowledge without making the code overly verbose is one of the core aspects of enabling correct local reasoning about code, and enabling folks to edit code correctly *without* global awareness of the code base is one of the core skills in learning to write maintainable code :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Wed Sep 21 16:39:33 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Sep 2016 13:39:33 -0700 Subject: [Python-ideas] from __pip__ import In-Reply-To: References: Message-ID: On Mon, Sep 19, 2016 at 9:25 AM, ????? wrote: > Many proposals to add something to stdlib are rejected here with the > suggestion to add such library to pypi first. As noted by someone, pypi is > not as reachable as stdlib, and one should install that package first, > which many people don't know how. Additionally, there is no natural > distinction between 3rd party dependencies and in-project imports (at least > in tiny projects). > > This can be made easier if the first line of the program will declare the > required library, and executing it will try to download and install that > library if it is not installed yet. > yes, these are issues, but having a module or script download an install something jsut to run is a really bad idea! I handle it with something like: try: import some_package except ImportError: print("""This package requires "some_package": if you do not have it installed, it can be installed with pip: $ python -m pip install my_package YOu can find more information aobut mypacakge at: http://github.com/my_package """) raise Where I've done this has been with ugly optional dependencies -- I don't want to force the user to install the dep simply to run the code, if they are not using features that require it, so I dont put it in teh requiremetns in setup.py (Or conda, or...), But it's nice to get a reasonable message if the user tries to use a feature that does require that dependency. note also that we don't want to force people to use pip to install packages -- they may use conda, or the system package manager, or want to install from source, or.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 22 13:19:12 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 17:19:12 +0000 Subject: [Python-ideas] Delay evaluation of annotations Message-ID: Hi all, Annotations of function parameters and variables are evaluated when encountered. This makes it necessary to use string representation for names that are not yet bound, which affects almost every class definition. It is also easy to forget, and the result might be a (very uninteresting) exception in certain untested paths, e.g. inside functions. Editors and IDEs also don't handle it well; for example, PyDev does not consider string-annotations as an occurrence of the name, and warns about unused imports. I propose delaying evaluation of annotation-expressions by either keeping the AST of the annotation, or turning it implicitly from EXP into "lambda: EXP". Inspection code that is interested in this information can access it be calling (or evaluating) it. It certainly isn't a backward compatible change, but I think it won't affect too much code. On the positive side, it will make annotated code much more pleasing to read, will be less surprising for beginners, and will help editors in syntax highlighting and name lookup. In short, it will shift the burden of handling type annotations from standard code to inspection code, which is where I believe it should rest. While this idea is quite obvious, I've found no clue in PEP-3107 as to why will it be a bad one (no such "rejected proposal" there), so I raise the question here. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Sep 22 13:50:30 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 22 Sep 2016 13:50:30 -0400 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: Message-ID: On Thu, Sep 22, 2016 at 1:19 PM, ????? wrote: > I propose delaying evaluation of annotation-expressions by either keeping > the AST of the annotation, or turning it implicitly from EXP into "lambda: > EXP". Inspection code that is interested in this information can access it > be calling (or evaluating) it. > > It certainly isn't a backward compatible change > It can be made almost backward compatible by making __annotations__ a property that calls your "lambda: EXP"s at the time when it is requested. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Sep 22 14:42:16 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2016 04:42:16 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: Message-ID: <20160922184215.GZ22471@ando.pearwood.info> On Thu, Sep 22, 2016 at 05:19:12PM +0000, ????? wrote: > Hi all, > > Annotations of function parameters and variables are evaluated when > encountered. Right, like all other Python expressions in general, and specifically like function parameter default arguments. > This makes it necessary to use string representation for names > that are not yet bound, which affects almost every class definition. Almost every class? Really? I find that implausible. Still, I can see it affecting *many* class definitions, so let's not quibble. > It is > also easy to forget, and the result might be a (very uninteresting) > exception in certain untested paths, e.g. inside functions. Unlikely, unless you're talking about functions nested inside other functions, or unusual (but legal and sometimes useful) conditional definitions: if condition: # forward reference to MyClass def f(arg:'MyClass'): ... else: # oops, untested path def f(arg:MyClass): ... class MyClass: ... But generally speaking, that sort of code is unusual, and besides, if you're doing this, either the static type checker won't be able to cope with it at all (in which case there's little point in annotating the function), or it will cope, and detect the invalid annotation. > Editors and > IDEs also don't handle it well; for example, PyDev does not consider > string-annotations as an occurrence of the name, and warns about unused > imports. I would call that a bug in PyDev. > I propose delaying evaluation of annotation-expressions by either keeping > the AST of the annotation, or turning it implicitly from EXP into "lambda: > EXP". Inspection code that is interested in this information can access it > be calling (or evaluating) it. -1 on complicating the simple Python model that expressions are evaluated when they are reached. You would also complicate the introspection of annotations. With your proposal, *every* annotation would be a function, and every(?) inspection would require calling the function to find out what the real annotation is. And what would that do to modules which use annotations for some other purpose? I know Guido is keen to discourage such alternative uses, but they're still legal, and a change like this would outright break them. Personally, I'm not convinced that it is a burden to expect people to remember to quote forward references. If they forget, they will nearly always get a NameError at runtime or a warning/error when they run the type checker. > It certainly isn't a backward compatible change, but I think it won't > affect too much code. On the positive side, it will make annotated code > much more pleasing to read, A bit more pleasant. It's not unpleasant to read 'MyClass' instead of MyClass. > will be less surprising for beginners, Only because said beginners aren't familar enough to be surprised by how surprising this is. > and will > help editors in syntax highlighting and name lookup. But will harm runtime introspection. -- Steve From elazarg at gmail.com Thu Sep 22 15:21:18 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 19:21:18 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <20160922184215.GZ22471@ando.pearwood.info> References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano wrote: > On Thu, Sep 22, 2016 at 05:19:12PM +0000, ????? wrote: > > Hi all, > > > > Annotations of function parameters and variables are evaluated when > > encountered. > > Right, like all other Python expressions in general, and specifically > like function parameter default arguments. > Just because you call it "expression", when for most purposes it isn't - it is an annotation. "Expression" is something that you need its value right now, and "annotation" is something that, well, annotates the code you see right now. Terminology is not the important thing, but that seems to be the basis your argument here. > This makes it necessary to use string representation for names > > that are not yet bound, which affects almost every class definition. > > Almost every class? Really? I find that implausible. Still, I can see it > affecting *many* class definitions, so let's not quibble. > > I retract this as a factual claim, I think we agree here. > It is > > also easy to forget, and the result might be a (very uninteresting) > > exception in certain untested paths, e.g. inside functions. > > Unlikely, unless you're talking about functions nested inside other > functions, or unusual (but legal and sometimes useful) conditional > definitions: > I was thinking about the former, but yeah, uncovered code will fail at runtime, possibly in production, for *no* real reason. I do not claim that this is common, but it is definitely unnecessary - unlike initialization expressions. (which I would have liked to see delayed too, but I can understand the reasons why this is strongly opposed; it *is* an expression. Sadly bugs are *much* more common there). > if condition: > # forward reference to MyClass > def f(arg:'MyClass'): ... > else: > # oops, untested path > def f(arg:MyClass): ... > > class MyClass: ... > > > But generally speaking, that sort of code is unusual, and besides, if > you're doing this, either the static type checker won't be able to cope > with it at all (in which case there's little point in annotating the > function), or it will cope, and detect the invalid annotation. > > Why would it detect invalid annotation here? It shouldn't. And of course the fact that I use annotated code does not necessarily mean I also use type checkers. > > Editors and > > IDEs also don't handle it well; for example, PyDev does not consider > > string-annotations as an occurrence of the name, and warns about unused > > imports. > > I would call that a bug in PyDev. > > Agreed - given the current situation, which calls for bugs in IDEs and requires unnecessary effort from any person or tool parsing the code. > > I propose delaying evaluation of annotation-expressions by either keeping > > the AST of the annotation, or turning it implicitly from EXP into > "lambda: > > EXP". Inspection code that is interested in this information can access > it > > be calling (or evaluating) it. > > -1 on complicating the simple Python model that expressions are > evaluated when they are reached. > > Again, should not be called "expressions" but "annotations", and every reader that understand types will not expect it to be evaluated. So the current "simple" model is actually surprising. It's only easier on the implementation side. You would also complicate the introspection of annotations. With > your proposal, *every* annotation would be a function, and every(?) > inspection would require calling the function to find out what the real > annotation is. > This argument was partially answered by Alexander before. Generally, introspection *libraries* will be tiny bit more complicated. Introspection user code will not. And people that write introspection code *must* understand the nitty-gritty details of the language, whereas people that read and write regular code need not. > And what would that do to modules which use annotations for some other > purpose? I know Guido is keen to discourage such alternative uses, but > they're still legal, and a change like this would outright break them. > I don't know any concrete examples. Can you give any? Are these examples use side effects on annotation evaluation? Lastly, do *you* consider it a good idea, one that should be accounted for? > Personally, I'm not convinced that it is a burden to expect people to > remember to quote forward references. If they forget, they will nearly > always get a NameError at runtime or a warning/error when they run the > type checker. > It's just another irritating inconvenience making the write-test cycle longer for no obvious reason (at least from the perspective of the user). > It certainly isn't a backward compatible change, but I think it won't > > affect too much code. On the positive side, it will make annotated code > > much more pleasing to read, > > A bit more pleasant. It's not unpleasant to read 'MyClass' instead of > MyClass. > It is to me, but that's only personal taste. > > will be less surprising for beginners, > > Only because said beginners aren't familar enough to be surprised by how > surprising this is. > I don't understand this answer at all. I am pretty familiar with Python - not as most of the people on this list, but possibly not less than anyone I know in person (sadly so). And this behavior still surprises me. It definitely surprises people coming from a statically-typed background. > > and will > > help editors in syntax highlighting and name lookup. > > But will harm runtime introspection. > Very little. And to quote Frank Miller, ?An old man dies, a little girl lives. Fair trade.? Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Sep 22 15:02:14 2016 From: mertz at gnosis.cx (David Mertz) Date: Thu, 22 Sep 2016 12:02:14 -0700 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <20160922184215.GZ22471@ando.pearwood.info> References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 11:42 AM, Steven D'Aprano wrote: > > This makes it necessary to use string representation for names > > that are not yet bound, which affects almost every class definition. > > Almost every class? Really? I find that implausible. Still, I can see it > affecting *many* class definitions, so let's not quibble. > I would say this affects a "rare class here and there." Almost all typing will be with things defined in the `typing` module (or built-ins). I guess once in a while we'll see e.g. `Sequence[CustomThing]`, but it will be uncommon for that typing involving `CutomThing` to be within CustomThing itself (well, unless you use much more recursion than Python encourages). > -1 on complicating the simple Python model that expressions are > evaluated when they are reached. > I think there is a decent argument for a more general concept of macros, or symbols, or simpler delayed evaluation than lambda for Python in general. I see places where this would be very nice for Pandas, for example, and for Dask (I work with the developers of both of those projects). In such a hypothetical future world we might come to allow, e.g. `Sequence[#CustomThing]` where some general lazy facility or indirection is indicated by the '#' (just a placeholder for this comment, not a proposal). But if that comes about, it should be available everywhere, not only in annotations. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 22 15:35:34 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 19:35:34 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 10:29 PM David Mertz wrote: > On Thu, Sep 22, 2016 at 11:42 AM, Steven D'Aprano > wrote: > >> > This makes it necessary to use string representation for names >> > that are not yet bound, which affects almost every class definition. >> >> Almost every class? Really? I find that implausible. Still, I can see it >> affecting *many* class definitions, so let's not quibble. >> > > I would say this affects a "rare class here and there." Almost all typing > will be with things defined in the `typing` module (or built-ins). I guess > once in a while we'll see e.g. `Sequence[CustomThing]`, but it will be > uncommon for that typing involving `CutomThing` to be within CustomThing > itself (well, unless you use much more recursion than Python encourages). > I think we're talking about different things here. I just referred to the common need to use the name of the current class in type annotation class A: def add(self, other: A) -> A: ... > > >> -1 on complicating the simple Python model that expressions are >> evaluated when they are reached. >> > > I think there is a decent argument for a more general concept of macros, > or symbols, or simpler delayed evaluation than lambda for Python in > general. I see places where this would be very nice for Pandas, for > example, and for Dask (I work with the developers of both of those > projects). > > In such a hypothetical future world we might come to allow, e.g. > `Sequence[#CustomThing]` where some general lazy facility or indirection is > indicated by the '#' (just a placeholder for this comment, not a > proposal). But if that comes about, it should be available everywhere, not > only in annotations. > I generally agree, but this future world must be very far and has many consequences, whereas the story of annotations is special in that it's not actually an expression, to the reader. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Sep 22 15:45:48 2016 From: mertz at gnosis.cx (David Mertz) Date: Thu, 22 Sep 2016 12:45:48 -0700 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 12:35 PM, ????? wrote: > In such a hypothetical future world we might come to allow, e.g. >> `Sequence[#CustomThing]` where some general lazy facility or indirection is >> indicated by the '#' (just a placeholder for this comment, not a >> proposal). But if that comes about, it should be available everywhere, not >> only in annotations. >> > > I generally agree, but this future world must be very far and has many > consequences, whereas the story of annotations is special in that it's not > actually an expression, to the reader. > The CPython developers (of whom I'm not one, but I've followed them closely for 18 years) place a high value on simplicity in the parser and interpreter. Adding a new custom type of thing that is an "annotation object" would be a special case with a high burden to show its utility. My feeling is that this burden is actually lower for a new "delayed eval object" that might conceivably be added at a syntax level. In some sense, this would add just as much complexity as a new annotation object, but it would be something that applies many places and hence perhaps be worth the added complexity. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Sep 22 15:58:50 2016 From: mertz at gnosis.cx (David Mertz) Date: Thu, 22 Sep 2016 12:58:50 -0700 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 12:35 PM, ????? wrote: > I think we're talking about different things here. I just referred to the > common need to use the name of the current class in type annotation > > class A: > def add(self, other: A) -> A: ... > Yeah, I find the need for using the string "A" here a wart. Rather than change the entire semantics of annotations, it feels like a placeholder for this meaning would be better. E.g.: class A: def __add__(self, other: CLS) -> CLS: ... A static checker could do the magic of recognizing that special name easily enough (no harder than recognizing the quoted string). At runtime 'CLS' could either just be a singleton with no other behavior... or perhaps it could be some sort of magic introspection object. It's more verbose, but you can also spell it now as: class A: def __add__(self, other: type(self)) -> type(self): ... That's a little ugly, but it expresses the semantics we want. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 22 15:59:05 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 19:59:05 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 10:45 PM David Mertz wrote: > On Thu, Sep 22, 2016 at 12:35 PM, ????? wrote: > >> In such a hypothetical future world we might come to allow, e.g. >>> `Sequence[#CustomThing]` where some general lazy facility or indirection is >>> indicated by the '#' (just a placeholder for this comment, not a >>> proposal). But if that comes about, it should be available everywhere, not >>> only in annotations. >>> >> >> I generally agree, but this future world must be very far and has many >> consequences, whereas the story of annotations is special in that it's not >> actually an expression, to the reader. >> > > The CPython developers (of whom I'm not one, but I've followed them > closely for 18 years) place a high value on simplicity in the parser and > interpreter. Adding a new custom type of thing that is an "annotation > object" would be a special case with a high burden to show its utility. > > My feeling is that this burden is actually lower for a new "delayed eval > object" that might conceivably be added at a syntax level. In some sense, > this would add just as much complexity as a new annotation object, but it > would be something that applies many places and hence perhaps be worth the > added complexity. > > I don't If this feature is "nice, but does not worth the complication", then so be it; I can't claim I know better. I only speculate that it does not necessarily requires a new custom type. A delayed eval object will be very useful for initilizers, for the very reason that the current behavior is surprising. -- This made me think about Steven's argument above: it is not true that expressions are evaluated when they are encountered, since x = lambda: print(1) prints nothing. So a colon before an expression hints about delayed evaluation. This includes annotations and lambda. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 22 16:02:17 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 20:02:17 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 10:58 PM David Mertz wrote: > On Thu, Sep 22, 2016 at 12:35 PM, ????? wrote: > >> I think we're talking about different things here. I just referred to the >> common need to use the name of the current class in type annotation >> >> class A: >> def add(self, other: A) -> A: ... >> > > Yeah, I find the need for using the string "A" here a wart. Rather than > change the entire semantics of annotations, it feels like a placeholder for > this meaning would be better. E.g.: > > class A: > def __add__(self, other: CLS) -> CLS: ... > > A static checker could do the magic of recognizing that special name > easily enough (no harder than recognizing the quoted string). At runtime > 'CLS' could either just be a singleton with no other behavior... or perhaps > it could be some sort of magic introspection object. It's more verbose, > but you can also spell it now as: > > class A: > def __add__(self, other: type(self)) -> type(self): ... > > That's a little ugly, but it expresses the semantics we want. > > Mypy fails on this, and version 4.4 fails very hard and ugly, with errors for each line to the end of the file + 1. It can be fixed of course, but it's worth noting: mypy does not expect arbitrary expression - because it is an annotation. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Sep 22 16:02:48 2016 From: mertz at gnosis.cx (David Mertz) Date: Thu, 22 Sep 2016 13:02:48 -0700 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 12:59 PM, ????? wrote: > I don't If this feature is "nice, but does not worth the complication", >> then so be it; I can't claim I know better. I only speculate that it does >> not necessarily requires a new custom type. >> > I don't mean a runtime type here (necessarily), but rather a new type for the parser. I.e. transform this actual expression into some sort of delayed expression when parsing. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Sep 22 16:02:57 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 22 Sep 2016 16:02:57 -0400 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 3:58 PM, David Mertz wrote: > It's more verbose, but you can also spell it now as: > > class A: > def __add__(self, other: type(self)) -> type(self): ... > No, you can't: >>> class A: ... def __add__(self, other: type(self)) -> type(self): ... ... Traceback (most recent call last): File "", line 1, in File "", line 2, in A NameError: name 'self' is not defined -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 22 16:04:15 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 20:04:15 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 11:02 PM David Mertz wrote: > On Thu, Sep 22, 2016 at 12:59 PM, ????? wrote: > >> I don't If this feature is "nice, but does not worth the complication", >>> then so be it; I can't claim I know better. I only speculate that it does >>> not necessarily requires a new custom type. >>> >> > I don't mean a runtime type here (necessarily), but rather a new type for > the parser. I.e. transform this actual expression into some sort of > delayed expression when parsing. > > I understood this, and that was my speculation. I will like an explanation as to why is this necessary, but it's probably off topic. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Sep 22 16:09:24 2016 From: mertz at gnosis.cx (David Mertz) Date: Thu, 22 Sep 2016 13:09:24 -0700 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: Doh! Yes, of course 'self' is only a scoped name within the body of the method, not in the signature. On Thu, Sep 22, 2016 at 1:02 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Thu, Sep 22, 2016 at 3:58 PM, David Mertz wrote: > >> It's more verbose, but you can also spell it now as: >> >> class A: >> def __add__(self, other: type(self)) -> type(self): ... >> > > No, you can't: > > >>> class A: > ... def __add__(self, other: type(self)) -> type(self): ... > ... > Traceback (most recent call last): > File "", line 1, in > File "", line 2, in A > NameError: name 'self' is not defined > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 22 16:29:21 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 20:29:21 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 11:02 PM David Mertz wrote: > On Thu, Sep 22, 2016 at 12:59 PM, ????? wrote: > >> I don't If this feature is "nice, but does not worth the complication", >>> then so be it; I can't claim I know better. I only speculate that it does >>> not necessarily requires a new custom type. >>> >> > I don't mean a runtime type here (necessarily), but rather a new type for > the parser. I.e. transform this actual expression into some sort of > delayed expression when parsing. > Just as a demonstration, the parser can transform `EXP` into `lambda: EXP` - and that's it. It will not solve everything (e.g. error messages and .__annotation__ access as Alexander says), but it demonstrates the fact that the change need not be so deep at all. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Sep 22 16:43:05 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 22 Sep 2016 16:43:05 -0400 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 4:37 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > On the second thought, why can't the parser simply replace A with 'A' in > annotations that appear in the body of class A? This will only break > somewhat pathological code that defines A before it is (re)defined by the > class statement. Then the default metaclass (type) can go over the annotations and replace 'A' with A that it just computed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Sep 22 16:37:58 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 22 Sep 2016 16:37:58 -0400 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 4:29 PM, ????? wrote: > Just as a demonstration, the parser can transform `EXP` into `lambda: EXP` > - and that's it. It will not solve everything (e.g. error messages and > .__annotation__ access as Alexander says), but it demonstrates the fact > that the change need not be so deep at all. On the second thought, why can't the parser a simply replace A with 'A' in annotations that appear in the body of class A? This will only break somewhat pathological code that defines A before it is (re)defined by the class statement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Sep 22 16:48:40 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 22 Sep 2016 16:48:40 -0400 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Thu, Sep 22, 2016 at 4:43 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Thu, Sep 22, 2016 at 4:37 PM, Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> On the second thought, why can't the parser simply replace A with 'A' in >> annotations that appear in the body of class A? This will only break >> somewhat pathological code that defines A before it is (re)defined by the >> class statement. > > > Then the default metaclass (type) can go over the annotations and replace > 'A' with A that it just computed. > On the third thought, this entire feature can be implemented in the metaclass by injecting A = 'A' in the dict in __prepare__. -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Thu Sep 22 17:05:21 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 22 Sep 2016 23:05:21 +0200 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On 22 September 2016 at 22:02, ????? wrote: > On Thu, Sep 22, 2016 at 10:58 PM David Mertz wrote: > >> On Thu, Sep 22, 2016 at 12:35 PM, ????? wrote: >> >>> I think we're talking about different things here. I just referred to >>> the common need to use the name of the current class in type annotation >>> >>> class A: >>> def add(self, other: A) -> A: ... >>> >> >> Yeah, I find the need for using the string "A" here a wart. Rather than >> change the entire semantics of annotations, it feels like a placeholder for >> this meaning would be better. E.g.: >> >> class A: >> def __add__(self, other: CLS) -> CLS: ... >> >> A static checker could do the magic of recognizing that special name >> easily enough (no harder than recognizing the quoted string). At runtime >> 'CLS' could either just be a singleton with no other behavior... or perhaps >> it could be some sort of magic introspection object. It's more verbose, >> but you can also spell it now as: >> >> class A: >> def __add__(self, other: type(self)) -> type(self): ... >> >> That's a little ugly, but it expresses the semantics we want. >> > Concerning the __add__ method, I think a more typical type for it is T = TypeVar('T', bound='A') class A: def __add__(self: T, other: T) -> T: ... There is a plan to support this in one of next releases of mypy. In general I think it is a bit early to judge what would be the best solution for forward references. (there are related issues like performance drop etc). More evidence is needed to decide a way forward. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Thu Sep 22 17:10:03 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 21:10:03 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 12:05 AM Ivan Levkivskyi wrote: > On 22 September 2016 at 22:02, ????? wrote: > >> On Thu, Sep 22, 2016 at 10:58 PM David Mertz wrote: >> >>> On Thu, Sep 22, 2016 at 12:35 PM, ????? wrote: >>> >>>> I think we're talking about different things here. I just referred to >>>> the common need to use the name of the current class in type annotation >>>> >>>> class A: >>>> def add(self, other: A) -> A: ... >>>> >>> >>> Yeah, I find the need for using the string "A" here a wart. Rather than >>> change the entire semantics of annotations, it feels like a placeholder for >>> this meaning would be better. E.g.: >>> >>> class A: >>> def __add__(self, other: CLS) -> CLS: ... >>> >>> A static checker could do the magic of recognizing that special name >>> easily enough (no harder than recognizing the quoted string). At runtime >>> 'CLS' could either just be a singleton with no other behavior... or perhaps >>> it could be some sort of magic introspection object. It's more verbose, >>> but you can also spell it now as: >>> >>> class A: >>> def __add__(self, other: type(self)) -> type(self): ... >>> >>> That's a little ugly, but it expresses the semantics we want. >>> >> > Concerning the __add__ method, I think a more typical type for it is > > T = TypeVar('T', bound='A') > > class A: > def __add__(self: T, other: T) -> T: ... > > There is a plan to support this in one of next releases of mypy. > > And now the reader should understand what is TypeVar, and you still have the string there. Not that I'm saying it's a bad solution, but it fits pyi files more than average-programmer-code. > In general I think it is a bit early to judge what would be the best > solution for forward references. > (there are related issues like performance drop etc). More evidence is > needed to decide a way forward. > The problem with waiting for more evidence is that more code will break if the change require such breakage. At least side-effect in annotation expressions should be "deprecated" and not guarantee when it happen, how many times, etc. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Sep 22 17:17:27 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Sep 2016 07:17:27 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 6:48 AM, Alexander Belopolsky wrote: > On the third thought, this entire feature can be implemented in the > metaclass by injecting A = 'A' in the dict in __prepare__. That would be the easiest, and least magical, solution. It simply means that the name of the current class is available as a pseudo-reference to itself, for typing purposes only. It parallels function recursion, which is done using the function's name: # Recursion in functions def spam(): return spam() # Recursion in type annotations class Spam: def make_spam() -> Spam: return self Clean and simple. And there's less magic here than super() - a lot less. It does mean that Spam.Spam == "Spam" forever afterwards, but I doubt that's going to break anything. It'd be just like __name__, except that currently, Spam.__name__ is set afterwards, so it's not available during class definition (and the module's name will be used instead). ChrisA From elazarg at gmail.com Thu Sep 22 17:33:58 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 22 Sep 2016 21:33:58 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 12:18 AM Chris Angelico wrote: > # Recursion in functions > def spam(): > return spam() > I just note that it *is* surprising, for most users, that you can't be sure that this is a recursion, yet. So it if you want a trusted-upon recursion you should write # spam: def spam(): def spam(): return spam() return spam() Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Sep 22 17:39:43 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Sep 2016 07:39:43 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 7:33 AM, ????? wrote: > On Fri, Sep 23, 2016 at 12:18 AM Chris Angelico wrote: >> >> # Recursion in functions >> def spam(): >> return spam() > > > I just note that it *is* surprising, for most users, that you can't be sure > that this is a recursion, yet. So it if you want a trusted-upon recursion > you should write > > # spam: > def spam(): > def spam(): > return spam() > return spam() > > Only surprising for people who want it _guaranteed_. It's the exact same problem as this: def helper(x): ... def spaminate(x, y): helper(x) helper(y) How do you know that a replacement helper hasn't been injected? You don't... but you trust that people aren't normally going to do that, and if they do, they're taking responsibility (maybe they're mocking helper for testing). ChrisA From ncoghlan at gmail.com Thu Sep 22 21:17:54 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2016 11:17:54 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On 23 September 2016 at 05:58, David Mertz wrote: > On Thu, Sep 22, 2016 at 12:35 PM, ????? wrote: >> >> I think we're talking about different things here. I just referred to the >> common need to use the name of the current class in type annotation >> >> class A: >> def add(self, other: A) -> A: ... > > > Yeah, I find the need for using the string "A" here a wart. Rather than > change the entire semantics of annotations, it feels like a placeholder for > this meaning would be better. E.g.: > > class A: > def __add__(self, other: CLS) -> CLS: ... > > A static checker could do the magic of recognizing that special name easily > enough (no harder than recognizing the quoted string). At runtime 'CLS' > could either just be a singleton with no other behavior... or perhaps it > could be some sort of magic introspection object. It's more verbose, but > you can also spell it now as: > > class A: > def __add__(self, other: type(self)) -> type(self): ... > > That's a little ugly, but it expresses the semantics we want. That doesn't work, as "self" hasn't been bound yet when the annotations are evaluated, just like A hasn't been bound yet (since it doesn't exist until *after* the class body finishes executing). As others have noted, the general idea of allowing either a placeholder name or the class name to refer to a suitable type annotation is fine, though - that would be a matter of implicitly injecting that name into the class namespace after calling __prepare__, and ensuring the compiler is aware of that behaviour, just as we inject __class__ as a nonlocal reference into method bodies that reference "super" or "__class__". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From elazarg at gmail.com Thu Sep 22 22:05:28 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 23 Sep 2016 02:05:28 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 4:17 AM Nick Coghlan wrote: ... > As others have noted, the general idea of allowing either a > placeholder name or the class name to refer to a suitable type > annotation is fine, though - that would be a matter of implicitly > injecting that name into the class namespace after calling > __prepare__, and ensuring the compiler is aware of that behaviour, > just as we inject __class__ as a nonlocal reference into method bodies > that reference "super" or "__class__". > > Just to be sure I understand, will the following work? class A: def repeat(n: int) -> List[A]: pass Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Sep 22 22:35:49 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2016 12:35:49 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: Message-ID: <20160923023549.GB22471@ando.pearwood.info> On Thu, Sep 22, 2016 at 09:33:58PM +0000, ????? wrote: > On Fri, Sep 23, 2016 at 12:18 AM Chris Angelico wrote: > > > # Recursion in functions > > def spam(): > > return spam() > > > > I just note that it *is* surprising, for most users, that you can't be sure > that this is a recursion, yet. Who are these "most users" of which you speak? Fortran programmers? C programmers? *Beginner* Python programmers? You should specify who you are referring about, rather than claim "most" without evidence. Experienced Python programmers should realise that recursion in Python is implemented by name lookup, like all other function calls, so if you rebind the name "spam" to something else, the function will call something else. This is no different from any other form of function call, including calls to built-ins. If you rebind or shadow a name, you will change which object is called. That shouldn't be a surprise, whether it involves recursion or not. > So it if you want a trusted-upon recursion > you should write > > # spam: > def spam(): > def spam(): > return spam() > return spam() *shrug* But if I do that, then I make it difficult or impossible to monkey-patch spam on the fly, for instance in the interactive interpreter. I wouldn't do it in production, but for interactive exploritory work, it is astonishing how often monkey-patching comes in handy. Just yesterday I played around with some code where I monkey-patched the built-in iter() so I could get a better idea of how the code worked. The straight-forward and simple way of writing a recursive spam() function surprises beginners, but they might go years or their entire career without running into a situation where they are caught by surprise. After all, it is rare for productuon code to rename functions, and rarer still to do it to recursive functions: func = spam spam = something_else() func() # why does the recursion not work??? In production code, that sort of thing almost never happens. On the other hand, your clever trick for preventing that surprise will surprise *me* and other experienced Pythonistas who know how recursion and function calls work in Python and expect to be able to take advantage of that when and if needed. In other words, in order to protect beginners from accidents which are extremely rare, you will take away power from experienced programmers who are more likely to want to make use of that power. I don't think that's a good tradeoff. For the avoidance of doubt: we're all adults here. If you personally want to write your recursive functions the "trusted" way, go right ahead. It will make them just a little bit less useful to experts, add an insignificant amount of safety, and require a bit more work on your part. But it's your code, and I don't intend to tell you not to do this. In the meantime, I'll usually just write my recursive functions the old-fashioned normal way. -- Steve From rosuav at gmail.com Thu Sep 22 22:54:14 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Sep 2016 12:54:14 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <20160923023549.GB22471@ando.pearwood.info> References: <20160923023549.GB22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 12:35 PM, Steven D'Aprano wrote: > The straight-forward and simple way of writing a recursive spam() > function surprises beginners, but they might go years or their entire > career without running into a situation where they are caught by > surprise. After all, it is rare for productuon code to rename functions, > and rarer still to do it to recursive functions: > > func = spam > spam = something_else() > func() # why does the recursion not work??? > > In production code, that sort of thing almost never happens. There's actually one very common technique involving rebinding functions. @count_calls def mergesort(lst): mid = len(lst) // 2 if not mid: return lst return merge(mergesort(lst[..mid]), mergesort(lst[mid..])) *Obviously* this is recursive. But if you used some magic that said "call the function that's currently being called", you'd be bypassing the count_calls decoration (which would presumably work by creating a wrapper function). Yes, it may defeat some potential optimizations (eg tail recursion optimization), but it enables all this flexibility. So we _need_ to have this kind of rebind available, and not just for experts. > In the meantime, I'll usually just write my recursive functions the > old-fashioned normal way. As will I. Of course, people are welcome to work differently, just as long as I never have to write tests for their code, or refactor anything into a decorator, or anything like that. I want the POWAH!!!!! :) ChrisA From steve at pearwood.info Thu Sep 22 23:06:16 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2016 13:06:16 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: <20160923030616.GC22471@ando.pearwood.info> On Thu, Sep 22, 2016 at 07:21:18PM +0000, ????? wrote: > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano wrote: > > > On Thu, Sep 22, 2016 at 05:19:12PM +0000, ????? wrote: > > > Hi all, > > > > > > Annotations of function parameters and variables are evaluated when > > > encountered. > > > > Right, like all other Python expressions in general, and specifically > > like function parameter default arguments. > > > > Just because you call it "expression", when for most purposes it isn't - it > is an annotation. It is *both*. It's an expression, because it's not a statement or a block. You cannot write: def func(arg1: while flag: sleep(1), arg2: raise ValueError): ... because the annotation must be a legal Python expression, not a code block or a statement. It's an annotation because that's the specific *purpose* of the expression in that context. As an analogy: would you argue that it is wrong to call the for-loop iterable an expression? for in : block I trust that you understand that the loop iterable can be any expression that evaluates to an iterable. Well, annotations can be any expression that evaluates to anything at all, but for the purposes of type checking, are expected to evaluate to a string or a type object. In the case of function annotations, remember that they can be any legal Python expression. They're not even guaranteed to be type annotations. Guido has expressed a strong preference that they are only used as type annotations, but he hasn't yet banned other uses (and I hope he doesn't), so any "solution" for a type annotation problem must not break other uses. > "Expression" is something that you need its value right > now, and "annotation" is something that, well, annotates the code you see > right now. Right. In the case of Python, function annotations **do** have a runtime effect: the expressions are evaluated, and the evaluated results are assigned in function.__annotations__ and made available for runtime introspection. Don't think that function annotations are **only** for the static type checker. Python is a much richer language than that! > > > It is > > > also easy to forget, and the result might be a (very uninteresting) > > > exception in certain untested paths, e.g. inside functions. > > > > Unlikely, unless you're talking about functions nested inside other > > functions, or unusual (but legal and sometimes useful) conditional > > definitions: > > I was thinking about the former, but yeah, uncovered code will fail at > runtime, possibly in production, for *no* real reason. I do not claim that > this is common, but it is definitely unnecessary - unlike initialization > expressions. Unnecessary? class MyClass: pass def function(arg: MyCalss): ... I want to see an immediate NameError here, thank you very much, even if I'm not running a static checker. I don't want to have to manually call: function.__annotations__['arg']() to see whether or not the annotation is valid. I accept that using strings as forward annotations is not a foolproof solution either: def function(arg: 'MyCalss'): ... but let's not jump into a "fix" that actually makes things worse. > > if condition: > > # forward reference to MyClass > > def f(arg:'MyClass'): ... > > else: > > # oops, untested path > > def f(arg:MyClass): ... > > > > class MyClass: ... > > > > > > But generally speaking, that sort of code is unusual, and besides, if > > you're doing this, either the static type checker won't be able to cope > > with it at all (in which case there's little point in annotating the > > function), or it will cope, and detect the invalid annotation. > > > Why would it detect invalid annotation here? It shouldn't. MyClass doesn't exist at that point, so it is in invalid annotation. > > > help editors in syntax highlighting and name lookup. > > > > But will harm runtime introspection. > > > > Very little. And to quote Frank Miller, ?An old man dies, a little girl > lives. Fair trade.? Not to the old man, and especially not if the little girl is a psychopath who grows up to become a mass murdering totalitarian dictator. -- Steve From tjreedy at udel.edu Thu Sep 22 23:07:13 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 22 Sep 2016 23:07:13 -0400 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160923023549.GB22471@ando.pearwood.info> Message-ID: On 9/22/2016 10:54 PM, Chris Angelico wrote: > On Fri, Sep 23, 2016 at 12:35 PM, Steven D'Aprano wrote: >> The straight-forward and simple way of writing a recursive spam() >> function surprises beginners, but they might go years or their entire >> career without running into a situation where they are caught by >> surprise. After all, it is rare for productuon code to rename functions, >> and rarer still to do it to recursive functions: >> >> func = spam >> spam = something_else() >> func() # why does the recursion not work??? >> >> In production code, that sort of thing almost never happens. > > There's actually one very common technique involving rebinding functions. > > @count_calls > def mergesort(lst): > mid = len(lst) // 2 > if not mid: return lst > return merge(mergesort(lst[..mid]), mergesort(lst[mid..])) > > *Obviously* this is recursive. But if you used some magic that said > "call the function that's currently being called", you'd be bypassing > the count_calls decoration (which would presumably work by creating a > wrapper function). Yes, it may defeat some potential optimizations (eg > tail recursion optimization), but it enables all this flexibility. @memoize is another decorator that depends on recursion by name. > So we _need_ to have this kind of rebind available, and not just for experts. > >> In the meantime, I'll usually just write my recursive functions the >> old-fashioned normal way. > > As will I. Of course, people are welcome to work differently, just as > long as I never have to write tests for their code, or refactor > anything into a decorator, or anything like that. I want the > POWAH!!!!! :) > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Terry Jan Reedy From ncoghlan at gmail.com Thu Sep 22 23:08:19 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2016 13:08:19 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: On 23 September 2016 at 12:05, ????? wrote: > On Fri, Sep 23, 2016 at 4:17 AM Nick Coghlan wrote: > ... >> >> As others have noted, the general idea of allowing either a >> placeholder name or the class name to refer to a suitable type >> annotation is fine, though - that would be a matter of implicitly >> injecting that name into the class namespace after calling >> __prepare__, and ensuring the compiler is aware of that behaviour, >> just as we inject __class__ as a nonlocal reference into method bodies >> that reference "super" or "__class__". >> > Just to be sure I understand, will the following work? > > class A: > def repeat(n: int) -> List[A]: pass Right now? No - you'll get a name error on the "A", just as you would if you tried to reference it as a default argument: >>> class A: ... def side_effects_ahead(arg: print(A) = print(A)) -> print(A): pass ... Traceback (most recent call last): File "", line 1, in File "", line 2, in A NameError: name 'A' is not defined And that's the problem with using the class name in method annotations in the class body: they're evaluated eagerly, so they'd fail at runtime, even if the typecheckers were updated to understand them. Rather than switching annotations to being evaluated lazilly in the general case, one of the solutions being suggested is that *by default*, the class name could implicitly be bound in the body of the class definition to some useful placeholder, which can already be done explicitly today: >>> class A: ... A = "placeholder" ... def side_effects_ahead(arg: print(A) = print(A)) -> print(A): pass ... placeholder placeholder placeholder Since method bodies don't see class level name bindings (by design), such an approach would have the effect of "A" referring to the placeholder in the class body (including for annotations and default arguments), but to the class itself in method bodies. I don't think this is an urgent problem (since the "A"-as-a-string spelling works today without any runtime changes), but it's worth keeping an eye on as folks gain more experience with annotations and the factors affecting their readability. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 22 23:23:52 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2016 13:23:52 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <20160923030616.GC22471@ando.pearwood.info> References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> Message-ID: On 23 September 2016 at 13:06, Steven D'Aprano wrote: > On Thu, Sep 22, 2016 at 07:21:18PM +0000, ????? wrote: >> "Expression" is something that you need its value right >> now, and "annotation" is something that, well, annotates the code you see >> right now. > > Right. In the case of Python, function annotations **do** have a runtime > effect: the expressions are evaluated, and the evaluated results are > assigned in function.__annotations__ and made available for runtime > introspection. > > Don't think that function annotations are **only** for the static type > checker. Python is a much richer language than that! If folks are after a simple non-type-checking related example of annotation usage, the "begins" CLI library is a decent one: https://pypi.python.org/pypi/begins That lets you supply command line help for parameters as annotations: ============ In Python3, any function annotations for a parameter become the command line option help. For example: >>> import begin >>> @begin.start # doctest: +SKIP ... def run(name: 'What, is your name?', ... quest: 'What, is your quest?', ... colour: 'What, is your favourite colour?'): ... pass Will generate command help like: usage: holygrail_py3.py [-h] -n NAME -q QUEST -c COLOUR optional arguments: -h, --help show this help message and exit -n NAME, --name NAME What, is your name? -q QUEST, --quest QUEST What, is your quest? -c COLOUR, --colour COLOUR What, is your favourite colour? ============ It's not a substitute for something like click or argparse when it comes to more complex argument parsing, but it's a good example of the kind of simple pseudo-DSL folks have long been able to create with annotations independently of the type hinting use case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Fri Sep 23 01:50:55 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Sep 2016 17:50:55 +1200 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: <57E4C2BF.7030100@canterbury.ac.nz> ????? wrote: > it feels like a > placeholder for this meaning would be better. E.g.: > > class A: > def __add__(self, other: CLS) -> CLS: ... That's fine for a class that refers to itself, but what about classes that refer to each other? This only addresses a small part of the problem. -- Greg From ncoghlan at gmail.com Fri Sep 23 02:23:08 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2016 16:23:08 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <57E4C2BF.7030100@canterbury.ac.nz> References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> Message-ID: On 23 September 2016 at 15:50, Greg Ewing wrote: > ????? wrote: >> >> it feels like a >> placeholder for this meaning would be better. E.g.: >> >> class A: >> def __add__(self, other: CLS) -> CLS: ... > > > That's fine for a class that refers to itself, but > what about classes that refer to each other? This > only addresses a small part of the problem. Same answer as with any other circular dependency: the code smell is the circular dependency itself, not the awkwardness of the syntax for spelling it. If the string based "circular reference here!" spelling really bothers you, refactor to eliminate the circularity (e.g. by extracting a base class or an implementation independent interface definition), rather than advocating to make the spelling less obnoxious. The difference between that and the "methods referring to the class they're defined in" case is that it's likely to be pretty normal to want to do the latter, so it may prove worthwhile to provide a cleaner standard spelling for it. The counter-argument is the general circularity one above: do you *really* need instances of the particular class being defined? Or is there a more permissive interface based type signature you could specify instead? Or perhaps no type signature at all, and let ducktyping sort it out implicitly at runtime? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Fri Sep 23 01:39:28 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Sep 2016 17:39:28 +1200 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: <57E4C010.3070808@canterbury.ac.nz> David Mertz wrote: > I guess once in a while we'll see e.g. > `Sequence[CustomThing]`, but it will be uncommon for that typing > involving `CutomThing` to be within CustomThing itself I think that depends on what kind of software you're writing. Anything involving any kind of trees or graphs will have classes that refer to themselves or each other. > (well, unless you > use much more recursion than Python encourages). Recursive data structures don't necessarily imply recursive code to process them, although recursion is often the most natural way to write that code. -- Greg From elazarg at gmail.com Fri Sep 23 05:59:35 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 23 Sep 2016 09:59:35 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160923023549.GB22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 5:54 AM Chris Angelico wrote: > On Fri, Sep 23, 2016 at 12:35 PM, Steven D'Aprano > wrote: > > The straight-forward and simple way of writing a recursive spam() > > function surprises beginners, but they might go years or their entire > > career without running into a situation where they are caught by > > surprise. After all, it is rare for productuon code to rename functions, > > and rarer still to do it to recursive functions: > > > > func = spam > > spam = something_else() > > func() # why does the recursion not work??? > > > > In production code, that sort of thing almost never happens. > > There's actually one very common technique involving rebinding functions. > > @count_calls > def mergesort(lst): > mid = len(lst) // 2 > if not mid: return lst > return merge(mergesort(lst[..mid]), mergesort(lst[mid..])) > > *Obviously* this is recursive. But if you used some magic that said > "call the function that's currently being called", you'd be bypassing > the count_calls decoration (which would presumably work by creating a > wrapper function). Yes, it may defeat some potential optimizations (eg > tail recursion optimization), but it enables all this flexibility. > > So we _need_ to have this kind of rebind available, and not just for > experts. > > I think you are mixing levels of abstraction because you know how this is implemented. The user only sees "A function named mergesort decorated by count_calls". She does not see "A function named mergesort passed to a higher order function named count_calls whose result is bound into the variable mergesort". Even if the latter is exactly what happens, declaratively the former is more accurate by intention. Ideally, the calls to mergesort will rebind to this _decorated_ function. not to the mutable global variable. Again, the argument that it will be very hard to implement it in a different way, or that is will break things, is a very strong argument, and I am not confronting it. > In the meantime, I'll usually just write my recursive functions the > > old-fashioned normal way. > > As will I. Of course, people are welcome to work differently, just as > long as I never have to write tests for their code, or refactor > anything into a decorator, or anything like that. I want the > POWAH!!!!! :) > As will I, simply because the old-fashioned way is more readable. And I will sadly accept the fact that I can't be 100% sure what's function is called at runtime. But _some_ people (medium-level, Steven, whose main language is probably not Python) will not even know this is the case. Tests are important and could have reworked into the system (through inspect, or by using a special import which allow monkey patching). I can't see why the ability to test must remain in production. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Fri Sep 23 06:17:15 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 23 Sep 2016 10:17:15 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <20160923030616.GC22471@ando.pearwood.info> References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 6:06 AM Steven D'Aprano wrote: > On Thu, Sep 22, 2016 at 07:21:18PM +0000, ????? wrote: > > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano > wrote: > > > > > On Thu, Sep 22, 2016 at 05:19:12PM +0000, ????? wrote: > > > > Hi all, > > > > > > > > Annotations of function parameters and variables are evaluated when > > > > encountered. > > > > > > Right, like all other Python expressions in general, and specifically > > > like function parameter default arguments. > > > > > > > Just because you call it "expression", when for most purposes it isn't - > it > > is an annotation. > > It is *both*. It's an expression, because it's not a statement or a > block. Did you just use a false-trichotomy argument? :) > You cannot write: > > def func(arg1: while flag: sleep(1), arg2: raise ValueError): > ... because the annotation must be a legal Python expression, not a code > block or a statement. This is the situation I'm asking to change > It's an annotation because that's the > specific *purpose* of the expression in that context. > > Exactly! Ergo, this is an annotation. > As an analogy: would you argue that it is wrong to call the for-loop > iterable an expression? > > for in : > block > > I trust that you understand that the loop iterable can be any expression > that evaluates to an iterable. Well, annotations can be any expression > that evaluates to anything at all, but for the purposes of type > checking, are expected to evaluate to a string or a type object. > > for-loop iterable is an expression, evaluated at runtime, _for_ the resulting value to be used in computation. A perfectly standard expression. Nothing fancy. > In the case of function annotations, remember that they can be any > legal Python expression. They're not even guaranteed to be type > annotations. Guido has expressed a strong preference that they are only > used as type annotations, but he hasn't yet banned other uses (and I > hope he doesn't), so any "solution" for a type annotation problem must > not break other uses. > > Must *allow* other use cases. My proposal allows: just evaluate them at the time of their use, instead at definition time. > > > "Expression" is something that you need its value right > > now, and "annotation" is something that, well, annotates the code you see > > right now. > > Right. In the case of Python, function annotations **do** have a runtime > effect: the expressions are evaluated, and the evaluated results are > assigned in function.__annotations__ and made available for runtime > introspection. > > Don't think that function annotations are **only** for the static type > checker. Python is a much richer language than that! > > function.__annotations__ can have the delayed value, be it a lambda, ast or string. It can also be computed at the time of access as suggested earlier. > > > > > It is > > > > also easy to forget, and the result might be a (very uninteresting) > > > > exception in certain untested paths, e.g. inside functions. > > > > > > Unlikely, unless you're talking about functions nested inside other > > > functions, or unusual (but legal and sometimes useful) conditional > > > definitions: > > > > I was thinking about the former, but yeah, uncovered code will fail at > > runtime, possibly in production, for *no* real reason. I do not claim > that > > this is common, but it is definitely unnecessary - unlike initialization > > expressions. > > Unnecessary? > > class MyClass: > pass > > > def function(arg: MyCalss): > ... > > I want to see an immediate NameError here, thank you very much > Two things to note here: A. IDEs will point at this NameError B. Type checkers catch this NameError C. Even the compiler can be made to catch this name error, since the name MyCalss is bound to builtins where it does not exist - you see, name lookup does happen at compile time anyway. I'm not really suggesting the compiler should make it error though. D. Really, where's the error here? if no tool looks at this signature, there's nothing wrong with it - As a human I understand perfectly. If a tool will look at it, it will warn or fail, exactly as I would liked it too. function.__annotations__['arg']() > > to see whether or not the annotation is valid. > > I accept that using strings as forward annotations is not a foolproof > solution either: > > > def function(arg: 'MyCalss'): > ... > but let's not jump into a "fix" that actually makes things worse. > > > That's not a "fix". I suggest always using the last form - which is already in common use - with a nicer syntax and semantics, since there's nothing wrong about it. It is there for a very natural reason. > > > if condition: > > > # forward reference to MyClass > > > def f(arg:'MyClass'): ... > > > else: > > > # oops, untested path > > > def f(arg:MyClass): ... > > > > > > class MyClass: ... > > > > > > > > > But generally speaking, that sort of code is unusual, and besides, if > > > you're doing this, either the static type checker won't be able to cope > > > with it at all (in which case there's little point in annotating the > > > function), or it will cope, and detect the invalid annotation. > > > > > Why would it detect invalid annotation here? It shouldn't. > > MyClass doesn't exist at that point, so it is in invalid annotation. > > > MyClass does not exist. The _name_ MyClass does, definitely. So it is an invalid _expression_ but a perfectly valid _annotation_ (as a concept). > > > > help editors in syntax highlighting and name lookup. > > > > > > But will harm runtime introspection. > > > > > > > Very little. And to quote Frank Miller, ?An old man dies, a little girl > > lives. Fair trade.? > > Not to the old man, and especially not if the little girl is a > psychopath who grows up to become a mass murdering totalitarian > dictator. > > :) Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Fri Sep 23 06:31:11 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 23 Sep 2016 10:31:11 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 6:24 AM Nick Coghlan wrote: > On 23 September 2016 at 13:06, Steven D'Aprano > wrote: > > On Thu, Sep 22, 2016 at 07:21:18PM +0000, ????? wrote: > >> "Expression" is something that you need its value right > >> now, and "annotation" is something that, well, annotates the code you > see > >> right now. > > > > Right. In the case of Python, function annotations **do** have a runtime > > effect: the expressions are evaluated, and the evaluated results are > > assigned in function.__annotations__ and made available for runtime > > introspection. > > > > Don't think that function annotations are **only** for the static type > > checker. Python is a much richer language than that! > > If folks are after a simple non-type-checking related example of > annotation usage, the "begins" CLI library is a decent one: > https://pypi.python.org/pypi/begins > > That lets you supply command line help for parameters as annotations: > > ============ > In Python3, any function annotations for a parameter become the > command line option help. For example: > > >>> import begin > >>> @begin.start # doctest: +SKIP > ... def run(name: 'What, is your name?', > ... quest: 'What, is your quest?', > ... colour: 'What, is your favourite colour?'): > ... pass > > Will generate command help like: > > usage: holygrail_py3.py [-h] -n NAME -q QUEST -c COLOUR > > optional arguments: > -h, --help show this help message and exit > -n NAME, --name NAME What, is your name? > -q QUEST, --quest QUEST > What, is your quest? > -c COLOUR, --colour COLOUR > What, is your favourite colour? > ============ > > It's not a substitute for something like click or argparse when it > comes to more complex argument parsing, but it's a good example of the > kind of simple pseudo-DSL folks have long been able to create with > annotations independently of the type hinting use case. > > That's a very nice use, and I was wrong - I did know it; I've found it not long ago when I wanted to implement it myself... And guess what? It does not require eager evaluation _at all_. No decorator-helped-annotation mechanism require eager evaluation built into the language. Lazy evaluation is more general than eager, in that it can always be forced (and not the other way around). def eager_annotation(f): f.__annotations__ = {k:v() for k, v in f.__annotations__} return f Use @eager_annotation wherever you like, or collapse it into other decorators. You don't need @eager_annotation for type annotations, or any other form of annotation without runtime semantics. On the other hand - if you do want side effect in this function's annotations, well there's better be some nice big @EAGER! decorator above it. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Sep 23 08:10:12 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2016 22:10:12 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> Message-ID: <20160923121004.GD22471@ando.pearwood.info> On Fri, Sep 23, 2016 at 10:17:15AM +0000, ????? wrote: > On Fri, Sep 23, 2016 at 6:06 AM Steven D'Aprano wrote: > > On Thu, Sep 22, 2016 at 07:21:18PM +0000, ????? wrote: > > > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano wrote: > > > > On Thu, Sep 22, 2016 at 05:19:12PM +0000, ????? wrote: > > > > > Hi all, > > > > > > > > > > Annotations of function parameters and variables are evaluated when > > > > > encountered. > > > > > > > > Right, like all other Python expressions in general, and specifically > > > > like function parameter default arguments. > > > > > > > > > > Just because you call it "expression", when for most purposes it isn't - > > > it is an annotation. > > > > It is *both*. It's an expression, because it's not a statement or a > > block. > > > Did you just use a false-trichotomy argument? :) No. You are the one trying to deny that annotations are expressions -- I'm saying that they are both annotations and expressions at the same time. There's no dichotomy here, since the two are not mutually exclusive. (The word here is dichotomy, not trichotomy, since there's only two things under discussion, not three.) > > You cannot write: > > > > def func(arg1: while flag: sleep(1), arg2: raise ValueError): > > ... > > > because the annotation must be a legal Python expression, not a code > > block or a statement. > > > This is the situation I'm asking to change That's a much bigger change than what you suggested earlier, changing function annotations to lazy evaluation instead of eager. Supporting non-expressions as annotations -- what's your use-case? Under what circumstances would you want to annotate an function parameter with a code block instead of an expression? > > It's an annotation because that's the > > specific *purpose* of the expression in that context. > > Exactly! Ergo, this is an annotation. I've never denied that annotations are annotations, or that annotations are used to annotate function parameters. I'm not sure why you are giving a triumphant cry of "Exactly!" here -- it's not under dispute that annotations are annotations. And it shouldn't be under dispute that annotations are expressions. They're not code blocks. They're not statements. What else could they be apart from expressions? The PEP that introduced them describes them as expressions: Function annotations are nothing more than a way of associating arbitrary Python EXPRESSIONS with various parts of a function at compile-time. [Emphasis added.] https://www.python.org/dev/peps/pep-3107/ and they are documented as an expression: parameter ::= identifier [":" expression] Parameters may have annotations of the form ?: expression? following the parameter name. ... These annotations can be any valid Python expression https://docs.python.org/3/reference/compound_stmts.html#function-definitions I think its time to give up arguing that annotations aren't expressions. > > As an analogy: would you argue that it is wrong to call the for-loop > > iterable an expression? > > > > for in : > > block > > > > I trust that you understand that the loop iterable can be any expression > > that evaluates to an iterable. Well, annotations can be any expression > > that evaluates to anything at all, but for the purposes of type > > checking, are expected to evaluate to a string or a type object. > > > > > for-loop iterable is an expression, evaluated at runtime, _for_ the > resulting value to be used in computation. A perfectly standard expression. > Nothing fancy. Right. And so are annotations. You want to make them fancy, give them super-powers, in order to solve the forward reference problem. I don't think that the problem is serious enough to justify changing the semantics of annotation evaluation and make them non-standard, fancy, lazy-evaluated expressions. > > In the case of function annotations, remember that they can be any > > legal Python expression. They're not even guaranteed to be type > > annotations. Guido has expressed a strong preference that they are only > > used as type annotations, but he hasn't yet banned other uses (and I > > hope he doesn't), so any "solution" for a type annotation problem must > > not break other uses. > > > > > Must *allow* other use cases. My proposal allows: just evaluate them at the > time of their use, instead at definition time. I meant what I said. Changing the evaluation model for annotations is a big semantic change, a backwards-incompatible change. It's not just adding new syntax for something that was a syntax error before, it would be changing the meaning of existing Python code. The transition from 3.6 to 3.7 is not like that from 2.x to 3.0 -- backwards compatibility is a hard requirement. Code that works a certain way in 3.6 is expected to work the same way in 3.7 onwards, unless we go through a deprecation period of at least one full release, and probably with a `from __future__ import ...` directive required. There may be a little bit of wiggle-room available for small changes in behaviour, under some circumstances -- but changing the evaluation model is unlikely to be judged to be a "small" change. In any case, before such a backwards-incompatible change would be allowed, you would have to prove that it was needed. [...] > > class MyClass: > > pass > > > > def function(arg: MyCalss): > > ... > > > > I want to see an immediate NameError here, thank you very much > > Two things to note here: > A. IDEs will point at this NameError Some or them might. Not everyone uses an IDE, it is not a requirement for Python programmers. Runtime exceptions are still, and always will be, the primary way of detecting such errors. > B. Type checkers catch this NameError Likewise for type checkers. > C. Even the compiler can be made to catch this name error, since the name > MyCalss is bound to builtins where it does not exist How do you know it doesn't exist? Any module, any function, any class, any attribute access, might have added something called MyCalss to this module's namespace, or to the built-ins. It's okay for a non-compulsory type-checker, linter or editor to make common-sense assumptions about built-ins. But the compiler cannot: it has no way of knowing *for sure* whether or not MyCalss exists until runtime. It has to actually do the name lookup, and see what happens. > - you see, name lookup does happen at compile time anyway. It really doesn't. You might be confusing function-definition time (which occurs at runtime) with compile time. When the function is defined, which occurs at runtime, the name MyCalss must exist or a NameError will occur. But that's not at compile time. > D. Really, where's the error here? if no tool looks at this signature, > there's nothing wrong with it - As a human I understand perfectly. class CircuitDC: ... class CircuitAC: ... def func(arg: CircuitSC): ... Do you still understand perfectly what I mean? -- Steve From elazarg at gmail.com Fri Sep 23 09:58:44 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Fri, 23 Sep 2016 13:58:44 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <20160923121004.GD22471@ando.pearwood.info> References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 3:11 PM Steven D'Aprano wrote: > On Fri, Sep 23, 2016 at 10:17:15AM +0000, ????? wrote: > > On Fri, Sep 23, 2016 at 6:06 AM Steven D'Aprano > wrote: > > > On Thu, Sep 22, 2016 at 07:21:18PM +0000, ????? wrote: > > > > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano wrote: > > > > > On Thu, Sep 22, 2016 at 05:19:12PM +0000, ????? wrote: > > > > > > Hi all, > > > > > > > > > > > > Annotations of function parameters and variables are evaluated > when > > > > > > encountered. > > > > > > > > > > Right, like all other Python expressions in general, and > specifically > > > > > like function parameter default arguments. > > > > > > > > > > > > > Just because you call it "expression", when for most purposes it > isn't - > > > > it is an annotation. > > > > > > It is *both*. It's an expression, because it's not a statement or a > > > block. > > > > > > Did you just use a false-trichotomy argument? :) > > No. > > You are the one trying to deny that annotations are expressions -- I'm > saying that they are both annotations and expressions at the same time. > There's no dichotomy here, since the two are not mutually exclusive. > (The word here is dichotomy, not trichotomy, since there's only two > things under discussion, not three.) > > The argument "It's an expression, because it's not a statement or a block" assumes that things must an expression, a statement or a block. Hence "trichotomy". And it is false. But I think we are getting lost in the terminology. Since I propose no change in what is considered valid syntax, > > > You cannot write: > > > > > > def func(arg1: while flag: sleep(1), arg2: raise ValueError): > > > ... > > > > > because the annotation must be a legal Python expression, not a code > > > block or a statement. > > > > > > This is the situation I'm asking to change > > That's a much bigger change than what you suggested earlier, changing > function annotations to lazy evaluation instead of eager. > > Supporting non-expressions as annotations -- what's your use-case? Under > what circumstances would you want to annotate an function parameter with > a code block instead of an expression? > > It indeed came out different than I meant. I don't suggest allowing anything that is not already allowed, syntactically. I only propose giving the current syntax a slightly different meaning, in a way that I'm sure matches how Python coders already understand the code. > > It's an annotation because that's the > > > specific *purpose* of the expression in that context. > > > > Exactly! Ergo, this is an annotation. > > I've never denied that annotations are annotations, or that annotations > are used to annotate function parameters. I'm not sure why you are > giving a triumphant cry of "Exactly!" here -- it's not under dispute > that annotations are annotations. > > :( this kind of fighting over terminology takes us nowhere indeed. What other context you see where the result of an expression is not intended to be used at all? Well there's Expression statements, which are evaluated for side effect. There's docstrings, which are a kind of annotations. What else? The only other that comes to mind is reveal_type(exp)... surely I don't need evaluation there. And it shouldn't be under dispute that annotations are expressions. > They're not code blocks. They're not statements. What else could they be > apart from expressions? > > Now it is a false dichotomy as question. The answer is "annotation" as an independent concept, closely related to expressions, but not evaluated when encountered. Very similar to E in `lambda: E` except that lambda are there mainly for the resulting value (hence "expression") and annotations are there mainly for being there. In the code. > The PEP that introduced them describes them as expressions: > > Function annotations are nothing more than a way of associating > arbitrary Python EXPRESSIONS with various parts of a function at > compile-time. [Emphasis added.] > > https://www.python.org/dev/peps/pep-3107/ > > Syntactically, yes. Just like X in "a = lambda: X" is an expression, but you don't see it evaluated, do you? And this is an _actual_ expression, undeniably so, that is intended to be evaluated and used at runtime. > and they are documented as an expression: > > parameter ::= identifier [":" expression] > > Parameters may have annotations of the form ?: expression? following > the parameter name. ... These annotations can be any valid Python > expression > > > https://docs.python.org/3/reference/compound_stmts.html#function-definitions > > I think its time to give up arguing that annotations aren't expressions. > > I don't care if you call them expressions, delayed-expressions, or flying monkeys. The allowed syntax is exactly that of an expression (like inside a lambda). The time of binding of names to scope is the same (again like a lambda) but the evaluation time is unknown to the non-reflecting-developer. Decorators may promise time of evaluation, if they want to. "Unknown evaluation time" is scary. _for expressions_, which might have side effects (one of which is running time). But annotations must be pure by convention (and tools are welcome to warn about it). I admit that I propose breaking the following code: def foo(x: print("defining foo!")): pass Do you know anyone who would dream about writing such code? > > > As an analogy: would you argue that it is wrong to call the for-loop > > > iterable an expression? > > > > > > for in : > > > block > > > > > > I trust that you understand that the loop iterable can be any > expression > > > that evaluates to an iterable. Well, annotations can be any expression > > > that evaluates to anything at all, but for the purposes of type > > > checking, are expected to evaluate to a string or a type object. > > > > > > > > for-loop iterable is an expression, evaluated at runtime, _for_ the > > resulting value to be used in computation. A perfectly standard > expression. > > Nothing fancy. > > Right. And so are annotations. > > You want to make them fancy, give them super-powers, in order to solve > the forward reference problem. I don't think that the problem is serious > enough to justify changing the semantics of annotation evaluation and > make them non-standard, fancy, lazy-evaluated expressions. > > My proposal solves the forward reference problem, but I believe in it because I believe it is aligned with what the programmer see. > > > In the case of function annotations, remember that they can be any > > > legal Python expression. They're not even guaranteed to be type > > > annotations. Guido has expressed a strong preference that they are only > > > used as type annotations, but he hasn't yet banned other uses (and I > > > hope he doesn't), so any "solution" for a type annotation problem must > > > not break other uses. > > > > > > > > Must *allow* other use cases. My proposal allows: just evaluate them at > the > > time of their use, instead at definition time. > > I meant what I said. Changing the evaluation model for annotations is a > big semantic change, a backwards-incompatible change. It's not just > adding new syntax for something that was a syntax error before, it would > be changing the meaning of existing Python code. > > The transition from 3.6 to 3.7 is not like that from 2.x to 3.0 -- > backwards compatibility is a hard requirement. Code that works a certain > way in 3.6 is expected to work the same way in 3.7 onwards, unless we go > through a deprecation period of at least one full release, and probably > with a `from __future__ import ...` directive required. There may be a > little bit of wiggle-room available for small changes in behaviour, > under some circumstances -- but changing the evaluation model is > unlikely to be judged to be a "small" change. > > In any case, before such a backwards-incompatible change would be > allowed, you would have to prove that it was needed. > > I would like to see an example for a code that breaks under the Alexander's suggestion of forcing evaluation at `.__annotations__` access time. > [...] > > > class MyClass: > > > pass > > > > > > def function(arg: MyCalss): > > > ... > > > > > > I want to see an immediate NameError here, thank you very much > > > > Two things to note here: > > A. IDEs will point at this NameError > > Some or them might. Not everyone uses an IDE, it is not a requirement > for Python programmers. Runtime exceptions are still, and always will > be, the primary way of detecting such errors. How useful is the detection of this error at production? > > > B. Type checkers catch this NameError > > Likewise for type checkers. > > > > C. Even the compiler can be made to catch this name error, since the name > > MyCalss is bound to builtins where it does not exist > > How do you know it doesn't exist? Any module, any function, any class, > any attribute access, might have added something called MyCalss to this > module's namespace, or to the built-ins. > > It's okay for a non-compulsory type-checker, linter or editor to make > common-sense assumptions about built-ins. But the compiler cannot: it > has no way of knowing *for sure* whether or not MyCalss exists until > runtime. It has to actually do the name lookup, and see what happens. > > Yeah it was just a thought. I wouldn't really want the compiler to do that. > > - you see, name lookup does happen at compile time anyway. > > It really doesn't. > > You might be confusing function-definition time (which occurs at > runtime) with compile time. When the function is defined, which occurs > at runtime, the name MyCalss must exist or a NameError will occur. But > that's not at compile time. > > Can you repeat that? NameError indeed happens at runtime, but the scope in which MyCalss was looked up for is determined at compile time - as far as I know. The bytecode-based typechecker I wrote rely on this information being accessible statically in the bytecode. def foo(): locals()['MyType'] = str def bar(a : MyType): pass >>> foo() Traceback (most recent call last): File "", line 1, in File "", line 4, in foo NameError: name 'MyType' is not defined What do I miss? > D. Really, where's the error here? if no tool looks at this signature, > > there's nothing wrong with it - As a human I understand perfectly. > > class CircuitDC: > ... > > class CircuitAC: > ... > > def func(arg: CircuitSC): > ... > > > Do you still understand perfectly what I mean? > > No. def func(arg: CircuitAC): ... Do you understand what I mean? Code with small distance (hamming distance / edit distance) between related-but-different entities is prone to such errors, and NameError gives you very little assurance - if you erred this way, you get it; If you err that way, you don't. --- This way or the other, the very least that I hope, is explicitly forbidding reliance on side-effect or any other way to distinguish evaluation time of annotation expressions. Annotations must be pure, and the current promise of evaluation time should be deprecated. Additionally, before making it impossible to go back, we should make the new variable annotation syntax add its annotations to a special object __reflect__, so that __reflect__.annotations__ will allow forcing evaluation (since there is no mechanism to do this in a variable). Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Sep 23 11:58:59 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Sep 2016 01:58:59 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: On Fri, Sep 23, 2016 at 11:58 PM, ????? wrote: > What other context you see where the result of an expression is not intended > to be used at all? Well there's Expression statements, which are evaluated > for side effect. There's docstrings, which are a kind of annotations. What > else? The only other that comes to mind is reveal_type(exp)... surely I > don't need evaluation there. Function annotations ARE used. They're stored as function attributes, just as default argument values and docstrings are. (It's not the language's problem if you never use them.) >> The PEP that introduced them describes them as expressions: > > Syntactically, yes. Just like X in "a = lambda: X" is an expression, but you > don't see it evaluated, do you? And this is an _actual_ expression, > undeniably so, that is intended to be evaluated and used at runtime. And the X in "if False: X" is a statement, but you don't see it evaluated either. This is an actual expression that has to be evaluated and used just like any other does. >> I think its time to give up arguing that annotations aren't expressions. >> > > I don't care if you call them expressions, delayed-expressions, or flying > monkeys. The allowed syntax is exactly that of an expression (like inside a > lambda). The time of binding of names to scope is the same (again like a > lambda) but the evaluation time is unknown to the non-reflecting-developer. > Decorators may promise time of evaluation, if they want to. Thing is, literally every other expression in Python is evaluated at the point where it's hit. You can guard an expression with control flow statements or operators, but other than that, it will be hit when execution reaches its line: def func(x): expr # evaluated when function called if cond: expr # evaluated if cond is true [expr for x in range(n)] # evaluated if n > 0 (expr for x in [1]) # evaluated when genexp nexted expr if cond else "spam" # evaluated if cond is true lambda: expr # evaluated when function called def func(x=expr): pass # evaluated when function defined def func(x: expr): pass # evaluated when function defined Default arguments trip some people up because they expect them to be evaluated when the function's called, but it can easily be explained. Function annotations are exactly the same. Making them magically late-evaluate would have consequences for the grokkability of the language - they would be special. Now, that can be done, but as Rumplestiltskin keeps reminding us, all magic comes with a price, so it has to be strongly justified. (For instance, the no-arg form of super() is most definitely magical, but its justification is obvious when you compare Py2 inheritance with Py3.) > "Unknown evaluation time" is scary. _for expressions_, which might have side > effects (one of which is running time). But annotations must be pure by > convention (and tools are welcome to warn about it). I admit that I propose > breaking the following code: > > def foo(x: print("defining foo!")): pass > > Do you know anyone who would dream about writing such code? Yes, side effects make evaluation time scary. But so do rebindings, and any other influences on expression evaluation. Good, readable code generally follows the rule that the first instance of a name is its definition. That's why we put imports up the top of the script, and so on. Making annotations not work that way isn't going to improve readability; you'd have to search the entire project for the class being referenced. And since you can't probe them at definition time, you have to wait until, uhh, SOME time, to do that search - you never know where the actual name binding will come from. (It might even get injected from another file, so you can't statically search the one file.) >> You want to make them fancy, give them super-powers, in order to solve >> the forward reference problem. I don't think that the problem is serious >> enough to justify changing the semantics of annotation evaluation and >> make them non-standard, fancy, lazy-evaluated expressions. >> > > My proposal solves the forward reference problem, but I believe in it > because I believe it is aligned with what the programmer see. This is on par with a proposal to make default argument values late-bind, which comes up every now and then. It's just not worth making these expressions magical. >> > > class MyClass: >> > > pass >> > > >> > > def function(arg: MyCalss): >> > > ... >> > > >> > > I want to see an immediate NameError here, thank you very much >> > >> > Two things to note here: >> > A. IDEs will point at this NameError >> >> Some or them might. Not everyone uses an IDE, it is not a requirement >> for Python programmers. Runtime exceptions are still, and always will >> be, the primary way of detecting such errors. > > How useful is the detection of this error at production? The sooner you catch an error, the better. Always. > Can you repeat that? NameError indeed happens at runtime, but the scope in > which MyCalss was looked up for is determined at compile time - as far as I > know. The bytecode-based typechecker I wrote rely on this information being > accessible statically in the bytecode. > > def foo(): > locals()['MyType'] = str > def bar(a : MyType): pass > >>>> foo() > Traceback (most recent call last): > File "", line 1, in > File "", line 4, in foo > NameError: name 'MyType' is not defined > > What do I miss? That locals() is not editable (or rather, that mutations to it don't necessarily change the actual locals). This is equivalent to: def foo(): locals()['MyType'] = str print(MyType) >>> foo() Traceback (most recent call last): File "", line 1, in File "", line 3, in foo NameError: name 'MyType' is not defined In each case, you have to *call* foo() to see the NameError. It's happening at run time. > This way or the other, the very least that I hope, is explicitly forbidding > reliance on side-effect or any other way to distinguish evaluation time of > annotation expressions. Annotations must be pure, and the current promise of > evaluation time should be deprecated. Define "pure". Function decorator syntax goes to some lengths to ensure that this is legal: @deco(arg) def f(): pass PEP 484 annotations include subscripting, even nested: def inproduct(v: Iterable[Tuple[T, T]]) -> T: so you'd have to accept some measure of run-time evaluation. It's worth reiterating, too, that function annotations have had the exact same semantics since Python 3.0, in 2008. Changing that now would potentially break up to eight years' worth of code, not all of which follows PEP 484. When Steve mentioned 'not breaking other uses of annotations', he's including this large body of code that might well not even be visible to us, much less under python.org control. Changing how annotations get evaluated is a *major, breaking change*, so all you can really do is make a style guide recommendation that "annotations should be able to be understood with minimal external information" or something. > Additionally, before making it impossible to go back, we should make the new > variable annotation syntax add its annotations to a special object > __reflect__, so that __reflect__.annotations__ will allow forcing evaluation > (since there is no mechanism to do this in a variable). Wow, lots of magic needed to make this work. Here's my counter-proposal. In C++, you can pre-declare a class like this: class Mutual2; //Pre-declare Mutual2 class Mutual1 { Mutual2 *ptr; }; class Mutual2 { Mutual1 *ptr; } Here's how you could do it in Python: Mutual2 = "Mutual2" # Pre-declare Mutual2 class Mutual1: def spam() -> Mutual2: pass class Mutual2: def spam() -> Mutual1: pass Problem solved, no magic needed. ChrisA From ncoghlan at gmail.com Fri Sep 23 12:38:05 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Sep 2016 02:38:05 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> Message-ID: On 23 September 2016 at 20:31, ????? wrote: > On Fri, Sep 23, 2016 at 6:24 AM Nick Coghlan wrote: >> It's not a substitute for something like click or argparse when it >> comes to more complex argument parsing, but it's a good example of the >> kind of simple pseudo-DSL folks have long been able to create with >> annotations independently of the type hinting use case. >> > > That's a very nice use, and I was wrong - I did know it; I've found it not > long ago when I wanted to implement it myself... > > And guess what? It does not require eager evaluation _at all_. No > decorator-helped-annotation mechanism require eager evaluation built into > the language. Lazy evaluation is more general than eager, in that it can > always be forced (and not the other way around). The problem it poses for your proposal isn't that a library like begins couldn't be updated to work with lazy annotations (as you say, it clearly could be), it's that it demonstrates the idea of switching to lazy annotations involves a language level *compatibility break* for a feature that has been around and in use for almost 8 years now, and those need incredibly strong justifications. While I personally have some sympathy for the perspective that using strings for forward references in type hints feels a bit clunky, it still doesn't come close to reaching that deliberately high bar. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Sep 23 13:00:56 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Sep 2016 03:00:56 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: On 24 September 2016 at 01:58, Chris Angelico wrote: > Default arguments trip some people up because they expect them to be > evaluated when the function's called, but it can easily be explained. > Function annotations are exactly the same. Making them magically > late-evaluate would have consequences for the grokkability of the > language - they would be special. Now, that can be done Folks have been assuming that's straightforward, but the way class namespaces work actually makes it significantly harder than it first appears. Using lambda and default arguments to illustrate the problem: >>> class Example: ... attr = 10 ... @staticmethod ... def good_method(eager=attr): ... return eager ... @staticmethod ... def bad_method(lazy=(lambda:attr)): ... return lazy() ... >>> Example().good_method() 10 >>> Example().bad_method() Traceback (most recent call last): File "", line 1, in File "", line 8, in bad_method File "", line 7, in NameError: name 'attr' is not defined By design, function scopes can't see attributes defined in containing class scopes, and we don't currently have any other kind of scope that supports delayed evaluation (unlike function bodies, class bodies are evaluated eagerly at class definition time, and all the other delayed evaluation constructs are syntactic sugar for some particular flavour of function scope definition - even generators and coroutines use the same basic name resolution scheme as regular functions, they just use different execution models). If it was still 2006 or 2007 and Python 3.0 hadn't been released yet, lazy annotations could seriously be considered as an option. It's 2016 though, eager annotations have been out in the wild since December 2008, and the existing "string literals are Python's de facto lazy evaluation syntax" approach works well enough for the purpose, since type checkers can say they're actually going to parse those string literals when they appear to be type hints. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From fcktrll89 at gmail.com Sat Sep 24 04:44:59 2016 From: fcktrll89 at gmail.com (=?UTF-8?B?0KHQstGP0YLQvtGB0LvQsNCyINCc0L7Qu9C+0LTRgdC60LjRhQ==?=) Date: Sat, 24 Sep 2016 12:44:59 +0400 Subject: [Python-ideas] Python-ideas Digest, Vol 118, Issue 124 In-Reply-To: References: Message-ID: 23 ????. 2016 ?. 20:00 ???????????? ???????: > > Send Python-ideas mailing list submissions to > python-ideas at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request at python.org > > You can reach the person managing the list at > python-ideas-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > Today's Topics: > > 1. Re: Delay evaluation of annotations (Chris Angelico) > > > ---------- ???????????? ????????? ---------- > From: Chris Angelico > To: python-ideas > Cc: > Date: Sat, 24 Sep 2016 01:58:59 +1000 > Subject: Re: [Python-ideas] Delay evaluation of annotations > On Fri, Sep 23, 2016 at 11:58 PM, ????? wrote: > > What other context you see where the result of an expression is not intended > > to be used at all? Well there's Expression statements, which are evaluated > > for side effect. There's docstrings, which are a kind of annotations. What > > else? The only other that comes to mind is reveal_type(exp)... surely I > > don't need evaluation there. > > Function annotations ARE used. They're stored as function attributes, > just as default argument values and docstrings are. (It's not the > language's problem if you never use them.) > > >> The PEP that introduced them describes them as expressions: > > > > Syntactically, yes. Just like X in "a = lambda: X" is an expression, but you > > don't see it evaluated, do you? And this is an _actual_ expression, > > undeniably so, that is intended to be evaluated and used at runtime. > > And the X in "if False: X" is a statement, but you don't see it > evaluated either. This is an actual expression that has to be > evaluated and used just like any other does. > > >> I think its time to give up arguing that annotations aren't expressions. > >> > > > > I don't care if you call them expressions, delayed-expressions, or flying > > monkeys. The allowed syntax is exactly that of an expression (like inside a > > lambda). The time of binding of names to scope is the same (again like a > > lambda) but the evaluation time is unknown to the non-reflecting-developer. > > Decorators may promise time of evaluation, if they want to. > > Thing is, literally every other expression in Python is evaluated at > the point where it's hit. You can guard an expression with control > flow statements or operators, but other than that, it will be hit when > execution reaches its line: > > def func(x): > expr # evaluated when function called > > if cond: > expr # evaluated if cond is true > > [expr for x in range(n)] # evaluated if n > 0 > (expr for x in [1]) # evaluated when genexp nexted > expr if cond else "spam" # evaluated if cond is true > lambda: expr # evaluated when function called > > def func(x=expr): pass # evaluated when function defined > def func(x: expr): pass # evaluated when function defined > > Default arguments trip some people up because they expect them to be > evaluated when the function's called, but it can easily be explained. > Function annotations are exactly the same. Making them magically > late-evaluate would have consequences for the grokkability of the > language - they would be special. Now, that can be done, but as > Rumplestiltskin keeps reminding us, all magic comes with a price, so > it has to be strongly justified. (For instance, the no-arg form of > super() is most definitely magical, but its justification is obvious > when you compare Py2 inheritance with Py3.) > > > "Unknown evaluation time" is scary. _for expressions_, which might have side > > effects (one of which is running time). But annotations must be pure by > > convention (and tools are welcome to warn about it). I admit that I propose > > breaking the following code: > > > > def foo(x: print("defining foo!")): pass > > > > Do you know anyone who would dream about writing such code? > > Yes, side effects make evaluation time scary. But so do rebindings, > and any other influences on expression evaluation. Good, readable code > generally follows the rule that the first instance of a name is its > definition. That's why we put imports up the top of the script, and so > on. Making annotations not work that way isn't going to improve > readability; you'd have to search the entire project for the class > being referenced. And since you can't probe them at definition time, > you have to wait until, uhh, SOME time, to do that search - you never > know where the actual name binding will come from. (It might even get > injected from another file, so you can't statically search the one > file.) > > >> You want to make them fancy, give them super-powers, in order to solve > >> the forward reference problem. I don't think that the problem is serious > >> enough to justify changing the semantics of annotation evaluation and > >> make them non-standard, fancy, lazy-evaluated expressions. > >> > > > > My proposal solves the forward reference problem, but I believe in it > > because I believe it is aligned with what the programmer see. > > This is on par with a proposal to make default argument values > late-bind, which comes up every now and then. It's just not worth > making these expressions magical. > > >> > > class MyClass: > >> > > pass > >> > > > >> > > def function(arg: MyCalss): > >> > > ... > >> > > > >> > > I want to see an immediate NameError here, thank you very much > >> > > >> > Two things to note here: > >> > A. IDEs will point at this NameError > >> > >> Some or them might. Not everyone uses an IDE, it is not a requirement > >> for Python programmers. Runtime exceptions are still, and always will > >> be, the primary way of detecting such errors. > > > > How useful is the detection of this error at production? > > The sooner you catch an error, the better. Always. > > > Can you repeat that? NameError indeed happens at runtime, but the scope in > > which MyCalss was looked up for is determined at compile time - as far as I > > know. The bytecode-based typechecker I wrote rely on this information being > > accessible statically in the bytecode. > > > > def foo(): > > locals()['MyType'] = str > > def bar(a : MyType): pass > > > >>>> foo() > > Traceback (most recent call last): > > File "", line 1, in > > File "", line 4, in foo > > NameError: name 'MyType' is not defined > > > > What do I miss? > > That locals() is not editable (or rather, that mutations to it don't > necessarily change the actual locals). This is equivalent to: > > def foo(): > locals()['MyType'] = str > print(MyType) > > >>> foo() > Traceback (most recent call last): > File "", line 1, in > File "", line 3, in foo > NameError: name 'MyType' is not defined > > In each case, you have to *call* foo() to see the NameError. It's > happening at run time. > > > This way or the other, the very least that I hope, is explicitly forbidding > > reliance on side-effect or any other way to distinguish evaluation time of > > annotation expressions. Annotations must be pure, and the current promise of > > evaluation time should be deprecated. > > Define "pure". Function decorator syntax goes to some lengths to > ensure that this is legal: > > @deco(arg) > def f(): pass > > PEP 484 annotations include subscripting, even nested: > > def inproduct(v: Iterable[Tuple[T, T]]) -> T: > > so you'd have to accept some measure of run-time evaluation. > > It's worth reiterating, too, that function annotations have had the > exact same semantics since Python 3.0, in 2008. Changing that now > would potentially break up to eight years' worth of code, not all of > which follows PEP 484. When Steve mentioned 'not breaking other uses > of annotations', he's including this large body of code that might > well not even be visible to us, much less under python.org control. > Changing how annotations get evaluated is a *major, breaking change*, > so all you can really do is make a style guide recommendation that > "annotations should be able to be understood with minimal external > information" or something. > > > Additionally, before making it impossible to go back, we should make the new > > variable annotation syntax add its annotations to a special object > > __reflect__, so that __reflect__.annotations__ will allow forcing evaluation > > (since there is no mechanism to do this in a variable). > > Wow, lots of magic needed to make this work. Here's my > counter-proposal. In C++, you can pre-declare a class like this: > > class Mutual2; //Pre-declare Mutual2 > class Mutual1 { > Mutual2 *ptr; > }; > class Mutual2 { > Mutual1 *ptr; > } > > Here's how you could do it in Python: > > Mutual2 = "Mutual2" # Pre-declare Mutual2 > class Mutual1: > def spam() -> Mutual2: pass > class Mutual2: > def spam() -> Mutual1: pass > > Problem solved, no magic needed. > > ChrisA > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Sep 24 15:07:05 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 25 Sep 2016 04:07:05 +0900 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160923023549.GB22471@ando.pearwood.info> Message-ID: <22502.52953.598853.980473@turnbull.sk.tsukuba.ac.jp> ????? writes: > But _some_ people (medium-level, Steven, whose main language is > probably not Python) will not even know [function names are looked > up at runtime, and so the called function may not be why you think > it is] is the case. And the dinosaurs will have returned by independent evolution by the time it matters to them (unless it's a deliberate attack, in which case people at that level would be toast anyway). But I think you're completely missing what people are trying to tell you. You shouldn't be so concerned with refuting their arguments because it doesn't matter. No matter how many points you amass for technique, you're going to get creamed on style points anyway. It's like this: (1) Python is a "consenting adults" language, and that is presumed by its development culture. The goal is not to stop people from creating "functions that look like recursions but aren't" on purpose; it's to make it easy for them to write recursive functions if they want to. From your example, that goal is obviously satisfied. Nobody who matters wants to go farther than that in Python. The reason one can create "functions that look like recursions but aren't" is because another of Python's goals is to ensure that all things -- specifically including functions -- are objects that can be manipulated "the same way" where appropriate -- in this case, saving off the original function object somewhere then rebinding the original name to something else.[1] Granted, we don't go so far as Lisp where expressions are lists that you can manipulate like any other list, but aside from the fact that the code itself is an opaque object, functions are no different from other objects. Even builtins: Python 3.6.0a4 (default, Sep 3 2016, 19:21:32) >>> def help(*ignored, **paid_no_attention): ... print("Ouch, you just shot off your foot!") ... >>> help(help) Ouch, you just shot off your foot! >>> Shooting off your opposite extremity by redefining builtin classes is left as an exercise for the reader. All of this is a matter of the general attitude of pragmatism and bias toward simplicity of implementation (both enshrined in the Zen of Python). (2) You keep talking about others being lost in terminology, but in the context of Python discussions, you have a really big problem yourself. You use the phrase "just an annotation" as though that means something, but there is nothing like a "just an " in Python discourse, not in the sense that "once we introduce s, they can be anything we want". The Language Reference defines what things are possible, and truly new ones are rarely added. This is deliberate. Another design principle is Occam's Razor, here applied as "new kinds of thing shall not spring up like whiskers on Barry's chin." Yes, function annotations need new syntax and so are a new kind of thing to that extent. *Their values don't need to be,* and even the annotations themselves are implemented in the preferred way for "new things" (a dunder on an existing type). Since it's new syntax, it's language-level, and so the values are going to be something already defined in the language reference. "Expression resolving to object to be saved in an attribute on the function" seems to be as close to "anything you want" as you're gonna get without a new kind of thing. (3) Python has a very simple model of expressions. The compiler turns them into code. The interpreter executes that code, except in the case where it is "quoted" by the "def" or "lambda" keywords, in which case it's stored in an object (and in the case of "def", registered in a namespace). As Nick admits, you could indeed argue that initializations and annotation values *could* consistently be turned into "thunks" (stored code objects, we already have those) in attributes on the function object. But (1) that's an extension of the model (admittedly slight since functions, which already do that for their bodies, are involved -- but see Nick's reply for the hidden difficulties due to normal handling of namespaces in Python), and (2) it's a clear pessimization in the many cases where those values are immutable or very rarely mutated, and the use case (occasional) of keeping state in mutable values. The thunk approach is more complex, for rather small benefit. Re "small benefit", IMHO YMMV, but at least with initialization Guido is on record saying it's the RightThang[tm] (despite a propensity of new users to write buggy initializations). (4) Chris argues that "compile to thunk" is incoherent, that expressions in function bodies are no different than anywhere else -- they're evaluated when flow of control reaches them. AFAICS that *still* doesn't rule out having the compiler recognize the syntax and produce code that returns thunks instead of ordinary values, but Chris's point makes that seem way too magical to me. (5) This points up the fact that Python is thoroughly dynamic. It's not just that types adhere to objects rather than variables, but the whole attitude toward language design and implementation is. A variable not defined because it's on the path not taken, or even a function: they just don't exist as far as the interpreter is concerned -- there's no way to find them from Python. That's not true in say C: if you have a powerful enough debugger, you can even call a function defined, but never referenced, in the code. So while we'd be happy for people familiar with "statically-typed languages" to enjoy the benefits of using Python for some of their work, we can't help them if they can't shake off that attitude when using Python. Making things seem intuitive (which here translates to "familiar", as usual) to them is just misleading. Python doesn't work that way, and often enough, that matters. (6) As you point out: of course, thunks are more general than values (in fact, without instructions that move them around, in computers a value is just a tree falling in a forest with noone to hear). But maximum generality is not necessarily an important goal, even if it makes some things prettier. Allow me to quote the late Saunders Mac Lane: "[G]ood general theory does not search for the maximum generality, but for the right generality." (7) Re Nick's comment about backward compatibility on the high bar having a G degree of difficulty, I'm sure you don't disagree with the principle of avoiding compatibility breaks. But while that's probably the argument that defeats this proposal here, I think that even looking forward from the time before releasing Python 3.0, the decision would be the same. That is, I think the decision to go with the simpler "evaluate to object" model was one of the right decisions at that time, for the reasons above. Your proposal of "evaluate to thunk" (possibly incorporating the property-based magic Alexander proposed) might be right *too*, but it's far from obviously better to me. I see nothing there that would be likely to have dissuaded the authors of PEP 3107, or Guido when he designed default initialization, from "evaluate to object". If you don't like that philosophy, or somehow don't think it applies here, keep trying, you may have a point. But in this thread, IMO you're trying to ski up a slope that's barely able to hold snow, and wasting everybody's time. And even if I'm wrong about wasting time with the feature, you'll be way more persuasive if you argue in terms of Python as it is designed (mostly deliberately so), which is the way most of us mostly like it. Although we do change our mind every 18 months. :-) Footnotes: [1] For maximum humor, rebind it to a different recursive function! From elazarg at gmail.com Sat Sep 24 15:21:13 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sat, 24 Sep 2016 19:21:13 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <22502.52953.598853.980473@turnbull.sk.tsukuba.ac.jp> References: <20160923023549.GB22471@ando.pearwood.info> <22502.52953.598853.980473@turnbull.sk.tsukuba.ac.jp> Message-ID: Thank you all for your feedback. I will try to respond to it in a way that will not waste your time, But to do that I still need an example for the stromgest issue raised - bacwards compatibility. It is not just the mere change that is incompatible, since _any_ visible change is incompatible in some way, or otherwise it wasn't visible. Again, I assume ".__annotations__" access evaluates them in the original context. I couldn't find any useful example yet. Elazar ?????? ???, 24 ????' 2016, 22:07, ??? Stephen J. Turnbull ?< turnbull.stephen.fw at u.tsukuba.ac.jp>: > ????? writes: > > > But _some_ people (medium-level, Steven, whose main language is > > probably not Python) will not even know [function names are looked > > up at runtime, and so the called function may not be why you think > > it is] is the case. > > And the dinosaurs will have returned by independent evolution by the > time it matters to them (unless it's a deliberate attack, in which > case people at that level would be toast anyway). > > But I think you're completely missing what people are trying to tell > you. You shouldn't be so concerned with refuting their arguments > because it doesn't matter. No matter how many points you amass for > technique, you're going to get creamed on style points anyway. It's > like this: > > (1) Python is a "consenting adults" language, and that is presumed by > its development culture. The goal is not to stop people from > creating "functions that look like recursions but aren't" on > purpose; it's to make it easy for them to write recursive > functions if they want to. From your example, that goal is > obviously satisfied. Nobody who matters wants to go farther than > that in Python. > > The reason one can create "functions that look like recursions but > aren't" is because another of Python's goals is to ensure that all > things -- specifically including functions -- are objects that can > be manipulated "the same way" where appropriate -- in this case, > saving off the original function object somewhere then rebinding > the original name to something else.[1] Granted, we don't go so > far as Lisp where expressions are lists that you can manipulate > like any other list, but aside from the fact that the code itself > is an opaque object, functions are no different from other > objects. Even builtins: > > Python 3.6.0a4 (default, Sep 3 2016, 19:21:32) > >>> def help(*ignored, **paid_no_attention): > ... print("Ouch, you just shot off your foot!") > ... > >>> help(help) > Ouch, you just shot off your foot! > >>> > > Shooting off your opposite extremity by redefining builtin classes > is left as an exercise for the reader. > > All of this is a matter of the general attitude of pragmatism and > bias toward simplicity of implementation (both enshrined in the > Zen of Python). > > (2) You keep talking about others being lost in terminology, but in > the context of Python discussions, you have a really big problem > yourself. You use the phrase "just an annotation" as though that > means something, but there is nothing like a "just an " > in Python discourse, not in the sense that "once we introduce > s, they can be anything we want". The Language > Reference defines what things are possible, and truly new ones are > rarely added. > > This is deliberate. Another design principle is Occam's Razor, > here applied as "new kinds of thing shall not spring up like > whiskers on Barry's chin." Yes, function annotations need new > syntax and so are a new kind of thing to that extent. *Their > values don't need to be,* and even the annotations themselves are > implemented in the preferred way for "new things" (a dunder on an > existing type). Since it's new syntax, it's language-level, and > so the values are going to be something already defined in the > language reference. "Expression resolving to object to be saved > in an attribute on the function" seems to be as close to "anything > you want" as you're gonna get without a new kind of thing. > > (3) Python has a very simple model of expressions. The compiler turns > them into code. The interpreter executes that code, except in the > case where it is "quoted" by the "def" or "lambda" keywords, in > which case it's stored in an object (and in the case of "def", > registered in a namespace). > > As Nick admits, you could indeed argue that initializations and > annotation values *could* consistently be turned into "thunks" > (stored code objects, we already have those) in attributes on the > function object. But > > (1) that's an extension of the model (admittedly slight since > functions, which already do that for their bodies, are > involved -- but see Nick's reply for the hidden difficulties > due to normal handling of namespaces in Python), and > > (2) it's a clear pessimization in the many cases where those > values are immutable or very rarely mutated, and the use case > (occasional) of keeping state in mutable values. The thunk > approach is more complex, for rather small benefit. Re "small > benefit", IMHO YMMV, but at least with initialization Guido is > on record saying it's the RightThang[tm] (despite a propensity > of new users to write buggy initializations). > > (4) Chris argues that "compile to thunk" is incoherent, that > expressions in function bodies are no different than anywhere else > -- they're evaluated when flow of control reaches them. AFAICS > that *still* doesn't rule out having the compiler recognize the > syntax and produce code that returns thunks instead of ordinary > values, but Chris's point makes that seem way too magical to me. > > (5) This points up the fact that Python is thoroughly dynamic. It's > not just that types adhere to objects rather than variables, but > the whole attitude toward language design and implementation is. > A variable not defined because it's on the path not taken, or even > a function: they just don't exist as far as the interpreter is > concerned -- there's no way to find them from Python. That's not > true in say C: if you have a powerful enough debugger, you can > even call a function defined, but never referenced, in the code. > > So while we'd be happy for people familiar with "statically-typed > languages" to enjoy the benefits of using Python for some of their > work, we can't help them if they can't shake off that attitude > when using Python. Making things seem intuitive (which here > translates to "familiar", as usual) to them is just misleading. > Python doesn't work that way, and often enough, that matters. > > (6) As you point out: of course, thunks are more general than values > (in fact, without instructions that move them around, in computers > a value is just a tree falling in a forest with noone to hear). > But maximum generality is not necessarily an important goal, even > if it makes some things prettier. Allow me to quote the late > Saunders Mac Lane: "[G]ood general theory does not search for the > maximum generality, but for the right generality." > > (7) Re Nick's comment about backward compatibility on the high bar > having a G degree of difficulty, I'm sure you don't disagree with > the principle of avoiding compatibility breaks. > > But while that's probably the argument that defeats this proposal > here, I think that even looking forward from the time before > releasing Python 3.0, the decision would be the same. That is, I > think the decision to go with the simpler "evaluate to object" > model was one of the right decisions at that time, for the reasons > above. Your proposal of "evaluate to thunk" (possibly > incorporating the property-based magic Alexander proposed) might > be right *too*, but it's far from obviously better to me. I see > nothing there that would be likely to have dissuaded the authors > of PEP 3107, or Guido when he designed default initialization, > from "evaluate to object". > > If you don't like that philosophy, or somehow don't think it applies > here, keep trying, you may have a point. But in this thread, IMO > you're trying to ski up a slope that's barely able to hold snow, and > wasting everybody's time. And even if I'm wrong about wasting time > with the feature, you'll be way more persuasive if you argue in terms > of Python as it is designed (mostly deliberately so), which is the way > most of us mostly like it. Although we do change our mind every 18 > months. :-) > > > Footnotes: > [1] For maximum humor, rebind it to a different recursive function! > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Sep 24 20:43:40 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 25 Sep 2016 09:43:40 +0900 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160923023549.GB22471@ando.pearwood.info> <22502.52953.598853.980473@turnbull.sk.tsukuba.ac.jp> Message-ID: <22503.7612.27511.437657@turnbull.sk.tsukuba.ac.jp> ????? writes: > But to do that I still need an example for the stromgest issue > raised - bacwards compatibility. Given that Nick has misgivings about the ease of actually implementing this, I think you need to present an implementation, and then we talk about how closely it approximates backward compatibility. > Again, I assume ".__annotations__" access evaluates them in the original > context. You don't get to assume that, without an implementation that shows how you work around the "def can't see names of lambda arguments" issue. At the very least you need to define "evaluate in original context" operationally -- the "original context" as I understand it is what is visible the current implementation, but that is clearly not what you mean. Of course an implementation would serve to define that. From rosuav at gmail.com Sat Sep 24 23:10:45 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Sep 2016 13:10:45 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: On Sun, Sep 25, 2016 at 11:55 AM, ????? wrote: > Short-ish version: > > 1. Please consider disallowing the use of side effects of any kind in > annotations, in that it is not promised when it will happen, if at all. So > that a change 3 years from now will be somewhat less likely to break things. > Please consider doing this for version 3.6; it is feature-frozen, but this > is not (yet) a feature, and I got the feeling it is hardly controversial. > > I really have no interest in wasting the time of anybody here. If this > request is not something you would ever consider, please ignore the rest of > this email. I don't think Python has any concept of *disallowing* side effects. As soon as arbitrary objects can be called and/or subscripted, arbitrary code can be executed. However, a style guide may *discourage* extensive side effects, and this I would agree with - not for reasons of future change, but for reasons of simplicity and readability. > 3. The main benefit from my proposal is that contracts (examples, > explanations, assertions, and types) are naturally expressible as (almost) > arbitrary Python expressions, but not if they are evaluated or evaluatable, > at definition time, by the interpreter. Why: because it is really written in > a different language - *always*. This is the real reason behind the > existence, and the current solutions, of the forward reference problem. In > general it is much more flexible than current situation. So, basically, you want annotations to be able to make use of names defined in the object they're annotating. That's a reasonable summary of the idea, if I have this correct. I'll trim out a ton of quoted material that digs into details. Ultimately, though, you're asking to change something that has been this way since *Python 3.0*. You're not asking for a tiny tweak to a feature that's new in 3.6. If you were, perhaps this could be done, despite feature freeze; but you're breaking compat with eight years of Pythons, and that's almost certainly not going to happen. > Chris: > > On Fri, Sep 23, 2016 at 6:59 PM Chris Angelico wrote: >> Good, readable code >> generally follows the rule that the first instance of a name is its >> definition. > > > No, it isn't. I guess that even the code you write or consider to be > excellent and readable still contains functions that use entities defined > only later in the code. It is only when you follow execution path that you > should be already familiar with the names. Actually, no, I do generally stick to this pattern, builtins aside. Obviously there are times when you can't (mutually exclusive functions, for instance), but those are pretty rare. Here's an example program of mine: https://github.com/Rosuav/LetMeKnow/blob/master/letmeknow.py There is one star-import, which breaks this pattern (the global name CLIENT_SECRET comes from keys.py), and which I consider to be a failing under this principle; but it's better than most of the alternatives, and like all style recommendations, "define before use" is a rule that can be broken. >> The sooner you catch an error, the better. Always. >> > > No. No. No. If a code in production will fail at my client's site because of > a mispelled annotation (unused by runtime tools), I will be mad. *On the > language*. It is just as reasonable as failing because of mispled > documentation. (My suggestion does not prevent it completely of course. > Nothing will. I only say this is unhelpful). Then I strongly disagree. If it's going to fail at the client's site, I want it to first fail on my computer. >> It's worth reiterating, too, that function annotations have had the >> exact same semantics since Python 3.0, in 2008. > > > When was this semantics decided and for what purposes, if I may ask? because > the PEP (2006) explicitly states that "this PEP makes no attempt to > introduce any kind of standard semantics". Decorators also have clearly defined local semantics and completely undefined overall semantics. If you see this in a .py file: @spaminate(ham=1) def frobber(): pass you know exactly what's happening, on a mechanical level: first spaminate(ham=1) will be called, and then the result will be called with frobber as an argument, and the result of that bound to the name frobber. But exactly what the spaminate decorator does is not Python's business. It might make frobber externally callable (cf routing decorators in Flask), or it might modify frobber's behaviour (eg require that ham be 1 before it'll be called), or it might trigger some sort of run-time optimization (memoization being an easy one, and actual code modifications being also possible). Annotations are the same. There's a clearly defined local syntactic handling: @multicall def frobber(ham: [10,20,30]): pass but nothing in the language that says what this truly means. In this case, I'm envisioning a kind of special default argument handling that says "if you don't provide a ham argument, call frobber three times with the successive values from ham's annotation". But you can be absolutely certain that, on the mechanical level, what happens is that the expression "[10,20,30]" gets evaluated, and the result gets stashed into the function's __annotations__. In contrast, function default arguments have *both* forms of semantics clearly defined. The expression is evaluated and the result stashed away; and then, when the function is called, if there's no argument, the default is used. > But why is "deprecating side effects in annotation's > definition-time-execution" considered a breaking change? It is just a > documentation. Everything will work as always has. Even edge cases. I would > think this is possible even for the feature-freezed 3.6. Like saying "We've > found a loophole in the language; it might get fixed in the future. Don't > count on it." Deprecating in the sense of "style guides recommend against this" is fine. PEP 8 has been updated periodically, and it doesn't break anyone's code (except MAYBE linters, and even then they're not broken, just not up-to-date). But an actual code change that means that Python 3.7 will reject code that Python 3.5 accepted? That's a breaking change. And the purpose of your documentation-only deprecation is exactly that, or possibly Python 3.8 or 3.9, but timeframe doesn't change the fact that it will break code. >> Here's my counter-proposal. > >> Mutual2 = "Mutual2" # Pre-declare Mutual2 >> class Mutual1: >> def spam() -> Mutual2: pass >> class Mutual2: >> def spam() -> Mutual1: pass >> >> Problem solved, no magic needed. > > Problem not solved. Your counter proposal solves only certain forward > references, and requires keeping on more thing in sync, in particular > adapting the scope of the "forward declaration" to the scope of the later > definition, which may change over time and is in violation of DRY. Oh and > lastly, type checkers will scream or will work very hard to allow this > idiom. Type checkers that comply with PEP 484 are already going to support this notation, because "Mutual2" is a valid annotation. All I've done differently is make a simple assignment, in the same way that typevars get assigned. > Keeping an AST without evaluation at all is still a clear pessimization? The AST for an expression usually takes up more memory than the result of it, yeah. > And here's my attempt at presenting the "because maths" argument you > probably don't want to hear: it will allow natural and well-based way to > express contracts and dependent types, which is a much more natural and > flexible way to type dynamically-typed languages such as Python. It is > nothing new really; it is based on a 40-years old understanding that types > are propositions *and propositions are types*. And I want to use it. From > simple to complex: > > @typecheck > def to_float(x: int or str) -> float: > ... Please let's not go down this path. Already I have to explain to my students that this won't work: if response == "yes" or "y": If it *does* work in annotations but doesn't work everywhere else, that would be extremely confusing. > @typecheck > def __add__(self, x: int and float) -> float: > ... > > This should help resolve a real problem in type checkers regarding overloads > and overlapping types. Except @typecheck can only see the object "float". > And they did not invent Intersection[] yet. Bummer, but fixable. I'm not sure what the intersection of int and float would be, but perhaps you mean this more like Java's interfaces - something that "implements X" and "implements Y" is the intersection of the types X and Y. > Let's define contracts > > @contract > def divmod(x: int, y: int and y != 0) -> (x//y, x % y): > return # an optimized version > > NameError again? :( Now, this is where stuff starts to get interesting. You want to be able to define an assertion in terms of the variables you're creating here. In effect, you have something like this: def divmod(x, y): assert isinstance(x, int) assert isinstance(y, int) and y != 0 ... # optimized calculation assert ret == (x // y, x % y) return ret As ideas go, not a bad one. Not really compatible with annotations, though, and very difficult to adequately parse. If you want to flesh this out as your proposal, I would suggest setting this thread aside and starting over, explaining (a) why actual assertions aren't good enough, and (b) how annotations could be used without breaking compatibility. > What if I want to specify "a class that subclasses Abstract but can be > istantiated? I need it because otherwise mypy resorts to allowing unsafe > code: > > def create(cls: typing.Type[Abstract] and cls(...) ) -> Base: > return cls() > > NameError again. Why? Not because _you_ (people) don't understand it. Actually, I don't understand exactly what this should do. Does it assert that cls can be instantiated with some unknown args? Because you then instantiate it with no args. What does cls(...) require? > And what if I want "x: type(x)==int"? It has its uses. Then explicitly assert that. I don't see why you should type-declare that something is "this and not a subclass" - the whole point of subclassing is that it still is an instance of the superclass. Maybe what Python needs is a simple syntax for "AST-for-this-expression". We have lambda, which means "function which evaluates this expression"; this would operate similarly. >>> expr = kappa: x + y >>> ast.dump(expr) Module(body=[Expr(value=BinOp(left=Name(id='x', ctx=Load()), op=Add(), right=Name(id='y', ctx=Load())))]) It'd be exactly the same as ast.parse("x + y"), but might be able to make use of the existing parsing operation, and would be more easily syntax highlighted. (Or maybe it'd start at the Expr node instead - so it'd be equiv to ast.parse("x + y").body[0].) Open to suggestions as to an actual name. With that change, your proposals could all be added in a 100% backward compatible way. Annotations, as a feature, wouldn't change; you'd just kappafy your contracts: @contract def divmod(x: int, y: kappa: int and y != 0) -> kappa: (x//y, x % y): ... And then you could define contract() as either the identity function (optimized mode, no checks done), or a wrapper function that does run-time checks. Maybe that, rather than making annotations magical, would solve the problem? ChrisA From ncoghlan at gmail.com Sun Sep 25 12:07:01 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2016 02:07:01 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: On 25 September 2016 at 11:55, ????? wrote: > I promised not to bother you, but I really can't. So here's what I felt I > have to say. This email is quite long. Please do not feel obliged to read > it. You might find some things you'll want to bash at the end though :) > > Short-ish version: > > 1. Please consider disallowing the use of side effects of any kind in > annotations, in that it is not promised when it will happen, if at all. This may be part of the confusion, as Python is a language with a *reference implementation*, rather than relying solely on a documented language specification. Unless we specifically call something out in the language reference and/or the test suite as a CPython implementation detail, then "what CPython does" should be taken as the specification. While we're fairly permissive in allowing alternative implementations to deviate a bit and still call themselves Python, and sometimes alternate implementation authors point out quirky behaviours and we declare them to be bugs in CPython, "CPython correctly implements the Python language specification" is still the baseline assumption. So the order of evaluation for annotations with side effects has been defined since 3.0 came out: >>> def func(a:print("A1")=print("D1"), b:print("A2")=print("D2")) -> print("Areturn"): ... pass ... D1 D2 A1 A2 Areturn That is, at function definition time: - default values are evaluated from left to right - annotations are evaluated from left to right > So > that a change 3 years from now will be somewhat less likely to break things. > Please consider doing this for version 3.6; it is feature-frozen, but this > is not (yet) a feature, and I got the feeling it is hardly controversial. > > I really have no interest in wasting the time of anybody here. If this > request is not something you would ever consider, please ignore the rest of > this email. I don't think you're wasting anyone's time - this is a genuinely complex topic, and some of it relates to design instinct about what keeps a language relatively easy to learn. However, I do think we're talking past each other a bit. I suspect the above point regarding the differences between languages that are formally defined by a written specification and those like Python that let a particular implementation (in our case, CPython) fill in the details not otherwise written down may be a contributing factor to that Another may be that there are some things (like advanced metaprogramming techniques) where making them easy isn't actually a goal we necessarily pursue: we want to ensure they're *possible*, as in some situations they really are the best available answer, but we also want to guide folks towards simpler alternatives when those simpler alternatives are sufficient. PEP 487 is an interesting example of that, as that has the express goal of taking two broad categories of use cases that currently require a custom metaclass (implicitly affecting the definition of subclasses and letting descriptors know the attribute name they're bound to), and making them standard parts of the default class definition protocol. Ideally, this will lead to *fewer* custom metaclasses being defined in the future, with folks being able to instead rely on normal class definitions and those simpler extracted patterns. > 2. A refined proposal for future versions of the language: the ASTs of the > annotation-expressions will be bound to __raw_annotations__. > * This is actually more in line to what PEP-3107 was about ("no assigned > semantics"; except for a single sentence, it is only about expressions. Not > objects). PEP 3107 came with a reference implementation, it wasn't just the written PEP content: https://www.python.org/dev/peps/pep-3107/#implementation > * This is helpful even if the expression is evaluated at definition time, > and can help in smoothing the transformation. We talk about the idea of expression quoting and AST preservation fairly often, but it's not easy to extract from the archives unless you already know roughly what you're looking for - it tends to come up as a possible solution to *other* problems, and each time we either decide to leave the problem unsolved, or find a simpler alternative to letting the "syntactic support for AST metaprogramming" genie out of the bottle. Currently, the only supported interfaces for this are using the ast.parse() helper, or passing the ast.PyCF_ONLY_AST flag to the compile() builtin. This approach gives alternative implementations a fair bit of flexibility to *not* use that AST internally if it doesn't help their particular implementation. Once you start tying it in directly to language level features, though, it starts to remove a lot of that implementation flexibility. > 3. The main benefit from my proposal is that contracts (examples, > explanations, assertions, and types) are naturally expressible as (almost) > arbitrary Python expressions, but not if they are evaluated or evaluatable, > at definition time, by the interpreter. Why: because it is really written in > a different language - *always*. This is the real reason behind the > existence, and the current solutions, of the forward reference problem. In > general it is much more flexible than current situation. "More flexible" is only a virtue if you have concrete use cases in mind that can't otherwise be addressed today. Since you mention design-by-contract, you may want to take a look at https://www.python.org/dev/peps/pep-0316/ which is an old deferred proposal to support DBC by way of a particular formatting convention in docstrings, especially as special formatting in docstrings was one of the main ways folks did type annotations before PEP 3107 added dedicated syntax for them. > 4. For compatibility, a new raw_annotations() function will be added, and a > new annotations() function will be used to get the eval()ed version of them. Nothing *new* can ever be added for compatibility reasons: by definition, preserving backwards compatibility means old code continuing to run *without modification*. New interfaces can be added to simplify migration of old code, but it's not the same thing as actually preserving backwards compatibility. > Similarly to dir(), locals() and globals(). > * Accessing __annotations__ should work like calling annotations(), but > frowned upon, as it might disappear in the far future. > * Of course other `inspect` functions should give the same results as > today. > * Calling annotations()['a'] is like a eval(raw_annotations()['a']) which > resembles eval(raw_input()). > > I believe the last point has a very good reason, as explained later: it is > an interpretation of a different language, foreign to the interpreter, > although sometimes close enough to be useful. It is of course well formed, > so the considerations are not really security-related. Here you're getting into the question of expression quoting, and for a statement level version of that, you may want to explore the thread at https://mail.python.org/pipermail/python-ideas/2011-April/009765.html (I started that thread because I'd had an idea I needed to share so I could stop thinking about it, but I also think more syntactic sugar for metaprogramming isn't really something the vast majority of Python developers actually need) Mython, which was built as a variant of Python 2 with more metaprogramming features is also worth a look: http://mython.org/ > I am willing to do any hard work that will make this proposal happen > (evaluating existing libraries, implementing changes to CPython, etc) given > a reasonable chance for acceptance. I think the two basic road blocks you're running into are: - the order of evaluation for annotations with side effects is already well defined and has been since Python 3.0. It's just defined by the way CPython works as the reference implementation, rather than in English prose anywhere. - delayed evaluation already has two forms in Python (function scopes and quoted strings) and adding a third is a *really* controversial prospect, but if you don't add a third, you run into the fact that all function scopes inside a class scope are treated as methods by the compiler Stephen's post went into more detail on *why* that second point is so controversial: because it's a relatively major increase in the underlying complexity of the runtime execution model. The most recent run at it that I recall was my suggestion to extend f-strings (which are eagerly evaluated) to a more general purpose namespace capturing capability in https://www.python.org/dev/peps/pep-0501/ That's deferred pending more experience with f-strings between now and the 3.7 beta, but at this point I'll honestly be surprised if the simple expedient of "lambda: " doesn't turn out to be sufficient to cover any delayed evaluation needs that arise in practice (folks tend not to put complex logic in their class bodies). > Stephen - I read your last email only after writing this one; I think I have > partially addressed the lookup issue (with ASTs and scopes), and partially > agree: if there's a problem implementing this feature, I should look deeper > into it. But I want to know that it _might_ be considered seriously, _if_ it > is implementable. I also think that Nick refuted the claim that evaluation > time and lookup *today* are so simple to explain. I know I have hard time > explaining them to people. Most folks coming from pre-compiled languages like C++, C# & Java struggle with the fact that Python doesn't have separate compile time constructs (which deal with function, class and method declarations) and runtime constructs (which are your traditional control flow statements). Instead, Python just has runtime statements, and function and class definition are executed when encountered, just like any other statement. This fundamentally changes the relationship between compile time, definition time, and call time, most significantly by having "definition time" be something that happens during the operation of the program itself. > Nick, I have read your blog post about the high bar required for > compatibility break, and I follow this mailing list for a while. So I agree > with the reasoning (from my very, very little experience); I only want to > understand where is this break of compatibility happen, because I can't see > it. This code works as a doctest today: >>> def func(a: "Expected output"): ... pass ... >>> print(func.__annotations__["a"]) Expected output Any change that breaks that currently valid doctest is necessarily a compatibility break for the way annotations are handled at runtime. It doesn't matter for that determination how small the change to fix the second command is, it only matters that it *would* have to change in some way. In particular, switching to delayed evaluation would break all the introspection tools that currently read annotations at runtime, both those in the standard library (like inspect.signature() and pydoc), and those in third party tools (like IDEs). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From elazarg at gmail.com Sun Sep 25 12:50:03 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sun, 25 Sep 2016 16:50:03 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: Thanks for the references. I will read them. In general, I am against magic in code. I am for magic in specification, with appropriate hints (e.g. the explicit name in the decorator, as pointed to me by Chris) and with taste. The most important part about specification is being naturally understood by human. The second most important is being understood by tools. What's not important: being understood by the interpreter. CPython as a reference implementation has a very, very specific behavior, changing at every minor release. Of course not every tiny detail of this behavior is promised. It is understood by users that e.g. they cannot rely on their code taking 0.6ms to execute in such and such settings, since real-time con?traints are not promised even if some part of some version of CPython happens to run this fast deterministically. The implementation specifies the behavior, except when common sense or documentation says otherwise. Am I wrong? On Sun, Sep 25, 2016 at 7:07 PM Nick Coghlan wrote: > On 25 September 2016 at 11:55, ????? wrote: > > Nick, I have read your blog post about the high bar required for > > compatibility break, and I follow this mailing list for a while. So I > agree > > with the reasoning (from my very, very little experience); I only want to > > understand where is this break of compatibility happen, because I can't > see > > it. > > This code works as a doctest today: > > >>> def func(a: "Expected output"): > ... pass > ... > >>> print(func.__annotations__["a"]) > Expected output > > Any change that breaks that currently valid doctest is necessarily a > compatibility break for the way annotations are handled at runtime. It > doesn't matter for that determination how small the change to fix the > second command is, it only matters that it *would* have to change in > some way. > > In particular, switching to delayed evaluation would break all the > introspection tools that currently read annotations at runtime, both > those in the standard library (like inspect.signature() and pydoc), > and those in third party tools (like IDEs). > But my intention is that this code will work just fine. As is any other access using __annotations__ or any existing API. The only visible change should be that of expressions with visible side effect, so this is the kind of break I am looking for. The following will break def foo(a: print(1)): pass But nobody (yet) claimed it to be a reasonable example of code we don't want to break. There can hardly be any, since any side effect can be placed right before the definition. So just like star-imports that are broken every time a function is added to some library, IIUC, you don't care about breaking them, because they are strongly and explicitly discouraged. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Sat Sep 24 21:55:09 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sun, 25 Sep 2016 01:55:09 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: I promised not to bother you, but I really can't. So here's what I felt I have to say. This email is quite long. Please do not feel obliged to read it. You might find some things you'll want to bash at the end though :) Short-ish version: 1. Please consider disallowing the use of side effects of any kind in annotations, in that it is not promised when it will happen, if at all. So that a change 3 years from now will be somewhat less likely to break things. Please consider doing this for version 3.6; it is feature-frozen, but this is not (yet) a feature, and I got the feeling it is hardly controversial. I really have no interest in wasting the time of anybody here. If this request is not something you would ever consider, please ignore the rest of this email. 2. A refined proposal for future versions of the language: the ASTs of the annotation-expressions will be bound to __raw_annotations__. * This is actually more in line to what PEP-3107 was about ("no assigned semantics"; except for a single sentence, it is only about expressions. Not objects). * This is helpful even if the expression is evaluated at definition time, and can help in smoothing the transformation. 3. The main benefit from my proposal is that contracts (examples, explanations, assertions, and types) are naturally expressible as (almost) arbitrary Python expressions, but not if they are evaluated or evaluatable, at definition time, by the interpreter. Why: because it is really written in a different language - *always*. This is the real reason behind the existence, and the current solutions, of the forward reference problem. In general it is much more flexible than current situation. 4. For compatibility, a new raw_annotations() function will be added, and a new annotations() function will be used to get the eval()ed version of them. Similarly to dir(), locals() and globals(). * Accessing __annotations__ should work like calling annotations(), but frowned upon, as it might disappear in the far future. * Of course other `inspect` functions should give the same results as today. * Calling annotations()['a'] is like a eval(raw_annotations()['a']) which resembles eval(raw_input()). I believe the last point has a very good reason, as explained later: it is an interpretation of a different language, foreign to the interpreter, although sometimes close enough to be useful. It is of course well formed, so the considerations are not really security-related. I am willing to do any hard work that will make this proposal happen (evaluating existing libraries, implementing changes to CPython, etc) given a reasonable chance for acceptance. Thank you, Elazar --- Long version: Stephen - I read your last email only after writing this one; I think I have partially addressed the lookup issue (with ASTs and scopes), and partially agree: if there's a problem implementing this feature, I should look deeper into it. But I want to know that it _might_ be considered seriously, _if_ it is implementable. I also think that Nick refuted the claim that evaluation time and lookup *today* are so simple to explain. I know I have hard time explaining them to people. Nick, I have read your blog post about the high bar required for compatibility break, and I follow this mailing list for a while. So I agree with the reasoning (from my very, very little experience); I only want to understand where is this break of compatibility happen, because I can't see it. Chris: On Fri, Sep 23, 2016 at 6:59 PM Chris Angelico wrote: > On Fri, Sep 23, 2016 at 11:58 PM, ????? wrote: > "Unknown evaluation time" is scary. _for expressions_, which might have > side > > effects (one of which is running time). But annotations must be pure by > > convention (and tools are welcome to warn about it). I admit that I > propose > > breaking the following code: > > > > def foo(x: print("defining foo!")): pass > > > > Do you know anyone who would dream about writing such code? > > Yes, side effects make evaluation time scary. But so do rebindings, > and any other influences on expression evaluation. Good, readable code > generally follows the rule that the first instance of a name is its > definition. No, it isn't. I guess that even the code you write or consider to be excellent and readable still contains functions that use entities defined only later in the code. It is only when you follow execution path that you should be already familiar with the names. I think rebinding is only scary when it is combined with side effect or when the name lookup is not clear. And why do you call it _re_binding? > >> > > class MyClass: > >> > > pass > >> > > > >> > > def function(arg: MyCalss): > >> > > ... > >> > > > >> > > I want to see an immediate NameError here, thank you very much > >> > > >> > Two things to note here: > >> > A. IDEs will point at this NameError > >> > >> Some or them might. Not everyone uses an IDE, it is not a requirement > >> for Python programmers. Runtime exceptions are still, and always will > >> be, the primary way of detecting such errors. > > > > How useful is the detection of this error at production? > > The sooner you catch an error, the better. Always. > > No. No. No. If a code in production will fail at my client's site because of a mispelled annotation (unused by runtime tools), I will be mad. *On the language*. It is just as reasonable as failing because of mispled documentation. (My suggestion does not prevent it completely of course. Nothing will. I only say this is unhelpful). It's worth reiterating, too, that function annotations have had the > exact same semantics since Python 3.0, in 2008. When was this semantics decided and for what purposes, if I may ask? because the PEP (2006) explicitly states that "this PEP makes no attempt to introduce any kind of standard semantics". The main relevant paragraph reads (I quote the PEP, my own emphasis): "2. Function annotations are nothing more than a way of associating arbitrary Python EXPRESSIONS with various parts of a function at compile-time. By itself, Python does not attach ANY PARTICULAR MEANING or significance to annotations. Left to its own, Python simply makes these EXPRESSIONS available as described in Accessing Function Annotations below. The only way that annotations take on meaning is when they are interpreted by third-party libraries. These annotation consumers can do anything they want with a function's annotations." Amen to that! Word by word as my suggestion. Why aren't these _expressions_ available to me, as promised? Sadly, a few paragraphs later, the PEP adds that "All annotation expressions are evaluated when the function definition is executed, just like default values." - Now please explain to me how is that attaching "no particular meaning or significance to annotations". You practically evaluate them, for heavens sake! this is plain and simple "attached meaning" and "standard semantics". Unusefully so. I put there an expression, and all I got is a lousy object. > Changing that now > would potentially break up to eight years' worth of code, not all of > which follows PEP 484. When Steve mentioned 'not breaking other uses > of annotations', he's including this large body of code that might > well not even be visible to us, much less under python.org control. > Changing how annotations get evaluated is a *major, breaking change*, > so all you can really do is make a style guide recommendation that > "annotations should be able to be understood with minimal external > information" or something. > As I said, it is a strong argument - given an example of such a potential break for non-convoluted code. I want to see such an example. But why is "deprecating side effects in annotation's definition-time-execution" considered a breaking change? It is just a documentation. Everything will work as always has. Even edge cases. I would think this is possible even for the feature-freezed 3.6. Like saying "We've found a loophole in the language; it might get fixed in the future. Don't count on it." > Here's my counter-proposal. > Mutual2 = "Mutual2" # Pre-declare Mutual2 > class Mutual1: > def spam() -> Mutual2: pass > class Mutual2: > def spam() -> Mutual1: pass > > Problem solved, no magic needed. Problem not solved. Your counter proposal solves only certain forward references, and requires keeping on more thing in sync, in particular adapting the scope of the "forward declaration" to the scope of the later definition, which may change over time and is in violation of DRY. Oh and lastly, type checkers will scream or will work very hard to allow this idiom. My proposal asks for no magic at all. Unless you consider dir() and locals() magical (they are. a bit). Stephen: Regarding the terminology, I said "But I think we are getting lost in the terminology." including myself. On Sat, Sep 24, 2016 at 10:07 PM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Python has a very simple model of expressions. The compiler turns them into code. The interpreter executes that code, except in the case where it is "quoted" by the "def" or "lambda" keywords, in which case it's stored in an object (and in the case of "def", registered in a namespace). Here's the version for annotations: The compiler turns them into an AST. The interpreter does nothing with them, except attaching them to the __annotations__ object. Are you the average programmer? then just read them, they should be helpful. How simple is that? > Another design principle is Occam's Razor, here applied as "new kinds of thing shall not spring up like whiskers on Barry's chin." Yes, function annotations need new syntax and so are a new kind of thing to that extent. *Their values don't need to be,* Their values don't need to be there at all. All is needed is their structure. The AST. Their value is of very little use, actually. And I'm familiar with the typing module; it is bent here and there to match this need to have a "value" or an object when you actually need only the structure. I don't invent "new thing" any more than is already there. I have a strong belief that _already there_ is a new thing. In Python it is called "types", "annotations" and some forms of "documentations". In other languages it is called in other names. I don't mind the merging of this concept with the concept of Expression - I actually think it is brilliant. Sadly, the _interpreter_ does not understand it. It is (brilliantly again) admitted in the PEP, but sadly the interpreter does not seem to get it, and it tries to execute it anyway. So people shush it with a quote. Well done. Now where is that Expression thing? And why can't my editor highlight it? > (2) it's a clear pessimization in the many cases where those > values are immutable or very rarely mutated, and the use case > (occasional) of keeping state in mutable values. The thunk > approach is more complex, for rather small benefit. Re "small > benefit", IMHO YMMV, but at least with initialization Guido is > on record saying it's the RightThang[tm] (despite a propensity > of new users to write buggy initializations). Keeping an AST without evaluation at all is still a clear pessimization? > Chris argues that "compile to thunk" is incoherent, that > expressions in function bodies are no different than anywhere else > -- they're evaluated when flow of control reaches them. I argue that flow of control should not reach annotations at all. Not the control of the interpreter. It does not understand what's written there, and should not learn. > (5) > A variable not defined because it's on the path not taken, or even > a function: they just don't exist as far as the interpreter is > concerned -- there's no way to find them from Python See my previous comment. The interpreter should not look for them. You know what? It can try to give a hint in name resolution. As a favor. If the tools can't find it, it's their business to report. > "[G]ood general theory does not search for the maximum generality, but for the right generality." I believe this is the right generality "because maths" and my own intuition. This should not convince you at all. > Your proposal of "evaluate to thunk" (possibly > incorporating the property-based magic Alexander proposed) might > be right *too*, but it's far from obviously better to me As you've noticed, I refined my proposal to "don't evaluate". And here's my attempt at presenting the "because maths" argument you probably don't want to hear: it will allow natural and well-based way to express contracts and dependent types, which is a much more natural and flexible way to type dynamically-typed languages such as Python. It is nothing new really; it is based on a 40-years old understanding that types are propositions *and propositions are types*. And I want to use it. From simple to complex: @typecheck def to_float(x: int or str) -> float: ... (Oh, @typecheck can't access this, so they invented Union. Oh well. TODO: explain to junior what a Union[int, str] is) @typecheck def __add__(self, x: int and float) -> float: ... This should help resolve a real problem in type checkers regarding overloads and overlapping types. Except @typecheck can only see the object "float". And they did not invent Intersection[] yet. Bummer, but fixable. @dependent def call_foo_twice(x: x.foo()) -> None: x.foo() x.foo() Uh, I don't know. Perhaps x.foo() has side effect and I'm not sure how "dependent" works, and perhaps it is not so clear. Try again: @dependent def call_foo_twice(x: hasattr(x, "foo") and is_callable(x.foo)) -> None: x.foo() x.foo() But why NameError? :( Let's define contracts @contract def divmod(x: int, y: int and y != 0) -> (x//y, x % y): return # an optimized version NameError again? :( Not every function _should_ be annotated this way, but why can't you _allow_ this kind of annotation? These make a needless syntax error for an obviously meaningful expression. What if I want to specify "a class that subclasses Abstract but can be istantiated? I need it because otherwise mypy resorts to allowing unsafe code: def create(cls: typing.Type[Abstract] and cls(...) ) -> Base: return cls() NameError again. Why? Not because _you_ (people) don't understand it. No. It is because the _interpreter_ claims to understand it, but it doesn't. It cannot, because Python is not intended to be a specification language, and probably should not be. Even with simple types it doesn't, really. It just happen to look up the right _class_ which is not a type but is useful nonetheless; when I write "x: int" I actually mean "x: isinstance(x, int)", but the interpreter doesn't get it and should not get it. And what if I want "x: type(x)==int"? It has its uses. Regarding my "not an expression" claim: this is a proposition, an assumption, a precondition, or explanation - different incarnation of the same idea - as is any annotation system I have seen (except possibly injection, which will not break). Including Nick's "begin". Now notice this nice command-line-parser - the annotations there can still be strings, combined easily with type checkers. Why? because it targets human users, that's why. And the interpreter does not try to understand them because it does not understand English. Well, it does not understand Type-ish or Contract-ish, or any other Spec-ish. There are external tools for that, thank you very much. Just give them those things that you (rightly) consider to be expressions. I like the raw. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Sep 25 13:42:18 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 26 Sep 2016 02:42:18 +0900 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > This code works as a doctest today: > > >>> def func(a: "Expected output"): > ... pass > ... > >>> print(func.__annotations__["a"]) > Expected output > > Any change that breaks that currently valid doctest is necessarily a > compatibility break for the way annotations are handled at runtime. It > doesn't matter for that determination how small the change to fix the > second command is, it only matters that it *would* have to change in > some way. This is a bit unfair to ?????, although it's been a long thread so I can understand why some of his ideas have gone missing. His proposals have gotten a bit incoherent because he has been answering all the different objections one by one rather than organizing things into a single design, but I think eventually he would organize it as follows: (1) Add __raw_annotations__ and save the thunked expressions there, whether as code objects or AST. (2) Turn __annotations__ into a property which evaluates (and memoizes?) the thunks and returns them. (First explicitly suggested by Alexander Belopol, I think.) He claims that (2) solves the backward compatibility problem, I don't have the knowledge to figure out whether it is that simple or not. It seems plausible to me, so I'd love to hear an explanation. New ideas like DBC would of course be supported by the new __raw_annotations__ since there's no backward compatibility issue there. I'm still -1 on the whole thing on the principle "although sometimes never is better than *right* now". I think the aClass = "aClass" trick described by Chris is perfectly serviceable to deal with the PEP 484 forward type reference issue. The "let's turn all the annotations into expressions" idea can be practically exploited with ast.parse(). I'm guessing a decorator could be used to provide __raw_annotations__ and __annotations__ per (1) and (2) above (although I'm not sure how to do it myself: copying __annotations__ to __raw_annotations__ and then propertizing __annotations__ could be a bit tricky, I guess). From elazarg at gmail.com Sun Sep 25 13:58:28 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sun, 25 Sep 2016 17:58:28 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: Thank you Stephen. You have phrased my proposal better than I did. As per the using quoted strings, the problems are: 1. The well-formedness of the expression is not checked by the compiler. 2. It is not naturally supported by syntax highlighters and IDEs. They can be made to support it, but most will not. Partly because 3. There is no natural way to distinguish quoted expressions from actual human-readable text (as in the begins library). 4. (My own taste): this is ugly and inconsistent, and there's 2 meaningless characters there :) (6 if multiline) On Sun, Sep 25, 2016 at 8:42 PM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Nick Coghlan writes: > > > This code works as a doctest today: > > > > >>> def func(a: "Expected output"): > > ... pass > > ... > > >>> print(func.__annotations__["a"]) > > Expected output > > > > Any change that breaks that currently valid doctest is necessarily a > > compatibility break for the way annotations are handled at runtime. It > > doesn't matter for that determination how small the change to fix the > > second command is, it only matters that it *would* have to change in > > some way. > > This is a bit unfair to ?????, although it's been a long thread so I > can understand why some of his ideas have gone missing. His proposals > have gotten a bit incoherent because he has been answering all the > different objections one by one rather than organizing things into a > single design, but I think eventually he would organize it as follows: > > (1) Add __raw_annotations__ and save the thunked expressions there, > whether as code objects or AST. > (2) Turn __annotations__ into a property which evaluates (and > memoizes?) the thunks and returns them. (First explicitly > suggested by Alexander Belopol, I think.) > > He claims that (2) solves the backward compatibility problem, I don't > have the knowledge to figure out whether it is that simple or not. It > seems plausible to me, so I'd love to hear an explanation. New ideas > like DBC would of course be supported by the new __raw_annotations__ > since there's no backward compatibility issue there. > > I'm still -1 on the whole thing on the principle "although sometimes > never is better than *right* now". I think the aClass = "aClass" trick > described by Chris is perfectly serviceable to deal with the PEP 484 > forward type reference issue. The "let's turn all the annotations > into expressions" idea can be practically exploited with ast.parse(). > I'm guessing a decorator could be used to provide __raw_annotations__ > and __annotations__ per (1) and (2) above (although I'm not sure how > to do it myself: copying __annotations__ to __raw_annotations__ and > then propertizing __annotations__ could be a bit tricky, I guess). > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Sep 25 14:04:05 2016 From: mertz at gnosis.cx (David Mertz) Date: Sun, 25 Sep 2016 11:04:05 -0700 Subject: [Python-ideas] Fwd: Re: Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: > 1. Please consider disallowing the use of side effects of any kind in annotations, in that it is not promised when it will happen, if at all. So that a change 3 years from now will be somewhat less likely to break things. Please consider doing this for version 3.6; it is feature-frozen, but this is not (yet) a feature, and I got the feeling it is hardly controversial. What you are proposing is a very large and controversial feature change. Just repeating that you think there documented 10-year old behavior is an accident doesn't make it so. This is ABSOLUTELY impossible for 3.6. my feeling is is probably oppose it for 3.7, bit that is the soonest timeframe that is even conceivable. Here's a quick, only slightly contrived, example where i want an annotation to have a side effect: def foo(a, b) -> logdef(): ... This might be used as part of a tracing, cod coverage, or timing library. Obviously it would need to do some magic to get more info about the function being logged, bit that's possible. And yes, I can think of other ways to achieve this effect (decorators maybe), but you want to make change to prohibit this use that might be in production for ten years. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Sep 25 14:28:13 2016 From: mertz at gnosis.cx (David Mertz) Date: Sun, 25 Sep 2016 11:28:13 -0700 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sep 25, 2016 10:59 AM, "?????" wrote: > 2. It is not naturally supported by syntax highlighters and IDEs. They can be made to support it, but most will not. This is a complete red herring. Having a highlight rule of "apply highlights in string annotations" is straightforward in modern editors. This is like arguing Python should do because Notepad.exe doesn't do something smart with it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Sun Sep 25 14:54:21 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sun, 25 Sep 2016 18:54:21 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sun, Sep 25, 2016 at 9:28 PM David Mertz wrote: > On Sep 25, 2016 10:59 AM, "?????" wrote: > > 2. It is not naturally supported by syntax highlighters and IDEs. They > can be made to support it, but most will not. > > This is a complete red herring. Having a highlight rule of "apply > highlights in string annotations" is straightforward in modern editors. > This is like arguing Python should do because Notepad.exe > doesn't do something smart with it. > Not that I think it's a killer argument, but why a red herring? Quick search does not find such an explicit option in Gedit, PyDev and yes, Notepad++.exe. It is not a common or default option. Having such a rule by default amounts to admitting that these are not essentially strings, and the quotes there are overloaded. It also means that actual strings are not understood as such, and are incorrectly highlighted. But please let's not delve into this: it is of some importance, but should not affect an actual decision. IDEs are more important. Renaming facilities do over-renaming or under-renaming because of this need to rename inside some strings, but not inside others. Similarly code search facilities, and warnings from IDEs about inlining variables. I have encountered real bugs caused by such an imperfect renaming (and I hope your answer is not "don't do renaming"). A prefix like code"foo()" might help of course, but it is not really used as a string. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Sep 25 14:58:17 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Sep 2016 04:58:17 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: On Mon, Sep 26, 2016 at 4:54 AM, ????? wrote: > IDEs are more important. Renaming facilities do over-renaming or > under-renaming because of this need to rename inside some strings, but not > inside others. Similarly code search facilities, and warnings from IDEs > about inlining variables. I have encountered real bugs caused by such an > imperfect renaming (and I hope your answer is not "don't do renaming"). Which is why refactoring CANNOT be fully automated. Ever. String literals and comments may need to be updated, but they also may be coincidental. However, this is entirely beside the point. ChrisA From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Sep 25 15:00:47 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 26 Sep 2016 04:00:47 +0900 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: <22504.7903.90253.116922@turnbull.sk.tsukuba.ac.jp> ????? writes: > But nobody (yet) claimed it to be a reasonable example of code we > don't want to break. "Reasonable example" is not the standard. The ideal is that *nobody*'s code breaks unless it's necessary to to fix a bug. The current implementation conforms to the specification[1], and therefore the proposed change is not a bugfix. The Yale Book of Quotations quotes English judge Robert Megarry as follows: "Whereas in England all is permitted that is not expressly prohibited, it has been said that in Germany all is prohibited unless expressly permitted and in France all is permitted that is expressly prohibited. In the European Common Market no-one knows what is permitted and it all costs more." http://freakonomics.com/2009/10/29/quotes-uncovered-death-and-statistics/ Python, of course, follows the principle of English law. That's what we mean by "consenting adults". The rules about change are more flexible in the stdlib, but even there we get reports every release about breakage due to improvements in various modules. This is the language definition, so "if you can do it in vX.Y, it should do the same in vX.(Y+1)" is a strict rule.[2] Footnotes: [1] Assuming, as I do, that in PEP 3107 "expression" refers only to the syntax specification and does not at all imply adding a new expression type to the language. What is stored in __annotations__ is thus implied to be the object that is the value of the expression, following the precedent of initialization, and the general Pythonic approach of evaluating expressions when encountered. And that semantics is stated explicitly in PEP 3107. [2] The definition of "do the same" does not necessarily mean "produce identical output", eg, in the case of "dir()" in the bare interpreter with no imports. From mertz at gnosis.cx Sun Sep 25 15:14:01 2016 From: mertz at gnosis.cx (David Mertz) Date: Sun, 25 Sep 2016 12:14:01 -0700 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sep 25, 2016 11:54 AM, "?????" wrote: > Not that I think it's a killer argument, but why a red herring? Quick search does not find such an explicit option in Gedit, PyDev and yes, Notepad++.exe. It is not a common or default option. This would equally be an argument against any syntax change, any new keyword, and likewise any new built-in. Depending on what your editor does, it would perhaps be an argument against new common names in the standard library. All of those might promote changes in tools. But the tools exist to aid working with a language, not the other way around. I would not want my editor to do that much highlighting distinction, but assuming you do, all your comments defeat your own point. If an expression being within an annotation merits different evaluation semantics, it is a vastly smaller change to say a string being in an annotation merits different syntax highlighting. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Sun Sep 25 15:41:38 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sun, 25 Sep 2016 19:41:38 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sun, Sep 25, 2016 at 10:14 PM David Mertz wrote: > All of those might promote changes in tools. But the tools exist to aid > working with a language, not the other way around. > > I can't think of a way to know that this string is code and that string is an actual string. And figuring out means finding context dependency. I would not want my editor to do that much highlighting distinction, but > assuming you do, all your comments defeat your own point. If an expression > being within an annotation merits different evaluation semantics, it is a > vastly smaller change to say a string being in an annotation merits > different syntax highlighting. > It is a smaller change, but of a very different kind. I think annotations *theoretically* deserve different syntax (e.g. using the arrow notation) but keeping it an Expression keeps things simpler. Just like type(type) is not type, *theoretically*, but merging these objects is a very smart decision. So yes, I think it should have (and already has) different semantics; but its useful to have the same syntax (and related semantics). Simple keyword-highlighting "except inside quotes" should still be very useful. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Sep 25 22:14:44 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Sep 2016 12:14:44 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> Message-ID: <20160926021444.GI22471@ando.pearwood.info> On Sun, Sep 25, 2016 at 01:55:09AM +0000, ????? wrote: > 1. Please consider disallowing the use of side effects of any kind in > annotations, That is *simply not possible* in Python. Actually, no, that's not quite correct. One way to prohibit side-effects would be to make all annotations string literals, and ONLY string literals. Or possibly bare names (assuming current semantics for local variable name lookup): def func(arg:'no possible side effects here') -> OrHere: ... But as soon as allow such things as union types and lists, then all bets are off: def func(arg:Sequence[list]): ... There is no way of prohibiting side effects in type(Sequence).__getitem__ once it is called. Nor would we want to. The ability to shadow or monkey-patch types for mocking, testing, debugging etc, including the ability to have them call print, or perform logging, is a feature beyond price. We don't need it often, but when we do, the ability to replace Sequence with a mock that may have side-effects is really useful. > in that it is not promised when it will happen, if at all. So > that a change 3 years from now will be somewhat less likely to break > things. Please consider doing this for version 3.6; it is feature-frozen, > but this is not (yet) a feature, It has been a feature since Python 3.0 that annotations are evaluated at runtime. And that means the possibility of side-effects. So, yes, it is already a feature. Even if you get the behaviour that you want, the absolute earliest it could happen would be after a deprecation period of at least one point release. That means: * 3.7 introduces a DeprecationWarning whenever you use annotations which aren't simple names or strings; * and possibly a __future__ import to give the new behaviour; * and 3.8 would be the earliest it could be mandatory. Forget about 3.6 -- that's already frozen apart from bug fixes, and this is not a bug. > and I got the feeling it is hardly controversial. It is extremely controversial. The fact that you can say that it isn't suggests that you're not really paying attention to what we're saying. Even if what you ask for is easy (it isn't), or even possible, it still goes completely and utterly against the normal semantics of Python and the philosophy of the language. No, under normal circumstances nobody is going to write: def func(arg: mylist.append(value) or int): ... in production code. That's simply bad style. But we don't ban things just because they are bad style. Circumstances are not always normal, sometimes it is useful to use dirty hacks (but hopefully not in production code), and Python is not a B&D language where everything is prohibited unless explicitly allowed. > I really have no interest in wasting the time of anybody here. And yet, despite receiving virtually no interest from any other person, you continue to loudly and frequently argue for this proposal. [...] > 2. A refined proposal for future versions of the language: the ASTs of the > annotation-expressions will be bound to __raw_annotations__. > * This is actually more in line to what PEP-3107 was about ("no assigned > semantics"; except for a single sentence, it is only about expressions. Not > objects). All expressions evaluate to a value. And all values in Python are objects. I don't understand what distinction you think you are making here. Are you suggesting that Python should gain some sort of values which aren't objects? > * This is helpful even if the expression is evaluated at definition > time, and can help in smoothing the transformation. > > 3. The main benefit from my proposal is that contracts (examples, > explanations, assertions, and types) are naturally expressible as (almost) > arbitrary Python expressions, but not if they are evaluated or evaluatable, > at definition time, by the interpreter. Why: because it is really written > in a different language - *always*. Wrong. Not always. The proof is that Python exists. Contracts, types, assertions etc in Python *are* written in Python. That's the end of the story. You cannot argue that "contracts are written in a different language" because that is untrue. Contracts are written in Python, and we wouldn't have it any other way. > This is the real reason behind the > existence, and the current solutions, of the forward reference problem. In > general it is much more flexible than current situation. The forward reference problem still exists in languages where type declarations are a separate language, e.g. Pascal, C++, Java, etc. http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4754974 http://stackoverflow.com/questions/951234/forward-declaration-of-nested-types-classes-in-c etc. There are many ways around it. One way is to make the language so simple that forward declarations aren't relevant. Another is to make multiple passes over the source code. Another is to introduce an explicit "forward" declaration, as in some dialects of Pascal. Python uses strings. > I believe the last point has a very good reason, as explained later: it is > an interpretation of a different language, But it *isn't* such a thing, nor should it be. > foreign to the interpreter, > although sometimes close enough to be useful. Sometimes close enough to be useful. Does that mean it is usually useless? *wink* > It is of course well formed, > so the considerations are not really security-related. You've talked about eval'ing the contents of __raw_annotations__. That means if somebody can fool you into storing arbitrary values into __raw_annotations__, then get you to call annotations() or use inspect, they can execute arbitrary code. How is this not a security concern? It might be hard to exploit, since it requires the victim to do something like: myfunc.__raw_annotations__['arg'] = something_untrusted but if exploited, the consequences are major: full eval of arbitrary code. In comparison, the only similar threat with annotations today is if the victim is fooled into building a string containing a def with annotations, then passing it to exec: annot = something_untrusted code = """def func(arg: %s): ... """ % annot exec(code) but if you're using exec on an untrusted string you have already lost. So annotations as they exist now aren't adding any new vulnerabilities. Still, the important thing here is not the (hard to exploit) potential vulerability, but the fact that your proposal would lead to a massive increase in the complexity of the language (a whole new compiler/ iterpreter for the second, types-only, mini-language) and an equally major *decrease* in useful functionality. Have I mentioned that I'm against this? If not, I'm against it. -- Steve From ncoghlan at gmail.com Mon Sep 26 00:57:36 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2016 14:57:36 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: On 26 September 2016 at 03:42, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > This code works as a doctest today: > > > > >>> def func(a: "Expected output"): > > ... pass > > ... > > >>> print(func.__annotations__["a"]) > > Expected output > > > > Any change that breaks that currently valid doctest is necessarily a > > compatibility break for the way annotations are handled at runtime. It > > doesn't matter for that determination how small the change to fix the > > second command is, it only matters that it *would* have to change in > > some way. > > This is a bit unfair to ?????, although it's been a long thread so I > can understand why some of his ideas have gone missing. His proposals > have gotten a bit incoherent because he has been answering all the > different objections one by one rather than organizing things into a > single design, but I think eventually he would organize it as follows: > > (1) Add __raw_annotations__ and save the thunked expressions there, > whether as code objects or AST. > (2) Turn __annotations__ into a property which evaluates (and > memoizes?) the thunks and returns them. (First explicitly > suggested by Alexander Belopol, I think.) > > He claims that (2) solves the backward compatibility problem, I don't > have the knowledge to figure out whether it is that simple or not. It > seems plausible to me, so I'd love to hear an explanation. New ideas > like DBC would of course be supported by the new __raw_annotations__ > since there's no backward compatibility issue there. OK, that does indeed make more sense, and significantly reduces the scope for potential runtime compatibility breaks related to __annotations__ access. Instead, it changes the discussion to focus on the following main challenges: - the inconsistency introduced between annotations (lazily evaluated) and default arguments (eagerly evaluated) - the remaining compatibility breaks (depending on implementation details) - the runtime overhead of lazy evaluation - the debugging challenges of lazy evaluation The inconsistency argument is simply that people will be even more confused than they are today if default arguments are evaluated at definition time while annotations aren't. There is a lot of code out there that actively relies on eager evaluation of default arguments, so changing that is out of the question, which then provides a strong consistency argument in favour of keeping annotations eagerly evaluated as well. There would likely still be some compatibility breaks around name access in method annotation definitions, and compatibility would also break for any code that actually did expect to trigger a side-effect at definition time. This is a much smaller scope for breakage than breaking __annotations__ access, but we can't assume it won't affect anyone as there's a lot of code out there that we'd judge to be questionable from the point of view of maintainability and good design aesthetics that nevertheless still solves the problem the author was aiming to solve. The runtime memory overhead of lazy evaluation isn't trivial. Using a naive function based approach: >>> import sys >>> sys.getsizeof("") 49 >>> sys.getsizeof(lambda: "") 136 And that's only the function object itself - it's not counting all the other objects hanging off the function object like the attribute dictionary. A more limited thunk type could reduce that overhead, but it's still going to be larger in most cases than just storing the evaluation result. The impact on runtime speed overhead is less certain, but also likely to be a net negative - defining functions isn't particularly cheap (especially compared to literal references or a simple name lookup), and calling them if you actually access __annotations__ isn't going to be particularly cheap either. The debugging challenge is the same one that arises with any form of delayed evaluation: by default, the traceback you get will point you to the location where the delayed evaluation took place *not* the location where the flawed expression was found. That problem can be mitigated through an exception chaining design that references the likely location of the actual error, but it's never going to be as easy to figure out as cases where the traceback points directly at the code responsible for the problem. So I'm still -1 on the idea, but it's not as straightforward as the argument against the naive version of the proposal that also broke __annotations__ lookup. Cheers, Nick. P.S. As an illustration of that last point, the PEP 487 implementation currently makes problems with __set_name__ attribute definitions quite hard to figure out since the traceback points at the class definition header, rather than the offending descriptor assignment: http://bugs.python.org/issue28214 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From elazarg at gmail.com Mon Sep 26 06:20:37 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 26 Sep 2016 10:20:37 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: Thank you all. I think this thread is pretty much close by now. I understand at least most of your concerns and I will take time to shape my idea. I wanted to note one last thing, though, regarding my claim that annotations are not actually standard expressions: Guido had once expressed his concerns regarding the performance hit of using cast(), since it is not easily (or at all) optimized away. This performance hit should not be there in the first place, if the distinction between annotations and evaluatable expressions was kept - i.e. by allowing the attachment of annotations to expressions (as I believe was proposed several times). Now, I understand that there are very good reasons not to allow it; keeping the language simple and familiar would be my first guess - but note how the semantics of the "main" language is hindered by the complexities of its specification-related syntactic subset, which is not due, in my opinion. If you want to specify things, the syntactic hit is unavoidable, but the semantic hit is not. (BTW why isn't it written cast[T](exp) ?) Thank you again for this discussion Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Sep 26 08:46:57 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Sep 2016 22:46:57 +1000 Subject: [Python-ideas] Thunks (lazy evaluation) [was Re: Delay evaluation of annotations] In-Reply-To: References: <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> Message-ID: <20160926124657.GJ22471@ando.pearwood.info> Let's talk about lazy evaluation in a broader sense that just function annotations. If we had syntax for lazy annotation -- let's call them thunks, after Algol's thunks -- then we could use them in annotations as well as elsewhere. But if we special case annotations only, the Zen has something to say about special cases. On Mon, Sep 26, 2016 at 02:57:36PM +1000, Nick Coghlan wrote: [...] > OK, that does indeed make more sense, and significantly reduces the > scope for potential runtime compatibility breaks related to > __annotations__ access. Instead, it changes the discussion to focus on > the following main challenges: > > - the inconsistency introduced between annotations (lazily evaluated) > and default arguments (eagerly evaluated) > - the remaining compatibility breaks (depending on implementation details) > - the runtime overhead of lazy evaluation > - the debugging challenges of lazy evaluation Default arguments are a good use-case for thunks. One of the most common gotchas in Python is early binding of function defaults: def func(arg=[]): ... Nine times out of ten, that's probably not what you want. Now, to avoid all doubt, I do not want to change function defaults to late binding. I've argued repeatedly on comp.lang.python and elsewhere that if a language only offers one of early binding or late binding, it should offer early binding as Python does. The reason is, given early binding, it it trivial to simulate something like late binding: def func(arg=None): if arg is None: arg = [] ... but given late binding, it is ugly and inconvenient to get a poor substitute for early binding when that's what you want. So, please, let's not have a debate over the behaviour of function defaults. But what if we could have both? Suppose we use backticks `...` to make a thunk, then we could write: def func(arg=`[]`): ... to get the late binding result wanted. Are there other uses for thunks? Potentially, they could be used for Ruby-like code blocks: result = function(arg1, arg2, block=```# triple backticks do_this() do_that() while condition: do_something_else() print('Done') ```, another_arg=1) but then I'm not really sure what advantage code blocks have over functions. > The inconsistency argument is simply that people will be even more > confused than they are today if default arguments are evaluated at > definition time while annotations aren't. There is a lot of code out > there that actively relies on eager evaluation of default arguments, > so changing that is out of the question, which then provides a strong > consistency argument in favour of keeping annotations eagerly > evaluated as well. Indeed. There are only (to my knowledge) only two places where Python delays evaluation of code: - functions (def statements and lambda expressions); - generator expressions; where the second can be considered to be syntactic sugar for a generator function (def with yield). Have I missed anything? In the same way that Haskell is fundamentally built on lazy evaluation, Python is fundamentally built on eager evaluation, and I don't think we should change that. Until now, the only way to delay the evaluation of code (other than the body of a function, of course) is to write it as a string, then pass it to eval/exec. Thunks offer an alternative for delayed evaluation that makes it easier for editors to apply syntax highlighting: don't apply it to ordinary strings, but do apply it to thunks. I must admit that I've loved the concept of thunks for years now, but I'm still looking for the killer use-case for them, the one clear justification for why Python should include them. - Late-bound function default arguments? Nice to have, but we already have a perfectly serviceable way to get the equivalent behaviour. - Code blocks? Maybe a Ruby programmer can explain why they're so important, but we have functions, including lambda. - Function annotations? I'm not convinced thunks are needed or desirable for annotations. - A better way to write code intended for delayed execution? Sounds interesting, but not critical. Maybe somebody else can think of the elusive killer use-case for thunks, because I've been pondering this question for many years now and I'm no closer to an answer. > There would likely still be some compatibility breaks around name > access in method annotation definitions, and compatibility would also > break for any code that actually did expect to trigger a side-effect > at definition time. This is a much smaller scope for breakage than > breaking __annotations__ access, but we can't assume it won't affect > anyone as there's a lot of code out there that we'd judge to be > questionable from the point of view of maintainability and good design > aesthetics that nevertheless still solves the problem the author was > aiming to solve. It's not just published code. It's also one-off throw-away code, including code executed in the interactive interpreter then thrown away. It is really helpful to be able to monkey-patch or shadow builtins, insert some logging code or even a few print statements, or perhaps something that modifies and global variable, for debugging or to learn how something works. Could it be equally useful inside annotations? I expect so... complicated only by the fact that one needs to monkey-patch the *metaclass*, not the type itself. It may be that I'm completely off-base here and this is a stupid thing to do. But I say that until the community has more experience with annotations, we shouldn't rule it out. (Just to be clear: I'm mostly talking about interactive exploration of code, not production code. Python is not Ruby and we don't encourage heavy use of monkey-patching in production code. But it has its uses.) > The runtime memory overhead of lazy evaluation isn't trivial. Using a > naive function based approach: > > >>> import sys > >>> sys.getsizeof("") > 49 > >>> sys.getsizeof(lambda: "") > 136 > > And that's only the function object itself - it's not counting all the > other objects hanging off the function object like the attribute > dictionary. A more limited thunk type could reduce that overhead, but > it's still going to be larger in most cases than just storing the > evaluation result. This is better: py> sys.getsizeof((lambda: "").__code__) 80 > The impact on runtime speed overhead is less certain, but also likely > to be a net negative - defining functions isn't particularly cheap > (especially compared to literal references or a simple name lookup), > and calling them if you actually access __annotations__ isn't going to > be particularly cheap either. > > The debugging challenge is the same one that arises with any form of > delayed evaluation: by default, the traceback you get will point you > to the location where the delayed evaluation took place *not* the > location where the flawed expression was found. That problem can be > mitigated through an exception chaining design that references the > likely location of the actual error, but it's never going to be as > easy to figure out as cases where the traceback points directly at the > code responsible for the problem. Nevertheless, an explicit thunk syntax will make this a matter of consenting adults: if you choose to shoot your foot off with a hard-to-debug thunk, you have nobody to blame but yourself. Or whoever wrote the library that you're using. *wink* -- Steve From steve at pearwood.info Mon Sep 26 09:05:07 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Sep 2016 23:05:07 +1000 Subject: [Python-ideas] Thunks (lazy evaluation) [was Re: Delay evaluation of annotations] In-Reply-To: <20160926124657.GJ22471@ando.pearwood.info> References: <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> <20160926124657.GJ22471@ando.pearwood.info> Message-ID: <20160926130507.GK22471@ando.pearwood.info> On Mon, Sep 26, 2016 at 10:46:57PM +1000, Steven D'Aprano wrote: > Let's talk about lazy evaluation in a broader sense that just function > annotations. > > If we had syntax for lazy annotation -- let's call them thunks, after > Algol's thunks Er, that should be lazy evaluation. Or at least delayed evaluation. > -- then we could use them in annotations as well as > elsewhere. But if we special case annotations only, the Zen has > something to say about special cases. -- Steve From sjoerdjob at sjoerdjob.com Mon Sep 26 09:27:03 2016 From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus) Date: Mon, 26 Sep 2016 15:27:03 +0200 Subject: [Python-ideas] Thunks (lazy evaluation) [was Re: Delay evaluation of annotations] In-Reply-To: <20160926124657.GJ22471@ando.pearwood.info> References: <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> <20160926124657.GJ22471@ando.pearwood.info> Message-ID: <20160926132703.GG1775@sjoerdjob.com> On Mon, Sep 26, 2016 at 10:46:57PM +1000, Steven D'Aprano wrote: > Let's talk about lazy evaluation in a broader sense that just function > annotations. > > If we had syntax for lazy annotation -- let's call them thunks, after > Algol's thunks -- then we could use them in annotations as well as > elsewhere. But if we special case annotations only, the Zen has > something to say about special cases. > > > On Mon, Sep 26, 2016 at 02:57:36PM +1000, Nick Coghlan wrote: > [...] > > OK, that does indeed make more sense, and significantly reduces the > > scope for potential runtime compatibility breaks related to > > __annotations__ access. Instead, it changes the discussion to focus on > > the following main challenges: > > > > - the inconsistency introduced between annotations (lazily evaluated) > > and default arguments (eagerly evaluated) > > - the remaining compatibility breaks (depending on implementation details) > > - the runtime overhead of lazy evaluation > > - the debugging challenges of lazy evaluation > > > Default arguments are a good use-case for thunks. One of the most common > gotchas in Python is early binding of function defaults: > > def func(arg=[]): > ... > > Nine times out of ten, that's probably not what you want. Now, to avoid > all doubt, I do not want to change function defaults to late binding. > I've argued repeatedly on comp.lang.python and elsewhere that if a > language only offers one of early binding or late binding, it should > offer early binding as Python does. The reason is, given early binding, > it it trivial to simulate something like late binding: > > def func(arg=None): > if arg is None: > arg = [] > ... > > but given late binding, it is ugly and inconvenient to get a poor > substitute for early binding when that's what you want. So, please, > let's not have a debate over the behaviour of function defaults. > > But what if we could have both? Suppose we use backticks `...` to make a > thunk, then we could write: > > def func(arg=`[]`): > ... > > to get the late binding result wanted. > > Are there other uses for thunks? Potentially, they could be used for > Ruby-like code blocks: > > result = function(arg1, arg2, block=```# triple backticks > do_this() > do_that() > while condition: > do_something_else() > print('Done') > ```, > another_arg=1) > > > but then I'm not really sure what advantage code blocks have over > functions. > > > > The inconsistency argument is simply that people will be even more > > confused than they are today if default arguments are evaluated at > > definition time while annotations aren't. There is a lot of code out > > there that actively relies on eager evaluation of default arguments, > > so changing that is out of the question, which then provides a strong > > consistency argument in favour of keeping annotations eagerly > > evaluated as well. > > Indeed. There are only (to my knowledge) only two places where Python > delays evaluation of code: > > - functions (def statements and lambda expressions); > - generator expressions; > > where the second can be considered to be syntactic sugar for a generator > function (def with yield). Have I missed anything? > > In the same way that Haskell is fundamentally built on lazy evaluation, > Python is fundamentally built on eager evaluation, and I don't think we > should change that. > > Until now, the only way to delay the evaluation of code (other than the > body of a function, of course) is to write it as a string, then pass it > to eval/exec. Thunks offer an alternative for delayed evaluation that > makes it easier for editors to apply syntax highlighting: don't apply it > to ordinary strings, but do apply it to thunks. > > I must admit that I've loved the concept of thunks for years now, but > I'm still looking for the killer use-case for them, the one clear > justification for why Python should include them. > > - Late-bound function default arguments? Nice to have, but we already > have a perfectly serviceable way to get the equivalent behaviour. > > - Code blocks? Maybe a Ruby programmer can explain why they're so > important, but we have functions, including lambda. > > - Function annotations? I'm not convinced thunks are needed or desirable > for annotations. > > - A better way to write code intended for delayed execution? Sounds > interesting, but not critical. > > Maybe somebody else can think of the elusive killer use-case for thunks, > because I've been pondering this question for many years now and I'm no > closer to an answer. Well, there's a use-case I have been pondering for a long while now which could be satisfied by this: enumerated generator displays. So suppose you have a composite boolean value, composed by the 'and' of many conditions (which all take long to compute), and you want to short-circuit. Let's take the following example. valid = True valid &= looks_like_emailaddress(username) valid &= more_than_8_characters(password) valid &= does_not_exist_in_database(username) valid &= domain_name_of_emailaddress_has_mx_record(username) ... some more options ... (I forgot the exact use-case, but I still remember the functionality I wanted, so bear with me). Of course, the above is not short-circuiting, so it would be replaced by def check_valid(username, password): if not looks_like_emailaddress(username): return False if not more_than_8_characters(password): return False if not does_not_exist_in_database(username): return False if not domain_name_of_emailaddress_has_mx_record(username): return False ... return True valid = check_valid() or valid = True\ and looks_like_emailaddress(username)\ and more_than_8_characters(password)\ and does_not_exist_in_database(username)\ and domain_name_of_emailaddress_has_mx_record(username) But in all reality, I want to write something like: valid = all(@@@ looks_like_emailaddress(username), more_than_8_characters(password), does_not_exist_in_database(username), domain_name_of_emailaddress_has_mx_record(username), @@@) With `@@@` designating the beginning/ending of the enumerated generator display. Now, this is currently not possible, but if we had some kind of thunk syntax that would become possible, without needing an enumerated generator display. However the problem I see with the concept of `thunk` is: When does it get un-thunked? In which of the following cases? 1. When getting an attribute on it? 2. When calling it? --> See 1. with `__call__`. 3. When subindexing it? --> See 1. with `__getitem__`. 4. When assigning it to a name? It shouldn't have to be un-thunked, I think. 5. When adding it to a list? No un-thunking should be necessary, I think. However, the problem with thunks is (I think) that to make that happen either - *all* objects need to include yet another level of redirection, or - a thunk needs to get allocated the maximum size of the value it could possibly store. (But a `unicode` object could have an arbitrary size) or - there needs to be some way to 'notify' objects holding the thunk that its value got updated. For a dict/list/tuple this could readily grow into O(n) behaviour when un-thunking a thunk. or - any C-level functionality needs to learn how to deal with thunks. For instance, `Py_TYPE` would have to *resolve* the thunk, and then return the type of the value. or - I'm running out of ideas here, but maybe creating a custom type object for each thunk that does pass-through to a wrapped item? Thunked objects would work *exactly* the same as normal objects, but at a (small) indirection for any action taken. Still, somehow `Py_TYPE` and `Py_SIZE` and any other macros would still have to force evaluation. Kind regards, Sjoerd Job From joejev at gmail.com Mon Sep 26 10:04:52 2016 From: joejev at gmail.com (Joseph Jevnik) Date: Mon, 26 Sep 2016 10:04:52 -0400 Subject: [Python-ideas] Thunks (lazy evaluation) [was Re: Delay evaluation of annotations] In-Reply-To: <20160926132703.GG1775@sjoerdjob.com> References: <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> <20160926124657.GJ22471@ando.pearwood.info> <20160926132703.GG1775@sjoerdjob.com> Message-ID: Hello everyone, this idea looks like something I have tried building already: https://github.com/llllllllll/lazy_python. This project implements a `thunk` class which builds up a deferred computation which is evaluated only when needed. One use case I have had for this project is building up a larger expression so that it may be simplified and then computed concurrently with dask: http://daisy-python.readthedocs.io/en/latest/. By building up a larger expression (and making the tree accessible) users have the ability to remove common subexpressions or remove intermediate objects. In numpy chained expressions often make lots of allocations which are quickly thrown away which is why projects like numexpr ( https://github.com/pydata/numexpr) can be such a serious speed up. These intermediates are required because the whole expression isn't known at the start so it must be evaluated as written. Things to consider about when to evaluate: 1. Functions which branch on their input need to know which branch to select. 2. Iteration is really hard to defer in a way that is efficient. lazy_python just eagerly evaluates at iteration time but builds thunks in the body. 3. Stateful operations like IO which normally have an implied order of operation now need some explicit ordering. Regarding the `Py_TYPE` change: I don't think that is correct unless we made a thunk have the same binary representation as the underlying object. A lot of code does a type check and then calls macros that act on the actual type like `PyTuple_GET_ITEM` so we cannot fool C functions very easily. On Mon, Sep 26, 2016 at 9:27 AM, Sjoerd Job Postmus wrote: > On Mon, Sep 26, 2016 at 10:46:57PM +1000, Steven D'Aprano wrote: > > Let's talk about lazy evaluation in a broader sense that just function > > annotations. > > > > If we had syntax for lazy annotation -- let's call them thunks, after > > Algol's thunks -- then we could use them in annotations as well as > > elsewhere. But if we special case annotations only, the Zen has > > something to say about special cases. > > > > > > On Mon, Sep 26, 2016 at 02:57:36PM +1000, Nick Coghlan wrote: > > [...] > > > OK, that does indeed make more sense, and significantly reduces the > > > scope for potential runtime compatibility breaks related to > > > __annotations__ access. Instead, it changes the discussion to focus on > > > the following main challenges: > > > > > > - the inconsistency introduced between annotations (lazily evaluated) > > > and default arguments (eagerly evaluated) > > > - the remaining compatibility breaks (depending on implementation > details) > > > - the runtime overhead of lazy evaluation > > > - the debugging challenges of lazy evaluation > > > > > > Default arguments are a good use-case for thunks. One of the most common > > gotchas in Python is early binding of function defaults: > > > > def func(arg=[]): > > ... > > > > Nine times out of ten, that's probably not what you want. Now, to avoid > > all doubt, I do not want to change function defaults to late binding. > > I've argued repeatedly on comp.lang.python and elsewhere that if a > > language only offers one of early binding or late binding, it should > > offer early binding as Python does. The reason is, given early binding, > > it it trivial to simulate something like late binding: > > > > def func(arg=None): > > if arg is None: > > arg = [] > > ... > > > > but given late binding, it is ugly and inconvenient to get a poor > > substitute for early binding when that's what you want. So, please, > > let's not have a debate over the behaviour of function defaults. > > > > But what if we could have both? Suppose we use backticks `...` to make a > > thunk, then we could write: > > > > def func(arg=`[]`): > > ... > > > > to get the late binding result wanted. > > > > Are there other uses for thunks? Potentially, they could be used for > > Ruby-like code blocks: > > > > result = function(arg1, arg2, block=```# triple backticks > > do_this() > > do_that() > > while condition: > > do_something_else() > > print('Done') > > ```, > > another_arg=1) > > > > > > but then I'm not really sure what advantage code blocks have over > > functions. > > > > > > > The inconsistency argument is simply that people will be even more > > > confused than they are today if default arguments are evaluated at > > > definition time while annotations aren't. There is a lot of code out > > > there that actively relies on eager evaluation of default arguments, > > > so changing that is out of the question, which then provides a strong > > > consistency argument in favour of keeping annotations eagerly > > > evaluated as well. > > > > Indeed. There are only (to my knowledge) only two places where Python > > delays evaluation of code: > > > > - functions (def statements and lambda expressions); > > - generator expressions; > > > > where the second can be considered to be syntactic sugar for a generator > > function (def with yield). Have I missed anything? > > > > In the same way that Haskell is fundamentally built on lazy evaluation, > > Python is fundamentally built on eager evaluation, and I don't think we > > should change that. > > > > Until now, the only way to delay the evaluation of code (other than the > > body of a function, of course) is to write it as a string, then pass it > > to eval/exec. Thunks offer an alternative for delayed evaluation that > > makes it easier for editors to apply syntax highlighting: don't apply it > > to ordinary strings, but do apply it to thunks. > > > > I must admit that I've loved the concept of thunks for years now, but > > I'm still looking for the killer use-case for them, the one clear > > justification for why Python should include them. > > > > - Late-bound function default arguments? Nice to have, but we already > > have a perfectly serviceable way to get the equivalent behaviour. > > > > - Code blocks? Maybe a Ruby programmer can explain why they're so > > important, but we have functions, including lambda. > > > > - Function annotations? I'm not convinced thunks are needed or desirable > > for annotations. > > > > - A better way to write code intended for delayed execution? Sounds > > interesting, but not critical. > > > > Maybe somebody else can think of the elusive killer use-case for thunks, > > because I've been pondering this question for many years now and I'm no > > closer to an answer. > > Well, there's a use-case I have been pondering for a long while now > which could be satisfied by this: enumerated generator displays. > > So suppose you have a composite boolean value, composed by the 'and' of > many conditions (which all take long to compute), and you want to > short-circuit. Let's take the following example. > > valid = True > valid &= looks_like_emailaddress(username) > valid &= more_than_8_characters(password) > valid &= does_not_exist_in_database(username) > valid &= domain_name_of_emailaddress_has_mx_record(username) > ... some more options ... > > (I forgot the exact use-case, but I still remember the functionality I > wanted, so bear with me). > > Of course, the above is not short-circuiting, so it would be replaced by > > def check_valid(username, password): > if not looks_like_emailaddress(username): return False > if not more_than_8_characters(password): return False > if not does_not_exist_in_database(username): return False > if not domain_name_of_emailaddress_has_mx_record(username): return > False > ... > return True > > > valid = check_valid() > > or > > valid = True\ > and looks_like_emailaddress(username)\ > and more_than_8_characters(password)\ > and does_not_exist_in_database(username)\ > and domain_name_of_emailaddress_has_mx_record(username) > > But in all reality, I want to write something like: > > valid = all(@@@ > looks_like_emailaddress(username), > more_than_8_characters(password), > does_not_exist_in_database(username), > domain_name_of_emailaddress_has_mx_record(username), > @@@) > > With `@@@` designating the beginning/ending of the enumerated generator > display. > > Now, this is currently not possible, but if we had some kind of thunk > syntax that would become possible, without needing an enumerated > generator display. > > However the problem I see with the concept of `thunk` is: When does it > get un-thunked? In which of the following cases? > > 1. When getting an attribute on it? > 2. When calling it? --> See 1. with `__call__`. > 3. When subindexing it? --> See 1. with `__getitem__`. > 4. When assigning it to a name? It shouldn't have to be un-thunked, I > think. > 5. When adding it to a list? No un-thunking should be necessary, I > think. > > However, the problem with thunks is (I think) that to make that happen > either > > - *all* objects need to include yet another level of redirection, > or > - a thunk needs to get allocated the maximum size of the value it could > possibly store. (But a `unicode` object could have an arbitrary size) > or > - there needs to be some way to 'notify' objects holding the thunk that > its value got updated. For a dict/list/tuple this could readily grow > into O(n) behaviour when un-thunking a thunk. > or > - any C-level functionality needs to learn how to deal with thunks. For > instance, `Py_TYPE` would have to *resolve* the thunk, and then return > the type of the value. > or > - I'm running out of ideas here, but maybe creating a custom type object > for each thunk that does pass-through to a wrapped item? Thunked > objects would work *exactly* the same as normal objects, but at a > (small) indirection for any action taken. Still, somehow `Py_TYPE` and > `Py_SIZE` and any other macros would still have to force evaluation. > > Kind regards, > Sjoerd Job > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Mon Sep 26 12:04:06 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 26 Sep 2016 16:04:06 +0000 Subject: [Python-ideas] Thunks (lazy evaluation) [was Re: Delay evaluation of annotations] In-Reply-To: References: <20160923030616.GC22471@ando.pearwood.info> <20160923121004.GD22471@ando.pearwood.info> <22504.3194.984800.442940@turnbull.sk.tsukuba.ac.jp> <20160926124657.GJ22471@ando.pearwood.info> <20160926132703.GG1775@sjoerdjob.com> Message-ID: You already know I want this for contracts etc.. Here some things that I consider important: 1. There should be some way to bind the names to function parameters, as in @contract def invert(x: `x != 0`) -> float: return 1 / x @contract def invertdiff(x: int, y: `x != y`) -> float: return 1 / (x-y) 2. For this and other reasons, the AST should be available. I think it can be a single AST per place in code, but it should be immutable. 3. Backticks are problematic because they cannot be nested. I suggest (name: ) or ('name': expression). This name can be googled. def compose(f: `such_that: pure(f)`, g: `such_that: pure(g)`): return lambda x: f(g(x)) 4. I think it's a bad idea to use thunks as DSL (different semantics than standard expressions), except in annotations and for specification purposes. In short, I want this thing. But for only annotations, assertions, and possibly default arguments as an ad-hoc fix. Elazar On Mon, Sep 26, 2016 at 5:05 PM Joseph Jevnik wrote: > Hello everyone, this idea looks like something I have tried building > already: https://github.com/llllllllll/lazy_python. This project > implements a `thunk` class which builds up a deferred computation which is > evaluated only when needed. One use case I have had for this project is > building up a larger expression so that it may be simplified and then > computed concurrently with dask: > http://daisy-python.readthedocs.io/en/latest/. By building up a larger > expression (and making the tree accessible) users have the ability to > remove common subexpressions or remove intermediate objects. In numpy > chained expressions often make lots of allocations which are quickly thrown > away which is why projects like numexpr (https://github.com/pydata/numexpr) > can be such a serious speed up. These intermediates are required because > the whole expression isn't known at the start so it must be evaluated as > written. > > Things to consider about when to evaluate: > > 1. Functions which branch on their input need to know which branch to > select. > 2. Iteration is really hard to defer in a way that is efficient. > lazy_python just eagerly evaluates at iteration time but builds thunks in > the body. > 3. Stateful operations like IO which normally have an implied order of > operation now need some explicit ordering. > > Regarding the `Py_TYPE` change: I don't think that is correct unless we > made a thunk have the same binary representation as the underlying object. > A lot of code does a type check and then calls macros that act on the > actual type like `PyTuple_GET_ITEM` so we cannot fool C functions very > easily. > > On Mon, Sep 26, 2016 at 9:27 AM, Sjoerd Job Postmus < > sjoerdjob at sjoerdjob.com> wrote: > >> On Mon, Sep 26, 2016 at 10:46:57PM +1000, Steven D'Aprano wrote: >> > Let's talk about lazy evaluation in a broader sense that just function >> > annotations. >> > >> > If we had syntax for lazy annotation -- let's call them thunks, after >> > Algol's thunks -- then we could use them in annotations as well as >> > elsewhere. But if we special case annotations only, the Zen has >> > something to say about special cases. >> > >> > >> > On Mon, Sep 26, 2016 at 02:57:36PM +1000, Nick Coghlan wrote: >> > [...] >> > > OK, that does indeed make more sense, and significantly reduces the >> > > scope for potential runtime compatibility breaks related to >> > > __annotations__ access. Instead, it changes the discussion to focus on >> > > the following main challenges: >> > > >> > > - the inconsistency introduced between annotations (lazily evaluated) >> > > and default arguments (eagerly evaluated) >> > > - the remaining compatibility breaks (depending on implementation >> details) >> > > - the runtime overhead of lazy evaluation >> > > - the debugging challenges of lazy evaluation >> > >> > >> > Default arguments are a good use-case for thunks. One of the most common >> > gotchas in Python is early binding of function defaults: >> > >> > def func(arg=[]): >> > ... >> > >> > Nine times out of ten, that's probably not what you want. Now, to avoid >> > all doubt, I do not want to change function defaults to late binding. >> > I've argued repeatedly on comp.lang.python and elsewhere that if a >> > language only offers one of early binding or late binding, it should >> > offer early binding as Python does. The reason is, given early binding, >> > it it trivial to simulate something like late binding: >> > >> > def func(arg=None): >> > if arg is None: >> > arg = [] >> > ... >> > >> > but given late binding, it is ugly and inconvenient to get a poor >> > substitute for early binding when that's what you want. So, please, >> > let's not have a debate over the behaviour of function defaults. >> > >> > But what if we could have both? Suppose we use backticks `...` to make a >> > thunk, then we could write: >> > >> > def func(arg=`[]`): >> > ... >> > >> > to get the late binding result wanted. >> > >> > Are there other uses for thunks? Potentially, they could be used for >> > Ruby-like code blocks: >> > >> > result = function(arg1, arg2, block=```# triple backticks >> > do_this() >> > do_that() >> > while condition: >> > do_something_else() >> > print('Done') >> > ```, >> > another_arg=1) >> > >> > >> > but then I'm not really sure what advantage code blocks have over >> > functions. >> > >> > >> > > The inconsistency argument is simply that people will be even more >> > > confused than they are today if default arguments are evaluated at >> > > definition time while annotations aren't. There is a lot of code out >> > > there that actively relies on eager evaluation of default arguments, >> > > so changing that is out of the question, which then provides a strong >> > > consistency argument in favour of keeping annotations eagerly >> > > evaluated as well. >> > >> > Indeed. There are only (to my knowledge) only two places where Python >> > delays evaluation of code: >> > >> > - functions (def statements and lambda expressions); >> > - generator expressions; >> > >> > where the second can be considered to be syntactic sugar for a generator >> > function (def with yield). Have I missed anything? >> > >> > In the same way that Haskell is fundamentally built on lazy evaluation, >> > Python is fundamentally built on eager evaluation, and I don't think we >> > should change that. >> > >> > Until now, the only way to delay the evaluation of code (other than the >> > body of a function, of course) is to write it as a string, then pass it >> > to eval/exec. Thunks offer an alternative for delayed evaluation that >> > makes it easier for editors to apply syntax highlighting: don't apply it >> > to ordinary strings, but do apply it to thunks. >> > >> > I must admit that I've loved the concept of thunks for years now, but >> > I'm still looking for the killer use-case for them, the one clear >> > justification for why Python should include them. >> > >> > - Late-bound function default arguments? Nice to have, but we already >> > have a perfectly serviceable way to get the equivalent behaviour. >> > >> > - Code blocks? Maybe a Ruby programmer can explain why they're so >> > important, but we have functions, including lambda. >> > >> > - Function annotations? I'm not convinced thunks are needed or desirable >> > for annotations. >> > >> > - A better way to write code intended for delayed execution? Sounds >> > interesting, but not critical. >> > >> > Maybe somebody else can think of the elusive killer use-case for thunks, >> > because I've been pondering this question for many years now and I'm no >> > closer to an answer. >> >> Well, there's a use-case I have been pondering for a long while now >> which could be satisfied by this: enumerated generator displays. >> >> So suppose you have a composite boolean value, composed by the 'and' of >> many conditions (which all take long to compute), and you want to >> short-circuit. Let's take the following example. >> >> valid = True >> valid &= looks_like_emailaddress(username) >> valid &= more_than_8_characters(password) >> valid &= does_not_exist_in_database(username) >> valid &= domain_name_of_emailaddress_has_mx_record(username) >> ... some more options ... >> >> (I forgot the exact use-case, but I still remember the functionality I >> wanted, so bear with me). >> >> Of course, the above is not short-circuiting, so it would be replaced by >> >> def check_valid(username, password): >> if not looks_like_emailaddress(username): return False >> if not more_than_8_characters(password): return False >> if not does_not_exist_in_database(username): return False >> if not domain_name_of_emailaddress_has_mx_record(username): return >> False >> ... >> return True >> >> >> valid = check_valid() >> >> or >> >> valid = True\ >> and looks_like_emailaddress(username)\ >> and more_than_8_characters(password)\ >> and does_not_exist_in_database(username)\ >> and domain_name_of_emailaddress_has_mx_record(username) >> >> But in all reality, I want to write something like: >> >> valid = all(@@@ >> looks_like_emailaddress(username), >> more_than_8_characters(password), >> does_not_exist_in_database(username), >> domain_name_of_emailaddress_has_mx_record(username), >> @@@) >> >> With `@@@` designating the beginning/ending of the enumerated generator >> display. >> >> Now, this is currently not possible, but if we had some kind of thunk >> syntax that would become possible, without needing an enumerated >> generator display. >> >> However the problem I see with the concept of `thunk` is: When does it >> get un-thunked? In which of the following cases? >> >> 1. When getting an attribute on it? >> 2. When calling it? --> See 1. with `__call__`. >> 3. When subindexing it? --> See 1. with `__getitem__`. >> 4. When assigning it to a name? It shouldn't have to be un-thunked, I >> think. >> 5. When adding it to a list? No un-thunking should be necessary, I >> think. >> >> However, the problem with thunks is (I think) that to make that happen >> either >> >> - *all* objects need to include yet another level of redirection, >> or >> - a thunk needs to get allocated the maximum size of the value it could >> possibly store. (But a `unicode` object could have an arbitrary size) >> or >> - there needs to be some way to 'notify' objects holding the thunk that >> its value got updated. For a dict/list/tuple this could readily grow >> into O(n) behaviour when un-thunking a thunk. >> or >> - any C-level functionality needs to learn how to deal with thunks. For >> instance, `Py_TYPE` would have to *resolve* the thunk, and then return >> the type of the value. >> or >> - I'm running out of ideas here, but maybe creating a custom type object >> for each thunk that does pass-through to a wrapped item? Thunked >> objects would work *exactly* the same as normal objects, but at a >> (small) indirection for any action taken. Still, somehow `Py_TYPE` and >> `Py_SIZE` and any other macros would still have to force evaluation. >> >> Kind regards, >> Sjoerd Job >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Sep 27 03:29:00 2016 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 27 Sep 2016 00:29:00 -0700 (PDT) Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> Message-ID: <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> On Friday, September 23, 2016 at 2:23:58 AM UTC-4, Nick Coghlan wrote: > > On 23 September 2016 at 15:50, Greg Ewing > wrote: > > ????? wrote: > >> > >> it feels like a > >> placeholder for this meaning would be better. E.g.: > >> > >> class A: > >> def __add__(self, other: CLS) -> CLS: ... > > > > > > That's fine for a class that refers to itself, but > > what about classes that refer to each other? This > > only addresses a small part of the problem. > > Same answer as with any other circular dependency: the code smell is > the circular dependency itself, not the awkwardness of the syntax for > spelling it. If the string based "circular reference here!" spelling > really bothers you, refactor to eliminate the circularity (e.g. by > extracting a base class or an implementation independent interface > definition), rather than advocating to make the spelling less > obnoxious. > > The difference between that and the "methods referring to the class > they're defined in" case is that it's likely to be pretty normal to > want to do the latter, so it may prove worthwhile to provide a cleaner > standard spelling for it. The counter-argument is the general > circularity one above: do you *really* need instances of the > particular class being defined? Or is there a more permissive > interface based type signature you could specify instead? Or perhaps > no type signature at all, and let ducktyping sort it out implicitly at > runtime? > I agree that circularity should in general be avoided, but it's not always possible or elegant to do that. Sometimes you really need two classes to refer to each other. In that case, why not expose your placeholder idea to the user via a library? You have one function that generates placeholder singletons (generate_placeholder()), and another function to walks a class object and replaces a placeholder with a given value (replace_placeholder(placeholder, cls, value)). Best, Neil > Cheers, > Nick. > > -- > Nick Coghlan | ncog... at gmail.com | Brisbane, > Australia > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Sep 27 03:24:45 2016 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 27 Sep 2016 00:24:45 -0700 (PDT) Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> Message-ID: <87275730-d73a-42fe-b47c-9dcb38b300c1@googlegroups.com> I really like this idea, and in the rare case that someone adds an element to a class with the same name as the class that would shadow your definition, but that seems fine to me. On Thursday, September 22, 2016 at 11:09:28 PM UTC-4, Nick Coghlan wrote: > > On 23 September 2016 at 12:05, ????? > > wrote: > > On Fri, Sep 23, 2016 at 4:17 AM Nick Coghlan > wrote: > > ... > >> > >> As others have noted, the general idea of allowing either a > >> placeholder name or the class name to refer to a suitable type > >> annotation is fine, though - that would be a matter of implicitly > >> injecting that name into the class namespace after calling > >> __prepare__, and ensuring the compiler is aware of that behaviour, > >> just as we inject __class__ as a nonlocal reference into method bodies > >> that reference "super" or "__class__". > >> > > Just to be sure I understand, will the following work? > > > > class A: > > def repeat(n: int) -> List[A]: pass > > Right now? No - you'll get a name error on the "A", just as you would > if you tried to reference it as a default argument: > > >>> class A: > ... def side_effects_ahead(arg: print(A) = print(A)) -> print(A): > pass > ... > Traceback (most recent call last): > File "", line 1, in > File "", line 2, in A > NameError: name 'A' is not defined > > And that's the problem with using the class name in method annotations > in the class body: they're evaluated eagerly, so they'd fail at > runtime, even if the typecheckers were updated to understand them. > > Rather than switching annotations to being evaluated lazilly in the > general case, one of the solutions being suggested is that *by > default*, the class name could implicitly be bound in the body of the > class definition to some useful placeholder, which can already be done > explicitly today: > > >>> class A: > ... A = "placeholder" > ... def side_effects_ahead(arg: print(A) = print(A)) -> print(A): > pass > ... > placeholder > placeholder > placeholder > > Since method bodies don't see class level name bindings (by design), > such an approach would have the effect of "A" referring to the > placeholder in the class body (including for annotations and default > arguments), but to the class itself in method bodies. > > I don't think this is an urgent problem (since the "A"-as-a-string > spelling works today without any runtime changes), but it's worth > keeping an eye on as folks gain more experience with annotations and > the factors affecting their readability. > > Cheers, > Nick. > > -- > Nick Coghlan | ncog... at gmail.com | Brisbane, > Australia > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Sep 27 05:01:10 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2016 19:01:10 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> Message-ID: On 27 September 2016 at 17:29, Neil Girdhar wrote: > On Friday, September 23, 2016 at 2:23:58 AM UTC-4, Nick Coghlan wrote: >> The difference between that and the "methods referring to the class >> they're defined in" case is that it's likely to be pretty normal to >> want to do the latter, so it may prove worthwhile to provide a cleaner >> standard spelling for it. The counter-argument is the general >> circularity one above: do you *really* need instances of the >> particular class being defined? Or is there a more permissive >> interface based type signature you could specify instead? Or perhaps >> no type signature at all, and let ducktyping sort it out implicitly at >> runtime? > > I agree that circularity should in general be avoided, but it's not always > possible or elegant to do that. Sometimes you really need two classes to > refer to each other. In that case, why not expose your placeholder idea to > the user via a library? You have one function that generates placeholder > singletons (generate_placeholder()), and another function to walks a class > object and replaces a placeholder with a given value > (replace_placeholder(placeholder, cls, value)). Because the general case is already covered by using a quoted string instead of a name reference. "I don't like using strings to denote delayed evaluation" isn't a compelling argument, which is why alternative ideas have to offer some other significant benefit, or else be incredibly simple both to implement and to explain. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Sep 27 06:20:01 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 27 Sep 2016 19:20:01 +0900 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> Message-ID: <22506.18385.278656.503077@turnbull.sk.tsukuba.ac.jp> Neil Girdhar writes: > I agree that circularity should in general be avoided, but it's not always > possible or elegant to do that. Sometimes you really need two classes to > refer to each other. In that case, why not expose your placeholder idea to > the user via a library? Why not just expose it through a simple assignment? https://mail.python.org/pipermail/python-ideas/2016-September/042563.html Note that this also works for typechecking in PEP 484 checkers that allow forward reference via the stringified class name. See also https://mail.python.org/pipermail/python-ideas/2016-September/042544.html, which would allow eliding the assignment, but pollutes the class namespace. "Simple is better than complex." This feature is still looking for a persuasive use that needs it, and not something simpler. Steve From mistersheik at gmail.com Tue Sep 27 07:54:40 2016 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 27 Sep 2016 11:54:40 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <22506.18385.278656.503077@turnbull.sk.tsukuba.ac.jp> References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> <22506.18385.278656.503077@turnbull.sk.tsukuba.ac.jp> Message-ID: I don't understand why that would work and this clearly doesn't? Mutual2 = "Mutual2" # Pre-declare Mutual2 class Mutual1: def spam(self, x=Mutual2): print(type(x)) class Mutual2: def spam(self): pass Mutual1().spam() prints class "str" rather than "type". On Tue, Sep 27, 2016 at 6:20 AM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Neil Girdhar writes: > > > I agree that circularity should in general be avoided, but it's not > always > > possible or elegant to do that. Sometimes you really need two classes > to > > refer to each other. In that case, why not expose your placeholder > idea to > > the user via a library? > > Why not just expose it through a simple assignment? > > https://mail.python.org/pipermail/python-ideas/2016-September/042563.html > > Note that this also works for typechecking in PEP 484 checkers that > allow forward reference via the stringified class name. See also > https://mail.python.org/pipermail/python-ideas/2016-September/042544.html, > which would allow eliding the assignment, but pollutes the class namespace. > > "Simple is better than complex." > > This feature is still looking for a persuasive use that needs it, and > not something simpler. > > Steve > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Tue Sep 27 08:14:58 2016 From: phd at phdru.name (Oleg Broytman) Date: Tue, 27 Sep 2016 14:14:58 +0200 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> <22506.18385.278656.503077@turnbull.sk.tsukuba.ac.jp> Message-ID: <20160927121458.GA30270@phdru.name> On Tue, Sep 27, 2016 at 11:54:40AM +0000, Neil Girdhar wrote: > I don't understand why that would work and this clearly doesn't? > > Mutual2 = "Mutual2" # Pre-declare Mutual2 > > class Mutual1: > def spam(self, x=Mutual2): ^^^^^^^ - calculated at compile time, not at run time > print(type(x)) > > class Mutual2: > def spam(self): > pass > > Mutual1().spam() > > prints class "str" rather than "type". Try this: class Mutual1: def spam(self, x=None): if x is None: x = Mutual2 print(type(x)) class Mutual2: def spam(self): pass Mutual1().spam() Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From mistersheik at gmail.com Tue Sep 27 08:46:30 2016 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 27 Sep 2016 12:46:30 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: <20160927121458.GA30270@phdru.name> References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> <22506.18385.278656.503077@turnbull.sk.tsukuba.ac.jp> <20160927121458.GA30270@phdru.name> Message-ID: Yes, I understand that, but I don't see how that would help at all with annotations. Aren't annotations also evaluated at "compile time"? On Tue, Sep 27, 2016 at 8:14 AM Oleg Broytman wrote: > On Tue, Sep 27, 2016 at 11:54:40AM +0000, Neil Girdhar < > mistersheik at gmail.com> wrote: > > I don't understand why that would work and this clearly doesn't? > > > > Mutual2 = "Mutual2" # Pre-declare Mutual2 > > > > class Mutual1: > > def spam(self, x=Mutual2): > ^^^^^^^ - calculated at compile time, > not at run time > > print(type(x)) > > > > class Mutual2: > > def spam(self): > > pass > > > > Mutual1().spam() > > > > prints class "str" rather than "type". > > Try this: > > class Mutual1: > def spam(self, x=None): > if x is None: > x = Mutual2 > print(type(x)) > > class Mutual2: > def spam(self): > pass > > Mutual1().spam() > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Sep 27 08:50:12 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Sep 2016 13:50:12 +0100 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> <22506.18385.278656.503077@turnbull.sk.tsukuba.ac.jp> <20160927121458.GA30270@phdru.name> Message-ID: On 27 September 2016 at 13:46, Neil Girdhar wrote: > Yes, I understand that, but I don't see how that would help at all with > annotations. Aren't annotations also evaluated at "compile time"? Yes, but a string whose value is a class name is treated as being the same annotation (i.e., meaning the same) as the class itself. Paul From mistersheik at gmail.com Tue Sep 27 09:00:25 2016 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 27 Sep 2016 13:00:25 +0000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> Message-ID: On Tue, Sep 27, 2016 at 5:01 AM Nick Coghlan wrote: > On 27 September 2016 at 17:29, Neil Girdhar wrote: > > On Friday, September 23, 2016 at 2:23:58 AM UTC-4, Nick Coghlan wrote: > >> The difference between that and the "methods referring to the class > >> they're defined in" case is that it's likely to be pretty normal to > >> want to do the latter, so it may prove worthwhile to provide a cleaner > >> standard spelling for it. The counter-argument is the general > >> circularity one above: do you *really* need instances of the > >> particular class being defined? Or is there a more permissive > >> interface based type signature you could specify instead? Or perhaps > >> no type signature at all, and let ducktyping sort it out implicitly at > >> runtime? > > > > I agree that circularity should in general be avoided, but it's not > always > > possible or elegant to do that. Sometimes you really need two classes to > > refer to each other. In that case, why not expose your placeholder idea > to > > the user via a library? You have one function that generates placeholder > > singletons (generate_placeholder()), and another function to walks a > class > > object and replaces a placeholder with a given value > > (replace_placeholder(placeholder, cls, value)). > > Because the general case is already covered by using a quoted string > instead of a name reference. "I don't like using strings to denote > delayed evaluation" isn't a compelling argument, which is why > alternative ideas have to offer some other significant benefit, or > else be incredibly simple both to implement and to explain. > My motivation for something other than quoted strings is that there are other instances of circular dependencies. Currently, when I am forced into a circular dependency, I import the later class in the member functions of the first: # module x class X: def f(self): from y import Y # do something with Y # module y class Y: pass That's not ideal and I don't see how to extend this solution to use of "y" in class level definitions. Best, Neil > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Tue Sep 27 10:55:00 2016 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 27 Sep 2016 16:55:00 +0200 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> Message-ID: On Sun, Sep 11, 2016 at 12:28 PM, Bernardo Sulzbach wrote: > On 09/11/2016 06:36 AM, Dominik Gresch wrote: >> >> So I asked myself if a syntax as follows would be possible: >> >> for i in range(10) if i != 5: >> body >> >> Personally, I find this extremely intuitive since this kind of >> if-statement is already present in list comprehensions. >> >> What is your opinion on this? Sorry if this has been discussed before -- >> I didn't find anything in the archives. >> > > I find it interesting. > > I thing that this will likely take up too many columns in more convoluted > loops such as > > for element in collection if is_pretty_enough(element) and ...: > ... > > However, this "problem" is already faced by list comprehensions, so it is > not a strong argument against your idea. Sorry to re-raise this thread--I'm inclined to agree that the case doesn't really warrant new syntax. I just wanted to add that I think the very fact that this syntax is supported by list comprehensions is an argument *in its favor*. I could easily see a Python newbie being confused that they can write "for x in y if z" inside a list comprehension, but not in a bare for-statement. Sure they'd learn quickly enough that the filtering syntax is unique to list comprehensions. But to anyone who doesn't know the historical progression of the Python language that would seem highly arbitrary and incongruous I would think. Just $0.02 USD from a pedagogical perspective. Erik From ncoghlan at gmail.com Tue Sep 27 10:57:53 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Sep 2016 00:57:53 +1000 Subject: [Python-ideas] Delay evaluation of annotations In-Reply-To: References: <20160922184215.GZ22471@ando.pearwood.info> <57E4C2BF.7030100@canterbury.ac.nz> <77fe5cfb-89d6-4510-ba5c-e2b6f1f28aa7@googlegroups.com> <22506.18385.278656.503077@turnbull.sk.tsukuba.ac.jp> <20160927121458.GA30270@phdru.name> Message-ID: On 27 September 2016 at 22:46, Neil Girdhar wrote: > Yes, I understand that, but I don't see how that would help at all with > annotations. Aren't annotations also evaluated at "compile time"? This thread isn't about circular references in general, just circular references in the context of type hinting. For type hinting purposes, it already doesn't matter whether you use a variable name to refer to a type or a quoted string literal, as the typechecker ignores the quotation marks (and this is mandated by PEP 484). For runtime annotation use, the difference is visible, but the only required runtime behaviours for the typechecking use case are "doesn't throw an exception" and "doesn't take a prohibitive amount of time to evaluate when the function is defined". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 27 11:33:54 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Sep 2016 01:33:54 +1000 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> Message-ID: On 28 September 2016 at 00:55, Erik Bray wrote: > On Sun, Sep 11, 2016 at 12:28 PM, Bernardo Sulzbach > wrote: >> On 09/11/2016 06:36 AM, Dominik Gresch wrote: >>> >>> So I asked myself if a syntax as follows would be possible: >>> >>> for i in range(10) if i != 5: >>> body >>> >>> Personally, I find this extremely intuitive since this kind of >>> if-statement is already present in list comprehensions. >>> >>> What is your opinion on this? Sorry if this has been discussed before -- >>> I didn't find anything in the archives. >>> >> >> I find it interesting. >> >> I thing that this will likely take up too many columns in more convoluted >> loops such as >> >> for element in collection if is_pretty_enough(element) and ...: >> ... >> >> However, this "problem" is already faced by list comprehensions, so it is >> not a strong argument against your idea. > > Sorry to re-raise this thread--I'm inclined to agree that the case > doesn't really warrant new syntax. I just wanted to add that I think > the very fact that this syntax is supported by list comprehensions is > an argument *in its favor*. > > I could easily see a Python newbie being confused that they can write > "for x in y if z" inside a list comprehension, but not in a bare > for-statement. Sure they'd learn quickly enough that the filtering > syntax is unique to list comprehensions. But to anyone who doesn't > know the historical progression of the Python language that would seem > highly arbitrary and incongruous I would think. > > Just $0.02 USD from a pedagogical perspective. This has come up before, and it's considered a teaching moment regarding how the comprehension syntax actually works: it's an *arbitrarily deep* nested chain of if statements and for statements. That is: [f(x,y,z) for x in seq1 if p1(x) for y in seq2 if p2(y) for z in seq3 if p3(z)] can be translated mechanically to the equivalent nested statements (with the only difference being that the loop variable leak due to the missing implicit scope): result = [] for x in seq1: if p1(x): for y in seq2: if p2(y): for z in seq3: if p3(z): result.append(f(x, y, z)) So while the *most common* cases are a single for loop (map equivalent), or a single for loop and a single if statement (filter equivalent), they're not only the forms folks may encounter in the wild. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brakhane at googlemail.com Tue Sep 27 11:39:52 2016 From: brakhane at googlemail.com (Dennis Brakhane) Date: Tue, 27 Sep 2016 17:39:52 +0200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: Message-ID: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> I don't know if it works on Windows, but at least in Linux pressing Ctrl-L will do exactly what you describe (as long as the REPL uses readline) On 17.09.2016 12:51, Jo?o Matos wrote: > Hello, > > I would like to suggest adding a clear command (not function) to Python. > It's simple purpose would be to clear the REPL screen, leaving the >>> > prompt at the top left of the screen. > > This is something very basic but also very useful for newbies learning > Python from the REPL. > After some trial and errors it is best to start with a clean screen. > Clearing the screen helps clear your mind. > > Historically it is a common command in interpreted languages. > > Best regards, > > JM > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From erik.m.bray at gmail.com Tue Sep 27 11:54:23 2016 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 27 Sep 2016 17:54:23 +0200 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> Message-ID: On Tue, Sep 27, 2016 at 5:33 PM, Nick Coghlan wrote: > On 28 September 2016 at 00:55, Erik Bray wrote: >> On Sun, Sep 11, 2016 at 12:28 PM, Bernardo Sulzbach >> wrote: >>> On 09/11/2016 06:36 AM, Dominik Gresch wrote: >>>> >>>> So I asked myself if a syntax as follows would be possible: >>>> >>>> for i in range(10) if i != 5: >>>> body >>>> >>>> Personally, I find this extremely intuitive since this kind of >>>> if-statement is already present in list comprehensions. >>>> >>>> What is your opinion on this? Sorry if this has been discussed before -- >>>> I didn't find anything in the archives. >>>> >>> >>> I find it interesting. >>> >>> I thing that this will likely take up too many columns in more convoluted >>> loops such as >>> >>> for element in collection if is_pretty_enough(element) and ...: >>> ... >>> >>> However, this "problem" is already faced by list comprehensions, so it is >>> not a strong argument against your idea. >> >> Sorry to re-raise this thread--I'm inclined to agree that the case >> doesn't really warrant new syntax. I just wanted to add that I think >> the very fact that this syntax is supported by list comprehensions is >> an argument *in its favor*. >> >> I could easily see a Python newbie being confused that they can write >> "for x in y if z" inside a list comprehension, but not in a bare >> for-statement. Sure they'd learn quickly enough that the filtering >> syntax is unique to list comprehensions. But to anyone who doesn't >> know the historical progression of the Python language that would seem >> highly arbitrary and incongruous I would think. >> >> Just $0.02 USD from a pedagogical perspective. > > This has come up before, and it's considered a teaching moment > regarding how the comprehension syntax actually works: it's an > *arbitrarily deep* nested chain of if statements and for statements. > > That is: > > [f(x,y,z) for x in seq1 if p1(x) for y in seq2 if p2(y) for z in > seq3 if p3(z)] > > can be translated mechanically to the equivalent nested statements > (with the only difference being that the loop variable leak due to the > missing implicit scope): > > result = [] > for x in seq1: > if p1(x): > for y in seq2: > if p2(y): > for z in seq3: > if p3(z): > result.append(f(x, y, z)) > > So while the *most common* cases are a single for loop (map > equivalent), or a single for loop and a single if statement (filter > equivalent), they're not only the forms folks may encounter in the > wild. Thanks for pointing this out Nick. Then following my own logic it would be desirable to also allow the nested for loop syntax of list comprehensions outside them as well. That's a slippery slope to incomprehensibility (they're bad enough in list comprehensions, though occasionally useful). This is a helpful way to think about list comprehensions though--I'll remember it next time I teach them. Thanks, Erik From p.f.moore at gmail.com Tue Sep 27 12:34:48 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Sep 2016 17:34:48 +0100 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> Message-ID: On 27 September 2016 at 16:54, Erik Bray wrote: > Then following my own logic it > would be desirable to also allow the nested for loop syntax of list > comprehensions outside them as well. I'd say that it's a case where we should either allow arbitrary concatenation outside of comprehensions, or we allow none. And as arbitrary concatenation is obviously bad, the only sane choice is no nesting :-) Paul From jcrmatos at gmail.com Tue Sep 27 13:05:16 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Matos?=) Date: Tue, 27 Sep 2016 10:05:16 -0700 (PDT) Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> Message-ID: <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> Hello, It doesn't work in Windows. Best regards, JM ter?a-feira, 27 de Setembro de 2016 ?s 16:40:42 UTC+1, Dennis Brakhane via Python-ideas escreveu: > I don't know if it works on Windows, but at least in Linux pressing > Ctrl-L will do exactly what you describe (as long as the REPL uses > readline) > > On 17.09.2016 12:51, Jo?o Matos wrote: > > Hello, > > > > I would like to suggest adding a clear command (not function) to Python. > > It's simple purpose would be to clear the REPL screen, leaving the >>> > > prompt at the top left of the screen. > > > > This is something very basic but also very useful for newbies learning > > Python from the REPL. > > After some trial and errors it is best to start with a clean screen. > > Clearing the screen helps clear your mind. > > > > Historically it is a common command in interpreted languages. > > > > Best regards, > > > > JM > > > > _______________________________________________ > > Python-ideas mailing list > > Python... at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Sep 27 17:36:22 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 28 Sep 2016 10:36:22 +1300 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> Message-ID: <57EAE656.1070103@canterbury.ac.nz> Erik Bray wrote: > Then following my own logic it > would be desirable to also allow the nested for loop syntax of list > comprehensions outside them as well. The only use for such a syntax would be to put an inadvisable amount of stuff on one line. When describing a procedural series of steps, the Pythonic style encourages putting each step on its own line. Other areas of the language also nudge one in this direction, e.g. the fact that mutating operations usually return None, which discourages chaining them together into a single expression. -- Greg From steve at pearwood.info Wed Sep 28 22:04:28 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 29 Sep 2016 12:04:28 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> Message-ID: <20160929020428.GU22471@ando.pearwood.info> On Tue, Sep 27, 2016 at 10:05:16AM -0700, Jo?o Matos wrote: > Hello, > > > It doesn't work in Windows. What is "it"? Are you talking about Ctrl-L to clear the screen? Perhaps we should start by adding Ctrl-L as a standard way to clear the Python REPL, in the same way that Ctrl-C is the standard way to interrupt the interpreter regardless of whether you are using Linux, Mac or Windows. (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but Windows is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) -- Steve From phd at phdru.name Wed Sep 28 23:34:21 2016 From: phd at phdru.name (Oleg Broytman) Date: Thu, 29 Sep 2016 05:34:21 +0200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160929020428.GU22471@ando.pearwood.info> References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> Message-ID: <20160929033421.GA11964@phdru.name> On Thu, Sep 29, 2016 at 12:04:28PM +1000, Steven D'Aprano wrote: > (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but Windows > is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) I don't think consistency should go that far. Consistency inside the platform is more important than consistency between platforms, and other w32 console programs understand [Ctrl]+[Z] as EOF and as far as I know only [Ctrl]+[Z]. > -- > Steve Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From rosuav at gmail.com Wed Sep 28 23:36:03 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 29 Sep 2016 13:36:03 +1000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160929020428.GU22471@ando.pearwood.info> References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> Message-ID: On Thu, Sep 29, 2016 at 12:04 PM, Steven D'Aprano wrote: > (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but Windows > is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) Sadly, I suspect not. If you're running in the default Windows terminal emulator (the one a normal user will get by invoking cmd.exe), you're running under a lot of restrictions, and I believe one of them is that you can't get Ctrl-D without an enter. But it would be very nice to support both. ChrisA From elazarg at gmail.com Wed Sep 28 23:41:06 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Thu, 29 Sep 2016 03:41:06 +0000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> Message-ID: "Bash on Ubuntu on windows" responds to CTRL+D just fine. I don't really know how it works, but it looks like it is based on the Windows terminal emulator. Elazar ?????? ??? ??, 29 ????' 2016, 06:36, ??? Chris Angelico ?: > On Thu, Sep 29, 2016 at 12:04 PM, Steven D'Aprano > wrote: > > (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but Windows > > is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) > > Sadly, I suspect not. If you're running in the default Windows > terminal emulator (the one a normal user will get by invoking > cmd.exe), you're running under a lot of restrictions, and I believe > one of them is that you can't get Ctrl-D without an enter. > > But it would be very nice to support both. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Thu Sep 29 02:50:32 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Matos?=) Date: Wed, 28 Sep 2016 23:50:32 -0700 (PDT) Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <20160929020428.GU22471@ando.pearwood.info> References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> Message-ID: Hello, Yes, Ctrl-L doesn't clear the screen on Windows. Making Ctrl-L clear the screen would be a good solution (no need for a clear screen command). Best regards, JM quinta-feira, 29 de Setembro de 2016 ?s 03:06:26 UTC+1, Steven D'Aprano escreveu: > On Tue, Sep 27, 2016 at 10:05:16AM -0700, Jo?o Matos wrote: > > Hello, > > > > > > It doesn't work in Windows. > > What is "it"? Are you talking about Ctrl-L to clear the screen? > > > Perhaps we should start by adding Ctrl-L as a standard way to clear the > Python REPL, in the same way that Ctrl-C is the standard way to > interrupt the interpreter regardless of whether you are using Linux, Mac > or Windows. > > (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but Windows > is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Thu Sep 29 03:08:25 2016 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 29 Sep 2016 09:08:25 +0200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> Message-ID: Hi all, I just tried with this official Python binary: Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 bit (Intel)] on win32 and CTRL-L for sure does clear the window. It just doesn't then move the prompt to the top, so you end up with a bunch of empty lines, followed by the prompt. Stephan 2016-09-29 8:50 GMT+02:00 Jo?o Matos : > Hello, > > Yes, Ctrl-L doesn't clear the screen on Windows. > Making Ctrl-L clear the screen would be a good solution (no need for a > clear screen command). > > > Best regards, > > JM > > quinta-feira, 29 de Setembro de 2016 ?s 03:06:26 UTC+1, Steven D'Aprano > escreveu: > >> On Tue, Sep 27, 2016 at 10:05:16AM -0700, Jo?o Matos wrote: >> > Hello, >> > >> > >> > It doesn't work in Windows. >> >> What is "it"? Are you talking about Ctrl-L to clear the screen? >> >> >> Perhaps we should start by adding Ctrl-L as a standard way to clear the >> Python REPL, in the same way that Ctrl-C is the standard way to >> interrupt the interpreter regardless of whether you are using Linux, Mac >> or Windows. >> >> (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but Windows >> is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Sep 29 03:51:18 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 29 Sep 2016 08:51:18 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> Message-ID: On 29 September 2016 at 08:08, Stephan Houben wrote: > I just tried with this official Python binary: > Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 bit > (Intel)] on win32 > > and CTRL-L for sure does clear the window. It just doesn't then move the > prompt to the top, so you end up with a bunch of empty lines, followed by > the prompt. Do you have pyreadline installed? I believe that intercepts things like CTRL-L. Paul From eryksun at gmail.com Thu Sep 29 03:53:47 2016 From: eryksun at gmail.com (eryk sun) Date: Thu, 29 Sep 2016 07:53:47 +0000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> Message-ID: On Thu, Sep 29, 2016 at 7:08 AM, Stephan Houben wrote: > > I just tried with this official Python binary: > Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 bit > (Intel)] on win32 > > and CTRL-L for sure does clear the window. It just doesn't then move the > prompt to the top, so you end up with a bunch of empty lines, followed by > the prompt. You probably have pyreadline installed. It calls ReadConsoleInputW to read low-level input records, bypassing the console's normal cooked read. See the following file that defines the key binding: https://github.com/pyreadline/pyreadline/blob/1.7/pyreadline/configuration/pyreadlineconfig.ini#L18 Unfortunately pyreadline is broken for non-ASCII input. It ignores the Alt+Numpad record sequences used for non-ASCII characters. Without having to implement readline module for Windows (personally, I don't use it), support for Ctrl+L can be added relatively easily in 3.6+. ReadConsoleW takes a parameter to specify a mask of ASCII control characters that terminate a read. The control character is left in the buffer, so code just has to be written that looks for various control characters to implement features such as a Ctrl+L clear screen. From stephanh42 at gmail.com Thu Sep 29 03:56:37 2016 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 29 Sep 2016 09:56:37 +0200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <39f3f406-6978-4865-ba8c-b1506e9fdf70@googlegroups.com> References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> <39f3f406-6978-4865-ba8c-b1506e9fdf70@googlegroups.com> Message-ID: Hi JM, Windows 7 Enterprise "Microsoft Windows [Version 6.1.7601]" I am running Python directly from the shortcut created on installation. But starting it from cmd.exe has the same effect. Codepage is 437 , this may be relevant? I just tried it on a Windows 10 PC, there it has the same effect. Stephan 2016-09-29 9:12 GMT+02:00 Jo?o Matos : > Hello, > > I tried on Python 2.7.10 and Python 3.5.2 and Ctrl-L doesn't work on both. > I tried on 2 PCs with Windows 7 and none of them worked. > > What is your Windows version? Are you trying on the cmd.exe console or PS? > > > Best regards, > > JM > > quinta-feira, 29 de Setembro de 2016 ?s 08:09:13 UTC+1, Stephan Houben > escreveu: > >> Hi all, >> >> I just tried with this official Python binary: >> Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 >> bit (Intel)] on win32 >> >> and CTRL-L for sure does clear the window. It just doesn't then move the >> prompt to the top, so you end up with a bunch of empty lines, followed by >> the prompt. >> >> Stephan >> >> 2016-09-29 8:50 GMT+02:00 Jo?o Matos : >> >>> Hello, >>> >>> Yes, Ctrl-L doesn't clear the screen on Windows. >>> Making Ctrl-L clear the screen would be a good solution (no need for a >>> clear screen command). >>> >>> >>> Best regards, >>> >>> JM >>> >>> quinta-feira, 29 de Setembro de 2016 ?s 03:06:26 UTC+1, Steven D'Aprano >>> escreveu: >>> >>>> On Tue, Sep 27, 2016 at 10:05:16AM -0700, Jo?o Matos wrote: >>>> > Hello, >>>> > >>>> > >>>> > It doesn't work in Windows. >>>> >>>> What is "it"? Are you talking about Ctrl-L to clear the screen? >>>> >>>> >>>> Perhaps we should start by adding Ctrl-L as a standard way to clear the >>>> Python REPL, in the same way that Ctrl-C is the standard way to >>>> interrupt the interpreter regardless of whether you are using Linux, >>>> Mac >>>> or Windows. >>>> >>>> (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but >>>> Windows >>>> is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) >>>> >>>> >>>> -- >>>> Steve >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python... at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python... at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Thu Sep 29 03:59:24 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Thu, 29 Sep 2016 08:59:24 +0100 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> <39f3f406-6978-4865-ba8c-b1506e9fdf70@googlegroups.com> Message-ID: <3c659a22-476b-ddca-6b7e-1e117e37296e@gmail.com> Hello, You must have pyreadline installed (it isn't installed in the default CPython distribution). Best regards, JM On 29-09-2016 08:56, Stephan Houben wrote: > Hi JM, > > Windows 7 Enterprise > "Microsoft Windows [Version 6.1.7601]" > > I am running Python directly from the shortcut created on installation. > But starting it from cmd.exe has the same effect. > > Codepage is 437 , this may be relevant? > > I just tried it on a Windows 10 PC, there it has the same effect. > > Stephan > > > 2016-09-29 9:12 GMT+02:00 Jo?o Matos >: > > Hello, > > I tried on Python 2.7.10 and Python 3.5.2 and Ctrl-L doesn't work > on both. > I tried on 2 PCs with Windows 7 and none of them worked. > > What is your Windows version? Are you trying on the cmd.exe > console or PS? > > > Best regards, > > JM > > quinta-feira, 29 de Setembro de 2016 ?s 08:09:13 UTC+1, Stephan > Houben escreveu: > > Hi all, > > I just tried with this official Python binary: > Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC > v.1900 32 bit (Intel)] on win32 > > and CTRL-L for sure does clear the window. It just doesn't > then move the prompt to the top, so you end up with a bunch of > empty lines, followed by the prompt. > > Stephan > > 2016-09-29 8:50 GMT+02:00 Jo?o Matos : > > Hello, > > Yes, Ctrl-L doesn't clear the screen on Windows. > Making Ctrl-L clear the screen would be a good solution > (no need for a clear screen command). > > > Best regards, > > JM > > quinta-feira, 29 de Setembro de 2016 ?s 03:06:26 UTC+1, > Steven D'Aprano escreveu: > > On Tue, Sep 27, 2016 at 10:05:16AM -0700, Jo?o Matos > wrote: > > Hello, > > > > > > It doesn't work in Windows. > > What is "it"? Are you talking about Ctrl-L to clear > the screen? > > > Perhaps we should start by adding Ctrl-L as a standard > way to clear the > Python REPL, in the same way that Ctrl-C is the > standard way to > interrupt the interpreter regardless of whether you > are using Linux, Mac > or Windows. > > (Also, it seems a shame that Ctrl-D is EOF in Linux > and Mac, but Windows > is Ctrl-Z + Return. Can that be standardized to Ctrl-D > everywhere?) > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Thu Sep 29 03:12:58 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Matos?=) Date: Thu, 29 Sep 2016 00:12:58 -0700 (PDT) Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> Message-ID: <39f3f406-6978-4865-ba8c-b1506e9fdf70@googlegroups.com> Hello, I tried on Python 2.7.10 and Python 3.5.2 and Ctrl-L doesn't work on both. I tried on 2 PCs with Windows 7 and none of them worked. What is your Windows version? Are you trying on the cmd.exe console or PS? Best regards, JM quinta-feira, 29 de Setembro de 2016 ?s 08:09:13 UTC+1, Stephan Houben escreveu: > Hi all, > > I just tried with this official Python binary: > Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 > bit (Intel)] on win32 > > and CTRL-L for sure does clear the window. It just doesn't then move the > prompt to the top, so you end up with a bunch of empty lines, followed by > the prompt. > > Stephan > > 2016-09-29 8:50 GMT+02:00 Jo?o Matos >: > >> Hello, >> >> Yes, Ctrl-L doesn't clear the screen on Windows. >> Making Ctrl-L clear the screen would be a good solution (no need for a >> clear screen command). >> >> >> Best regards, >> >> JM >> >> quinta-feira, 29 de Setembro de 2016 ?s 03:06:26 UTC+1, Steven D'Aprano >> escreveu: >> >>> On Tue, Sep 27, 2016 at 10:05:16AM -0700, Jo?o Matos wrote: >>> > Hello, >>> > >>> > >>> > It doesn't work in Windows. >>> >>> What is "it"? Are you talking about Ctrl-L to clear the screen? >>> >>> >>> Perhaps we should start by adding Ctrl-L as a standard way to clear the >>> Python REPL, in the same way that Ctrl-C is the standard way to >>> interrupt the interpreter regardless of whether you are using Linux, Mac >>> or Windows. >>> >>> (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but Windows >>> is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) >>> >>> >>> -- >>> Steve >>> _______________________________________________ >>> Python-ideas mailing list >>> Python... at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Thu Sep 29 04:20:54 2016 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 29 Sep 2016 10:20:54 +0200 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <3c659a22-476b-ddca-6b7e-1e117e37296e@gmail.com> References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> <39f3f406-6978-4865-ba8c-b1506e9fdf70@googlegroups.com> <3c659a22-476b-ddca-6b7e-1e117e37296e@gmail.com> Message-ID: Hi JM, Yes indeed, I have that installed. I was unaware it activates itself in the normal interpreter. (I have it installed because IPython asks for it). Stephan 2016-09-29 9:59 GMT+02:00 Jo?o Matos : > Hello, > > You must have pyreadline installed (it isn't installed in the default > CPython distribution). > > > Best regards, > > JM > > > > On 29-09-2016 08:56, Stephan Houben wrote: > > Hi JM, > > Windows 7 Enterprise > "Microsoft Windows [Version 6.1.7601]" > > I am running Python directly from the shortcut created on installation. > But starting it from cmd.exe has the same effect. > > Codepage is 437 , this may be relevant? > > I just tried it on a Windows 10 PC, there it has the same effect. > > Stephan > > > 2016-09-29 9:12 GMT+02:00 Jo?o Matos : > >> Hello, >> >> I tried on Python 2.7.10 and Python 3.5.2 and Ctrl-L doesn't work on both. >> I tried on 2 PCs with Windows 7 and none of them worked. >> >> What is your Windows version? Are you trying on the cmd.exe console or PS? >> >> >> Best regards, >> >> JM >> >> quinta-feira, 29 de Setembro de 2016 ?s 08:09:13 UTC+1, Stephan Houben >> escreveu: >> >>> Hi all, >>> >>> I just tried with this official Python binary: >>> Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 >>> bit (Intel)] on win32 >>> >>> and CTRL-L for sure does clear the window. It just doesn't then move the >>> prompt to the top, so you end up with a bunch of empty lines, followed by >>> the prompt. >>> >>> Stephan >>> >>> 2016-09-29 8:50 GMT+02:00 Jo?o Matos : >>> >>>> Hello, >>>> >>>> Yes, Ctrl-L doesn't clear the screen on Windows. >>>> Making Ctrl-L clear the screen would be a good solution (no need for a >>>> clear screen command). >>>> >>>> >>>> Best regards, >>>> >>>> JM >>>> >>>> quinta-feira, 29 de Setembro de 2016 ?s 03:06:26 UTC+1, Steven D'Aprano >>>> escreveu: >>>> >>>>> On Tue, Sep 27, 2016 at 10:05:16AM -0700, Jo?o Matos wrote: >>>>> > Hello, >>>>> > >>>>> > >>>>> > It doesn't work in Windows. >>>>> >>>>> What is "it"? Are you talking about Ctrl-L to clear the screen? >>>>> >>>>> >>>>> Perhaps we should start by adding Ctrl-L as a standard way to clear >>>>> the >>>>> Python REPL, in the same way that Ctrl-C is the standard way to >>>>> interrupt the interpreter regardless of whether you are using Linux, >>>>> Mac >>>>> or Windows. >>>>> >>>>> (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but >>>>> Windows >>>>> is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?) >>>>> >>>>> >>>>> -- >>>>> Steve >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python... at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python... at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eryksun at gmail.com Thu Sep 29 05:53:24 2016 From: eryksun at gmail.com (eryk sun) Date: Thu, 29 Sep 2016 09:53:24 +0000 Subject: [Python-ideas] Suggestion: Clear screen command for the REPL In-Reply-To: <39f3f406-6978-4865-ba8c-b1506e9fdf70@googlegroups.com> References: <809a59bf-1fb5-77e0-029b-bdafc2ffaa79@googlemail.com> <4f80e18e-afda-4165-84d0-6b403e8121ea@googlegroups.com> <20160929020428.GU22471@ando.pearwood.info> <39f3f406-6978-4865-ba8c-b1506e9fdf70@googlegroups.com> Message-ID: On Thu, Sep 29, 2016 at 7:12 AM, Jo?o Matos wrote: > What is your Windows version? Are you trying on the cmd.exe console or PS? Are you talking about PowerShell ISE? That doesn't work for interactive console programs such as Python's REPL shell. Otherwise, FYI, there is no such thing as a cmd.exe or powershell.exe console. Those are shells, which can even run detached from a console using standard handles for the NUL device, pipes, or disk files. A shell that's attached to a console is just another console client application. Here's a brief overview of the Windows console system. Each console window is hosted by an instance of conhost.exe, but you can't simply run conhost.exe and get a console window. It depends on specific command-line arguments and handle inheritance, and Microsoft doesn't publish the implementation details. (It could be reverse engineered but there's not much to be gained from that.) A process can attach to a single console and set its standard input, output, and error file handles to either the console's input buffer or one of its screen buffers (or a new screen buffer via CreateConsoleScreenBuffer). This will be set up automatically by Windows if the executable is flagged as a console program and isn't run as a detached process. Otherwise a program can manually allocate or attach to a console via AllocConsole or AttachConsole. It can detach via FreeConsole. Attaching and detaching needs to be handled with care when updating the C standard FILE streams. Getting it wrong can crash the process. From random832 at fastmail.com Fri Sep 30 12:52:57 2016 From: random832 at fastmail.com (Random832) Date: Fri, 30 Sep 2016 12:52:57 -0400 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: <57EAE656.1070103@canterbury.ac.nz> References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> <57EAE656.1070103@canterbury.ac.nz> Message-ID: <1475254377.680408.742129865.62D71D9D@webmail.messagingengine.com> On Tue, Sep 27, 2016, at 17:36, Greg Ewing wrote: > Erik Bray wrote: > > Then following my own logic it > > would be desirable to also allow the nested for loop syntax of list > > comprehensions outside them as well. > > The only use for such a syntax would be to put > an inadvisable amount of stuff on one line. The only difference between an inadvisable amount of stuff and a reasonable amount of stuff is the amount. Don't we already have a recommended column limit in PEP8? for y in range(25) for x in range(80): doesn't seem unreasonable to me; YMMV. Or maybe you want to put it on multiple lines, but not consuming indentation levels: for x in seq1 if p1(x)\ for y in seq2 if p2(y)\ for z in seq3 if p3(z): result.append(f(x, y, z)) From python at 2sn.net Fri Sep 30 20:25:29 2016 From: python at 2sn.net (Alexander Heger) Date: Sat, 1 Oct 2016 10:25:29 +1000 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: <1475254377.680408.742129865.62D71D9D@webmail.messagingengine.com> References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> <57EAE656.1070103@canterbury.ac.nz> <1475254377.680408.742129865.62D71D9D@webmail.messagingengine.com> Message-ID: > > for x in seq1 if p1(x)\ > for y in seq2 if p2(y)\ > for z in seq3 if p3(z): > result.append(f(x, y, z)) > It think it is not uncommon to have nested loops, maybe with an if/continue statement at the beginning, with only one logical block that then should go down only one indentation level rather than three or four. An alternate way may be from itertools import product for x,y,z in filter(lambda x:p1(x[0]) and p2(x[1]) and p3(x[2]), \ product(seq1, seq3, seq3)): do_stuff(x,y,z) Doable, but seems kind of clunky. As a note, in general p2 could depend on x and y and p3 and x,y, and z; seq2 could depend on x and seq3 and x and y. The latter is something my example could not cover easily but only when explicitly writing out the loop, either in comprehension-style or "classic" loops. I think wasting of indentation levels for a single logical block should be avoided if possible to make the code more legible, otherwise one hits the suggested line length limit too fast - suppose this is now inside a method, you already lose at least 8 char ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Sep 30 23:34:56 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 1 Oct 2016 13:34:56 +1000 Subject: [Python-ideas] if-statement in for-loop In-Reply-To: References: <5ee6a817-bc68-06f6-3ecf-75fe0c7a4139@gmail.com> <57EAE656.1070103@canterbury.ac.nz> <1475254377.680408.742129865.62D71D9D@webmail.messagingengine.com> Message-ID: On 1 October 2016 at 10:25, Alexander Heger wrote: > I think wasting of indentation levels for a single logical block should be > avoided if possible to make the code more legible, otherwise one hits the > suggested line length limit too fast - suppose this is now inside a method, > you already lose at least 8 char ... Hence generators, which allow the nested loops to be readily factored out into a named operation. def iter_interesting_triples(seq1, seq2, seq3): for x in seq1: if p1(x): for y in seq2: if p2(x, y): for z in seq3: if p3(x, y, z): yield x, y, z for x, y, z in iter_interesting_triples(seq1, seq2, seq3): f(x, y, z) As you pointed out, the simple cases with no filtering, or only filtering that depends on the value of "z", are already covered by itertools.product: for x, y, z in itertools.product(seq1, seq2, seq3): f(x, y, z) for x, y, z in itertools.product(seq1, seq2, seq3): if p(x, y, z): f(x, y, z) And that step of separating the process of generating the candidate triples from actually doing the work on them also opens up a whole new world of possibilities when it comes to your execution model, like using concurrent.futures to process them in parallel in multiple threads or processes. There's no question that modeling algorithms as purely procedural code starts breaking down beyond a certain level of complexity. The fact that *not* breaking up our Python code into more modular units gets painful as we start hitting those limits as the author of the code in question is a side effect of that - it's a warning that we have an imminent readability problem, just as written documents start needing section headings to aid reader navigation as they get longer. As a result, much of the art of developing maintainable software lies in building out the refactoring tools we have available to us when we hit those limits, and in the case of Python, that means using features like functions to factor out request/response operations, generators to factor out a result series, classes to factor out structurally related data, context managers to separate out before/after structures, couroutines to separate out interactive agents, etc. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia